diff --git a/.github/workflows/cfn-publish.yml b/.github/workflows/cfn-publish.yml new file mode 100644 index 0000000..549304d --- /dev/null +++ b/.github/workflows/cfn-publish.yml @@ -0,0 +1,57 @@ +name: Publish CloudFormation template + +# The Launch Stack button requires the template at a public S3 HTTPS URL +# (CloudFormation quick-create cannot read from GitHub). This uploads +# deploy/cloudformation/compass-quickstart.yaml to the template bucket on +# every change and every release tag. +# +# One-time setup (repo admin): +# 1. Create an S3 bucket (default name: compass-cloudformation, us-east-1) +# with a bucket policy allowing public s3:GetObject on *.yaml. +# 2. Repo settings -> Secrets: CFN_PUBLISH_AWS_ACCESS_KEY_ID / +# CFN_PUBLISH_AWS_SECRET_ACCESS_KEY (an IAM user with PutObject on the +# bucket only). Variables: CFN_TEMPLATE_BUCKET (the bucket name). +# Without the secrets the job skips cleanly — forks stay green. + +on: + push: + branches: [main] + paths: + - "deploy/cloudformation/**" + tags: + - "v*" + workflow_dispatch: + +permissions: + contents: read + +jobs: + publish: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + + - name: Lint template + run: pipx install cfn-lint && cfn-lint deploy/cloudformation/compass-quickstart.yaml + + - name: Check publish credentials + id: creds + env: + KEY: ${{ secrets.CFN_PUBLISH_AWS_ACCESS_KEY_ID }} + run: | + if [ -n "$KEY" ]; then echo "have=true" >> "$GITHUB_OUTPUT"; + else echo "have=false" >> "$GITHUB_OUTPUT"; echo "::notice::CFN publish secrets not configured — lint-only run"; fi + + - name: Upload to the template bucket + if: steps.creds.outputs.have == 'true' + env: + AWS_ACCESS_KEY_ID: ${{ secrets.CFN_PUBLISH_AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.CFN_PUBLISH_AWS_SECRET_ACCESS_KEY }} + AWS_DEFAULT_REGION: us-east-1 + BUCKET: ${{ vars.CFN_TEMPLATE_BUCKET || 'compass-cloudformation' }} + run: | + aws s3 cp deploy/cloudformation/compass-quickstart.yaml "s3://$BUCKET/compass-quickstart.yaml" + # Versioned copy on tags so old buttons keep working forever. + if [[ "$GITHUB_REF" == refs/tags/v* ]]; then + aws s3 cp deploy/cloudformation/compass-quickstart.yaml "s3://$BUCKET/${GITHUB_REF#refs/tags/}/compass-quickstart.yaml" + fi diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d7926d..73097b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **One-click AWS deploy**: a Launch Stack button (README header) opens the AWS console with `deploy/cloudformation/compass-quickstart.yaml` pre-loaded — one Create click provisions a self-contained serverless Compass (minimal VPC, private encrypted S3 bucket retained on stack deletion, least-privilege task role with zero long-lived keys via native ECS task credentials, Fargate task running the published image). Template is cfn-lint clean and stack-tested (17 resources to CREATE_COMPLETE, bucket survives stack delete); `.github/workflows/cfn-publish.yml` lints and publishes it to the template bucket. + - **One-command deployment** (`deploy/`): a kustomize tree for any Kubernetes cluster — serving StatefulSet with a PVC per replica (warm restarts), stateless writer Deployment, optional cold-serving tier, hardened pod security (non-root, seccomp, no capabilities), startup probes sized for cold rebuilds — plus Terraform for the AWS storage half (private KMS-encrypted S3 bucket with a TLS-only policy, least-privilege IAM, IRSA role or access key). A self-contained `minio-dev` overlay brings the whole serverless topology up on kind/k3s/minikube in one `kubectl apply -k`. Verified end-to-end on a live k3s cluster (all tiers serving, cross-tier read-your-writes, PVC persistence across pod replacement) and `terraform apply` verified against an AWS API emulator (idempotent re-apply; destroy refuses while data exists). - **Published container images**: `ghcr.io/runcaptain/compass` (multi-arch amd64+arm64) on every release tag via `.github/workflows/docker.yml`. The Docker builder base moved to `rust:1.88-trixie` — bookworm's gcc-12 could not compile a dependency's ARM feature probes, which had made arm64 images impossible. diff --git a/README.md b/README.md index bfc14a7..3ee3d25 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ [![Rust](https://img.shields.io/badge/rust-1.88%2B-orange.svg)](https://www.rust-lang.org) [![Image](https://img.shields.io/badge/ghcr.io-runcaptain%2Fcompass-2496ED?logo=docker&logoColor=white)](https://github.com/runcaptain/compass/pkgs/container/compass) [![Deploy](https://img.shields.io/badge/%E2%98%81%20one--command%20deploy-Kubernetes%20%2F%20AWS-326CE5?logo=kubernetes&logoColor=white)](#deploy-to-kubernetes--aws-one-command) +
+ Launch Stack on AWS Built by [Captain](https://runcaptain.com) for high-throughput retrieval in on-prem enterprise deployments where customer data cannot leave their VPC. @@ -427,9 +429,19 @@ docker run -p 4001:4001 -v ./data:/app/data compass ## Deploy to Kubernetes / AWS (one command) -One command deploys the full serverless topology — serving nodes on -persistent volumes, stateless writers, an optional cold tier, and a bundled -dev S3 — onto **any** Kubernetes cluster, no clone required: +**AWS, zero tools installed** — the Launch Stack button opens the AWS console +with everything pre-filled; one Create click gives you a running serverless +Compass (Fargate task + private encrypted S3 bucket + least-privilege IAM, +no long-lived keys — the task role is picked up automatically): + +Launch Stack on AWS + +The bucket is retained when the stack is deleted — your data outlives the +compute, always. Template source: [`deploy/cloudformation/compass-quickstart.yaml`](deploy/cloudformation/compass-quickstart.yaml). + +**Kubernetes** — one command deploys the full multi-node topology — serving +nodes on persistent volumes, stateless writers, an optional cold tier, and a +bundled dev S3 — onto **any** cluster, no clone required: ```bash kubectl apply -k "https://github.com/runcaptain/compass//deploy/kubernetes/overlays/minio-dev?ref=main" diff --git a/deploy/README.md b/deploy/README.md index aaa6a08..fa8cdc5 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,8 +1,21 @@ # Deploying Compass -Three paths, in increasing order of ceremony. All of them are the same +Four paths, in increasing order of ceremony. All of them are the same binary; topology background lives in [docs/deployment.md](../docs/deployment.md). +## 0. One click (AWS CloudFormation) + +Launch Stack on AWS + +[`cloudformation/compass-quickstart.yaml`](cloudformation/compass-quickstart.yaml) +creates a self-contained stack: minimal VPC, private encrypted S3 bucket +(**retained on stack deletion** — data outlives compute), least-privilege +task role (no keys; the engine picks up ECS task credentials natively), and +one Fargate task running `ghcr.io/runcaptain/compass`. Parameters: image +tag, API key, allowed CIDR, CPU/memory. The stack outputs a one-liner to +fetch the task's public IP. Built for evaluation and small single-node +workloads — graduate to the Kubernetes topology below for fleets. + ## 1. One machine (docker compose) ```bash diff --git a/deploy/cloudformation/compass-quickstart.yaml b/deploy/cloudformation/compass-quickstart.yaml new file mode 100644 index 0000000..d1dfe59 --- /dev/null +++ b/deploy/cloudformation/compass-quickstart.yaml @@ -0,0 +1,257 @@ +AWSTemplateFormatVersion: "2010-09-09" +Description: >- + Compass quick-start: a serverless single-node deployment on AWS in one + click. Creates a private encrypted S3 bucket (the database), a + least-privilege task role, and one Fargate task running the published + Compass image inside a minimal self-contained VPC. No long-lived keys + anywhere - the container picks up the task role automatically. Intended + for evaluation and small workloads; for multi-node fleets use + deploy/kubernetes + deploy/terraform/aws from the repository. + +Parameters: + ImageTag: + Type: String + Default: latest + Description: Tag of ghcr.io/runcaptain/compass to run (e.g. v0.4.1). + ApiKey: + Type: String + Default: "" + NoEcho: true + Description: >- + Bearer token required on every API call. Leave empty for an + UNAUTHENTICATED instance - only do that with a restrictive AllowedCidr. + AllowedCidr: + Type: String + Default: 0.0.0.0/0 + AllowedPattern: '^(\d{1,3}\.){3}\d{1,3}/\d{1,2}$' + Description: >- + CIDR allowed to reach port 4001. The default is the whole internet - + set your IP (x.x.x.x/32) or set ApiKey (ideally both). + Cpu: + Type: Number + Default: 1024 + AllowedValues: [512, 1024, 2048, 4096] + Description: Fargate CPU units (1024 = 1 vCPU). + Memory: + Type: Number + Default: 4096 + AllowedValues: [1024, 2048, 4096, 8192, 16384] + Description: Fargate memory (MiB). Must be a valid pairing with Cpu. + +Conditions: + HasApiKey: !Not [!Equals [!Ref ApiKey, ""]] + +Resources: + # ── Minimal VPC (two public subnets, so this template needs zero inputs) ── + Vpc: + Type: AWS::EC2::VPC + Properties: + CidrBlock: 10.42.0.0/24 + EnableDnsSupport: true + EnableDnsHostnames: true + Tags: [{Key: Name, Value: !Sub "${AWS::StackName}-vpc"}] + InternetGateway: + Type: AWS::EC2::InternetGateway + VpcGatewayAttachment: + Type: AWS::EC2::VPCGatewayAttachment + Properties: + VpcId: !Ref Vpc + InternetGatewayId: !Ref InternetGateway + SubnetA: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref Vpc + CidrBlock: 10.42.0.0/25 + AvailabilityZone: !Select [0, !GetAZs ""] + MapPublicIpOnLaunch: true + SubnetB: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref Vpc + CidrBlock: 10.42.0.128/25 + AvailabilityZone: !Select [1, !GetAZs ""] + MapPublicIpOnLaunch: true + RouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref Vpc + DefaultRoute: + Type: AWS::EC2::Route + DependsOn: VpcGatewayAttachment + Properties: + RouteTableId: !Ref RouteTable + DestinationCidrBlock: 0.0.0.0/0 + GatewayId: !Ref InternetGateway + SubnetARouteAssoc: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref SubnetA + RouteTableId: !Ref RouteTable + SubnetBRouteAssoc: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref SubnetB + RouteTableId: !Ref RouteTable + + # ── The bucket: this IS the database ───────────────────────────────────── + DataBucket: + Type: AWS::S3::Bucket + # Survives stack deletion: the bucket holds every collection, and + # CloudFormation refuses to delete non-empty buckets anyway. Clean up + # deliberately when you mean it. + DeletionPolicy: Retain + UpdateReplacePolicy: Retain + Properties: + BucketEncryption: + ServerSideEncryptionConfiguration: + - ServerSideEncryptionByDefault: + SSEAlgorithm: AES256 + PublicAccessBlockConfiguration: + BlockPublicAcls: true + BlockPublicPolicy: true + IgnorePublicAcls: true + RestrictPublicBuckets: true + LifecycleConfiguration: + Rules: + - Id: abort-incomplete-multipart + Status: Enabled + AbortIncompleteMultipartUpload: + DaysAfterInitiation: 7 + + # ── IAM: least privilege, no keys ───────────────────────────────────────── + TaskRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: {Service: ecs-tasks.amazonaws.com} + Action: sts:AssumeRole + Policies: + - PolicyName: compass-s3 + PolicyDocument: + Version: "2012-10-17" + Statement: + - Sid: CompassObjects + Effect: Allow + Action: + - s3:GetObject + - s3:PutObject + - s3:DeleteObject + - s3:AbortMultipartUpload + - s3:ListMultipartUploadParts + Resource: !Sub "${DataBucket.Arn}/*" + - Sid: CompassList + Effect: Allow + Action: + - s3:ListBucket + - s3:ListBucketMultipartUploads + - s3:GetBucketLocation + Resource: !GetAtt DataBucket.Arn + ExecutionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: {Service: ecs-tasks.amazonaws.com} + Action: sts:AssumeRole + ManagedPolicyArns: + - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy + + # ── Compute ─────────────────────────────────────────────────────────────── + Cluster: + Type: AWS::ECS::Cluster + Properties: + ClusterName: !Sub "${AWS::StackName}" + LogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/compass/${AWS::StackName}" + RetentionInDays: 30 + SecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Compass API access + VpcId: !Ref Vpc + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 4001 + ToPort: 4001 + CidrIp: !Ref AllowedCidr + TaskDefinition: + Type: AWS::ECS::TaskDefinition + Properties: + Family: !Sub "${AWS::StackName}-compass" + RequiresCompatibilities: [FARGATE] + NetworkMode: awsvpc + Cpu: !Ref Cpu + Memory: !Ref Memory + TaskRoleArn: !GetAtt TaskRole.Arn + ExecutionRoleArn: !GetAtt ExecutionRole.Arn + RuntimePlatform: + CpuArchitecture: X86_64 + OperatingSystemFamily: LINUX + ContainerDefinitions: + - Name: compass + Image: !Sub "ghcr.io/runcaptain/compass:${ImageTag}" + PortMappings: + - ContainerPort: 4001 + Protocol: tcp + Environment: + - Name: COMPASS_STORAGE + Value: !Sub "s3://${DataBucket}" + - Name: AWS_REGION + Value: !Ref AWS::Region + - !If + - HasApiKey + - {Name: COMPASS_API_KEY, Value: !Ref ApiKey} + - !Ref AWS::NoValue + LogConfiguration: + LogDriver: awslogs + Options: + awslogs-group: !Ref LogGroup + awslogs-region: !Ref AWS::Region + awslogs-stream-prefix: compass + Service: + Type: AWS::ECS::Service + DependsOn: DefaultRoute + Properties: + ServiceName: compass + Cluster: !Ref Cluster + LaunchType: FARGATE + DesiredCount: 1 + NetworkConfiguration: + AwsvpcConfiguration: + AssignPublicIp: ENABLED + Subnets: [!Ref SubnetA, !Ref SubnetB] + SecurityGroups: [!Ref SecurityGroup] + TaskDefinition: !Ref TaskDefinition + +Outputs: + BucketName: + Description: The S3 bucket holding every collection (the database). + Value: !Ref DataBucket + ClusterName: + Description: ECS cluster running Compass. + Value: !Ref Cluster + FindTheEndpoint: + Description: >- + The task gets a public IP (no load balancer keeps this template cheap; + the IP changes if the task is replaced). Fetch it with this command, + then hit http://IP:4001/health. + Value: !Sub >- + aws ecs list-tasks --cluster ${Cluster} --query 'taskArns[0]' --output text | + xargs -I{} aws ecs describe-tasks --cluster ${Cluster} --tasks {} + --query 'tasks[0].attachments[0].details[?name==`networkInterfaceId`].value' + --output text | xargs -I{} aws ec2 describe-network-interfaces + --network-interface-ids {} + --query 'NetworkInterfaces[0].Association.PublicIp' --output text + DataDurability: + Description: What survives what. + Value: >- + The bucket is the source of truth: kill the task, delete the stack - + data survives in S3 (bucket is retained on stack deletion). Point any + future Compass at the same bucket to recover everything.