Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions .github/workflows/cfn-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Publish CloudFormation template

# The Launch Stack button requires the template at a public S3 HTTPS URL
# (CloudFormation quick-create cannot read from GitHub). This uploads
# deploy/cloudformation/compass-quickstart.yaml to the template bucket on
# every change and every release tag.
#
# One-time setup (repo admin):
# 1. Create an S3 bucket (default name: compass-cloudformation, us-east-1)
# with a bucket policy allowing public s3:GetObject on *.yaml.
# 2. Repo settings -> Secrets: CFN_PUBLISH_AWS_ACCESS_KEY_ID /
# CFN_PUBLISH_AWS_SECRET_ACCESS_KEY (an IAM user with PutObject on the
# bucket only). Variables: CFN_TEMPLATE_BUCKET (the bucket name).
# Without the secrets the job skips cleanly — forks stay green.

on:
push:
branches: [main]
paths:
- "deploy/cloudformation/**"
tags:
- "v*"
workflow_dispatch:

permissions:
contents: read

jobs:
publish:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4

- name: Lint template
run: pipx install cfn-lint && cfn-lint deploy/cloudformation/compass-quickstart.yaml

- name: Check publish credentials
id: creds
env:
KEY: ${{ secrets.CFN_PUBLISH_AWS_ACCESS_KEY_ID }}
run: |
if [ -n "$KEY" ]; then echo "have=true" >> "$GITHUB_OUTPUT";
else echo "have=false" >> "$GITHUB_OUTPUT"; echo "::notice::CFN publish secrets not configured — lint-only run"; fi

- name: Upload to the template bucket
if: steps.creds.outputs.have == 'true'
env:
AWS_ACCESS_KEY_ID: ${{ secrets.CFN_PUBLISH_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.CFN_PUBLISH_AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-1
BUCKET: ${{ vars.CFN_TEMPLATE_BUCKET || 'compass-cloudformation' }}
run: |
aws s3 cp deploy/cloudformation/compass-quickstart.yaml "s3://$BUCKET/compass-quickstart.yaml"
# Versioned copy on tags so old buttons keep working forever.
if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
aws s3 cp deploy/cloudformation/compass-quickstart.yaml "s3://$BUCKET/${GITHUB_REF#refs/tags/}/compass-quickstart.yaml"
fi
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Added

- **One-click AWS deploy**: a Launch Stack button (README header) opens the AWS console with `deploy/cloudformation/compass-quickstart.yaml` pre-loaded — one Create click provisions a self-contained serverless Compass (minimal VPC, private encrypted S3 bucket retained on stack deletion, least-privilege task role with zero long-lived keys via native ECS task credentials, Fargate task running the published image). Template is cfn-lint clean and stack-tested (17 resources to CREATE_COMPLETE, bucket survives stack delete); `.github/workflows/cfn-publish.yml` lints and publishes it to the template bucket.

- **One-command deployment** (`deploy/`): a kustomize tree for any Kubernetes cluster — serving StatefulSet with a PVC per replica (warm restarts), stateless writer Deployment, optional cold-serving tier, hardened pod security (non-root, seccomp, no capabilities), startup probes sized for cold rebuilds — plus Terraform for the AWS storage half (private KMS-encrypted S3 bucket with a TLS-only policy, least-privilege IAM, IRSA role or access key). A self-contained `minio-dev` overlay brings the whole serverless topology up on kind/k3s/minikube in one `kubectl apply -k`. Verified end-to-end on a live k3s cluster (all tiers serving, cross-tier read-your-writes, PVC persistence across pod replacement) and `terraform apply` verified against an AWS API emulator (idempotent re-apply; destroy refuses while data exists).
- **Published container images**: `ghcr.io/runcaptain/compass` (multi-arch amd64+arm64) on every release tag via `.github/workflows/docker.yml`. The Docker builder base moved to `rust:1.88-trixie` — bookworm's gcc-12 could not compile a dependency's ARM feature probes, which had made arm64 images impossible.

Expand Down
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
[![Rust](https://img.shields.io/badge/rust-1.88%2B-orange.svg)](https://www.rust-lang.org)
[![Image](https://img.shields.io/badge/ghcr.io-runcaptain%2Fcompass-2496ED?logo=docker&logoColor=white)](https://github.com/runcaptain/compass/pkgs/container/compass)
[![Deploy](https://img.shields.io/badge/%E2%98%81%20one--command%20deploy-Kubernetes%20%2F%20AWS-326CE5?logo=kubernetes&logoColor=white)](#deploy-to-kubernetes--aws-one-command)
<br>
<a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/quickcreate?stackName=compass&templateURL=https://compass-cloudformation.s3.amazonaws.com/compass-quickstart.yaml"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack on AWS"></a>
</div>

Built by [Captain](https://runcaptain.com) for high-throughput retrieval in on-prem enterprise deployments where customer data cannot leave their VPC.
Expand Down Expand Up @@ -427,9 +429,19 @@ docker run -p 4001:4001 -v ./data:/app/data compass

## Deploy to Kubernetes / AWS (one command)

One command deploys the full serverless topology — serving nodes on
persistent volumes, stateless writers, an optional cold tier, and a bundled
dev S3 — onto **any** Kubernetes cluster, no clone required:
**AWS, zero tools installed** — the Launch Stack button opens the AWS console
with everything pre-filled; one Create click gives you a running serverless
Compass (Fargate task + private encrypted S3 bucket + least-privilege IAM,
no long-lived keys — the task role is picked up automatically):

<a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/quickcreate?stackName=compass&templateURL=https://compass-cloudformation.s3.amazonaws.com/compass-quickstart.yaml"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack on AWS"></a>

The bucket is retained when the stack is deleted — your data outlives the
compute, always. Template source: [`deploy/cloudformation/compass-quickstart.yaml`](deploy/cloudformation/compass-quickstart.yaml).

**Kubernetes** — one command deploys the full multi-node topology — serving
nodes on persistent volumes, stateless writers, an optional cold tier, and a
bundled dev S3 — onto **any** cluster, no clone required:

```bash
kubectl apply -k "https://github.com/runcaptain/compass//deploy/kubernetes/overlays/minio-dev?ref=main"
Expand Down
15 changes: 14 additions & 1 deletion deploy/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,21 @@
# Deploying Compass

Three paths, in increasing order of ceremony. All of them are the same
Four paths, in increasing order of ceremony. All of them are the same
binary; topology background lives in [docs/deployment.md](../docs/deployment.md).

## 0. One click (AWS CloudFormation)

<a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/quickcreate?stackName=compass&templateURL=https://compass-cloudformation.s3.amazonaws.com/compass-quickstart.yaml"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack on AWS"></a>

[`cloudformation/compass-quickstart.yaml`](cloudformation/compass-quickstart.yaml)
creates a self-contained stack: minimal VPC, private encrypted S3 bucket
(**retained on stack deletion** — data outlives compute), least-privilege
task role (no keys; the engine picks up ECS task credentials natively), and
one Fargate task running `ghcr.io/runcaptain/compass`. Parameters: image
tag, API key, allowed CIDR, CPU/memory. The stack outputs a one-liner to
fetch the task's public IP. Built for evaluation and small single-node
workloads — graduate to the Kubernetes topology below for fleets.

## 1. One machine (docker compose)

```bash
Expand Down
257 changes: 257 additions & 0 deletions deploy/cloudformation/compass-quickstart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,257 @@
AWSTemplateFormatVersion: "2010-09-09"
Description: >-
Compass quick-start: a serverless single-node deployment on AWS in one
click. Creates a private encrypted S3 bucket (the database), a
least-privilege task role, and one Fargate task running the published
Compass image inside a minimal self-contained VPC. No long-lived keys
anywhere - the container picks up the task role automatically. Intended
for evaluation and small workloads; for multi-node fleets use
deploy/kubernetes + deploy/terraform/aws from the repository.

Parameters:
ImageTag:
Type: String
Default: latest
Description: Tag of ghcr.io/runcaptain/compass to run (e.g. v0.4.1).
ApiKey:
Type: String
Default: ""
NoEcho: true
Description: >-
Bearer token required on every API call. Leave empty for an
UNAUTHENTICATED instance - only do that with a restrictive AllowedCidr.
AllowedCidr:
Type: String
Default: 0.0.0.0/0
AllowedPattern: '^(\d{1,3}\.){3}\d{1,3}/\d{1,2}$'
Description: >-
CIDR allowed to reach port 4001. The default is the whole internet -
set your IP (x.x.x.x/32) or set ApiKey (ideally both).
Cpu:
Type: Number
Default: 1024
AllowedValues: [512, 1024, 2048, 4096]
Description: Fargate CPU units (1024 = 1 vCPU).
Memory:
Type: Number
Default: 4096
AllowedValues: [1024, 2048, 4096, 8192, 16384]
Description: Fargate memory (MiB). Must be a valid pairing with Cpu.

Conditions:
HasApiKey: !Not [!Equals [!Ref ApiKey, ""]]

Resources:
# ── Minimal VPC (two public subnets, so this template needs zero inputs) ──
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.42.0.0/24
EnableDnsSupport: true
EnableDnsHostnames: true
Tags: [{Key: Name, Value: !Sub "${AWS::StackName}-vpc"}]
InternetGateway:
Type: AWS::EC2::InternetGateway
VpcGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref Vpc
InternetGatewayId: !Ref InternetGateway
SubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
CidrBlock: 10.42.0.0/25
AvailabilityZone: !Select [0, !GetAZs ""]
MapPublicIpOnLaunch: true
SubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
CidrBlock: 10.42.0.128/25
AvailabilityZone: !Select [1, !GetAZs ""]
MapPublicIpOnLaunch: true
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref Vpc
DefaultRoute:
Type: AWS::EC2::Route
DependsOn: VpcGatewayAttachment
Properties:
RouteTableId: !Ref RouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
SubnetARouteAssoc:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref SubnetA
RouteTableId: !Ref RouteTable
SubnetBRouteAssoc:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref SubnetB
RouteTableId: !Ref RouteTable

# ── The bucket: this IS the database ─────────────────────────────────────
DataBucket:
Type: AWS::S3::Bucket
# Survives stack deletion: the bucket holds every collection, and
# CloudFormation refuses to delete non-empty buckets anyway. Clean up
# deliberately when you mean it.
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
LifecycleConfiguration:
Rules:
- Id: abort-incomplete-multipart
Status: Enabled
AbortIncompleteMultipartUpload:
DaysAfterInitiation: 7

# ── IAM: least privilege, no keys ─────────────────────────────────────────
TaskRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal: {Service: ecs-tasks.amazonaws.com}
Action: sts:AssumeRole
Policies:
- PolicyName: compass-s3
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: CompassObjects
Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
- s3:DeleteObject
- s3:AbortMultipartUpload
- s3:ListMultipartUploadParts
Resource: !Sub "${DataBucket.Arn}/*"
- Sid: CompassList
Effect: Allow
Action:
- s3:ListBucket
- s3:ListBucketMultipartUploads
- s3:GetBucketLocation
Resource: !GetAtt DataBucket.Arn
ExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal: {Service: ecs-tasks.amazonaws.com}
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

# ── Compute ───────────────────────────────────────────────────────────────
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Sub "${AWS::StackName}"
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub "/compass/${AWS::StackName}"
RetentionInDays: 30
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Compass API access
VpcId: !Ref Vpc
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 4001
ToPort: 4001
CidrIp: !Ref AllowedCidr
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Sub "${AWS::StackName}-compass"
RequiresCompatibilities: [FARGATE]
NetworkMode: awsvpc
Cpu: !Ref Cpu
Memory: !Ref Memory
TaskRoleArn: !GetAtt TaskRole.Arn
ExecutionRoleArn: !GetAtt ExecutionRole.Arn
RuntimePlatform:
CpuArchitecture: X86_64
OperatingSystemFamily: LINUX
ContainerDefinitions:
- Name: compass
Image: !Sub "ghcr.io/runcaptain/compass:${ImageTag}"
PortMappings:
- ContainerPort: 4001
Protocol: tcp
Environment:
- Name: COMPASS_STORAGE
Value: !Sub "s3://${DataBucket}"
- Name: AWS_REGION
Value: !Ref AWS::Region
- !If
- HasApiKey
- {Name: COMPASS_API_KEY, Value: !Ref ApiKey}
- !Ref AWS::NoValue
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: compass
Service:
Type: AWS::ECS::Service
DependsOn: DefaultRoute
Properties:
ServiceName: compass
Cluster: !Ref Cluster
LaunchType: FARGATE
DesiredCount: 1
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets: [!Ref SubnetA, !Ref SubnetB]
SecurityGroups: [!Ref SecurityGroup]
TaskDefinition: !Ref TaskDefinition

Outputs:
BucketName:
Description: The S3 bucket holding every collection (the database).
Value: !Ref DataBucket
ClusterName:
Description: ECS cluster running Compass.
Value: !Ref Cluster
FindTheEndpoint:
Description: >-
The task gets a public IP (no load balancer keeps this template cheap;
the IP changes if the task is replaced). Fetch it with this command,
then hit http://IP:4001/health.
Value: !Sub >-
aws ecs list-tasks --cluster ${Cluster} --query 'taskArns[0]' --output text |
xargs -I{} aws ecs describe-tasks --cluster ${Cluster} --tasks {}
--query 'tasks[0].attachments[0].details[?name==`networkInterfaceId`].value'
--output text | xargs -I{} aws ec2 describe-network-interfaces
--network-interface-ids {}
--query 'NetworkInterfaces[0].Association.PublicIp' --output text
DataDurability:
Description: What survives what.
Value: >-
The bucket is the source of truth: kill the task, delete the stack -
data survives in S3 (bucket is retained on stack deletion). Point any
future Compass at the same bucket to recover everything.
Loading