Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@
# The leading slash is required — unanchored patterns would also match the
# per-piece lockfiles under packages/pieces/.
/bun.lock
/docs/
/tsconfig.base.json
/brain/
28 changes: 26 additions & 2 deletions .github/workflows/continuous-delivery-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,23 @@ name: Continuous Delivery — Canary
on:
workflow_dispatch:
workflow_call:
inputs:
image_tag:
description: 'Deploy this already-built tag instead of building a new image'
type: string
required: false
default: ''
skip_migration_check:
description: 'Deploy even when a pending migration is marked breaking'
type: boolean
required: false
default: false
schedule:
- cron: '0 9 * * *' # Daily 9 AM UTC — scheduled runs always use the default branch

jobs:
build-image:
if: ${{ !inputs.image_tag }}
runs-on: ubuntu-24.04
permissions:
contents: read
Expand Down Expand Up @@ -54,6 +66,10 @@ jobs:

check-migrations:
needs: build-image
if: |
always() &&
needs.build-image.result != 'failure' &&
inputs.skip_migration_check != true
runs-on: ubuntu-latest
outputs:
has_breaking: ${{ steps.check.outputs.has_breaking }}
Expand Down Expand Up @@ -114,10 +130,18 @@ jobs:

deploy-canary:
needs: [build-image, check-migrations]
if: |
always() &&
needs.build-image.result != 'failure' &&
needs.check-migrations.result != 'failure'
runs-on: ubuntu-latest
environment:
name: canary
steps:
- name: Resolve image tag
id: image
run: echo "image_tag=${{ inputs.image_tag || needs.build-image.outputs.image_tag }}" >> $GITHUB_OUTPUT

- name: Configure SSH
run: |
mkdir -p ~/.ssh/
Expand All @@ -137,9 +161,9 @@ jobs:

- name: Deploy canary app
run: |
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ needs.build-image.outputs.image_tag }} --config-file=config/app-canary.yml --skip-push; exit"'
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ steps.image.outputs.image_tag }} --config-file=config/app-canary.yml --skip-push; exit"'

- name: Deploy canary workers
run: |
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ needs.build-image.outputs.image_tag }} --config-file=config/worker-canary.yml --skip-push; exit"'
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ steps.image.outputs.image_tag }} --config-file=config/worker-canary.yml --skip-push; exit"'

25 changes: 8 additions & 17 deletions .github/workflows/continuous-delivery-cloud.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,22 +73,23 @@ jobs:
tags: ghcr.io/activepieces/activepieces-cloud:${{ steps.set-tag.outputs.image_tag }}

deploy-canary:
needs: [build-image]
needs: [guard, build-image]
if: |
always() &&
github.event_name != 'workflow_dispatch' &&
needs.build-image.result == 'skipped'
needs.guard.result != 'failure' &&
(needs.build-image.result == 'success' || needs.build-image.result == 'skipped')
uses: ./.github/workflows/continuous-delivery-canary.yml
with:
image_tag: ${{ needs.build-image.outputs.image_tag }}
skip_migration_check: ${{ github.event_name == 'workflow_dispatch' }}
secrets: inherit

promote-to-production:
needs: [build-image, deploy-canary]
if: |
always() &&
(
(needs.build-image.result == 'success' && github.event_name == 'workflow_dispatch') ||
(needs.deploy-canary.result == 'success')
)
needs.deploy-canary.result == 'success' &&
(needs.build-image.result == 'success' || needs.build-image.result == 'skipped')
runs-on: ubuntu-latest
permissions:
contents: write
Expand Down Expand Up @@ -125,16 +126,6 @@ jobs:
echo "image_tag=release-candidate" >> $GITHUB_OUTPUT
fi

- name: Deploy App to canary
if: github.event_name == 'workflow_dispatch'
run: |
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ steps.image.outputs.image_tag }} --config-file=config/app-canary.yml --skip-push; exit"'

- name: Deploy Workers to canary
if: github.event_name == 'workflow_dispatch'
run: |
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ steps.image.outputs.image_tag }} --config-file=config/worker-canary.yml --skip-push; exit"'

- name: Deploy App to production
run: |
ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy --version ${{ steps.image.outputs.image_tag }} --config-file=config/app.yml --skip-push; exit"'
Expand Down
8 changes: 8 additions & 0 deletions brain/knowledge/engineering/architecture-spine.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@ Activepieces: open-source AI-first workflow automation platform (self-hosted or

**Don't `.max()` a business limit on a request body — cap server-side.** A `.max()` on a request-body field rejects the *whole* request with a 400 the moment a user crosses it, so a user editing a list that reaches 50 items loses their entire save. Reserve `.max()` for a true trust-boundary DoS guard (Fastify's global body limit already covers gross abuse) and let business limits just *apply*: accept the input and `slice(0, MAX)` in the service layer, so the write always succeeds with the limit quietly enforced. Surfaced 2026-07 on `POST /v1/chat/memory`, where the schema's `.max(50)`/`.max(280)` duplicated a `slice` the save helper already did — redundant *and* a data-loss bug.

**`unique()` from `core-utils` is O(n²) over `JSON.stringify` — never put it on a hot path.** It is `filter` + `findIndex` with a `JSON.stringify` on *both* sides of every comparison, so it blocks the event loop: 1k items → 42ms, 5k → 889ms, 10k → 3.6s, during which health checks, websockets and webhook dispatch all stall. It exists for deep-equality dedupe of objects; for primitives use `[...new Set(xs)]`. Found 2026-07 as the first statement of the bulk record delete the same PR was trying to speed up (GIT-1652).

**`DeleteResult.affected` is `undefined` on PGlite — don't count rows with it.** TypeORM's `PostgresQueryRunner` only sets `affected` when the driver result carries `rowCount`; PGlite reports `affectedRows` instead and `typeorm-pglite` doesn't map it. So `result.affected ?? 0` is correct on `pg` and silently `0` on every PGlite deployment and test — the worst failure mode, since CI is green. Use `.returning('id')` and count the rows.

**Migration timestamps are hand-picked, so two PRs in flight will collide.** `postgres-connection.ts` uses round numbers (`1815000000000`, `1816000000000`, …), not `Date.now()`, and TypeORM orders migrations by the 13-digit suffix of the class name. Two branches both taking "the next one" produce duplicate keys, and ordering — including `rollback-migrations.ts` — silently falls back to `getMigrations()` array order. Check `git ls-tree main packages/server/api/src/app/database/migration/postgres/` for the number before you commit, and re-check after any rebase.

**`CREATE INDEX CONCURRENTLY IF NOT EXISTS` can record success over a permanently invalid index.** `CONCURRENTLY` requires `transaction = false`, so nothing rolls back an interrupted build — it leaves an `indisvalid = false` index. `IF NOT EXISTS` then matches on *name only*, so the retry skips it with a NOTICE and TypeORM marks the migration applied: the query the index was meant to fix stays slow, with a green migration log. An invalid index is not inert either — still maintained on every insert, still blocks HOT updates. Use `DROP INDEX CONCURRENTLY IF EXISTS` before the create, and assert `pg_index.indisvalid` after. The existing `1810`/`1815`/`1818` index migrations all carry this shape.

**TypeORM soft-delete (`@DeleteDateColumn`) is not canary/rollback-safe on a shared DB.** TypeORM only appends `WHERE "deleted" IS NULL` for code whose entity *declares* the column, so any two versions sharing one Postgres — every canary window (canary shares prod's DB), every rollback — means old code reads soft-deleted rows as live. During canary a row deleted by new code reappears live and editable on old-code requests; on rollback every soft-deleted row returns permanently. Partially unrecoverable, too: old code's delete is a hard `DELETE`, so it can destroy a resurrected row the new restore feature could otherwise bring back. Partial indexes (`WHERE deleted IS NULL`) also stop serving old queries → seq scans. Do it expand-contract: ship the column and make **all** read paths filter on it first, roll that out everywhere, and only then flip the write path to `softDelete()`. The additive column is fine — it's the read-semantics change that can't run split across versions, and the same applies to any migration where old code must interpret a column it doesn't know about. Seen in PR #14219 (feat: chat core).

**Canary doesn't proxy websockets — only broadcasts reach canary users.** Canary is a worker group that *also* has its own app tier (`CANARY_APP_URL`, `IS_CANARY_APP`), sharing prod's Postgres and Redis. The prod app is the ingress and `canaryRoutingMiddleware` HTTP-proxies a platform whose `workerGroupId === 'canary'` to the canary app — but the middleware is registered inside the `/api` scope, so only `/api/*` is proxied (the SPA is served at root from the baked-in bundle) and it bails on upgrades: `if (request.headers.upgrade === 'websocket') return`. A canary platform therefore runs the **prod** frontend, and its websocket is terminated by **prod (old code)** while its HTTP and flow jobs run on canary. Across a version split, server→client broadcasts still work (socket.io's Redis adapter relays canary's `emit` name-agnostically), but inbound handlers — `LOCK_RESOURCE`/`UNLOCK_RESOURCE`, presence — run on old prod code and silently degrade. The fix, verified 2026-07: point canary-platform websockets at the already-live `canary.activepieces.com` by making the frontend socket URL a runtime value from an authenticated `/api` flag (that call *is* proxied, so canary answers `wss://canary.activepieces.com` and prod answers same-origin) and deferring socket creation until it resolves. Cross-origin is fine (`cors:{origin:'*'}`, token in `socket.auth`, not cookies), and it closes the inbound half of the seam too. kamal-proxy can't help — host/path routing only, no cookie/header routing — and `reply.from` is HTTP-only. Canary is the only worker group with a separate app tier; dedicated groups share the prod app, so their users' websockets already hit the right code. Workers are the mirror case: they carry `workerGroupId` in post-upgrade auth but use an explicit `socketUrl`, so canary workers must point at the canary app by config.
10 changes: 6 additions & 4 deletions brain/knowledge/engineering/cloud-deployment-paths.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,21 @@ icon: 🚀

# Cloud Deployment Paths

How code reaches `cloud.activepieces.com`. Two workflows in `.github/workflows/`: `continuous-delivery-canary.yml` and `continuous-delivery-cloud.yml`.
How code reaches `cloud.activepieces.com`. Two workflows in `.github/workflows/`: `continuous-delivery-canary.yml` and `continuous-delivery-cloud.yml`. Both cloud paths run the same job graph — `guard` → `build-image` → `deploy-canary` → `promote-to-production` — and both reach prod only after canary deploys cleanly.

## The normal path
Cloud's `workflow_call`/scheduled run calls the canary workflow as a job, then promotes the `release-candidate` tag to prod. Canary builds its own `.canary` image; prod deploys the `release-candidate` tag, not that image.
Cloud's `workflow_call`/scheduled run skips `build-image`, so `deploy-canary` gets an empty `image_tag` and the canary workflow builds its own `.canary` image and runs `check-migrations`. Prod then deploys the `release-candidate` tag, not that image.

## The override path
`Continuous Delivery — Cloud` → **Run workflow** → `cloud-hotfix` builds a `.beta` image from the current branch and deploys it to canary **and** prod, in that order, from the `promote-to-production` job. It never invokes the canary workflow. A `guard` job refuses the hotfix if the scheduled promotion is under an hour away.
`Continuous Delivery — Cloud` → **Run workflow** → `cloud-hotfix` builds one `.beta` image from the current branch and hands its tag to the canary workflow via `image_tag`, so canary deploys the exact artifact prod is about to get instead of rebuilding it. It also passes `skip_migration_check: true`. A `guard` job refuses the hotfix if the scheduled promotion is under an hour away, and a rejected guard skips canary too.

## Staging (upstream of both)
`continuous-delivery-stg.yml` builds every push to `main` and deploys it with **Kamal**, not Kubernetes: it SSHes to the devops box and runs `kamal deploy --config-file=config/{app,worker}.yml` from `/root/mrsk/stg`. App containers live on one host, workers on another; `kubectl`'s `stg` context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in `config/app.yml` under `env.clear` (secrets are name-listed under `env.secret` and read from `.kamal/secrets`). The Thursday job retags whatever staging is running as `release-candidate`, which is what cloud promotes.

## Gotchas
- **Kamal reads its config at deploy time, so editing `config/app.yml` during an in-flight CD run silently misses.** The build takes ~5 min and the deploy job reads the file when *it* starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-run `kamal deploy --version <same tag> --config-file=config/app.yml --skip-push` after editing, and verify with `docker inspect <container> --format '{{range .Config.Env}}...'` on **every** container, not one. The command needs a TTY (`ssh -tt`); without it Kamal exits 0 having done nothing ("the input device is not a TTY").
- **`check-migrations` gates canary *and* the scheduled cloud promotion.** The canary workflow fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`). Because the scheduled cloud run calls that workflow, one breaking migration blocks both. The escape hatch is `cloud-hotfix`: it deploys canary and prod without touching the canary workflow, so it bypasses the gate by construction — there is no skip flag, and there is no canary-only override.
- **`check-migrations` gates canary *and* the cloud promotion that calls it.** The canary workflow fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`), and the scheduled cloud run inherits that gate. `cloud-hotfix` passes `skip_migration_check: true` to bypass it: a deliberate cloud release ships the breaking migration to prod anyway, so blocking canary on it only leaves canary behind prod. There is still no canary-only override — dispatch the cloud workflow.
- **Skipping a `needs` job reads the same as a rejected one.** `build-image` is skipped both when the run is scheduled *and* when `guard` fails a hotfix, so `needs.build-image.result == 'skipped'` alone would deploy canary for a hotfix the guard just refused. `deploy-canary` therefore also needs `guard` and checks `needs.guard.result != 'failure'`.
- **`docker pull … error from registry: denied` on the ops host can be transient.** Seen 2026-08-12 pulling a `.beta` tag to the canary host; re-running the same job with no other change pulled and deployed fine. The tag existed in GHCR the whole time. Re-run once before suspecting the host's registry credential — GHCR does answer `GET /v2/` for a token that cannot pull, so login-succeeds/pull-denied is *consistent* with an expired credential, but it is not evidence of one.
- **Never put a BuildKit cache mount on `/var/cache/apt` or `/var/lib/apt`.** The `node:*-bullseye-slim` base ships `/etc/apt/apt.conf.d/docker-clean`, which wipes downloaded `.deb`s and sets `Keep-Downloaded-Packages "false"` — so the mount caches nothing, but it does persist stale `apt` lists and `partial/` leftovers across Depot builds. When `bullseye-security` republishes and old `.deb`s rotate out, the next build dies on `Hash Sum mismatch` / `Unable to fetch some archives` → `exit code: 100`, which reads like a missing package but isn't. Removed from both Dockerfiles on 2026-08-12; a plain `apt-get update && apt-get install` is what works. Also: bullseye `main` has been frozen since Aug 2025 and Debian 11 LTS ends Aug 2026, so the base image needs a bookworm bump.
- **`breaking = true` on a migration and the `⛓️‍💥 breaking-change` PR label are different axes.** The migration flag is about rollback safety and is what stops deploys; the label is about self-hoster upgrade impact and is enforced by `breaking-change-check.yml` on PRs. Neither implies the other.
4 changes: 2 additions & 2 deletions docs/build-pieces/building-pieces/create-action.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ export const getIcecreamFlavor = createAction({
method: HttpMethod.GET,
url: 'https://cloud.activepieces.com/api/v1/webhooks/RGjv57ex3RAHOgs0YK6Ja/sync',
headers: {
Authorization: context.auth, // Pass API key in headers
Authorization: context.auth.secret_text, // Pass API key in headers
},
});
return res.body;
Expand All @@ -73,7 +73,7 @@ The `run` function is the function that is called when the action is executed. I
The `run` function utilizes the httpClient.sendRequest function to make a GET request, fetching a random ice cream flavor. It incorporates API key authentication in the request headers. Finally, it returns the response body.

<Tip>
You can describe how an action's output is presented in the builderincluding readable labels for outputs keyed by opaque IDs or returned as arrays by declaring an [Output Schema](/build-pieces/piece-reference/output-schema).
You can describe how an action's output is presented in the builder, including readable labels for outputs keyed by opaque IDs or returned as arrays, by declaring an [Output Schema](/build-pieces/piece-reference/output-schema).
</Tip>

## Expose The Definition
Expand Down
2 changes: 1 addition & 1 deletion docs/build-pieces/building-pieces/create-trigger.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ const polling: Polling<
method: HttpMethod.GET,
url: 'https://cloud.activepieces.com/api/v1/webhooks/aHlEaNLc6vcF1nY2XJ2ed/sync',
headers: {
authorization: auth,
authorization: auth.secret_text,
},
};
const res = await httpClient.sendRequest(request);
Expand Down
4 changes: 2 additions & 2 deletions docs/build-pieces/building-pieces/development-setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ icon: 'circle-2'

## Prerequisites

- Node.js v18+
- Node.js v22.15+ or v24
- npm v9+

## Instructions
Expand All @@ -18,7 +18,7 @@ node tools/setup-dev.js

2. Start the environment

This command will start activepieces with sqlite3 and in memory queue.
This command will start activepieces with PGLite and an in-memory queue.

```bash
npm start
Expand Down
2 changes: 2 additions & 0 deletions docs/build-pieces/building-pieces/piece-definition.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,10 @@ import { PieceAuth, createPiece } from '@activepieces/pieces-framework';

export const gelato = createPiece({
displayName: 'Gelato',
description: '',
logoUrl: 'https://cdn.activepieces.com/pieces/gelato.png',
auth: PieceAuth.None(),
minimumSupportedRelease: '0.36.1',
authors: [],
actions: [],
triggers: [],
Expand Down
2 changes: 1 addition & 1 deletion docs/build-pieces/building-pieces/setup-fork.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ If you are on windows, please install [WSL](https://learn.microsoft.com/en-us/wi
1. Go to the repository page at https://github.com/activepieces/activepieces.
2. Click the `Fork` button located in the top right corner of the page.

![Fork Repository](/resources/screenshots/fork-repository.jpg)
![Fork Repository](/resources/screenshots/fork-repository.png)

3. Clone your fork using a shallow clone for faster setup:

Expand Down
Loading
Loading