[pull] main from activepieces:main

.github/workflows/continuous-delivery-cloud.yml

@@ -160,6 +160,17 @@ jobs:
           git push origin "$BRANCH" --force
           echo "Created $BRANCH at $SHA"
 
+      - name: Notify on-call channel
+        env:
+          WEBHOOK: ${{ secrets.DISCORD_ON_CALL_WEBHOOK }}
+          IMAGE_TAG: ${{ steps.image.outputs.image_tag }}
+          DEPLOY_KIND: ${{ github.event_name == 'workflow_dispatch' && 'hotfix' || 'scheduled promotion' }}
+        run: |
+          curl -H "Content-Type: application/json" \
+            -X POST \
+            -d "{\"content\": \"🚀 **Cloud production deployed** ($DEPLOY_KIND)\\n\\n**Version:** \`$IMAGE_TAG\`\\n**Changelog:** ${{ github.server_url }}/${{ github.repository }}/releases/latest\\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\"}" \
+            "$WEBHOOK"
+
   # Run smoke test after scheduled promotion
   smoke-test:
     needs: promote-to-production

.github/workflows/release-self-hosted.yml

@@ -115,6 +115,17 @@ jobs:
           image: ghcr.io/activepieces/activepieces:${{ inputs.tag }}
           version: ${{ inputs.tag }}
 
+      - name: Notify on-call channel
+        if: ${{ !contains(inputs.tag, '-rc') || inputs.publish_rc_release }}
+        env:
+          WEBHOOK: ${{ secrets.DISCORD_ON_CALL_WEBHOOK }}
+          TAG: ${{ inputs.tag }}
+        run: |
+          curl -H "Content-Type: application/json" \
+            -X POST \
+            -d "{\"content\": \"📦 **Self-hosted $TAG released**\\n\\n**Docker image:** \`activepieces/activepieces:$TAG\` (also on ghcr.io)\\n**Changelog:** ${{ github.server_url }}/${{ github.repository }}/releases/tag/$TAG\\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\"}" \
+            "$WEBHOOK"
+
   sync-version-to-main:
     needs: release
     if: ${{ !cancelled() && needs.release.result == 'success' }}

brain/knowledge/ai-intelligence/ai-agents.md

@@ -28,6 +28,9 @@ A flow step type (backed by `@activepieces/piece-agent`) that runs an LLM-driven
 - **The enums and pure functions have exactly one home: `core/piece-types/src/lib/agents.ts`.** Do not re-declare `AgentToolType`, `McpAuthType`, `buildAuthHeaders`, `TASK_COMPLETION_TOOL_NAME`, or `mcpToolNameUtils` in `core-execution` — re-export them. They used to be duplicated byte-for-byte across both packages, which was silently load-bearing: if `createToolName` drifted, the tool names `migrate-v16` persisted would stop matching runtime names and every piece/flow/MCP call on a migrated flow would degrade to `ToolCallType.UNKNOWN`. `mcp-tool-name-util.test.ts` asserts both entry points resolve to the *same object*, so a re-fork fails the test rather than shipping.
 - The four `core/execution/src/lib/agents/` files are **not** uniform. `mcp-tool-name-util.ts` and `mcp.ts` are pure re-export shims (1 and 6 lines). `index.ts` and `tools.ts` re-export the canonical enums and functions but still **own** the execution-side plain-`zod` schema definitions — `tools.ts` declares the `AgentTool` union and the `McpAuth*` schemas, `index.ts` declares `AgentOutputField`, `MarkdownContentBlock`, `ToolCallContentBlock` and `AgentStepBlock`. Adding a field to one of those schemas means editing it there *and* in the `zod/mini` twin in `agents.ts`.
 - **A flow-step run must not reuse chat's resolution logic.** Four separate production failures came from this one assumption while moving the step server-side, each looking like its own bug. `resolveChatProvider` made a step need Chat's provider configured before it would run at all, so an instance that never uses Chat could not run an agent step — and it bit twice, because `resolveFastModel` reached the same helper underneath, so every *configured piece tool* failed with a bare `ENTITY_NOT_FOUND` long after the main model had been fixed. Grep for the transitive callers, not just the direct ones. `resolveModelIdForProvider` treats its argument as a *tier* id and falls back to the tier default when it is not in the curated chat list — a step configured for `claude-sonnet-4.5` silently ran `4.6`, because a step names a concrete model while chat names a tier. And the chat tool set reaches an unattended run, where a tool that asks the user a question is worse than useless: the agent opened a connection picker, read the empty answer as a refusal, and stopped. When a value crosses between the two surfaces, check what it *means* on each side, not just that the types line up.
+- **A worker RPC failure reaches the worker as `error.message` and nothing else.** The envelope in `core/execution/src/lib/engine/rpc.ts` drops `ActivepiecesError.params` and the stack, so three unrelated causes (conversation gone, no chat-enabled provider, pinned provider has no row) all arrive as the same bare `ENTITY_NOT_FOUND` — unreadable in the failed-job list. `createRpcServer` logs the intact error on the app side; read *that* log, not the worker's.
+- **Whatever enqueues an agent run must pre-check the same thing the worker resolves.** The chat route asked "is any provider enabled for chat" while the worker looked up the run's *pinned* provider, and the flow-step route checked nothing at all — so a run enqueued fine and could only fail. Both now call `agentHelpers.assertRunProviderConfigured`, which mirrors the worker's lookup. A pre-check that answers a *different* question than the worker is worse than none: it makes the failure look impossible.
+- **Everything the agent job does before its try/catch has no recovery.** `getAgentConfig` used to run outside it, so a config failure sent no error to the chat client and never called `releaseFlowStep` — the flow run sat PAUSED until `AP_PAUSED_FLOW_TIMEOUT_DAYS`. Anything added above that block needs its own failure path, or a paused run leaks.
 - **Build the unattended tool set as an allow-list.** Removing chat tools by name failed three times running — display tools, then build-plan and phase tools, then `ap_discover_action_auth` and `ap_load_guide`, which live with the local tools and so survived a filter written by tool group. Grouping tracks where a tool was constructed, not whether it assumes someone is reading. A flow step gets exactly what it is listed: its configured piece actions, the public-web readers, and the structured-output tool. Anything added to chat later stays out by default.
 - A separate zod-free `agent-primitives.ts` holding those values was tried and **folded back** — don't re-create it. It bought no isolation: `core-execution` imports the `@activepieces/core-piece-types` **barrel**, which re-exports `agents.ts`, so `zod/mini` comes along whatever the values live in.
 - Only the zod *schemas* stay duplicated — the `zod` vs `zod/mini` split is a real bundle-size decision, and a schema drift breaks loudly where a function drift did not.

brain/knowledge/data-storage-observability/file-storage.md

@@ -33,8 +33,10 @@ The central service for persisting binary files, backing the execution engine an
 
 ### Gotchas
 - **On cloud the real key prefix is doubled — `<bucket>/<bucket>/…` — so ad-hoc CLI work against the bucket silently finds nothing.** Cloud's `AP_S3_ENDPOINT` embeds the bucket as a *path segment* (`https://<account>.r2.cloudflarestorage.com/ap-files-prod`), and `getS3Client` sets `forcePathStyle: true` whenever an endpoint is present, so the SDK appends the bucket again. An object the app stores as `pieces/x.tgz` actually lands at `ap-files-prod/pieces/x.tgz` inside bucket `ap-files-prod`. Symptom when you get it wrong: `aws s3 ls` returns `NoSuchKey` on a *prefix* listing (Aug 2026 — cost three attempts to spot). Strip the trailing `/<bucket>` from the endpoint for the CLI, then prepend the bucket name to the prefix; or better, do bulk work through `s3Helper` so the same client resolves the same paths. Note cloud's object store is **Cloudflare R2** while the piece CDN is a **DigitalOcean Space** — two different systems, easy to conflate.
+- **The live piece-bundle cache sits *inside* the legacy one — `pieces/v2/` is nested under `pieces/`, so a recursive delete of `pieces/` takes the active cache with it.** `S3_PIECES_PREFIX` in `piece-bundle.ts` is `pieces/v2/`; the bare `pieces/` keys beside it are pre-CDN tarballs left by the older writer. Combined with the doubled prefix above, the real keys are `ap-files-prod/pieces/…` (legacy) and `ap-files-prod/pieces/v2/…` (live). Probe both with `wrangler r2 object get` before any prefix-wide operation — wiping v2 used to be survivable because it refilled lazily, at the cost of a burst of cache misses on every piece. **Both prefixes are now dead storage and safe to sweep:** the `BUNDLE_PIECE` job and the S3 mirror were removed, so `resolve()` no longer reads or writes either prefix and registry pieces redirect straight to the CDN (else npm). The mirror was deleted because it was written from whichever source was preferred *at cache time* and then took precedence over the CDN forever — a bucket populated before the CDN became preferred kept serving the unbundled npm build, which is what fans out one `@activepieces/shared` copy per piece in the engine (see the Workers page).
 - **`deleteFiles` succeeding does not mean the objects are gone.** `DeleteObjectsCommand` reports per-object failures in `response.Errors` and does **not** throw, and `Quiet: true` only suppresses the success entries — so a request that "worked" can still have left objects behind. `deleteFiles` logs a warn naming the failure codes, which is all its callers (best-effort cleanup) need. Anything whose *correctness* depends on the prefix being empty afterwards would have to surface those keys and retry — but prefer not to need that at all: a reader that must not see the old objects should read from a new key prefix rather than race a delete against writers that may still be running old code.
 - Cleanup job runs hourly (`30 */1 * * *`), deletes stale execution files past `EXECUTION_DATA_RETENTION_DAYS`; processes ~4000/iteration, deletes S3 keys in batches of 100.
+- **Retention cleanup is row-driven, so any S3 object written without a `file` row is immortal.** The job walks `file` rows and deletes each one's `s3Key`; it never lists the bucket. The piece-tarball cache was exactly that shape — keyed by `<name>-<version>.tgz` under a bare prefix, with no row anywhere — so nothing has ever swept it and nothing structurally could, whatever the retention setting says. Checked Aug 2026: no migration or job has ever bulk-deleted the `pieces/` prefix, and the only `deleteFiles` callers are the row-driven cleanup and the health probe's own key. If you add a store path that bypasses the `file` table, you own its lifecycle by hand — prefer writing a row, or expect a manual `wrangler`/`aws s3` operation forever.
 - S3 deletes send CRC32C checksum (OCI rejects the SDK-default CRC32). `S3_ENDPOINT` set → SDK checksum/aws-chunked encoding disabled for S3-compatible providers.
 - `S3_USE_SIGNED_URLS=true` redirects downloads to 7-day pre-signed URLs instead of streaming through the app.
 

brain/knowledge/decisions/000028-piece-tarballs-are-never-mirrored-into-our-own-s3.md

@@ -0,0 +1,25 @@
+---
+status: accepted
+---
+
+# Piece tarballs are never mirrored into our own S3
+
+## Decision
+`pieceBundle.resolve()` hands out exactly two kinds of link for a registry piece: the CDN's self-contained bundle when it serves one, otherwise the npm tarball. It does not consult, populate, or fall back to an Activepieces-owned S3 copy, and there is no lazy caching job behind it — the `BUNDLE_PIECE` system job was deleted rather than fixed. `ARCHIVE` (custom) pieces are unaffected; they are platform-scoped rows in our own file store and are still streamed from it.
+
+## Context
+Until Aug 2026 `resolve()` checked an S3 mirror first (`pieces/v2/`), and on a miss enqueued `BUNDLE_PIECE` to populate it for next time. That job wrote **whichever source was preferred at cache time** — the CDN artifact when `AP_USE_CDN_FOR_BUNDLES` was on, the npm tarball otherwise.
+
+Because the read order put S3 ahead of the CDN, a bucket populated while the flag was off kept serving the *unbundled npm build* for that piece forever, and turning the flag on later changed nothing for any piece already mirrored. The job carried a comment warning about precisely this, which is the tell that the ordering was known to be load-bearing and fragile.
+
+The cost is not theoretical. An unbundled tarball declares its build-time dependencies, so each piece pulls its own `@activepieces/shared` and `pieces-framework`. Measured on a 0.88.1 dedicated cloud host: **7 resident `@activepieces/shared` copies holding 208 MB of a 642 MB engine heap** (~33 MB each, ~3,400–4,000 Zod schema objects per copy retained through `require.cache`), 12 distinct versions on disk, and only **7 of 45** installed piece folders carrying a CDN bundle.
+
+## Why
+A cache that can outrank its own upstream is a cache that pins a bug. The mirror's precedence made the *first* fetch permanent, so any later improvement to how pieces are built could not reach a piece that had already been cached — and the repair for a poisoned entry is a bucket operation, not a deploy. Deleting the mirror makes the CDN the single upstream, so what a worker installs is decided by the current release rather than by whenever that piece was first requested.
+
+The obvious alternative was to keep the mirror and only cache CDN artifacts, never npm. Rejected: it keeps a second source of truth whose contents still depend on flag state at write time, still needs a prefix bump every time the meaning of a cached object changes (`pieces/` → `pieces/v2/` was already the second generation), and buys little — the CDN is itself a CDN, so we were caching a cache. The mirror's real benefit was egress and independence from npm, and that was never actually delivered, because a cold entry always fetched from npm anyway.
+
+Not in scope, deliberately: `AP_USE_CDN_FOR_BUNDLES` still defaults to `false`. This decision removes what would *shadow* the CDN once that default flips; flipping it is a separate call.
+
+## Consequences
+Both S3 prefixes (`pieces/`, `pieces/v2/`) become dead storage and can be swept — and note `pieces/v2/` is nested inside `pieces/`, so a recursive delete hits both (see *File Storage*). Every piece install now depends on the CDN or npm being reachable, with no local buffer; a self-hoster with S3 configured no longer accumulates a private copy. During a rolling deploy, old instances keep enqueuing `bundle-piece` while new ones have no handler, so those jobs fail with `No handler for job bundle-piece` until the rollout finishes — they are cache-warming only, so nothing user-facing breaks. Reintroducing a mirror later means re-deciding the read order, and the rule to keep is that a mirror must never be consulted ahead of the source it was copied from.

brain/knowledge/engineering/engineering-handbook-playbooks.md

@@ -19,3 +19,6 @@ Run EE, building for self-hosting, setup BetterStack, releases, canary deploymen
 
 ## Postmortems & product
 Postmortems (Redis/queue overload, infra upgrade — March 2026) and product interface-design notes.
+
+## Gotchas
+- **`DISCORD_ON_CALL_WEBHOOK` is the single repo secret behind every on-call Discord notification** — cloud production deploys (`continuous-delivery-cloud.yml`), self-hosted releases (`release-self-hosted.yml`), and release-pieces failure alerts all post through it. Rotating it repoints all of them at once; there is no per-workflow webhook. Self-hosted release notifications skip `-rc` tags unless `publish_rc_release` is set, mirroring the release-drafter condition so the message never links a release page that doesn't exist.

brain/knowledge/engineering/server-module-anatomy.md

@@ -119,12 +119,16 @@ Register in `app.ts`, in the CE or EE section. EE-only modules live under `src/a
 
 Queued work: add to `SystemJobName` or `WorkerJobType` in shared, register the handler via `systemJobHandlers.registerJobHandler()` in `app.ts`.
 
+**Retiring a `SystemJobName` is two steps, and doing only the first orphans jobs forever.** Deleting the enum member removes it from `knownJobNames`, but `isDeprecated()` in `system-job.ts` is `!knownJobNames.includes(name) && deprecatedJobs.some(d => name.startsWith(d))` — so a name that is unknown *and* unlisted matches neither branch and is never swept. Whatever is already queued in Redis then survives every `init()`, and `getJobHandler` throws `No handler for job <name>` on each scan, forever. So also **add the string literal to the `deprecatedJobs` array** in the same file; the 14 names already there are the precedent. Seed one in `test/unit/app/helper/system-jobs/remove-deprecated-jobs.test.ts` — its assertions compare the whole remaining queue, so a seeded job is covered for free.
+
 ## Tests
 
 `packages/server/api/test/integration/ce/{feature}.test.ts`, using `setupTestEnvironment()` + `createTestContext(app)` → `ctx.post()` / `ctx.get()`. The DB is cleaned between tests.
 
 Verify with `npm run lint-dev` and `npm run test-api`.
 
+**`packages/server/api/test/unit/**` runs in no pipeline — do not trust it as a safety net.** The package defines a `test-unit` script, but CI only runs `turbo run test-ce test-ee test-cloud check-migrations --filter=api` (`ci.yml`), and the root `npm run test-unit` filters to `engine`/`shared`/`sandbox`/`core-utils`/`server-utils`/`pieces-framework`/`web`/`ee-embed-sdk` — `api` is not in that list. So those specs are only ever run by hand, and they rot: measured Aug 2026 on a clean `main`, **18 tests across 4 files already failed** (`workers/job-queue/job-broker`, `workers/machine/machine-service`, `core/canary/worker-group.service`, `knowledge-base/file-service-delete`). Two consequences: put a server test you actually want enforced under `test/integration/ce`, and when a local `test/unit` run goes red, check `main` before assuming your branch caused it.
+
 ## Gotchas
 
 - **`getEntities()` and `getMigrations()` are both manual.** Nothing is auto-discovered. A missing entity registration fails silently at runtime; a missing migration registration means the migration simply never runs.