[pull] main from activepieces:main

brain/knowledge/decisions/000027-an-agent-is-a-project-scoped-row-that-flow-steps-reference-live.md

@@ -0,0 +1,44 @@
+---
+title: An agent is a project-scoped row that flow steps reference live
+icon: 🎯
+status: proposed
+---
+
+## Decision
+
+An `agent` row holds instructions, tools, model, max steps and structured output. A Run Agent step
+stores only `agentId` and the server resolves the config at run start, so editing an agent changes the
+next run of every flow linking it with no republish. **Detach & customise** copies the config inline
+and clears `agentId` for steps that must differ.
+
+## Context
+
+Agreed 2026-08-12. The runtime already reads tools off the job payload rather than the flow version,
+and `flow_version.agentIds` + `extractAgentIds` survive from the deleted 2025 agents module — so a
+live reference costs no worker change and "which flows use this agent" comes free. Project scope
+follows the tools: flow tools resolve by `projectId` and connections match on
+`ArrayContains([projectId])`, so a platform-scoped agent would rebuild project scoping inside the row.
+
+## Why
+
+The point of naming an agent is to improve it once. A snapshot lets every flow drift independently and
+turns "make the agent better" back into per-flow editing — the pain the feature exists to remove. The
+rejected alternative was exactly that: dropdown prefills the step, link then gone.
+
+## Consequences
+
+- **It widens unattended authorisation.** Decision 000024 held that configuring a tool on an agent step
+  authorises it to run unattended; that authorisation now lives on the agent. We treat editing an agent
+  as editing a flow someone else published — `WRITE_AGENT` gates it, the edit is audit-logged, and the
+  editor shows "Used in N flows" before saving. If too weak in practice, the next move is a per-agent
+  "allow unattended writes" flag, not re-litigating the reference.
+- **Moving a flow between projects breaks the link** — the id is project-local, and git sync carries
+  connections but not agents. Hence `externalId` on the row from day one; project state must upsert
+  agents by `(projectId, externalId)`, and until it does the import must fail loudly.
+- Two orphaned tables (`agent`, `agent_run`) from the 2025 module are dropped so the entity can take
+  the obvious name. `breaking = true` for rollback safety, no `⛓️‍💥 breaking-change` label.
+- **Autonomy stays a flow** — "Run on a schedule" scaffolds a real flow with a linked step, so there is
+  one execution model and one observability path.
+- **Chatting with an agent is a third `AgentRunSource`**, not a nullable-`agentId` check: every gate
+  already branches on `source`, and it keeps agent conversations out of the Chat list for free. Unlike
+  a flow step it is **attended** — taint starts `false` and approval must not auto-decline.

brain/knowledge/engineering/server-module-anatomy.md

@@ -129,8 +129,11 @@ Verify with `npm run lint-dev` and `npm run test-api`.
 
 - **`getEntities()` and `getMigrations()` are both manual.** Nothing is auto-discovered. A missing entity registration fails silently at runtime; a missing migration registration means the migration simply never runs.
 - **The migration generator emits the wrong interface.** Every generated file must be patched from `MigrationInterface` to this repo's `Migration`, or CI rejects it. Never hand-write the SQL instead — generate from the entity diff, then patch.
+  - **Exception: the generator diffs against *your* database, so a surviving table of the same name produces an `ALTER`, not a `CREATE`.** Adding `AgentEntity` (table `agent`) emitted a mutation of the dead 2025 `agent` table — `DROP COLUMN systemPrompt`, then `ADD "iconKey" character varying NOT NULL` with no default, which fails on any table that has rows — and left `agent_run` untouched. When you are deliberately replacing an orphaned table, hand-write `DROP TABLE IF EXISTS … CASCADE` + `CREATE TABLE`, and check `pg_constraint` for FKs pointing at it first.
+- **`npm run check-migrations` can pass while your database is untouched.** It sources `.env.tests` (not `.env.dev`) and pipes `migration:run` to `/dev/null`, so it reported "No changes in database schema were found" against a database still holding the pre-migration table. Treat a green run as "the entity and *some* database agree", not as proof your migration executed. To actually verify, run `migration:run` against the dev DB with an explicit `AP_POSTGRES_HOST` and then inspect `information_schema.columns`, `pg_indexes` and `pg_constraint`.
+- **Migration timestamps collide across unmerged branches.** `migrations` is keyed by class name, so two branches can both claim `1824000000000` and only conflict at merge. Before picking a timestamp, check the applied ledger (`select name from migrations order by id desc limit 5`) as well as the files on `main` — a timestamp can already be in use by a branch you cannot see.
 - **PGlite has one connection, so `CONCURRENTLY` breaks it.** Guard on `system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITE` and issue a plain `CREATE INDEX` on that branch. When you do use `CONCURRENTLY`, set `transaction = false` on the migration class — PostgreSQL requires it outside a transaction.
-- **`UpdateResult.affected` is `undefined` on PGlite — never branch on it.** TypeORM's Postgres driver sets `affected` from `raw.rowCount`, and `typeorm-pglite` returns PGlite's `Results` (`{ rows, fields, affectedRows }`) with no `rowCount`. So the compare-and-set idiom `if (result.affected === 0) return null` is *always false* on PGlite and every predicate in the `WHERE` becomes decorative — the guard silently passes. This is not test-only: `AP_DB_TYPE=PGLITE` is the documented one-line Docker install (`docs/install/options/docker.mdx`). It hit MCP OAuth (`mcpOAuthCodeService.consume`), where it made authorization codes replayable, unbound to their client and redirect_uri, and immune to expiry. Use `.returning('*')` and test `updateResult.raw` for emptiness instead — that works on both drivers. Same idiom still live in `ee/agent/agent-rpc-handlers.ts` and `ee/projects/platform-project-service.ts`. Integration tests run on PGlite, so a test written against this idiom passes without proving anything.
+- **`UpdateResult.affected` is `undefined` on PGlite — never branch on it.** TypeORM's Postgres driver sets `affected` from `raw.rowCount`, and `typeorm-pglite` returns PGlite's `Results` (`{ rows, fields, affectedRows }`) with no `rowCount`. So the compare-and-set idiom `if (result.affected === 0) return null` is *always false* on PGlite and every predicate in the `WHERE` becomes decorative — the guard silently passes. This is not test-only: `AP_DB_TYPE=PGLITE` is the documented one-line Docker install (`docs/install/options/docker.mdx`). It hit MCP OAuth (`mcpOAuthCodeService.consume`), where it made authorization codes replayable, unbound to their client and redirect_uri, and immune to expiry. Use `.returning('*')` and test `updateResult.raw` for emptiness instead — that works on both drivers. Confirmed against the pinned `@electric-sql/pglite` 0.3.14: a plain `UPDATE` answers `{ rows, fields, affectedRows }` with **`rowCount: undefined`**, while the same statement with `RETURNING *` fills `rows` correctly (0 on no match, 1 on match). Note PGlite *does* report `affectedRows` — it is only `rowCount`, the field TypeORM reads, that is missing, so "PGlite loses the count" is the wrong mental model. The remaining call sites were converted in 2026-08 (`ee/agent/agent-rpc-handlers.ts`, `ee/projects/platform-project-service.ts`); a `.affected` that only feeds a log line was left alone. **Integration tests here run on Postgres** (`.env.tests` points at a real server), so they cannot catch this class at all — run the suite with `AP_DB_TYPE=PGLITE` prefixed to exercise it, which works today and is how the fix was proven red-to-green. Prefer `.returning('id')` over `.returning('*')`: on a table like `agent_conversation` the star form hauls the whole `messages` jsonb back on every write, and a row only has to be counted, not read.
 - **`breaking = true` is the rollback-safety flag, not the customer-facing one.** It marks destructive DDL (`DROP TABLE`/`DROP COLUMN`, `ADD ... NOT NULL` without a default) for `rollback-migrations.ts`. It does *not* by itself mean the PR needs the `⛓️‍💥 breaking-change` label — decide that from upgrade impact on self-hosters and API consumers.
 - **A new `AppSystemProp` needs three edits, not one.** Add the enum entry in `system-props.ts`, a default in `systemPropDefaultValues` (`system.ts`), *and* a validator in `systemPropValidators` (`system-validator.ts`). Miss the validator and `validateEnvPropsOnStartup` throws `systemPropValidators[prop] is not a function` at boot — every API test fails on setup, not just the new one. Document the var in `docs/install/reference/environment-variables.mdx` too.
 - **`permission: undefined` on `securityAccess.project(...)` silently allows any project member.** The argument is required in practice even though the type tolerates omitting it.

brain/knowledge/execution-runtime/index.md

@@ -41,7 +41,7 @@ The four calls a run emits to the app during execution: `updateRunProgress`, `up
 - **In-flight Run** — a worker is actively executing it; has a FlowRun row + checkpointed log in Postgres/S3, survives worker or Redis loss.
 
 ### ⚠️ Gotchas
-- **An agent tool's piece can go un-provisioned → `PieceNotFoundError` at runtime.** `extractAgentToolPieceRefs` (`flow-provisioning.ts`) strict-`safeParse`s each `agentTools` entry against `AgentPieceTool` and silently `return []`s on failure. `PredefinedInputsStructure` *requires* `fields`, but flow versions still carry the legacy flat `predefinedInput` (`{ auth, model, … }`) — those all fail to parse, so their pieces never get installed. The engine's `agentTools.tools()` does **no** validation and tolerates the legacy shape, so it happily tries to load the missing piece and the run dies `INTERNAL_ERROR` with an empty `failedStep`. Provisioning must not be stricter than the engine.
+- **A flow's sandbox never needs an agent tool's piece — do not re-add provisioning for it.** Since the agent step became a thin client (#14699, #14730) a configured piece tool runs outside the flow entirely: `agent-worker-tools.ts` → RPC `executePieceTool` → `piece-tool-runner.ts` → `flow-run-utils.ts` → `actionRunService` submits a **separate action run that resolves its own piece** from `pieceName@pieceVersion`. The flow bundle only ever needs `@activepieces/piece-ai`. `flow-provisioning.ts` used to scan `step.settings.input['agentTools']` and union the result into `resolvePieces` (`extractAgentToolPieceRefs`, deleted 2026-08); it was installing packages into a sandbox nothing loaded them from. The lesson it was written for still holds wherever a validate-then-provision pair exists: **provisioning must not be stricter than the engine.** It strict-`safeParse`d each entry against `AgentPieceTool` and silently `return []`ed on failure, while the engine tolerated the legacy flat `predefinedInput` shape — so pieces went un-provisioned and runs died `INTERNAL_ERROR` with an empty `failedStep`.
 - **A wrong Flow Bundle is sticky forever.** `parseManifest` only invalidates on `schemaVersion !== LATEST_FLOW_SCHEMA_VERSION`. A bundle published by buggy/older worker code stays "valid", keeps being served for that locked flow version, and short-circuits `resolvePieces` — so fixing the resolver code does **not** heal affected flows. Recovery is deleting the `FLOW_BUNDLE` file row (its id **is** the `flowVersionId`) + S3 object, or republishing the flow. Worth a bundle-format/generation field in the manifest.
 - **The piece-bundle CDN prefix moved, and the flag is off by default again.** `CDN_PIECES_URL` (`piece-bundle.ts`) points at `https://cdn.activepieces.com/pieces/bundled/` — a 2026-08-13 seeding of the *repackaged, self-contained* tarballs, anonymously readable (`200`). It replaces `pieces/retro/`, whose ~1735 objects all answered **`403 AccessDenied`** on both `cdn.activepieces.com` and the Spaces origin (object ACL, not the CDN); since `cdnBundleExists` counts only `2xx` as present, that tier silently bought nothing but a wasted `HEAD` per resolve. `AP_USE_CDN_FOR_BUNDLES` defaults to `false` — opt in per deployment. Two sharp edges survive the move: `release-pieces.yml` does **not** mirror to the bucket, so any version published after a seeding permanently misses; and `safeHttp.axios` sets no `timeout`, so an egress policy that blackholes the CDN hangs the existence check for the OS TCP connect timeout on the piece-install path instead of failing fast. Auditing a prefix means an **anonymous** `curl` against the exact URL the server builds — an authenticated `ls` proves only that the bytes exist. Verified end-to-end on staging 2026-08-13 with the flag on: 1745 objects / 746 pieces, anonymously listable and readable, and the tarball a worker caches at `cache/v14/common/pieces/<name>-<version>/bundle.tgz` is **byte-identical** (md5 == CDN ETag) to the public object and carries `src/bundle.cjs`. The seeding holds one version per minor line as of that date, so *latest* versions 404 and fall back to npm — the "published after a seeding permanently misses" edge is the common case, not the rare one.
 - **Turning `AP_USE_CDN_FOR_BUNDLES` on is a one-way door for every piece version resolved during the rollout.** The flag is per-app-container, and a rolling deploy runs flagged and unflagged containers side by side. An unflagged container that resolves a piece writes the **npm** tarball into `pieces/v2/`, and because `resolve()` checks S3 before the CDN that version is pinned to the unbundled copy permanently — it never re-resolves, so finishing the rollout does not heal it. Measured on staging with only *two* app containers (Aug 2026): `text-helper 0.5.1` came back as the 18 KB npm tarball (md5 `68334b5c…`) instead of the 396 KB CDN bundle (`fcdc62c9…`), while pieces resolved by the flagged container correctly logged `source:"cdn"`. Cloud prod is **35 app containers across 5 hosts**, so the window is far wider and lands on the hottest piece versions first. Deploying canary first surfaces it but does not avoid it; the only clean fixes are pre-seeding `pieces/v2/` from the CDN before flipping, or deleting the poisoned keys afterwards.

brain/knowledge/flows-execution/flow-runs.md

@@ -29,6 +29,7 @@ A Flow Run records one execution of a specific flow version, from trigger to ter
 - **Failed-trigger payload** survives past BullMQ job completion only because `buildFailedTriggerContext` writes it into the trigger step's `output` slot.
 - **The trigger step's status IS the raw-vs-extracted discriminator for retry** — there is no separate field (a `payload` field was tried and removed as redundant). `FAILED` means "`output` holds a raw event, re-run `run()` on it" (`executeTrigger: true`); `SUCCEEDED` means "`output` is already the trigger's result, replay as-is" (`executeTrigger: false`). So any code that *fabricates* a trigger step without the engine having run — the `QUOTA_EXCEEDED` admission gate is the first — must pick the status from where its payload came: raw for sync webhooks, extracted for anything sourced from the worker RPC `submitPayloads` (which passes post-`TriggerHookType.RUN` output). Get it wrong on a polling trigger and retry re-polls against an already-advanced `lastPoll` cursor, so the run gets `undefined` or an unrelated newer item *and* silently consumes those fresh items' own runs.
 - **Big step outputs**: over 32 KB inline → stored as a `LogSliceRef` pointer to a `FLOW_RUN_LOG_SLICE` file (`outputType === SLICE`); missing backing file throws `ENTITY_NOT_FOUND` (loud retry failure). Step *inputs* over 2 KB (`AP_FLOW_RUN_LOG_INPUT_TRUNCATE_THRESHOLD_KB`) become a display-only truncation placeholder.
+- **An INTERNAL_ERROR run does not record why it failed** — read the BullMQ job's `failedReason`, not the run. `reportFlowStatus` (`execute-flow.ts`) only forwards `logsFileId` when `data.logsFileId` is set, which is nil on every `BEGIN` run, so `engineRunCallbackService.uploadRunLog` skips persisting `internalError` into the log file. The run page and the log file both show nothing; the engine's actual error (plus its stderr) survives only in the job that `jobBroker.completeJob` moved to failed. On a dedicated worker group that job lives in `platform-<platformId>-jobs`, **not** `workerJobs` — pass `--queue` to `debug-failed-job.js` or it reports "job not found".
 - Retries only allowed on terminal states within `EXECUTION_DATA_RETENTION_DAYS`.
 - **Credit metering (Autumn)**: on terminal runs (paid editions), `onFinish` does two tryCatch-wrapped billing steps that never break run completion. (1) A PRODUCTION run not in `QUOTA_EXCEEDED` charges +1 apCredit via `billingProvider.trackCredits` with idempotency key `{runId}:run`. (2) `flowRunAiUsageTracker` pre-scans the flow version for `@activepieces/piece-ai` steps, extracts per-provider/model usage from step outputs (`flow-run-ai-usage-extractor` — recurses into loops, fetches `FLOW_RUN_LOG_SLICE` files, falls back to flow-version settings on `**REDACTED**` models), meters `Σ(messages × model credit weight) + toolCalls` to Autumn (`{runId}:ai`, plus `{runId}:appSumoAi` for the managed-ACTIVEPIECES AppSumo cap), then emits the `AI_USAGE_PER_RUN` PostHog event — the license key is only the PostHog distinctId, no longer a gate on metering.
 - **Credit gate is fail-open at admission**: the worker RPC `submitPayloads` checks `shouldBlockOnCredits` (blocks only when the platform is `billingEnforced` AND the cached balance is exhausted; CE default and Autumn-outage behavior is false). A blocked run is still admitted — as a `QUOTA_EXCEEDED` run with the trigger payload persisted in its log — so it stays retryable once credits return instead of being dropped. **`AP_EDITION=ee` skips the gate entirely** (`shouldBlockRunOnCredits` returns `false` before any provider call) so self-hosters pay no Redis/Autumn latency on admission — a temporary measure, see decision 000020.

packages/core/shared/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/shared",
-  "version": "0.131.0",
+  "version": "0.133.0",
   "type": "commonjs",
   "sideEffects": false,
   "main": "./dist/src/index.js",