[pull] main from activepieces:main

brain/knowledge/decisions/000029-the-engine-never-imports-a-piece-a-fresh-child-process-does.md

@@ -0,0 +1,31 @@
+---
+status: accepted
+---
+
+# The engine never imports a piece, a fresh child process does
+
+## Decision
+
+Nothing in the engine process may `import()` a piece package. A piece is loaded only inside a child process spawned per call and killed when that call returns (`packages/server/engine/src/lib/core/piece/piece-child.ts`, shipped as its own esbuild entry `piece-child.js`). The parent talks to it with exactly two requests — `describe` (piece metadata as JSON plus the list of paths that are functions) and `call` (`['actions', 'send_http', 'run']` and its arguments) — over `piece-runner.ts`.
+
+## Context
+
+The engine is long-lived and served many operations, each `import()`ing pieces into the same process. Resident piece modules (and their duplicated `@activepieces/shared` copies) never came back — measured as hundreds of MB of a single engine heap. Loading a piece to read its metadata, or just to discover an auth `validate` hook does not exist, cost the same permanent memory as running it. Measured after the change with `smoke-test/verify-memory.sh` (webhook → data-mapper → return-response, 2000 runs): the engine ends **28 MB below** its warm baseline, i.e. V8 gives the heap back because nothing from the pieces stays resident.
+
+## Why
+
+Process exit is the only reliable way to free a required module graph; a cache or a `delete require.cache` does not free native handles or the transitive graph. Everything the engine needs about a piece is data (props, auth, trigger type, `contextInfo`), so it can cross a process boundary — only *behaviour* has to run where the piece is loaded. Rejected: keeping metadata loading in-process and isolating only `run` (metadata loading is what most operations do, so the leak would remain), and a persistent piece process per version (it re-creates the leak with extra lifecycle).
+
+The child is a real bundled engine entry, not an inline `--eval` script, because file materialization must live with the engine's own file processor: `ApStreamingFile.body` is a `Readable` and cannot be structured-cloned.
+
+## Consequences
+
+- **The piece's context cannot be proxied back to the engine — the child has to build it.** Two hard constraints kill any marker/RPC bridge, and both fail silently: (1) parts of the context are **synchronous by contract** — `CreateWaitpointResult.buildResumeUrl` returns a `string` and pieces call it without `await`, which an RPC can only answer with a Promise; (2) pieces **mutate objects after handing them to a hook** — `return-response-and-wait-for-next-webhook` passes `response` to `createWaitpoint` and only then writes the resume URL into its `headers`, and in-process the engine sees that through the shared reference. A snapshot does not. Since almost every context function is just HTTP over scalars (`apiUrl`, `engineToken`, `projectId`, `flowId`) the child builds them itself; only the collectors the engine reads afterwards (`hookResponse` tags/stop/respond/paused/responseToSend, trigger `listeners` and `scheduleOptions`) travel back, as plain data on the result.
+- `describe` costs one extra spawn per piece per engine process (memoized by `name@version`), so a 10-step flow on one piece is 11 spawns, not 20.
+- **A piece call costs ~79 ms** (spawn + import the piece + build the context + run + IPC), measured on a warm cache against the built child bundle. The benchmark flow's three calls show up as `RUN=400ms` in `FlowRun.timeline` with `PROVISION=0ms, BOOT=0ms`. That is the price of never letting a piece into the engine heap; if the sync-webhook path ever needs it back, the upgrade is one child per *flow run* instead of per *call* — the process still dies at the end of the run, so nothing accumulates, and an N-step flow pays one spawn instead of N.
+- `fileProcessor` returns a `__apFileSource` marker and the child calls `materializeFile`, so nothing is fetched until the piece actually runs — a validation failure now opens zero connections. The cost: an unreachable file URL fails *in the child when the step runs* rather than in the parent's prop validation (same message, later stage), because you cannot check a remote file without fetching it.
+- Piece metadata reaches the engine JSON-round-tripped, so any *function* on a property (dropdown `options`, dynamic `props`) is addressable only by path, never callable in-process.
+- A sandbox gets the engine by **file-by-file copy**, not by copying a directory: `engineInstaller` (`packages/server/sandbox`) copies each bundle into the cache dir that isolate mounts at `/root/common`. A new engine entry point must be added to that list or it is simply absent at runtime — the Docker image, which copies all of `dist/packages/engine`, looks perfectly fine and hides it.
+- The child is a second esbuild entry, so anything that builds it (including `vitest.config.ts`, which builds it for tests) must reuse the same `alias` map as `esbuild.config.mjs` — miss it and tests bundle `@activepieces/*` from `dist` while production bundles from `src`, so a green suite proves nothing about the shipped child.
+- **The child inherits the engine's node flags, and its OOM is detected rather than prevented.** `engineNodeArgs` sets `--max-old-space-size` as the *only* bound on engine memory (isolate passes no `--mem`/`--cg-mem`), so spawning the child without it would leave it on V8's default heap — spawn it with `[...process.execArgv, entry]`. Engine + child can then together reach `AP_SANDBOX_MEMORY_LIMIT`, and that is accepted: the worker is the sandbox, so it dies and restarts. What must not happen is the failure being anonymous — `piece-runner` classifies the child's exit the way `sandbox.ts` classifies the engine's (V8 heap message, exit 134, SIGABRT, SIGKILL) and raises a user-level `PieceMemoryLimitError`. A budget split between the two processes was tried and reverted: it bought little, and a floor on each share silently overshot the limit on small configurations.
+- The child inherits no in-process guards the parent installs (e.g. the SSRF monkeypatches in `network/ssrf-guard.ts`). Whatever must apply to piece code has to be installed in `src/piece-child.ts`.

brain/knowledge/execution-runtime/index.md

@@ -48,6 +48,8 @@ The four calls a run emits to the app during execution: `updateRunProgress`, `up
 - **The S3 piece-tarball cache shadows the CDN, so changing *what* gets cached means bumping `S3_PIECES_PREFIX`, not purging it.** `resolve()` (`piece-bundle.ts`) checks S3 before the CDN, so whatever `BUNDLE_PIECE` wrote wins for every later request. Until Aug 2026 that job cached the **npm** tarball, which for versions published before piece repackaging still declares its build-time deps — measured cost: 12 resident `@activepieces/shared` versions holding 388 MB of a 554 MB engine heap on cloud. The job now prefers the CDN artifact, but fixing the writer does not fix the objects already written, and *purging* them cannot work: a rolling deploy leaves old app instances writing npm tarballs back into the prefix for the rest of the rollout, and the purge has no way to know when the last one is gone. So the prefix is versioned (`pieces/` → `pieces/v2/`) — old code can only write the old prefix, so the new one is reachable only by a CDN-preferring writer. Same reflex as `LATEST_CACHE_VERSION` on the worker: when the meaning of a cached value changes, move the key; the abandoned prefix is dead storage to be swept later, never a correctness dependency.
 - **`extractConnectionIds` misses agent-tool connections.** It only reads step/trigger `settings.input.auth`, never `agentTools[].pieceMetadata.predefinedInput.auth`, so `flowVersion.connectionIds` under-reports and "which flows use this connection" lies.
 - **A code-sandbox `functions` entry must be a standalone declaration, never an object-method shorthand.** The v8 isolate re-injects each entry as source via `const ${key} = ${value.toString()}` (`v8-isolate-code-sandbox.ts`). A standalone `function flattenNestedKeys(...) {...}` (as exported from `script-evaluator.ts`) stringifies to a valid RHS and keeps recursion working by its inner name; an inline object-method shorthand stringifies to `flattenNestedKeys(...) {...}`, a syntax error as a `const` RHS. Keep it a standalone `function` export, never a method. For the same reason do **not** relocate a sandbox-injected function behind a separately-built package boundary (e.g. `@activepieces/core-utils`): its serialized `.toString()` would then depend on that package's build/minify config staying isolate-friendly. The trap: `no-op-code-sandbox.ts` passes the function by reference and tolerates either form, so a test run that skips the isolated-vm suite ships the bug green. Related: the `functions` **key** is also the global name users type in flow inputs (`{{flattenNestedKeys(...)}}`), so it is a public contract string, not an implementation detail. Keep it a hardcoded literal (matched by `FLATTEN_NESTED_KEYS_PATTERN` in `props-resolver.ts`); never derive it from the function's `.name`, which mangles under minification and would wrongly couple the token to the JS identifier.
+- **The piece context is lazier and more mutable than it reads.** Three traps when assembling it anywhere new (they all surfaced when context assembly moved into the piece child process, `core/piece/piece-context-builder.ts`): `project.externalId` is a **function the piece calls**, not a value — resolving it while building the context fires a `/v1/worker/project` request on *every step*; the backward-compatibility wrapper (`backwardCompatabilityContextUtils.makeActionContextBackwardCompatible`) must wrap the finished context or pieces on older context versions die with `ctx.run.pause is not a function`; and the legacy pause shim calls `createWaitpoint()` **without awaiting it**, so whoever owns the context has to drain in-flight hook work before the process ends or the waitpoint POST never lands and the run hangs until timeout.
+- **An error loses its friendly HTTP details the moment it crosses a process boundary.** `formatPieceError` (`friendly-piece-error.ts`) reads `error.response.{status,body}`, `error.status`, and falls back to `error.constructor.name` for `errorName` — but on `HttpError` (`pieces-common`) `response` is a **prototype getter** and `name` is plain `'Error'`. Structured clone, `{...e}`, and `JSON.stringify` all copy own enumerable props only, so a child-process runner that ships an error back verbatim silently drops `status`, `apiMessage`, and the error name, and the step renders as an opaque JSON blob. Serialize errors explicitly: read the getter keys by name (`response`, `request`, `status`, `headers`, `body`, `error`) plus own props, and carry `constructor.name` as `name`. Same trap applies to the run **result**: it must be JSON round-tripped, or an unresolved promise/function anywhere in the returned object throws `could not be cloned` from `process.send` and fails the step.
 - **A props-resolver script session is per-`resolve()`, never shared or hoisted.** `getPropsResolver(...).resolve(...)` builds a fresh `PropsResolver` per call, creates the script session via `scriptEvaluator.initSession()`, and disposes it in `resolve`'s `finally`, so an instance is single-use. Freshness is load-bearing: `setGlobal` is no-overwrite (`v8-isolate-code-sandbox.ts`) and injects each referenced step view once per resolve, so a session reused across resolves serves **stale step views** as flow state advances, and a reused instance would run on an already-disposed session. When refactoring props-resolver, capture `getStepView` and `scriptSession` inside `resolve` (they depend on the per-call `executionState`), not at instance scope, and never behind a shared mutable variable.
 
 ---

brain/knowledge/flows-execution/flows.md

@@ -19,6 +19,7 @@ Flows are the core automation primitive: a versioned directed graph of trigger +
 - Sample data is captured per step (input+output) as File entities per flow version.
 
 ### Gotchas
+- **Step settings autosave from inside the form resolver, on every validating `setValue`.** `step-settings/index.tsx` runs `applyOperation(UPDATE_ACTION/UPDATE_TRIGGER)` in its `resolver` whenever the new values differ from the last saved snapshot — it is *not* gated on `isDirty` or on a submit. Any transient value a component writes with `shouldValidate: true` is therefore persisted immediately, including one it intends to overwrite a moment later from an async response.
 - **A CODE step's compiled size is its `packageJson`, not its code — bundling *inlines* `node_modules`, it does not exclude it.** This gets assumed backwards a lot: there is no `node_modules` at runtime precisely *because* esbuild inlines every dependency into the step's `index.js`. Measured on cloud, Aug 2026: a step with **1,870 characters** of user source and `{"pdfkit":"0.14.0","aws-sdk":"2.1531.0","uuid":"9.0.1"}` compiles to **24.13 MB**, of which 24.13 MB is `node_modules` — **21.16 MB of it `aws-sdk` alone**. v2 of that SDK resolves its ~200 service clients by dynamic `require`, so esbuild cannot tree-shake it and inlines all of them; `@aws-sdk/client-*` (v3) would be a few hundred KB. Fleet-wide there were 13,918 compiled steps totalling 1.4 GB, **61 of them over 10 MB**, and per *flow* the totals reach **190.7 MB across 40 code steps**. That per-flow number is the one that matters operationally, because `flowBundleStore.publish` holds a flow's entire compiled output in memory at once (three copies — see the OOM gotcha on [[workers]]). To find the offenders: esbuild leaves `// node_modules/<pkg>/…` markers in the output, so you can attribute bytes per package by summing the lines between markers.
 - **Step output nesting (schema v21+)**: every step output is wrapped as `{ output, error? }`; expressions must use the `['output']` accessor. The v20→v21 migration rewrites existing expressions via `expression-rewriter`.
 - **Continue on Failure**: CODE/PIECE steps with `continueOnFailure.value: true` carry `onSuccess`/`onFailure` sub-trees under `settings.errorHandlingOptions.continueOnFailureBranches`.

brain/knowledge/pieces-engine/pieces.md

@@ -24,6 +24,9 @@ The metadata catalog of automation integrations ("pieces") — each a named inte
 - Install and sync also enqueue a tool-search reindex, but only when `isToolSearchEnabled()`; no-op otherwise.
 - `delete` removes all versions sharing the name on that platform, and only for `CUSTOM` pieces the caller owns.
 - **A piece silently vanishes from the list when its `minimumSupportedRelease` is ahead of the root `package.json` version.** `fetchLatestPieces` filters every piece through `isSupportedRelease(apVersionUtil.getCurrentRelease(), piece)`. Pieces are routinely merged targeting the *next* release, so on `main` a couple dozen are invisible locally until the version bump lands. No warning is logged — it just isn't there.
+- **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.
+- `DynamicPropertiesContext` tracks loading by property name only, so two in-flight requests for the same property let the first completion clear the flag for both — briefly re-enabling Test Step while the value is still cleared.
+- **The frontend `POST /v1/pieces/options` client only rejects for DYNAMIC.** `piecesApi.options` (`packages/web/src/features/pieces/api/`) catches DROPDOWN failures, toasts, and *resolves* with a disabled-dropdown fallback — so for dropdowns every error path wired onto that mutation is dead: `usePieceOptions`' `onError` handlers, its `retry: 1`, and the `if (error) throw error` into `DynamicPropertiesErrorBoundary`. DYNAMIC must rethrow: a swallowed failure arrives as a *successful* empty schema, which resets the property's children to defaults and gets persisted by step-settings autosave.
 - **`AP_DEV_PIECES` shadows the DB registry copy by name**, so a dev piece failing the release gate removes the piece *entirely* rather than falling back to the published version. Dropping the name from `AP_DEV_PIECES` (or bumping the local root `package.json`) brings it back.
 
 ### Key files

packages/core/execution/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/core-execution",
-  "version": "0.13.0",
+  "version": "0.14.0",
   "type": "commonjs",
   "main": "./dist/src/index.js",
   "scripts": {

packages/core/execution/src/lib/engine/execution-errors.ts

@@ -84,6 +84,16 @@ export class PausedFlowTimeoutError extends ExecutionError {
     }
 }
 
+export class PieceMemoryLimitError extends ExecutionError {
+    constructor(heapLimitMb: string | undefined, standardError?: string, cause?: unknown) {
+        super('PieceMemoryLimitError', JSON.stringify({
+            message: 'The piece ran out of memory',
+            heapLimitMb,
+            standardError,
+        }), ExecutionErrorType.USER, cause)
+    }
+}
+
 export class FileSizeError extends ExecutionError {
     constructor(currentFileSize: number, maximumSupportSize: number, cause?: unknown) {
         super('FileSizeError', JSON.stringify({

packages/core/shared/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/shared",
-  "version": "0.138.0",
+  "version": "0.138.1",
   "type": "commonjs",
   "sideEffects": false,
   "main": "./dist/src/index.js",

packages/core/shared/src/lib/automation/app-connection/app-connection.ts

@@ -149,14 +149,13 @@ export const AppConnectionOwners = z.object({
 })
 
 export type AppConnectionOwners = z.infer<typeof AppConnectionOwners>
-/**i.e props: {projectId: "123"} and value: "{{projectId}}" will return "123" */
 export const resolveValueFromProps = (props: Record<string, unknown> | undefined, value: string)=>{
     let resolvedScope = value
     if (!props) {
         return resolvedScope
     }
     Object.entries(props).forEach(([key, value]) => {
-        resolvedScope = resolvedScope.replace(`{${key}}`, String(value))
+        resolvedScope = resolvedScope.replaceAll(`{${key}}`, () => String(value))
     })
     return resolvedScope
 }