Skip to content

feat: add PluginResult.executions structured control execution records - #141

Merged
araujof merged 3 commits into
0.1.xfrom
feat/execution-records-0.1.x
Jul 30, 2026
Merged

feat: add PluginResult.executions structured control execution records#141
araujof merged 3 commits into
0.1.xfrom
feat/execution-records-0.1.x

Conversation

@prakhar-singh1928

Copy link
Copy Markdown
Contributor

Summary

Adds PluginResult.executions — a list of structured ControlExecutionRecord objects returned on every invoke_hook call, giving consumers trusted per-plugin execution telemetry without parsing logs.

Closes: #130

Changes

  • Added ControlExecutionStatus enum (6 variants: completed, error, timeout, cancelled, skipped, disabled) and ControlExecutionRecord Pydantic model (13 fields) to cpex/framework/models.py
  • Added executions: list[ControlExecutionRecord] field to PluginResult — always present, empty when no plugins ran
  • Instrumented all five execution phases in PluginExecutor:
    • Sequential / Transform / Audit — monotonic per-plugin timing (duration_ns), derives matched / applied / effective_allow from result and mutation detection
    • Concurrent — record appended after each branch completes (duration_ns=0, no per-branch timing available)
    • Fire-and-forget — spawn-time record (status=completed, duration_ns=0); identify FAF records by mode == "fire_and_forget"
  • All identity fields (plugin_id, plugin_name, plugin_kind, mode) sourced from PluginRef.trusted_config — plugins cannot forge them
  • Security bounds: free-form strings capped at 256 bytes, config key lists capped at 64 entries, config values never stored
  • Exported ControlExecutionRecord and ControlExecutionStatus from cpex.framework
  • Updated CHANGELOG.md
  • Added 15 unit tests covering helpers, model defaults, serialisation, deny path, empty result, per-invocation isolation, and plugin_id hex format

Checks

  • make lint passes — note: ruff fails on max-statements-in-try in pyproject.toml; this is a pre-existing issue on 0.1.x unrelated to this PR
  • make test passes — 1962 passed, 0 failed
  • CHANGELOG updated

Notes

No breaking changes. PluginResult.executions is additive — existing CF call sites require zero changes. result.continue_processing and result.violation are unchanged.

Consumption pattern (CF side):

result, _ = await manager.invoke_hook("tool_pre_invoke", payload, global_context)

# Existing — unchanged
if not result.continue_processing:
    raise ToolDeniedError(result.violation)

# New — optional telemetry
for rec in result.executions:
    logger.info("control ran", plugin=rec.plugin_name, allowed=rec.effective_allow, duration_ns=rec.duration_ns)

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>
@araujof araujof self-assigned this Jul 29, 2026
@araujof araujof added this to CPEX Jul 29, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in CPEX Jul 29, 2026
@araujof araujof moved this from Backlog to In review in CPEX Jul 29, 2026
@araujof araujof added this to the 0.1.2 milestone Jul 29, 2026
@araujof araujof added the enhancement New feature or request label Jul 29, 2026

@araujof araujof left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The construction of the records object looks good. One blocking issue I found is that the instrumentation changed enforcement behavior.

Critical

The try/except around execute_plugin in _run_serial_phase (manager.py:642) swallows the PluginError raised for on_error=fail and returns continue_processing=True. That's a fail-open issue: a control that errors or times out now allows the request instead of blocking it — in the default enforcement mode. Record the error, then re-raise (the ignore/disable cases are already handled). Add a fail-closed regression test; there isn't one, which is how this slipped through.

Non-critical

  • Errors/timeouts are recorded as COMPLETED — the except asyncio.TimeoutError branch is dead, and a timeout under on_error=ignore looks like a clean allow.
  • "One record per plugin, always present" is false: disabled/skipped plugins and cancelled concurrent siblings get no record, so SKIPPED/DISABLED/CANCELLED are never emitted. Wire them up or fix the docs.
  • matched is None on a concurrent clean allow but False in serial phases — same outcome, contradicts the docstring.
  • Fire-and-forget records say COMPLETED at spawn and are never reconciled.
  • Tests only cover the sequential path; transform/audit/concurrent/FAF record-building is untested, and the docstring claims FAF coverage that isn't there.
  • _truncate exceeds its 256-byte cap by 3 (the ellipsis); the test was loosened to match.

Minor

  • config_keys/kind recomputed per plugin per hook though config is frozen — cache on PluginRef.
  • ControlExecutionRecord built at three drifting sites (already caused the matched bug) — extract a factory.
  • Cleanups: redundant except (PluginError, Exception), no-op requested_allow = x if x else False, a tautological test assertion, __all__ out of order.

Reusing the PluginResult to pass back the execution context (even though this could be passed in the metadata dictionary) is acceptable in 0.1.x. Let's revisit this in 0.2.x; I think we can leverage some of the work on security auditing that is being built in the Rust framework.

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>
Signed-off-by: Frederico Araujo <frederico.araujo@ibm.com>
@araujof

araujof commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Re-reviewed after 1a9baa9 — all earlier issues are addressed: fail-closed restored for on_error=fail, timeouts recorded as TIMEOUT, matched and _truncate fixed, record construction centralized, cleanups in, coverage across all five modes.

The fix did introduce one regression in the concurrent phase. execute_plugin now raises PluginTimeoutError for on_error=ignore/disable timeouts (it used to return continue_processing=True); the serial phase catches it but the concurrent as_completed loop did not, so a concurrent plugin timing out under on_error=ignore crashed the whole hook invocation instead of continuing.

Fixed in 9d6418e: the concurrent loop now catches it, records TIMEOUT, and continues — PluginError still propagates so fail stays fail-closed. Added a regression test; suite green.

Good to merge.

@araujof
araujof merged commit a636ce6 into 0.1.x Jul 30, 2026
@araujof
araujof deleted the feat/execution-records-0.1.x branch July 30, 2026 03:02
@github-project-automation github-project-automation Bot moved this from In review to Done in CPEX Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0.1.x enhancement New feature or request Python

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants