Looking for feedback: adds interceptor lifecycle hook

[skrawcz]

This proposes a new hook to allow one to remotely push execution of a Burr Action.

For example, if one wants to selectively push remote execution of an action to say Ray,
then one can now build an interceptor.

This also introduces companion hooks that would run pre & post on the remote to mirror
what Burr has today - -I thought it would be simpler to not try to reuse those hooks, and
instead make it explicit as to what is going on if you wanted to add something similar.

See tests for examples, but open to feedback here.

Note: we'd need to think through how this plays with parallelism functionality we have.
E.g. currently interceptors aren't propagated to sub applications...

Changes

• adds to internal hooks to intercept an action and remotely execute it
• adds ray example

How I tested this

• NEEDS EXTENSIVE TESTING
• Test sync
• Test async
• Test async streaming
• Test sync streaming
• I ran some ray examples
• need to write another interceptor for another backend, e.g. temporal to feel it out.

Notes

This change needs to have some thought and discussion.

Checklist

• PR has an informative and human-readable title (this will be pulled into the release notes)
• Changes are limited to a single goal (no scope creep)
• Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future TODOs are captured in comments
• Project documentation has been updated if adding/changing functionality.

[github-actions]

A preview of 105da95 is uploaded and can be seen here:

✨ https://burr.dagworks.io/pull/602 ✨

Changes may take a few minutes to propagate. Since this is a preview of production, content with draft: true will not be rendered. The source is here: https://github.com/apache/burr/tree/gh-pages/pull/602/

[goutamk09]

@skrawcz Thanks for opening this — I really like the direction of making remote execution explicit via lifecycle hooks.

Before diving into design feedback, I’m trying to fully understand the current execution flow vs the new interception path. A few things I’m exploring:

• Where in the action execution pipeline the new hook takes over execution
• How this behaves across sync / async / streaming paths
• Why tests are failing on 3.11 and 3.12 but not 3.9 / 3.10
• How this might interact with sub-applications and parallel execution (as noted in the PR)

I’ll follow up with more concrete observations once I’ve traced the execution path locally.

[goutamk09]

Hey @skrawcz
I’ve run the interceptor-specific integration tests locally on Python 3.11 and they are passing (sync, async, streaming, worker hooks).

Running the full test suite locally shows failures related to missing optional integration dependencies (redis, mongodb, asyncpg, langchain, etc.), not interceptor behavior.

From what I can see so far, the interceptor lifecycle implementation itself appears stable under 3.11.

I’m continuing to look into what might be causing the CI 3.11/3.12 failures — especially around async execution paths or example validation.

[goutamk09]

@skrawcz I ran the full core test suite locally under Python 3.11.

All execution, lifecycle, async, and persistence tests pass (264 passing).

The only failure is in test_graphviz_display, which requires the Graphviz system binary (dot) to be installed and available on PATH. This appears to be an environment/dependency issue rather than a behavioral regression in the interceptor or lifecycle implementation.

Given that all core execution paths pass under 3.11, I don’t see evidence of a Python-version-specific async or hook-related issue.

If CI failures are similar, it may be worth ensuring the Graphviz binary is installed in the CI environment or marking visualization tests as optional.

tests\core\test_state.py::test_register_field_serde_check_no_kwargs PASSED              [ 98%] 
tests\core\test_state.py::test_state_apply_keeps_typing_system PASSED                   [ 98%] 
tests\core\test_state.py::test_state_keys_returns_list PASSED                           [ 99%] 
tests\core\test_validation.py::test__assert_set PASSED                                  [ 99%] 
tests\core\test_validation.py::test__assert_set_unset PASSED                            [100%]

============================= 264 passed, 5 deselected in 8.63s ============================== 

[goutamk09]

Hey @skrawcz — quick update from my side:

I ran core tests locally on Python 3.11 and they pass (264 passed; only optional Graphviz dot env issue can fail if binary missing).

CI is currently failing for py3.11/py3.12 + validate-examples, but the logs expired, so we can’t see root cause.

Could you please re-run the failed jobs (or the full workflow) so we get fresh logs? Once logs are available, I’ll triage and propose a fix PR (workflow/env vs code).

[goutamk09]

@skrawcz — local test update (Windows, Python 3.12):

• Running python -m pytest -q from the repo root fails during collection because tests under examples/ expect local JSON fixtures (e.g. hypotheses_test_cases.json, prompt_for_more.json), and there’s also an import mismatch due to the duplicate test_application.py name in examples/ vs tests/.
• Running the main test suite with python -m pytest tests -q works after installing Graphviz (both the Python package and the dot system binary).
• Current local result: 444 passed, 3 skipped.

I did notice one PytestUnraisableExceptionWarning related to SQLitePersister.del, but it does not fail the suite.

If there are specific CI failures you’d like me to focus on, I’m happy to investigate further.

[skrawcz]

@goutamk09 sorry missed all these. will try to find some time this week to respond -- feel free to use the dev@ apache email list next time to discuss ...

[goutamk09]

Thanks @skrawcz — no worries at all. I’ll use the dev@ list next time for broader discussion.
I’ll keep an eye on the CI failures once logs are refreshed and happy to help if needed.

[goutamk09]

@skrawcz, I looked at the refreshed CI logs — failures are due to the new examples/remote-execution-ray not being covered in pyproject.toml [tool.flit.sdist] include/exclude, and validate-examples expects init.py + statemachine.png.
If you’re okay with it, I can open a small PR to:

• add examples/remote-execution-ray/init.py
• update pyproject.toml to exclude examples/remote-execution-ray/**
• and handle the statemachine.png requirement (either add it or update validator expectations) Let me know which direction you prefer for the PNG.

[github-actions]

This PR has had no activity for 94 days.
Closing to keep the PR list manageable.

Feel free to reopen when you are ready to continue.
If the branch has conflicts, a fresh PR against main might be easier.

[skrawcz]

Internal notes from own review:

Thanks for putting this up for feedback — the interceptor concept is a clean, opt-in way to push action execution to Ray/Temporal, and I like that it's genuinely vendor-neutral (core stays dependency-free, everything Ray/Temporal-specific is in examples/). Backwards compatibility is solid: all hooks/interceptors are additive, the private-function signature changes are defaulted, and the sync_hooks/async_hooks → property change is read-compatible (verified: 137 existing test_application.py tests still pass, and the 34 new tests pass).
Before this can move toward merge, a few things:

1. Correctness (blocking for merge): the synchronous execution paths (_run_function, _run_single_step_action, _run_single_step_streaming_action) don't guard against selecting an async interceptor the way the async paths guard against sync ones. Registering only an async interceptor and calling app.step() produces TypeError: argument of type 'coroutine' is not iterable (I reproduced it). Please mirror the inspect.iscoroutinefunction/isasyncgenfunction guard on the sync side and raise a clear error for the sync-vs-async mismatch. A test for this case is needed.
2. API design: the __INTERCEPTOR_NEW_STATE__ magic key is a smell for a public hook API — consider an explicit return contract before stabilizing.
3. Scope: the ~5000 lines of example markdown/notebook (excluded from validate_examples.py, so untested) is a lot for a WIP whose interface may still change. I'd split those out and land the hook + one validated example first.
   Also missing: error-path tests (interceptor raises / returns malformed result) and an explicit story (guard or test) for the parallelism/sub-application case you already flagged. Nice work on the decorator validation tests and the serde round-trip test.

---

Inline notes (line refs approximate — folded into body since this is a body-only draft):

• burr/core/application.py (~line 186, in _run_function) — [BLOCKING]
  The interceptor predicate here is lambda hook: hook.should_intercept(action=function) with no sync/async check, and interceptor.intercept_run(...) is called synchronously. If a user registered only an ActionExecutionInterceptorHookAsync, this calls a coroutine function without awaiting and later fails with TypeError: argument of type 'coroutine' is not iterable. Add a not inspect.iscoroutinefunction(hook.intercept_run) clause to the predicate (mirroring _arun_function), and raise a clear error if a sync step hits an async-only interceptor.
• burr/core/application.py (~line 355, in _run_single_step_action) — [BLOCKING]
  Same async-guard gap as _run_function. This is the exact spot the repro traceback lands (if "__INTERCEPTOR_NEW_STATE__" in result: on a coroutine). Guard the interceptor selection against async intercept_run.
• burr/core/application.py (~line 370, in _run_single_step_action) — [SUGGESTION]
  The __INTERCEPTOR_NEW_STATE__ sentinel key contract (result.pop(...)) is fragile for a public API. Prefer an explicit return type from intercept_run (e.g. (result_dict, Optional[State]) or a small dataclass) so the state hand-off isn't smuggled through a magic dict key.
• burr/lifecycle/internal.py (~line 315, in get_worker_adapter_set) — [NIT]
  This classifies "worker" adapters by crawling the MRO for a single SYNC_HOOK/ASYNC_HOOK attribute ending in _worker. Because base_hook sets exactly one such attribute per class, a class that mixes a _worker hook with a non-worker hook via multiple inheritance could be mis-detected. Low risk today, but a short comment documenting the assumption would help.
• burr/lifecycle/internal.py (~line 186, property change) — [NIT]
  Converting sync_hooks/async_hooks to read-only properties is fine for reads (verified against existing tests), but note it silently breaks any external code that assigned to these attributes. Worth a one-line changelog mention since LifecycleAdapterSet is semi-public.
• examples/validate_examples.py (line 47-48) — [SUGGESTION]
  Both new example dirs are added to the exclusion list, so the ~5000 lines of new example code/docs are never exercised by CI. Either make at least the core Ray example runnable-and-validated, or trim the speculative guides until the API is stable.
• tests/integration_tests/test_action_interceptor.py — [SUGGESTION]
  Great coverage of happy paths and "first wins." Please add: (a) an async-only interceptor invoked via sync app.step() (should error clearly, not TypeError), (b) an interceptor that raises, and (c) an interceptor returning a malformed/None result. These are the paths most likely to bite users.
• burr/lifecycle/base.py (PreStartStreamHookWorker / PostEndStreamHookWorker) — [NIT]
  These four stream-worker hooks have empty abstract bodies with no :param: docstrings, unlike the run-step worker hooks above them. Add matching param docs for consistency.