Skip to content

fix(profiling): fall back when PY_UNWIND is not a valid local event - #20003

Closed
vlad-scherbich wants to merge 1 commit into
mainfrom
vlad/315-profiling-py-unwind
Closed

fix(profiling): fall back when PY_UNWIND is not a valid local event#20003
vlad-scherbich wants to merge 1 commit into
mainfrom
vlad/315-profiling-py-unwind

Conversation

@vlad-scherbich

@vlad-scherbich vlad-scherbich commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Description

Delta vs #19269: adds a local-event fallback for PY_UNWIND. When sys.monitoring.set_local_events rejects that event, the profiling setup retries without it; the global callback remains configured.

This PR does not add the native build or change the asyncio task-tracking path.

Layer Meaning Which PR
Compiled Native prerequisite #19269
Armed Local-event fallback this PR
Observable No standalone support claim

Testing

Risks

Additional Notes

Sibling of the other runtime compatibility fixes. changelog/no-changelog.

@vlad-scherbich vlad-scherbich added the changelog/no-changelog A changelog entry is not required for this PR. label Sep 1, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/monitoring.py                                          @DataDog/apm-core-python
tests/internal/test_monitoring.py                                       @DataDog/apm-core-python

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 1, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 1, 2026

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 230 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 230 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.internal.ci_visibility.recorder -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)
ddtrace.llmobs._integrations.bedrock -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.internal.ci_visibility.api._base -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)
ddtrace.internal.ci_visibility.filters -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Pipelines

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

Changelog | Validate changelog

View more details · View in GitHub Actions

No changelog detected for PR. Remove the 'changelog/no-changelog' label to proceed with validation.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 99d3e63 | Docs | View more details | Give us feedback!

@vlad-scherbich
vlad-scherbich force-pushed the vlad/ddtracepy-315-profiling-native branch from 4a90e1a to a3f34cd Compare September 2, 2026 16:09
@vlad-scherbich
vlad-scherbich force-pushed the vlad/315-profiling-py-unwind branch from 89aed97 to 1b91da5 Compare September 2, 2026 16:09
@vlad-scherbich
vlad-scherbich force-pushed the vlad/ddtracepy-315-profiling-native branch from a3f34cd to be2ca2c Compare September 2, 2026 19:10
@vlad-scherbich
vlad-scherbich force-pushed the vlad/315-profiling-py-unwind branch 3 times, most recently from 5f77393 to d05bf3d Compare September 2, 2026 19:34
@vlad-scherbich
vlad-scherbich force-pushed the vlad/ddtracepy-315-profiling-native branch from df5e2f7 to 6d624f8 Compare September 2, 2026 20:15
@vlad-scherbich
vlad-scherbich force-pushed the vlad/315-profiling-py-unwind branch 2 times, most recently from 438d4ce to 8a65502 Compare September 3, 2026 02:10
@vlad-scherbich
vlad-scherbich force-pushed the vlad/ddtracepy-315-profiling-native branch from 79b8faf to ddba58a Compare September 4, 2026 18:57
@vlad-scherbich
vlad-scherbich force-pushed the vlad/315-profiling-py-unwind branch from 8a65502 to 326a2cb Compare September 4, 2026 19:01
Base automatically changed from vlad/ddtracepy-315-profiling-native to main September 9, 2026 14:25
@vlad-scherbich
vlad-scherbich force-pushed the vlad/315-profiling-py-unwind branch from 326a2cb to ccb19c1 Compare September 9, 2026 20:09
Pre-rc1 CPython 3.15 raises ValueError for local PY_UNWIND. Strip-only
retry would leave unwind dead; fall back to set_events and recompute on
unregister so handlers still fire. rc1+ keeps local-only enablement.
@pr-commenter

pr-commenter Bot commented Sep 9, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-09-09 20:38:43

Comparing candidate commit ccb19c1 in PR branch vlad/315-profiling-py-unwind with baseline commit 93fed02 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 2 performance regressions! Performance is the same for 571 metrics, 10 unstable metrics, 7 known flaky benchmarks, 17 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:iastaspects-title_aspect

  • 🟥 execution_time [+62.949µs; +68.199µs] or [+23.571%; +25.537%]

scenario:samplingrules-high_match

  • 🟥 execution_time [+12.276µs; +14.263µs] or [+7.797%; +9.059%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-693.400ns; +769.937ns] or [-6.293%; +6.988%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-32.105ns; +33.682ns] or [-5.255%; +5.514%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1646.646ns; +1631.554ns] or [-9.668%; +9.580%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1190.426ns; +1251.171ns] or [-9.134%; +9.600%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-325.482ns; +323.618ns] or [-8.857%; +8.807%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-251.828ns; +252.700ns] or [-8.594%; +8.624%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-71.717ns; +78.175ns] or [-6.253%; +6.816%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-4042.631ns; +3980.386ns] or [-9.819%; +9.668%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-739.823ns; +805.659ns] or [-9.092%; +9.901%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-57.483µs; +53.550µs] or [-9.847%; +9.173%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+2.196µs; +2.398µs] or [+13.127%; +14.338%]

scenario:iastaspects-casefold_noaspect

  • 🟥 execution_time [+42.887µs; +46.358µs] or [+17.922%; +19.372%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+127.021µs; +134.417µs] or [+32.425%; +34.313%]

scenario:iastaspectssplit-rsplit_aspect

  • 🟥 execution_time [+17.816µs; +20.490µs] or [+12.807%; +14.729%]

scenario:span-start

  • 🟥 execution_time [+1.532ms; +1.698ms] or [+10.823%; +11.996%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+484.791ns; +534.859ns] or [+17.645%; +19.468%]

scenario:tracer-small

  • 🟥 execution_time [+37.769µs; +40.506µs] or [+11.584%; +12.423%]

Known flaky benchmarks without significant changes:

  • scenario:errortrackingflasksqli-baseline
  • scenario:flasksimple-iast-get
  • scenario:iastaspects-casefold_aspect
  • scenario:iastaspects-index_aspect
  • scenario:iastaspects-ljust_noaspect
  • scenario:iastaspects-lower_aspect
  • scenario:iastaspects-replace_aspect
  • scenario:iastaspects-rstrip_aspect
  • scenario:iastaspects-swapcase_aspect
  • scenario:iastaspects-title_noaspect
  • scenario:iastaspects-translate_aspect
  • scenario:iastaspects-translate_noaspect
  • scenario:iastaspects-upper_noaspect
  • scenario:packagespackageforrootmodulemapping-cache_off
  • scenario:packagespackageforrootmodulemapping-cache_on
  • scenario:sethttpmeta-all-enabled
  • scenario:telemetryaddmetric-record-100-metrics

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change alters low-level sys.monitoring event arming behavior (including global event enablement) in ways that could have non-obvious runtime and performance impact and merits final human review.

Pull request overview

Adds a runtime-compatibility fallback in ddtrace.internal.monitoring for CPython 3.15 pre-rc1 builds where PY_UNWIND is rejected as a local event, ensuring unwind callbacks still fire by enabling PY_UNWIND globally only when needed.

Changes:

  • Detects set_local_events(..., PY_UNWIND) rejection and falls back to enabling PY_UNWIND via set_events() while keeping other events local.
  • Adjusts _on_py_unwind behavior in fallback mode to avoid returning sticky DISABLE for unregistered code objects.
  • Updates tests to accept either local-only or global-fallback arming and adds an explicit regression test for the fallback path.
File summaries
File Description
ddtrace/internal/monitoring.py Implements the local-event probe and global-event fallback for PY_UNWIND, plus global enable/disable recomputation.
tests/internal/test_monitoring.py Adds fixtures/helpers and new assertions/tests to cover both local and global fallback arming behavior.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +336 to +340
needed: int = 0
for handlers in _registry.values():
if _events_for(handlers) & _E.PY_UNWIND:
needed = _E.PY_UNWIND
break
@vlad-scherbich

Copy link
Copy Markdown
Contributor Author

Revisit if becomes a problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants