Skip to content

chore: wrapping context support for Python 3.15 - #17849

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 31 commits into
mainfrom
chore/315-wrapping-context
Aug 28, 2026
Merged

chore: wrapping context support for Python 3.15#17849
gh-worker-dd-mergequeue-cf854d[bot] merged 31 commits into
mainfrom
chore/315-wrapping-context

Conversation

@P403n1x87

@P403n1x87 P403n1x87 commented May 1, 2026

Copy link
Copy Markdown
Collaborator

prev: #19906 | next: #19910

Summary

Wrapping context + bytecode injection for 3.15. Delta vs #19906 only (monitoring + 3.15 assemblies).

Test plan

  • CI green on this branch
  • Stack merges cleanly into the next PR's base branch

Additional Notes

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented May 1, 2026

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

@P403n1x87 P403n1x87 added the changelog/no-changelog A changelog entry is not required for this PR. label May 1, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented May 1, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 2 jobs - 2 passed on retry View in Datadog

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7e34962 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented May 1, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-27 23:40:01

Comparing candidate commit 7e34962 in PR branch chore/315-wrapping-context with baseline commit 98304b0 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 7 performance regressions! Performance is the same for 579 metrics, 10 unstable metrics, 2 known flaky benchmarks, 16 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+2.453µs; +2.621µs] or [+14.784%; +15.799%]

scenario:iastaspects-add_aspect

  • 🟥 execution_time [+11.769µs; +13.842µs] or [+11.964%; +14.070%]

scenario:iastaspects-strip_aspect

  • 🟥 execution_time [+88.680µs; +93.167µs] or [+31.213%; +32.792%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+138.711µs; +146.635µs] or [+35.581%; +37.613%]

scenario:iastaspectssplit-rsplit_aspect

  • 🟥 execution_time [+38.210µs; +43.654µs] or [+27.055%; +30.909%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+621.161ns; +652.996ns] or [+23.087%; +24.270%]

scenario:tracer-small

  • 🟥 execution_time [+29.393µs; +31.478µs] or [+9.034%; +9.674%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-683.377ns; +780.795ns] or [-6.256%; +7.147%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-37.161ns; +29.100ns] or [-6.063%; +4.748%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1443.147ns; +1833.974ns] or [-8.576%; +10.899%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-811.921ns; +1718.374ns] or [-6.314%; +13.362%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-273.149ns; +372.135ns] or [-7.543%; +10.276%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-193.169ns; +313.524ns] or [-6.712%; +10.893%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-79.010ns; +67.963ns] or [-6.706%; +5.768%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-4207.281ns; +3727.764ns] or [-10.271%; +9.100%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-901.306ns; +662.155ns] or [-10.986%; +8.071%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-57.097µs; +60.310µs] or [-9.185%; +9.702%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:iastaspects-casefold_noaspect

  • 🟥 execution_time [+62.343µs; +67.407µs] or [+25.341%; +27.400%]

scenario:span-start

  • 🟥 execution_time [+1.272ms; +1.538ms] or [+9.165%; +11.085%]

Known flaky benchmarks without significant changes:

  • scenario:errortrackingflasksqli-baseline
  • scenario:flasksimple-iast-get
  • scenario:iastaspects-casefold_aspect
  • scenario:iastaspects-index_aspect
  • scenario:iastaspects-ljust_noaspect
  • scenario:iastaspects-lower_aspect
  • scenario:iastaspects-replace_aspect
  • scenario:iastaspects-swapcase_aspect
  • scenario:iastaspects-title_noaspect
  • scenario:iastaspects-translate_aspect
  • scenario:iastaspects-translate_noaspect
  • scenario:iastaspects-upper_noaspect
  • scenario:packagespackageforrootmodulemapping-cache_off
  • scenario:packagespackageforrootmodulemapping-cache_on
  • scenario:sethttpmeta-all-enabled
  • scenario:telemetryaddmetric-record-100-metrics

@vlad-scherbich

vlad-scherbich commented May 19, 2026

Copy link
Copy Markdown
Contributor

@P403n1x87 I've taken a look at the PR, and it makes sense to me for the most part. For the parts that I don't have much experience with, I've asked Claude for thoughts. Here are its findings, hopefully they will be helpful:

Must be fixed before merging

1. Hot-path callbacks iterate a mutable dict without synchronization (monitoring.py)

The callbacks (_on_py_start, _on_py_return, etc.) do:

entries = _registry.get(code)
# ...
for e in entries.values():
...

Meanwhile register() and unregister() mutate that same inner dict (entries[id(handler)] = entry / existing.pop(...)) while holding _registry_lock -- but the callbacks don't acquire the lock. If a register call adds or removes an entry between two iterations of the for loop (GIL can switch between next calls on the dict view), you get RuntimeError: dictionary changed size during iteration.

Fix options: (a) snapshot with list(entries.values()) in callbacks (one small allocation per event), (b) swap to a copy-on-write scheme where register/unregister replace the entire inner dict atomically (the reference assignment is GIL-atomic, so the callback always sees a consistent snapshot), or (c) hold the lock in callbacks (worst for latency).

Option (b) is the best tradeoff: zero allocation on the hot path, and register/unregister are cold.

Nice to have

2. Tool ID allocation starts at 0, competing with debuggers (monitoring.py)

for tid in range(6):
    try:
        sys.monitoring.use_tool_id(tid, "ddtrace")
...

IDs 0-2 are conventionally reserved (debugger, coverage, profiler). Starting from 0 means ddtrace could claim the debugger slot if no debugger is attached yet, then a later debugpy or pdb attach would fail to register. Starting from 3 (or iterating range(5, -1, -1) to prefer higher IDs) would be more neighborly.

3. _ENTER_FRAME_DEPTH = 3 is fragile (context.py)

_ENTER_FRAME_DEPTH = 3 if sys.version_info >= (3, 15) else 1

This assumes a fixed call-stack depth from the monitored function through the monitoring dispatch to enter. If CPython changes the monitoring callback invocation depth, or if the multiplexer adds/removes a level, this silently produces the wrong frame. Consider walking the stack looking for the code object that matches self.wrapped.code rather than assuming a depth.

@P403n1x87

Copy link
Copy Markdown
Collaborator Author

Awesome, thanks!

register() and unregister() mutate that same inner dict (entries[id(handler)] = entry / existing.pop(...)) while holding _registry_lock

The reasoning here was that it would be unlikely to have mutation while callbacks are invoked, because these are generally installed on enablement (boot). However RC might violate this assumption, so it won't cost us much to be a bit defensive here.

IDs 0-2 are conventionally reserved (debugger, coverage, profiler). Starting from 0 means ddtrace could claim the debugger slot if no debugger is attached yet, then a later debugpy or pdb attach would fail to register. Starting from 3 (or iterating range(5, -1, -1) to prefer higher IDs) would be more neighborly.

We are a debugger as a matter of fact 🙁 but I guess it doesn't matter where we start with the ID so we can just comply.

3. _ENTER_FRAME_DEPTH = 3 is fragile (context.py)

Deliberate choice. This value should be fixed for each Python release, so the cost is at most 1 update every year. Still much better than updating a whole bunch of opcodes 🙂

@gyuheon0h

This comment was marked as resolved.

Comment thread ddtrace/internal/monitoring.py Outdated
Comment thread ddtrace/appsec/_iast/_taint_tracking/aspects/helpers.h
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 7, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@vlad-scherbich
vlad-scherbich changed the base branch from main to ci/build-base-venvs-oom July 9, 2026 20:35
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 13, 2026
## Description

`build_base_venvs` is OOMKilled for cold-`ext_cache` Python versions. Currently this happens for `v3.15` only, which is new and has no warm cache. This was found while working on #17849.

**Root cause**
- It sets **no** `KUBERNETES_MEMORY_*` and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap **warm-cache** history.
- A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit.

### Changes

Pin CPU/memory and disable the VPA on `build_base_venvs`, matching what `.build_base`, `test sdist` already use for the same build targets:

```yaml
KUBERNETES_CPU_REQUEST: '6'
KUBERNETES_MEMORY_REQUEST: '10Gi'
KUBERNETES_MEMORY_LIMIT: '10Gi'
DD_DISABLE_VPA: 'true'
```

## Test plan

* [Next PR](#17849) passes [dd-gitlab/build_base_venvs: [3.15]](https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-py/builds/1846141002)

<img width="494" height="472" alt="Screenshot 2026-07-09 at 4 25 46 PM" src="https://github.com/user-attachments/assets/c622de21-a232-4f23-a085-728ad4461d25" />
 

## Additional Notes

* Warm builds are unaffected.
* The failing job logs also show the GitLab **runner S3 cache returning 403 AccessDenied**, so `ext_cache` restore fails on every run and forces cold compiles every time. That's a runner/`ddbuild` platform-side credential issue. Fixing this separately would restores warm-build speed.

Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
Base automatically changed from ci/build-base-venvs-oom to main July 13, 2026 20:51
@vlad-scherbich
vlad-scherbich changed the base branch from main to vlad/ci-auto-regen-requirements July 14, 2026 15:13
@vlad-scherbich
vlad-scherbich changed the base branch from vlad/ci-auto-regen-requirements to main July 14, 2026 15:26
Comment thread ddtrace/debugging/_function/store.py Outdated

@brettlangdon brettlangdon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits

Comment thread pyproject.toml Outdated
Comment thread mypy.ini
@P403n1x87

Copy link
Copy Markdown
Collaborator Author

/merge -m squash

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Aug 27, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-08-27 11:09:53 UTC ℹ️ Start processing command /merge -m squash


2026-08-27 11:10:01 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

  • Run /code blockers to see what is blocking it.
  • Run /remove to cancel it.

2026-08-27 15:44:18 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

@P403n1x87

Copy link
Copy Markdown
Collaborator Author

/merge -m squash

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Aug 28, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-08-28 09:49:09 UTC ℹ️ Start processing command /merge -m squash


2026-08-28 09:49:15 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).


2026-08-28 10:04:16 UTCMergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 2539cb9:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[3.15] Tracer wrapping 3.15

8 participants