chore(ci): enable Python 3.15 support for dramatiq - #19909
Open
wantsui wants to merge 140 commits into
Open
Conversation
Codeowners resolved asResolved from the full PR diff against |
Dependency direction analysis
|
Circular import analysis
|
Contributor
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 096a1fc | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-09-04 14:08:49 Comparing candidate commit 096a1fc in PR branch Found 0 performance improvements and 6 performance regressions! Performance is the same for 579 metrics, 10 unstable metrics, 2 known flaky benchmarks, 16 flaky benchmarks without significant changes.
|
The release_pypi job uploaded every artifact matching pywheels/ddtrace-*, which includes the cp315 wheels that "build linux" produces for internal S3 and the private prerelease index. Nothing prevented a Python 3.15 artifact from reaching PyPI other than the Requires-Python bound baked into the wheel metadata, which is a property of the artifact rather than a release guard. Filter cp315 out of the upload list and refuse to upload at all if the resulting list is empty, so a broken filter fails the job instead of silently uploading everything or nothing. twine check --strict still runs over the whole of pywheels/, so cp315 artifacts remain validated. The S3 and adms prerelease paths are unchanged: those are the intended consumers.
…is uploaded The comment used DEV(py-315), a marker form this repo has never used; main has three TODO(py-315) anchors and only a bare "# DEV:" convention. Build the upload list before "twine check" so both steps see the same distributions. --strict promotes warnings to errors, so checking the unfiltered directory let a malformed cp315 wheel from an allow_failure build fail a release whose supported wheels were all fine. release_pypi_prod depends on "ddtrace package", which carries no serverless artifacts, so the narrowed check loses no coverage beyond cp315.
release_pypi_prod is gated on .is_release, so a PR pipeline never runs its shell and the filter added in the previous commit had no coverage. Pull the script block out of .gitlab/release.yml and run it against fixture directories with aws and uvx stubbed on PATH, so the assertions cannot drift from the shipped YAML. Pins three invariants: cp315 wheels are withheld while 3.15 is unsupported and everything else including the sdist still uploads; twine check and twine upload receive the same list, so a malformed wheel from the allow_failure cp315 build cannot fail a release it is not part of; and an empty list is a hard error rather than a silent no-op.
Test docstrings are never Sphinx-rendered, and AGENTS.md (L43-50) treats rST inline markup in them as editor noise. Replace the double-backtick literals in tests/internal/test_release_pypi_filter.py with plain text.
PR #19880 filtered cp315 out of the PyPI upload only, on the stated premise that the other destination was "internal S3 and the private prerelease index". That is wrong: s3://dd-trace-py-builds is anonymously readable and listable and is registered in DataDog/cloud-inventory as a "Public bucket to host dev builds of dd-trace-py from GitLab to share with customers". The pipeline republished cp315 wheels to main/ on 2026-08-27, so the leak was live, not historical. Three mechanisms publish a ddtrace wheel outside the pipeline: twine to PyPI, aws s3 cp to that bucket, and adms to pypi-private-prereleases. All three now call .gitlab/scripts/prune-unsupported-wheels.sh first, which holds the withheld ABI tags in one place. Putting it inside upload-wheels-to-s3.sh covers every S3 caller and every index suffix, including "upload serverless", which no filter reached before. The PyPI-side shell in release.yml reverts to the pre-filter form plus one prune call, per review feedback: with cp315 gone from the "ddtrace package" artifact, twine check no longer needs a hand-built file list. tests/internal/test_unsupported_wheel_pruning.py replaces the PyPI-specific test. It pins the prune behaviour and asserts that every file under .gitlab/ running a publish command also calls the prune script, so a new upload path cannot silently reintroduce cp315.
Bump sitecustomize runtime allow-list max to 3.16 (strict less-than) and add 3.15 to dl_wheels supported_versions. Does not change pyproject.toml requires-python — PyPI still blocks pip install on 3.15 until the capstone packaging PR lands. Skip mypy/bandit pre-commit on lib-injection/ (untyped deployment scripts, outside ddtrace/). Closes #17813
Wire download_dependency_wheels for PYTHON_VERSION=3.15 using the 3.15.0rc1 mirror image tag (GA python:3.15.0 not published yet). Mark the matrix leg allow_failure until cp315 build linux wheels are stable. Completes the packaging path Copilot flagged: sitecustomize allow-list + dl_wheels CLI + CI artifact population.
Root cause: stale manylinux2014 derived images (d2b8243 / May 2026) fail cp315 wheel builds while rebuilt musllinux images (233089d) succeed, leaving S3 with musllinux-only cp315 wheels that prof-python-3.15 glibc images cannot install. - Route cp315 manylinux2014 builds through the pypa 2026.05.13-1 mirror until derived manylinux images are rebuilt - Split the build-linux matrix so cp315 skips the stale derived manylinux tag - Lift requires-python to <3.16 and bytecode>=0.18.1 for 3.15 (PyPI 3.15 classifier deferred until the package is PyPI-ready) - Add cp315 to ddtrace package validation and drop cp315 allow_failure
The dd mirror pypa tag 2026.05.13-1 still ships cp315 as 3.15.0b1 and the native build SIGSEGVs; use quay.io 2026.08.24-1 until the mirror catches up. Restore musllinux cp315 allow_failure, sync requirements.csv, and add a reno.
GitLab third-party-registry policy blocks quay.io. Revert to the mirrored pypa manylinux2014 2026.05.13-1 image and rebuild cp315 as 3.15.0rc1 in before_script because the mirror still ships 3.15.0b1, which SIGSEGVs during the native wheel build. Drop ensure-cp315-rc1.sh once DataDog/images mirrors >= 2026.08.04-1.
The runtime manylinux2014 image ships build-cpython.sh but not cosign, which is only copied into pypa's intermediate build_cpython stage.
finalize-one.sh bootstraps pip for the newly built interpreter by running /usr/local/bin/cpython3.15, a shim that execs /opt/python/cp315-cp315/bin/python, and then ends with a plain "ln -s" into /opt/python. Deleting that directory up front broke the pip bootstrap; leaving it in place would break the symlink step. Free only the /opt/python name, repoint the shim at the stale prefix for the bootstrap, and assert the result is rc/final so a wheel can never be built against 3.15.0b1. Also install the extra manylinux build deps best-effort: EPEL-7 aarch64 has no libzstd-devel, and zstd-sys vendors its own zstd source, so a missing package must not abort the job. Includes temporary stdout/NDJSON instrumentation for CI diagnosis.
"ddtrace package" demanded cp315 macOS and Windows wheels that no job produces, because cp315 went into PYTHON_TAGS and that list is a full cross-product over BASE_PLATFORMS. Require cp315 only on manylinux, and tolerate musllinux cp315 either way since it is still allow_failure: requiring it would have let a best-effort build fail the pipeline. Also drop the debug scaffolding this investigation left behind. setup.py is back to its upstream state, and the cp315 rebuild scripts keep the comments explaining the missing headers in the runtime image without the NDJSON probes. The now-unreferenced AARCH64_IMAGES and X86_64_IMAGES anchors go too.
openssl-devel and openssl-static were never doing anything here. manylinux's own install-build-packages.sh already lists openssl-devel in its yum COMPILE_DEPS, and build-openssl.sh then calls manylinux_pkg_remove openssl-devel before building OpenSSL into /opt/_internal/openssl-3.5 -- so we installed it twice and it was uninstalled either way. Nothing links the system openssl. src/native/Cargo.lock has no openssl-sys or native-tls; TLS is rustls with the ring backend, and the only openssl-named crate is openssl-probe, which is pure Rust CA-path discovery. CPython links the image's own /opt/_internal/openssl-*, which ensure-cp315-rc1.sh restores headers for explicitly. Split from the autotools/libzstd removal so a revert can be partial.
With openssl gone the script installed autoconf, automake, libtool and libzstd, and the image already provides all of them. manylinux builds autoconf, automake and libtool from source at image build time and copies the result into / (install-autoconf.sh, install-automake.sh and install-libtool.sh all end in `cp -rlf /manylinux-rootfs/* /`), so the yum packages were shadowed by newer versions already on PATH. libzstd is not needed either: the dd-trace-py manylinux2014_aarch64 Dockerfile never installed it, and aarch64 manylinux wheels have been building green all along. zstd reaches the native crate transitively via zstd-sys, which compiles its own vendored C source. Epistemic note: the openssl removal in the previous commit is verified from upstream script contents, and the autotools claim is verified from those install scripts. libzstd resting on "aarch64 was always green" is the weakest link here, so if this commit is what breaks CI, libzstd is the first thing to look at.
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
3.3.0 is the first *stable* release with compiled cp315 wheels; 3.3.0b1 shipped them too, but uv skips pre-releases by default. Also attribute the segfault to the wheel being built against the 3.15.0rc1 PyThreadState layout rather than implying 3.15.0b1 is broken, scope the pure-Python fallback to the manylinux2014 images it actually applies to, and name the pypa manylinux tag that first ships 3.15.0rc1 so the exit condition is checkable.
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1 (pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so rather than letting rc1 read as the version that introduced it. The wheel is still described as built against the rc1 layout, and the exit condition stays at >= 3.15.0rc1 deliberately. Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
requires-python widens from <3.15 to <3.16, which ships in published package metadata: pip on Python 3.15 stops refusing the install and starts attempting a source build. Add a release note so that lands with notice rather than as a silent change in failure mode. Also correct the validator comment on musllinux cp315. The package.yml allow_failure rule keys on PYTHON_TAG alone, so it covers manylinux cp315 identically and cannot be the reason musllinux is optional; the real reason is a policy choice about which platforms must produce cp315 wheels. Make the Phase 4 wheel-count breakdown add up to the expected total, and record what the bytecode 0.18.1 floor actually is.
…eels Widening requires-python to <3.16 advertises 3.15 support we do not have, and it is the only thing standing between users on 3.15 and the cp315 wheels already published to PyPI (4.14.0 shipped two, 4.13.0 shipped four). 279b0ba (#18094) reverted this same line for the same reason. Nothing in this PR needs it: uv build ignores requires-python, and the one cp315 install in CI is `uv pip install` on a direct wheel path, which does not enforce Requires-Python. The release note went with it, since it had no other subject. The validator no longer requires cp315 anywhere. Every cp315 leg of "build linux" is allow_failure, while "ddtrace package" is not and gates the release stage, so requiring a cp315 manylinux wheel turned a best-effort build into a release blocker. 4.14.0 shipped cp315 musllinux but no cp315 manylinux and would have been blocked by it.
The empty cp315 required-wheel list is coupled to the cp315 allow_failure rule in .gitlab/package.yml: both must flip together, and #17816 owns that flip.
This PR is the logging get_version layer; the list() snapshot is unrelated.
wantsui
changed the title
wantsui
marked this pull request as ready for review
September 3, 2026 21:34
wantsui
requested review from
a team,
crysmags,
dubloom,
juanjux and
vlad-scherbich
and removed request for
a team
September 3, 2026 21:34
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a22032800b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
The Dramatiq matrix replaces Python 3.9 through 3.14 with Python 3.15. This removes existing CI coverage instead of extending it.
🤖 Datadog Autotest · Commit a220328 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Address review feedback: replacing pys=["3.15"] with a singleton dropped CI coverage for the existing 3.10-3.14 dramatiq matrix (and left 3.9 testing only the pinned 1.10.0 package). Extend the existing selection instead of replacing it, and regenerate the pruned lockfiles. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
wantsui
marked this pull request as draft
September 4, 2026 02:29
Regenerating the dramatiq lockfiles picked up dramatiq 2.2.1 (previously 2.2.0), which the CI registry-sync check flags as out of date. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vlad-scherbich
force-pushed
the
vlad/315-peripheral-compat
branch
from
September 4, 2026 04:17
b29cf07 to
2483780
Compare
logging.__version__ is deprecated on 3.15+ with no replacement API. Match stdlib integrations (unittest, subprocess, urllib) instead of silencing the warning.
vlad-scherbich
force-pushed
the
vlad/315-peripheral-compat
branch
from
September 4, 2026 04:24
2483780 to
3e0f7dc
Compare
vlad-scherbich
force-pushed
the
vlad/315-peripheral-compat
branch
from
September 4, 2026 13:34
3e0f7dc to
c097711
Compare
wantsui
marked this pull request as ready for review
September 4, 2026 14:03
Contributor
There was a problem hiding this comment.
More details
The Python 3.15 Dramatiq settings match the generated lock file and support data. The coverage code selects the static import path because accurate import hooks are not available on Python 3.15.
🤖 Datadog Autotest · Commit 096a1fc · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Builds on #19879, which already enabled Dramatiq testing on Python 3.14 and added the
17f991elockfile. This PR will unblock the integrations CI for dramatiq (and later other integrations).This PR enables the next runtime only:
Related PRs:
ci(testrunner): install pip-tools for Python 3.15 #19977
ci: update testrunner image #19984
Testing
scripts/lint fmt -- riotfile.pyscripts/lint riotscripts/lint checksscripts/run-tests --list tests/contrib/dramatiq/confirms the existing Python 3.14 venv17f991eand new Python 3.15 venv41125444112544with Riot under Python 3.15; the lockfile was unchangedRisks
Low integration risk because this changes only Dramatiq Riot coverage and generated metadata. Python 3.15 remains enabled per integration while the global matrix stays capped at 3.14.
Additional Notes
vlad/315-peripheral-compat.PIP_IGNORE_REQUIRES_PYTHON=1override because the package metadata still declaresRequires-Python <3.15.Historical CI error for 3.15
Issue 1: Injection Assembly errors (fixed in this PR)
There's 174 failures at the moment:
Based on https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1995599108 , we can't use INJECTION_ASSEMBLY
Issue 2: Lock files
Regenerated after the pip-tools PR.