Expand framework and platform compatibility coverage

[anmarchenko]

What

• Fix Vitest 1.6 config-aware discovery by passing the argv shape expected by its parseCLI API and serializing empty CLI arguments as []. This prevents discovery from silently falling back to a generic filesystem glob and selecting the wrong test files.
• Extend the Mocha adapter to resolve both the .js CLI modules used by Mocha 8–11 and the renamed .cjs modules used by Mocha 12.
• Align DDTest's enforced Python minimum with the documented ddtrace>=4.11.0 requirement; 4.10.3 does not implement the discovery-mode environment contract DDTest uses.
• Move real-package integration tests into a dedicated internal/compatibility package with separate files per platform/framework, and expand CI across minimum, latest, and framework-specific boundary versions.
• Verify configuration-aware discovery, exact test metadata, setup preservation, and selected-test-only execution.

Why

The expanded end-to-end matrix exposed existing compatibility failures rather than test-only issues. Vitest 1.6's fallback could ignore project configuration and return the wrong file set, while Mocha 12 could not load the adapter after its private CLI modules changed extension. The Python implementation also still accepted ddtrace 4.10.3 even though the documented minimum and required discovery protocol begin at 4.11.0.

Keeping these fixes with the compatibility tests makes the advertised support boundaries executable and prevents Test Impact Analysis from planning or skipping the wrong tests.

E2E testing

1. Run make test and confirm all unit and integration packages pass.
2. Run make lint and confirm the repository reports no issues.
3. Run the Vitest compatibility test with 1.6.1 using a non-default --config file and confirm discovery returns both configured files while execution runs only the selected file.
4. Run the Mocha matrix from 8.4.0 through 12.0.0 and confirm configuration-aware discovery and selected-file execution pass.
5. Run every compatibility matrix entry in GitHub Actions and confirm all required checks pass.

[datadog-prod-us1-5]

 

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 66.67%
• Overall Coverage: 86.47% (+0.00%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1e81b6a | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T11:23:51.689242Z	a496890	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-prod-us1-5]

Vitest 1 discovery now removes the first custom command option. A command that starts with --config can select the wrong files or no files.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit a496890 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[anmarchenko]

@autotest review

[datadog-prod-us1-5]

More details

The Mocha code supports both module layouts. The Vitest and Python changes match the new compatibility limits.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 1e81b6a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[anmarchenko]

E2E Test Report: SUCCESS ✅

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local, playground-only testing with mockdog; existing CI checks were not repeated
• Revision: 1e81b6a01044e232f6b71edf52537e87f3d7d4cc
• PR branch: anmarchenko/framework-compatibility-tests

Results

Area	Playground validation	Result
Python / pytest	Flask: ddtest discovered and ran 22 files; current tracer completed 495 tests and all 40 mockdog assertions passed	✅
Python minimum tracer	Flask with ddtrace==4.11.0: 495 tests and all 40 mockdog assertions passed; the process returned nonzero only because that tracer/runtime combination emitted an unclosed-socket warning under Python 3.13 / pytest 9	✅ PR behavior; environment caveat
Mocha 12	Express: configuration-aware discovery found all 88 files and ddtest ran them in one isolated worker; 1,249 test events were captured	✅
Vitest	Vue core with the available Vitest 4.1.5: discovery found 182 files, split them across 3 workers (61/61/60), ran 3,558 passing tests, and all 33 assertions passed	✅

Issues Found

• Express had four test failures with Mocha 12.0.0. A direct run without ddtest reproduced the identical four failures, so they are baseline Express/Mocha/dd-trace behavior rather than a regression from this PR.
• ddtrace==4.11.0 emitted an unclosed-socket warning with Python 3.13 / pytest 9 after otherwise successful execution and telemetry assertions. The current tracer (4.15.0rc2) passed cleanly.
• The Vue playground currently provides Vitest 4.1.5, so this run validates the updated Vitest adapter in a real playground but does not directly exercise the Vitest 1.6 fallback path.

No PR-specific blocker was found.

Verification

• mockdog captured the expected test, suite, module, and session events, and all selected playground assertion scenarios passed.
• Datadog UI verification was not performed: these compatibility changes do not alter backend request shapes, and testing was intentionally scoped to local playgrounds.

Test Methodology

1. Injected the PR branch with --dep ddtest=anmarchenko/framework-compatibility-tests.
2. Ran each selected playground with --debug against mockdog.
3. Checked framework discovery, splitting, execution, event counts, and scenario assertions.
4. Reproduced anomalous Mocha failures in a direct non-ddtest baseline run to isolate their source.

---

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization