Skip to content

Cover OTLP export status in startup logs [dotnet@brian.marks/otlp-export-startup-log][nodejs@brian.marks/otlp-export-startup-log][python@brian.marks/otlp-export-startup-log][ruby@brian.marks/otlp-export-startup-log]#7376

Draft
bm1549 wants to merge 2 commits into
mainfrom
brian.marks/otlp-export-startup-log

Conversation

@bm1549

@bm1549 bm1549 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Motivation

Every tracer now reports three booleans in its startup diagnostic configuration that describe whether traces, metrics, and logs are exported over OTLP: otlp_traces_export_enabled, otlp_metrics_export_enabled, and otlp_logs_export_enabled. Nothing in system-tests covered them, so a regression in any tracer's startup diagnostic would go unnoticed.

Changes

Adds four parametric methods to Test_Startup_Logs in tests/parametric/test_startup_logs.py:

  • test_startup_logs_otlp_export_fields: with no OTLP env vars set, asserts all three fields are present, are real booleans, and default to false.
  • test_startup_logs_otlp_traces_export_enabled: asserts otlp_traces_export_enabled is true with OTEL_TRACES_EXPORTER=otlp.
  • test_startup_logs_otlp_metrics_export_enabled: asserts otlp_metrics_export_enabled is true with DD_METRICS_OTEL_ENABLED=true (plus DD_RUNTIME_METRICS_ENABLED=true, which .NET requires to flip the metrics field).
  • test_startup_logs_otlp_logs_export_enabled: asserts otlp_logs_export_enabled is true with DD_LOGS_OTEL_ENABLED=true.

The startup diagnostic is not serialized the same way across tracers: JSON with a - marker for Node.js, .NET, and Ruby; JSON with no separator for Go and Java; and a Python dict repr that is not valid JSON. A shared helper, _extract_otlp_export_fields, locates the config line by field name and reads each boolean with a quote-style- and casing-tolerant regex. It requires an unquoted boolean, so a string-encoded value would not match, which keeps the assertion that each field is emitted as a real boolean.

This PR also hardens _get_startup_logs. The previous one-shot container.logs() read the whole stderr in a single call. If a tracer emits an unbounded stderr stream (for example a logging feedback loop), that read chases a moving target and can hang the parametric suite. The read now streams the current snapshot (follow=False) and stops at the first of a line, byte, or wall-clock cap, each set far above any real tracer's startup output. Behavior is unchanged for a well-behaved tracer, and it warns rather than truncating silently when a cap is hit.

Manifest gating is added for the eight tracers that run these startup-log tests. Ruby's traces test is marked irrelevant (Ruby has no OTLP trace exporter), PHP incomplete_test_app, and Rust missing_feature, matching the existing startup-log entries.

Before merge

  • The manifest version floors are placeholders. They are set to current dev/snapshot versions and MUST be confirmed to the actual release versions before merge.
  • Java, Go, and PHP do not support the [lang@branch] override, so they build from their default branch. Their new-field tests may be red until the corresponding tracer changes merge and the manifest versions are set.
  • The [lang@branch] annotations in the title must be removed before merge. The "Fail if target branch is specified" guard job intentionally fails while an override is present.

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

Related PRs — cross-tracer OTLP startup-log effort

bm1549 and others added 2 commits July 23, 2026 19:35
Extend Test_Startup_Logs to assert the three OTLP-export booleans every
tracer now publishes in its startup diagnostic configuration:
otlp_traces_export_enabled, otlp_metrics_export_enabled,
otlp_logs_export_enabled.

Add a cross-language extraction helper that locates the config line by
field name and reads each boolean with a quote-style- and casing-tolerant
regex. This tolerates the per-tracer serialization differences: JSON with a
" - " marker (Node.js, .NET, Ruby "CORE" section), JSON with no separator
(Go, Java), and Python's dict repr (True/False, single quotes) which is not
valid JSON. Requiring an unquoted true/false/True/False keeps the assertion
that each field is a real boolean.

New tests:
- test_startup_logs_otlp_export_fields: all three present, boolean, and
  false by default (no OTLP env vars).
- test_startup_logs_otlp_traces_export_enabled: OTEL_TRACES_EXPORTER=otlp.
- test_startup_logs_otlp_metrics_export_enabled: DD_METRICS_OTEL_ENABLED=true
  (+ DD_RUNTIME_METRICS_ENABLED for .NET).
- test_startup_logs_otlp_logs_export_enabled: DD_LOGS_OTEL_ENABLED=true.

Manifest gating uses per-language placeholder next-version floors (the
tracer changes are unreleased). Ruby's traces test is marked irrelevant (no
OTLP trace exporter), PHP incomplete_test_app, Rust missing_feature,
consistent with the existing startup-log entries.

NOTE: version pins are placeholders that MUST be confirmed before merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…am can't hang

_get_startup_logs read the entire container stderr in one tail-less
container.logs() call. If a tracer misbehaves and emits an ever-growing
stderr stream (e.g. a logging feedback loop), that read chases a moving
target and never returns, hanging the whole parametric suite.

Make the non-.NET read bounded and defensive:
- Stream the current stderr snapshot (stream=True, follow=False), which
  self-terminates at end-of-snapshot in normal operation, so behavior is
  unchanged when the tracer is well-behaved.
- Cap the read by line count, byte count, and wall-clock time; the first
  cap reached stops the read and returns what was accumulated. This bounds
  both the runaway case and the no-marker "disabled" case.
- Accumulate raw byte chunks (docker log frames, not lines) and decode once
  at the end so a line split across chunks is handled, using
  errors="replace" for a multi-byte char truncated at a cap boundary.
- Close the stream in a finally so a mid-snapshot stop releases the socket.

Read the whole bounded snapshot rather than early-exiting at the first
startup-config marker: some tracers emit content the tests assert on after
that marker (Ruby logs "CONFIGURATION - CORE" at init, then
"CONFIGURATION - TRACING" and its "Agent Error" line only after a flush), so
stopping at the first marker would truncate content the OTLP-field and
diagnostic tests depend on. The startup logs are emitted early, so the caps
never truncate them in normal runs.

The .NET on-disk managed-log path and the stderr-only stream selection are
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bm1549 bm1549 added the ai-generated The pull request includes a significant amount of AI-generated code label Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/parametric/test_startup_logs.py                                   @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jul 24, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

🚦 17 Pipeline jobs failed

Testing the test | System Tests (cpp, dev) / parametric / parametric (2)   View in Datadog   GitHub Actions

🧪 4 Tests failed

tests.parametric.test_startup_logs.Test_Startup_Logs.test_startup_logs_otlp_export_fields[library_env0, parametric-cpp] from system_tests_suite   View in Datadog
  
assert None is not None

self = &lt;tests.parametric.test_startup_logs.Test_Startup_Logs object at 0x7fb1b8473a40&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb1b7e9c4a0&gt;

    @parametrize(
        &#34;library_env&#34;,
        [{&#34;DD_TRACE_STARTUP_LOGS&#34;: &#34;true&#34;}],
    )
...
tests.parametric.test_startup_logs.Test_Startup_Logs.test_startup_logs_otlp_logs_export_enabled[library_env0, parametric-cpp] from system_tests_suite   View in Datadog
  
assert None is not None

self = &lt;tests.parametric.test_startup_logs.Test_Startup_Logs object at 0x7fb1b81d3c20&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb1b7a0ed50&gt;

    @parametrize(
        &#34;library_env&#34;,
        [{&#34;DD_TRACE_STARTUP_LOGS&#34;: &#34;true&#34;, &#34;DD_LOGS_OTEL_ENABLED&#34;: &#34;true&#34;}],
    )
...
View all failed tests

Testing the test | System Tests (cpp, prod) / parametric / parametric (1)   View in Datadog   GitHub Actions

🧪 4 Tests failed

tests.parametric.test_startup_logs.Test_Startup_Logs.test_startup_logs_otlp_export_fields[library_env0, parametric-cpp] from system_tests_suite   View in Datadog
  
assert None is not None

self = &lt;tests.parametric.test_startup_logs.Test_Startup_Logs object at 0x7f55b4428e60&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f55b3c57980&gt;

    @parametrize(
        &#34;library_env&#34;,
        [{&#34;DD_TRACE_STARTUP_LOGS&#34;: &#34;true&#34;}],
    )
...
tests.parametric.test_startup_logs.Test_Startup_Logs.test_startup_logs_otlp_logs_export_enabled[library_env0, parametric-cpp] from system_tests_suite   View in Datadog
  
assert None is not None

self = &lt;tests.parametric.test_startup_logs.Test_Startup_Logs object at 0x7f55b4429a90&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f55b349c4a0&gt;

    @parametrize(
        &#34;library_env&#34;,
        [{&#34;DD_TRACE_STARTUP_LOGS&#34;: &#34;true&#34;, &#34;DD_LOGS_OTEL_ENABLED&#34;: &#34;true&#34;}],
    )
...
View all failed tests

Testing the test | System Tests (golang, dev) / parametric / parametric (2)   View in Datadog   GitHub Actions

🧪 4 Tests failed

tests.parametric.test_startup_logs.Test_Startup_Logs.test_startup_logs_otlp_export_fields[library_env0, parametric-golang] from system_tests_suite   View in Datadog
assert None is not None

self = &lt;tests.parametric.test_startup_logs.Test_Startup_Logs object at 0x7f8f30b97da0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f8f0099fb60&gt;

    @parametrize(
        &#34;library_env&#34;,
        [{&#34;DD_TRACE_STARTUP_LOGS&#34;: &#34;true&#34;}],
    )
    def test_startup_logs_otlp_export_fields(self, test_library: APMLibrary):
...
tests.parametric.test_startup_logs.Test_Startup_Logs.test_startup_logs_otlp_logs_export_enabled[library_env0, parametric-golang] from system_tests_suite   View in Datadog
assert None is not None

self = &lt;tests.parametric.test_startup_logs.Test_Startup_Logs object at 0x7fbab05ca720&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fbaaf9f5700&gt;

    @parametrize(
        &#34;library_env&#34;,
        [{&#34;DD_TRACE_STARTUP_LOGS&#34;: &#34;true&#34;, &#34;DD_LOGS_OTEL_ENABLED&#34;: &#34;true&#34;}],
    )
    def test_startup_logs_otlp_logs_export_enabled(self, test_library: APMLibrary):
...
View all failed tests

View all 17 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 3 jobs - 0 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4dede7f | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated The pull request includes a significant amount of AI-generated code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant