feat(telemetry): USDT probes for tracing spans end - #257
Open
jsitnicki wants to merge 2 commits into
Open
Conversation
TheJokr
reviewed
Aug 24, 2026
…ption Our code bases are already instrumented with tracing spans, but there is no easy way to attach a BPF tracer to a specific span's start/end and measure its duration: one would have to craft custom uprobes at specific code offsets, which shift with every build, and then match span start and end events to each other across await points and thread hops, since an async span can be suspended and resumed on a different thread. span_with_probe! (and its span_fn attribute counterpart, end_probe = true) emit a per-span USDT probe named span_end__<sanitized span name> which fires at span end with the span duration as an argument, so a tracer can attach to a single span by name while all other spans pay no cost. The probe is armed via a per-span semaphore only while a tracer is attached and works regardless of span sampling. The USDT provider defaults to the FOUNDATIONS_USDT_PROVIDER environment variable at compile time (settable per project via [env] in .cargo/config.toml), or "foundations" when unset, and can be overridden per call site with usdt_provider = "...".
Runs spans instrumented with span_with_probe! and span_fn's end_probe option in a loop, with distinct delay ranges so their durations can be told apart. Comes with a bpftrace script that attaches to the span end probes and prints duration histograms.
jsitnicki
force-pushed
the
jakub/PERF-2835-span-with-probe
branch
from
August 28, 2026 10:17
f84e8b1 to
7ec8942
Compare
Author
|
Per our discussion off-line, reworked the changes to support This called for a completely different approach than in Other changes:
Open question: Do we want to mark this as unstable and gate it be a crate What's next? As a follow-up I will work on
|
TheJokr
reviewed
Aug 28, 2026
| /// drops. Arming records the start timestamp unconditionally with respect | ||
| /// to sampling, so probes work even when span tracing is disabled. | ||
| #[doc(hidden)] | ||
| pub fn __arm_probe(&mut self, record_probe_start: bool, end_probe: fn(u64)) { |
Collaborator
There was a problem hiding this comment.
please make this #[inline(always)] to avoid the function call in the common (disabled) case. same for DualSpanScope::__arm_probe.
Author
There was a problem hiding this comment.
I should also just pull up the record_probe_start check into __arm_probe caller.
Collaborator
|
LGTM other than that 1 nit! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Code bases which use
foundationsare already instrumented with tracing spans, but there is no easy way to measure the duration of a specific span with a BPF-based tracer, likebpftraceorebpf_exporter. One would have to craft custom uprobes at code offsets that shift with every build, and match span start/end events across await points and thread hops.Solution
span_with_probe!("span::name")— a drop-in replacement fortracing::span("...")that additionally emits a per-span USDT probe at span end,span_end_<sanitized name>, receiving the span duration (ns) as an argument.Key features:
tracing::span. aarch64 support to follow.examples/span_with_probe, to show the intended use.