feat: agent-lifetime heartbeat sender#32
Merged
Conversation
A daemon thread pings POST /v1/heartbeat from init() until process exit, answering the one question traces cannot: is this agent process alive right now? Spans export only when they finish, so an idle or crashed agent is otherwise indistinguishable from a healthy quiet one. Payload v1: instance_id (UUID per process lifetime), agent_name (defaults to service_name so the agents and traces views agree), sent_at, sdk_language/version, and the currently-open ROOT trace ids (capped at 32, with the true count) — this is what lets the backend derive running vs ready from the payload alone. An OpenRootSpanTracker span processor maintains the open set. Emission: first ping immediately at init (the agent appears without waiting an interval), then every heartbeat_interval seconds (default 15, clamped [5,300] — the backend derives staleness from the interval, so the bounds are wire contract). Graceful shutdown (client.shutdown() or atexit) sends a final stopped:true ping; no signal handlers are installed on purpose — a library must not own process signals, and silence after SIGKILL is exactly what the backend's stale->gone path exists for. Forked children re-arm with a fresh instance_id. Never raises into user code: 3s timeout, no retries or queueing (a late heartbeat is misinformation), one warning per process then debug. Off by default this release (heartbeat=True / GLASSFLOW_HEARTBEAT): the ingest path ships separately; flipping the default is a coordinated later change. heartbeat_transport is injectable for tests, like span_exporter; includes a real-subprocess atexit test and a real HTTP server test. Also sweeps internal ticket ids out of code comments (public repo).
Security/robustness review findings on the heartbeat: - The final stopped ping runs inside atexit: against a dead or hanging endpoint it could hold the user's process exit for up to ~6s (3s in-flight join + 3s final ping). The final ping now gets a 1s budget (a missed stopped ping just reports gone instead of stopped), and both timeouts are constructor-injectable; a hanging-socket test pins the exit-latency bound. - os.register_at_fork callbacks can never be unregistered, so the per-sender registration leaked a callback + sender reference for every init() in long-lived apps. One module-level fork hook now dispatches to a WeakSet registry; stopped senders leave it, and atexit.unregister releases the shutdown hook reference too. - Documented that TLS verification is urllib's default and deliberately not configurable in the transport. Reviewed and clean without changes: no secrets or API key in logs or error paths, no hostname/PID in the payload (privacy by design), payload size bounded by the 32-entry open_traces cap, uuid4 (random, no MAC derivation) for instance identity.
Nuzhnov
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the agent-lifetime heartbeat: a daemon thread pinging
POST /v1/heartbeatfrominit()until process exit, so the platform can tell a live-but-idle agent from a vanished one — the question traces structurally cannot answer (spans export only on finish).Design (per the heartbeat spec, payload v1)
instance_id(UUID per process lifetime),agent_name(defaults toservice_name),sent_at,sdk_language/sdk_version,open_traces(open ROOT trace ids, capped 32) +open_trace_count— lets the backend derive running vs ready from the payload alone. Workspace is never in the payload (backend derives it from the API key).heartbeat_interval(default 15s, clamped[5,300]— staleness is derived from the interval, so the bounds are wire contract).client.shutdown()/atexitsend a finalstopped: trueping. No signal handlers — a library must not own process signals; silence after an unhandled SIGTERM/SIGKILL is what the backend's stale→gone path covers.instance_id.heartbeat=True/GLASSFLOW_HEARTBEAT) until the ingest path ships; flipping the default is a coordinated later change.Testing
TDD; 24 new tests including a real subprocess proving the atexit stopped ping and a real local HTTP server exercising the default transport (auth header, 204).
heartbeat_transportis injectable likespan_exporter— no mocking.Also sweeps internal ticket ids out of code comments (public repo).