Skip to content

feat: add OpenTelemetry tracing export (OTLP/HTTP) - #245

Open
LauJosefsen wants to merge 29 commits into
mainfrom
lejo/otel-telemetry
Open

feat: add OpenTelemetry tracing export (OTLP/HTTP)#245
LauJosefsen wants to merge 29 commits into
mainfrom
lejo/otel-telemetry

Conversation

@LauJosefsen

@LauJosefsen LauJosefsen commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds optional OpenTelemetry instrumentation to gitte, exported via OTLP/HTTP to an OTLP-compatible backend (e.g. Elastic APM). The goal is to make failures on developer machines observable and debuggable — what command ran, what failed, and why — without manual reproduction.

What gets exported

Traces — each gitte invocation is one trace with a structured hierarchy:

gitte run                      (root span: command + args)
├── startup                    (phase span)
│   ├── startup.check <name>   (one span per check)
│   └── …
├── gitops                     (phase span)
│   ├── gitops.sync <repo>     (branch, commit SHA, dirty)
│   └── …
└── actions                    (phase span)
    ├── build                  (action span)
    │   ├── action.run <task>  (command, exit code, feature gates, injected env)
    │   └── …
    └── up                     (action span; starts after build completes)
        └── action.run <task>

Standalone commands (gitte startup / gitops / actions) emit the relevant phase span directly under the command root.

Task span attributes include gitte.task, gitte.project, gitte.command, gitte.exit_code, gitte.features (enabled, in-scope feature gates) and gitte.env (gitte-injected KEY=VALUE env — project env, env_when, feature-gate env; not the process environment). Errors are recorded on the relevant span.

Logs — action and startup command output is shipped as OTEL logs, correlated to the span that produced each line (stdout → INFO, stderr → WARN; [HINT] lines tagged gitte.hint=true). Gitops output is not logged (its outcomes live in span attributes).

Configuration

telemetry:
  endpoint: https://apm.example.com:8200
  headers:                                   # arbitrary export headers (auth, etc.)
    Authorization: "ApiKey <base64(id:key)>" # or "Bearer <secret-token>"

Resolution precedence: GITTE_TELEMETRY=off (kill-switch) > GITTE_TELEMETRY_URL > config endpoint > standard OTEL_EXPORTER_OTLP_* env vars. The token may instead be supplied per-developer via OTEL_EXPORTER_OTLP_HEADERS to keep it out of shared config.

Env var Effect
GITTE_TELEMETRY=off Disable all telemetry
GITTE_TELEMETRY_URL Override the endpoint
GITTE_TELEMETRY_LOGS=off Disable OTEL log export (keep traces)
GITTE_TELEMETRY_DEBUG=1 Surface export errors to stderr (otherwise swallowed)
OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS Standard OTEL fallback when no gitte endpoint is set

Design / reliability

  • Never blocks or slows gitte: degrades to a no-op when disabled or on setup failure; export errors are swallowed (visible only under GITTE_TELEMETRY_DEBUG); batch processors with bounded queues (logs drop on overflow); flush on exit is bounded to ~1s; AlwaysSample.
  • No execution-semantics change: spans/logs are a pure side-channel — ordering, parallelism, retries, output, and error returns are unchanged. Action spans are derived from the executor's existing task hooks (skipped tasks correctly ignored).
  • No PII beyond identifying context: user.name/host.name (to identify the developer/machine), repo name (never the full remote URL). The process environment is never exported.
  • HTTP exporter (no gRPC transport selection). Note: the OTLP exporter stack pulls in grpc/protobuf transitively regardless; binary grows ~+6 MB stripped.

Dependencies

OpenTelemetry traces stack at v1.44.0, logs stack at v0.20.0 (go.opentelemetry.io/otel{,/sdk,/log,/sdk/log} + the otlptracehttp/otlploghttp exporters).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds optional OpenTelemetry tracing to the gitte CLI, exporting spans via OTLP/HTTP when configured (or when standard OTEL_EXPORTER_OTLP_* env vars are set), and threads the root span context through gitops/actions execution paths.

Changes:

  • Introduces a new telemetry package to resolve config/env settings and initialize a global OTLP/HTTP tracer provider.
  • Instruments gitops.syncProject and actions.runGroupTask with spans/attributes and error recording.
  • Adds telemetry configuration to config.GitteConfig, documentation to README, and unit tests for resolution/init + attribute helpers.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
telemetry/telemetry.go Telemetry resolution + global tracer provider initialization + root-span helper
telemetry/telemetry_test.go Tests for precedence resolution and init/shutdown behavior
cmd/root.go Initializes telemetry and creates root span per CLI invocation; flushes on exit
gitops/gitops.go Adds gitops sync span and git context attributes (branch/SHA/dirty)
gitops/telemetry_test.go Tests git context attribute helper behavior
actions/runner.go Adds action run span, action context attributes, and exit-code attribute
actions/telemetry_test.go Tests action attribute helper behavior
config/types.go Adds telemetry block to config types
config/types_test.go Tests YAML unmarshal for telemetry config
README.md Documents telemetry usage, env vars, and precedence
go.mod Adds OpenTelemetry SDK + OTLP/HTTP exporter dependencies
go.sum Updates sums for new/transitive dependencies

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread telemetry/telemetry.go Outdated
Comment on lines +77 to +83
func Init(ctx context.Context, cfg *config.GitteConfig, version string) (func(), error) {
otel.SetErrorHandler(noopErrorHandler{})

r := Resolve(cfg)
if !r.Enabled {
return func() {}, nil
}
Comment thread cmd/root.go
Comment on lines +102 to +104
// finishTelemetry records the final command status on the root span and flushes
// pending spans. Safe to call when telemetry is disabled (handles are nil).
func finishTelemetry(err error) {
Comment thread actions/runner.go Outdated
Comment thread actions/runner.go Outdated
Comment on lines +279 to +281
ctx, span := telemetry.Tracer().Start(ctx, "action.run")
setActionAttrs(span, taskName, projName, strings.Join(cmds, " "))
defer func() {
Comment thread gitops/telemetry_test.go Outdated
Comment on lines +3 to +16
import (
"context"
"testing"

sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"go.opentelemetry.io/otel"
)

func TestSetGitContextAttrs(t *testing.T) {
exp := tracetest.NewInMemoryExporter()
tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(exp))
otel.SetTracerProvider(tp)
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
Comment thread actions/telemetry_test.go Outdated
Comment on lines +12 to +16
func TestSetActionAttrs(t *testing.T) {
exp := tracetest.NewInMemoryExporter()
tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(exp))
otel.SetTracerProvider(tp)
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
Comment thread README.md Outdated
Comment on lines +174 to +178
## Telemetry

Gitte can export OpenTelemetry traces to an OTLP/HTTP endpoint (e.g. Elastic
APM) to help debug failures. Traces capture the command run, per-repo git
context (branch, commit SHA, dirty state), and per-task outcomes with errors.
Comment thread go.mod
Comment on lines +54 to +57
google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
google.golang.org/grpc v1.81.1 // indirect
google.golang.org/protobuf v1.36.11 // indirect
Comment thread README.md
…ecorded data

- Guard getHeadSHA behind span.IsRecording() so it never execs when telemetry is off (was running git rev-parse on every repo every sync regardless)
- Shorten flush timeout 3s -> 1s so an unreachable endpoint adds at most 1s on exit
- Document that CLI args and action command lines are recorded; advise keeping secrets in env (never exported)
- Drop redundant gitte.repo double-set; drop always-nil error return from Init
- Set the global OTEL error handler only when telemetry is enabled
- Correct finishTelemetry doc comment (handles are nil only when uninitialized)
- Restore the previous tracer provider in gitops/actions attribute tests
- Document the telemetry config block in docs/config.md
@LauJosefsen
LauJosefsen force-pushed the lejo/otel-telemetry branch from a78dd95 to 0fa6e11 Compare July 13, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants