feat(webhook): add optional X-Ray instrumentation for GitHub event delivery latency

[wadherv]

Description

The webhook Lambda (lambdas/functions/webhook/src/webhook/index.ts) is the entry point for every GitHub workflow_job event, but its X-Ray instrumentation only covers Lambda-internal and downstream-AWS-SDK timing. There is no visibility into the delivery/ingestion latency between when GitHub generated the event (workflow_job.created_at) and when the webhook Lambda actually began processing it — the gap could stem from GitHub's own delivery queue, network transit, API Gateway, or ingestion-side queuing/cold-start, and this instrumentation can't attribute it to either side by itself.

We hit this directly: a workflow run was triggered by GitHub at 23:28:59 but wasn't processed by our webhook Lambda until 23:37:15 — an ~8 minute gap — with no GitHub-side outage reported for that window. Diagnosing it required manually correlating the workflow run's GitHub timestamp against CloudWatch logs.

This PR adds instrumentGithubLatency(), called from readWorkflowJobEvent() (shared by both publishForRunners and publishOnEventBridge), which:

• Computes the lag between workflow_job.created_at and Date.now().
• Adds it as an X-Ray annotation (event_lag_ms) on the current segment via the shared tracer singleton (@aws-github-runner/aws-powertools-util), so it's queryable/alertable directly.
• Creates a remote-namespace subsegment named github, backdated to the event's created_at, so the X-Ray service map shows a distinct github node feeding into the webhook Lambda with a span representing the real end-to-end delay.

The entire feature is disabled by default and gated behind a new boolean, webhook_xray_github_latency_enabled (env var WEBHOOK_XRAY_GITHUB_LATENCY_ENABLED), since backdating a subsegment's start_time is an unusual X-Ray pattern not every consumer of the module will want on by default. When the flag is false, instrumentGithubLatency() returns immediately — no annotation, no subsegment, zero behavior change for existing consumers.

The new variable is threaded through the full config chain: root variables.tf/main.tf → modules/webhook → both modules/webhook/direct and modules/webhook/eventbridge submodules → the Lambda's environment block, and also through modules/multi-runner for consumers of that module.

Test Plan

• terraform fmt -check -recursive clean.
• terraform validate passing on the root module and modules/multi-runner (pre-existing, unrelated deprecation warnings only).
• Manually verified via a standalone test Lambda that X-Ray accepts a backdated subsegment (rather than rejecting/clipping it): trace Duration correctly rolled up to the full backdated span, IsPartial: False, no fault/error flags, and the subsegment was promoted to a standalone linked segment in the trace (what renders as a distinct node in the Service Map).
• Did not run the lambda's vitest/ESLint suite locally in this environment (no node_modules installed) — flagging for CI/reviewer to confirm.

Related Issues

Addresses a gap identified in incident investigation; no visibility currently exists into GitHub-to-webhook delivery latency. (Link to feature-request issue to be added.)

[edersonbrilhante]

@wadherv your PR is removing some tf resource that are needed. Can you check please

[wadherv]

@wadherv your PR is removing some tf resource that are needed. Can you check please

@edersonbrilhante issue is fixed, can you please review again

[edersonbrilhante]

I think this PR is addressing a broader observability problem with a feature-specific workaround.

workflow_job.created_at represents job creation time, not webhook delivery time. If GitHub redelivers the event, the timestamp remains unchanged, so the calculated value is job age, not delivery latency. GitHub confirms that the delivery GUID remains the same across redeliveries (documentation).

Please rename this value to workflow_job_age_ms and avoid event_lag_ms.

I also recommend removing WEBHOOK_XRAY_GITHUB_LATENCY_ENABLED and implementing reusable Middy middleware with the existing Powertools tracer. It should capture API Gateway ingress time, Lambda processing time, forwarding time, event type, and X-GitHub-Delivery.

The middleware should be composed at the Lambda handler boundary and exported with the wrapped handler, rather than being initialized as a side effect. This keeps the instrumentation separate from the webhook business logic and makes it reusable for other event types.

X-Ray should represent the actual Lambda invocation and downstream SQS/EventBridge calls. The synthetic backdated GitHub subsegment may produce misleading trace semantics.

[wadherv]

@edersonbrilhante updated the code to use Middy middleware