Skip to content

feat: add structured error logging for webhook retry exhaustion#594

Merged
Chucks1093 merged 3 commits into
accesslayerorg:mainfrom
dubemoyibe-star:feature/webhook-retry-exhaustion-logging
Jul 23, 2026
Merged

feat: add structured error logging for webhook retry exhaustion#594
Chucks1093 merged 3 commits into
accesslayerorg:mainfrom
dubemoyibe-star:feature/webhook-retry-exhaustion-logging

Conversation

@dubemoyibe-star

Copy link
Copy Markdown
Contributor

PR: Webhook Retry Exhaustion Error Logging

Summary

Updated the webhook delivery retry-exhaustion log path to emit a single, structured error-level log entry with full operator context. The log is emitted exactly once when a webhook exhausts all retry attempts and is flagged as failing. Individual retry failures continue to use the existing warn-level log.

Motivation

When a webhook exhausts all delivery retries and is flagged as failing, operators need enough structured context to identify the webhook and manually investigate the callback URL. The previous log entry was missing event_type and used attempt_number instead of total_attempts, and did not include a flagged_at timestamp. This made it harder to correlate with other system events and to determine when the webhook was suspended.

Changes

Modified Files

  • src/modules/webhooks/webhook.service.ts
  • src/modules/webhooks/webhook.service.test.ts

Behavior Changes

Before After
Log field attempt_number Log field total_attempts
No event_type in log event_type included
No flagged_at in log flagged_at included (ISO 8601 timestamp)
No validation that log emits exactly once Test asserts logger.error called exactly once
Callback URL absent from exhaustion log Callback URL remains absent (never added)

Structured Error Log Fields (on exhaustion)

Field Type Description
webhook_id string The webhook registration ID.
creator_id string The creator whose webhook failed.
event_type string The event type being delivered ("BUY" or "SELL").
total_attempts number Total number of delivery attempts made.
last_error_code string The last error message from the failed delivery.
flagged_at string ISO 8601 timestamp when the webhook was flagged as failing.

Callback URL Privacy

The callback URL never appears in the exhaustion log. The existing retry warn log continues to use maskWebhookUrl() to log only the origin (e.g., https://hooks.example.com).

Implementation Details

The change is confined to the attemptDelivery() function in webhook.service.ts. When retries are exhausted (attempt >= maxAttempts):

  1. The webhook record is updated (isFailing = true).
  2. All pending WebhookEvent records are marked as FAILED.
  3. A single logger.error() call emits the structured log with the fields listed above.
  4. The log is emitted exactly once per exhaustion event, after all retries have failed.

Individual retry failures continue to emit logger.warn() with backoff details and a masked callback URL.

Formatting

Ran pnpm run format to format the codebase in prettier style

Testing

Updated webhook.service.test.ts to assert:

  • logger.error is called exactly once.
  • The log object contains webhook_id, creator_id, event_type, total_attempts, last_error_code, and flagged_at.
  • total_attempts equals WEBHOOK_RETRY_MAX_ATTEMPTS.
  • flagged_at is a valid string (checked via expect.any(String)).
  • Callback URL is not present in the exhaustion log fields.

Verification

Run the targeted test:

pnpm test -- src/modules/webhooks/webhook.service.test.ts

Closes #587

@Chucks1093
Chucks1093 merged commit 3f0f041 into accesslayerorg:main Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add structured log for webhook flagged as failing after all retries exhausted

2 participants