Skip to content

Show "Latest Outages" with Check Plugin Output in SLA Reports #1403

Description

@sva-ew

Is your feature request related to a problem? Please describe.

The current SLA reports (Host SLA, Service SLA) in Icinga DB Web show only the SLA percentage per host/service for the selected time range. When a service has an SLA below 100% (or below the threshold), there is no way to see why it was unavailable — the check plugin output (e.g. HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.012 second response time) is not shown.

Users migrating from Thruk/Icinga 1 reporting are used to a "Latest Outages" block that lists, for each outage:

  • Start time – End time (and duration)
  • The check plugin output at the time the outage started

This context is essential for operations teams to understand SLA violations and to communicate them to stakeholders in monthly/quarterly reports. A bare "99.96%" without the reason is of limited value when investigating incidents.

Describe the solution you'd like

Add an opt-in "Latest Outages" block to the SLA report, appended below the existing SLA table, that lists the most recent outages within the selected time range with:

Host Service Start End Duration State Output
host1 HTTP vHost [...] 2026-07-23 11:09:56 2026-07-23 11:27:49 0d 0h 17m 53s CRITICAL HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 237 bytes in 0.125 second response time

Proposed configuration options

  • "Show Latest Outages" (checkbox, default off) — opt-in to append the outages block to the report
  • "Max. Outages to Show" (number, default 20, max 100) — limit the number of outages shown (most recent first)

Outage definition

An outage is a hard state transition from OK (0) to any non-OK state (WARNING/CRITICAL/UNKNOWN). The outage ends at the next transition back to OK, or at the end of the selected time range if the outage is still ongoing. Transitions between non-OK states (e.g. WARNING → CRITICAL) are part of the same outage and do not start a new one; the original output is preserved.

Data source

The data is already available in the IcingaDB state_history table (output, long_output, hard_state, previous_hard_state, event_time columns). No schema change is required.

Describe alternatives you've considered

  • Manual investigation: Opening the history view for each service with SLA < 100% and scrolling through state changes. This is tedious and not feasible for monthly reports covering many services.
  • Custom external reporting: Building a separate tool that queries state_history directly. This duplicates effort and doesn't integrate with the Icinga Web 2 reporting framework (scheduling, PDF export, templates).
  • Thruk alongside Icinga DB Web: Running Thruk just for this report. This adds operational overhead and a second monitoring UI.

Additional context

  • The IcingaDB state_history table has an index on (environment_id, event_time) but not on (host_id, service_id, event_time). For large environments with long time ranges, the outage query should be filtered to only fetch outage-relevant transitions (hard_state > 0 AND previous_hard_state = 0 OR hard_state = 0 AND previous_hard_state > 0) rather than all hard state changes, to keep the result set manageable.
  • History retention may limit the availability of older outage data. For "Last Year" reports, outages older than the retention period will not be shown. This is a data constraint, not a code issue — the same applies to the SLA calculation itself.
  • The related (but different) icingaweb2-module-idoreports PR Fix incompatible hook usage with 3rd party modules #45 "Add report Host Outage" (open since Feb 2021) adds a cumulative outage-time column for the old IDO-based reporting, but does not show individual outage events or check output.

Mockup

┌─────────────────────────────────────────────────────────────────────────────┐
│ SLA Report: Service SLA                                                     │
│ ┌──────────────┬──────────────────────────┬─────────┬────────┐              │
│ │ Hostname     │ Service Name             │ Month   │ SLA in %│             │
│ ├──────────────┼──────────────────────────┼─────────┼────────┤              │
│ │ host1        │ HTTP vHost [checkbuy...] │ 2026-07 │ 99.96  │              │
│ │ ...          │ ...                      │ ...     │ ...    │              │
│ │ Total (180 Services)                    │         │ 99.87  │              │
│ └──────────────┴──────────────────────────┴─────────┴────────┘              │
│                                                                             │
│ Latest Outages                                                              │
│ ┌────────┬───────────┬─────────────────────┬─────────────────────┬─────────┬──────────┬──────────────────────────────────┐
│ │ Host   │ Service   │ Start               │ End                 │ Duration│ State    │ Output                           │
│ ├────────┼───────────┼─────────────────────┼─────────────────────┼─────────┼──────────┼──────────────────────────────────┤
│ │ host1  │ HTTP vHost│ 2026-07-23 11:09:56 │ 2026-07-23 11:27:49 │ 17m 53s │ CRITICAL │ HTTP CRITICAL: HTTP/1.1 500...   │
│ │ host2  │ ping4     │ 2026-07-15 08:02:11 │ 2026-07-15 08:05:44 │  3m 33s │ CRITICAL │ PING CRITICAL - Packet loss = ...│
│ └────────┴───────────┴─────────────────────┴─────────────────────┴─────────┴──────────┴──────────────────────────────────┘
└─────────────────────────────────────────────────────────────────────────────┘

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions