Record only what CRAB says about a job, keyed by job id

[kandrosov]

What went wrong

The Run3_2023BPix production is failing 565 of 4800 jobs, and 551 of those are two single broken worker nodes: comp-g-001.local at T2_EE_Estonia (311 of 480 jobs sent there) and node38-4.wn.iihe.ac.be at T2_BE_IIHE (240 of 628). The Estonia node is the same host that failed 258 jobs on 2026-08-27, and it fails the same way — /usr/bin/base64: Input/output error, so law's _render.py comes out truncated, SyntaxError: invalid syntax, input file rendering failed, job exit code 5, 23 seconds in. Nothing to fix in the production chain: those jobs are retried and succeed elsewhere.

The site quarantine exists precisely to stop that, and it did not fire. The record it judges from said this:

T2_CH_CERN     1773 events  1773 failed
T2_UK_London_IC 342 events   342 failed
T1_DE_KIT      1196 events  1188 failed
...
T2_EE_Estonia   342 events   342 failed     quarantined = no

Every site at ~100 % failure, while 4235 jobs were running happily. 8285 of those "failures" were recorded in a single poll on 08-31 08:00 — the moment a resumed run flipped all 8300 RunProd jobs of the previous era to retry with initially missing task outputs (#20). Each was counted against whatever site the job had last run at. A killed task does the same: its jobs come back failed, and crab kill was used repeatedly that day.

With the baseline at ~100 % everywhere, a site failing 65 % of its jobs looks better than average, so rate >= relative_factor * baseline can never hold and nothing is ever quarantined. The protection was inert exactly when it was needed.

The fix

Harvest from the CRAB status response in the job manager, instead of from job_data after the poll. Two problems disappear at once:

• law's bookkeeping cannot reach the record. initially missing task outputs is a status the proxy assigns after the query; a status response only ever carries what happened to the job. A job that ended without a job-level error code (Error absent — killed, or never started) is skipped too, so crab kill no longer counts against a site.
• the site can no longer land on the wrong job. law syncs per-job extra onto job_data positionally (law/workflow/remote.py), so with more than one live CRAB project the site_history of one job can be attributed to another. The parsed result is keyed by job id. This is the same reasoning as the FLAF CRAB layer (Harden the CRAB backend with the DSProd production lessons FLAF#311), which harvests in its job manager for this reason.

What still enters the record: a job that finished, and a job that failed with an exit code — with jobs in flight counted as the denominator, as before.

Changes

• dsprod/crab.py — DSProdCrabJobManager.harvest_site_stats, called on every successful query; _collect_site_stats and the poll-callback hook removed; the record is handed to the manager in crab_create_job_manager.
• test/test_site_stats_harvest.py — 11 tests: what is recorded (finished, failed-with-code), what is not (failed without a code, no site history, unreadable response), in-flight as the denominator and combined across projects, keying by job id, and two end-to-end quarantine tests — a black hole is quarantined against a healthy baseline, and is not against the poisoned one, which is the incident in a single assertion.
• docs/concepts/backends.md — what may enter the record and why, and that a record which looks like the above can simply be deleted.

Testing

28 tests pass (17 existing, 11 new); mkdocs build --strict and run_tools/apply_format.sh --dry-run clean.

Not yet exercised against a live CRAB poll — the change is on the polling path of a running production, so it wants a real run before the next campaign. The quarantine thresholds themselves are unchanged.

Operational note

The existing data/crab_site_stats.json in the production area still holds the poisoned events. Its window is 24 h so it heals on its own, but deleting the file makes the quarantine effective immediately — it is advisory and rebuilds within a poll or two.