Feat/eval tracking support

[camilobrownpinilla]

Summary

• Adds support for evaluation experiment tracking via tracking backends in KF core. Future support for MLFlow will be a 1 line fix.
• Adds benchmark manifest to keep track of benchmark specific keys related to tracking

Testing

• uv run ruff check kempnerforge/ tests/ passes
• uv run ruff format --check kempnerforge/ tests/ scripts/ passes
• uv run pyright kempnerforge/ passes (0 errors)
• uv run pytest tests/unit/ -v --timeout=60 passes

Closes #161

[Copilot]

Pull request overview

This PR extends the VLM evaluation harness to optionally log evaluation metrics via KempnerForge’s existing metrics backends, and adds core checkpoint utilities to safely extract checkpoint “extra” metadata (e.g., a training run identifier) without restoring RNG state.

Changes:

• Added load_train_state_extras() plus a shared TRAIN_STATE_STANDARD_KEYS constant to support reading checkpoint “extra” metadata without restoring RNG.
• Enhanced examples/vlm-evaluation/vlm_eval_harness.py to accept dotted config overrides, derive a logging step, attach eval logs to a training run when possible, and emit structured eval metrics via a new benchmark manifest.
• Added new benchmark manifest implementation and unit tests for checkpoint extras, benchmark manifest behavior, harness tracking glue, and adapter config loading.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/test_checkpoint_extras.py	Adds unit coverage for the new checkpoint “extras” accessor, including RNG non-application and ownership-gate behavior.
kempnerforge/checkpoint/state.py	Introduces TRAIN_STATE_STANDARD_KEYS and reuses it when extracting “extra” keys on restore.
kempnerforge/checkpoint/manager.py	Exposes load_train_state_extras() backed by the existing trusted _load_train_state() path.
kempnerforge/checkpoint/init.py	Exports load_train_state_extras as part of the public checkpoint API.
examples/vlm-evaluation/vlm_eval_harness.py	Adds config override forwarding, checkpoint step derivation, run-id attachment, and opt-in experiment tracking for eval results.
examples/vlm-evaluation/tests/unit/test_harness_tracking.py	Adds unit tests for the harness’s run-id resolution and tracking failure behavior.
examples/vlm-evaluation/tests/unit/test_benchmark_manifest.py	Adds thorough unit tests for benchmark aggregate resolution + metric flattening.
examples/vlm-evaluation/tests/unit/test_adapter.py	Ensures the harness can pass a pre-merged JobConfig through to the adapter without re-parsing.
examples/vlm-evaluation/README.md	Documents the new experiment tracking workflow and metric key scheme.
examples/vlm-evaluation/benchmark_manifest.py	Implements benchmark metric registry + normalized aggregate resolution and raw/perf metric flattening.
examples/vlm-evaluation/adapter.py	Allows config to be either a TOML path or an already-loaded JobConfig so CLI overrides reach the evaluated model.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.