Skip to content

Expose observation metadata through simulator interfaces - #340

Open
AlexanderFengler wants to merge 4 commits into
codex/observation-producer-registryfrom
codex/observation-producer-interfaces
Open

Expose observation metadata through simulator interfaces#340
AlexanderFengler wants to merge 4 commits into
codex/observation-producer-registryfrom
codex/observation-producer-interfaces

Conversation

@AlexanderFengler

Copy link
Copy Markdown
Member

Depends on #339.

Scope

  • resolve explicit observation metadata from registered model names without sampling
  • expose a fresh descriptor through Simulator.get_observation_metadata()
  • attach validated schema declarations to HSSM-compatible simulator wrappers
  • resolve supported *_deadline variants through the same model-config path as the simulator, preserving exact base-model response labels
  • preserve the existing unknown-model warning and fallback behavior

Compatibility boundary

This is additive, out-of-band metadata. It does not change simulator(),
Simulator.simulate(), hssm_sim_wrapper, result dictionaries, result metadata, or the
three-value validate_simulator_fun() contract.

The descriptor's obs_dim is semantic and derived from the schema. The existing wrapper
attribute remains a legacy output-width contract. Consequently, a response-only
inverse-temperature-softmax model reports semantic width one while its unchanged HSSM
wrapper still returns the historical two-column dummy-RT/response array.

Metadata lookup remains fail-closed: unknown producers and incomplete declarations do
not acquire schemas from nchoices, legacy obs_dim, result keys, sampled values, or
the HSSM fallback labels.

Review hardening

  • isolate the built-in metadata census from mutable custom registrations, which may
    intentionally omit schema declarations
  • preserve opaque producer-config extensions without deep-copying unrelated runtime
    state; copied response labels and schema declarations remain isolated
  • cover binary DDM and LBA deadline variants so direct, class-based, and wrapper
    introspection agree

Verification

  • full package suite: 1,243 passed, 134 skipped
  • focused metadata, HSSM-support, registry, and simulator-class suite: 158 passed
  • repository-wide Ruff check and format check
  • git diff --check
  • actual deterministic ddm_deadline wrapper execution
  • HSSM consumer canaries against this worktree: legacy simulator and random-variable
    construction, 2 passed with candidate provenance asserted
  • independent static and ecosystem reviews after the deadline and test-order fixes

Deferred

RLSSM ModelConfig/AssembledModel adoption, the broader downstream compatibility
matrix, contributor documentation, and any consumer switch to normalized results remain
separate stacked work.

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e9faab02-b4f1-4a0f-a8b9-e786e60cb814

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
unittests 94.39% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
ssms/basic_simulators/observation_metadata.py 100.00% <100.00%> (ø)
ssms/basic_simulators/simulator_class.py 83.06% <100.00%> (+0.37%) ⬆️
ssms/hssm_support.py 96.03% <100.00%> (+0.53%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant