Add cluster_id filtering and provenance.db dump to scan - #20
Merged
Conversation
mldag-query scan now accepts a repeatable --cluster-id to filter to a
specific set of clusters, and --db <path> to write results into
provenance.db's condor_history table instead of/alongside printing.
condor_history (task-28) already stores per-job summary data queried
from HTCondor's own history; scan's event-log-derived summaries are
the same shape of data from a different source, so they share the
table, distinguished by a new source column ('condor_history' /
'event_log'). This also surfaced and fixed the same job-array bug
condor_history had inherited from before we knew about it: its
primary key was cluster_id alone, but Schedd.history() returns one
ClassAd per proc, not per cluster. It's now (cluster_id, proc_id),
matching event_log_scan.py's fix from task-29. A status column, backed
by each source's own vocabulary (JobStatus for condor_history rows,
event_log_scan's own status string for event_log rows), lets both be
queried uniformly.
No migration path was written for existing condor_history rows under
the old schema: provenance.db is documented as a rebuildable cache
over source-of-truth files, so upgrading means deleting and rebuilding
it, not migrating in place.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
INSERT OR REPLACE blindly replaced the whole row, so writing from one source (condor_history or event_log) after the other had already written the same (cluster_id, proc_id) silently NULLed out whichever columns only the first write populated. Reworked to ON CONFLICT DO UPDATE with COALESCE(excluded.col, condor_history.col) per column, so a write only overwrites the columns it actually knows about. Splits the raw payload into condor_history_json/event_log_json (each nullable, only ever written by its own source, so neither can stomp the other's raw record) and unions `source` into a sorted comma-set instead of overwriting it, so it correctly reflects every source that has contributed once both have. Also fixes enrich_from_condor_history's "already enriched" check, which previously matched any row regardless of source -- a cluster_id touched only by scan --db was being treated as already condor_history-enriched and silently skipped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a third source into condor_history: enrich_from_jobad_events() mirrors job.assigned events (jobad.py's in-job $_CONDOR_JOB_AD capture, emitted immediately at job start -- see pretrain_local.sh) that are already in provenance.db's `events` table, tagged source='jobad'. This is additive, not a replacement -- job.assigned keeps being written exactly as before, available immediately for whatever consumes it right after the job starts; enrich-jobad just also lets that same data land in the same enrichment table as the other two sources, joinable against them. capture_job_ad_fields() now always includes cluster_id/proc_id when present in the ad (needed as the join key), which flows into job.assigned's existing payload with no changes to pretrain_local.sh itself. Deliberately never writes remote_wall_clock_s/cpus_usage/ memory_usage_mb/gpus_usage from this source, even though capture_job_ad_fields() can return them: $_CONDOR_JOB_AD is captured at job start, before the job has run, so those would be near-zero placeholders, not real usage -- enrich-history/scan are the sources of truth for those fields. Wired as `mldag-query db enrich-jobad`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mldag-query scanaccepts a repeatable--cluster-idto filter to a specific set of clusters (every proc of a matched cluster is kept).mldag-query scan --db <path>writes results intoprovenance.db'scondor_historytable (source='event_log'), reusing the table added in Add event-log scanner and condor_history enrichment #16 fordb enrich-history(source='condor_history').condor_historyitself: its primary key wascluster_idalone, butSchedd.history()returns one ClassAd per proc. Now(cluster_id, proc_id).statuscolumn populated from both sources' own vocabularies, so it's queryable uniformly.condor_historyrows —provenance.dbis a rebuildable cache, so upgrading means deleting and rebuilding it.Test plan
uv run pytest tests/ -q— 274 passed, 2 skippedruff check/ty checkclean on touched files--cluster-idfiltering,--dbdump)🤖 Generated with Claude Code