Consolidate manual-job recovery in check-jobs and add timeit-driven Condor scheduling - #558
Open
mondalspandan wants to merge 28 commits into
Open
Consolidate manual-job recovery in check-jobs and add timeit-driven Condor scheduling#558mondalspandan wants to merge 28 commits into
timeit-driven Condor scheduling#558mondalspandan wants to merge 28 commits into
Conversation
The babysitter loop only caught KeyboardInterrupt, so any unguarded parse killed a session that may have run for days. Guard the known crash sites: - bump_jobqueue: `queues.index(jf)` raised ValueError for a non-lxplus flavour, and `return next_jf` raised NameError when the .sub had no +JobFlavour line (e.g. rubin's +MaxRuntime). Fall back to the longest known queue for an unknown flavour and return None when there is no flavour line. - The SYSTEM_PERIODIC_REMOVE check did `glob(...logs/job_*.log)[0]` -> IndexError before any log exists; skip when there are no logs yet. - XRootD-error log parsing indexed `c[iln+1]`/`c[iln+3]` and `.split()[-1]` without bounds checks -> IndexError on a truncated log; guard both. - Add a broad except around the loop that reports the error clearly instead of dumping a raw traceback. Adds offline tests for bump_jobqueue (normal, capped, unknown flavour, no flavour line). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ate-jobs The runner/executor `--recreate-jobs` path and `check-jobs --resubmit` duplicated a lot of logic (two `find_other_file`, two queue-bump helpers + two `queues` lists, two fileset-rewrite + condor-resubmit loops) while operating on the same jobs_dir on-disk contract. Fold all of it into `check-jobs` so there is a single consumer, and remove the runner path. check-jobs gains a one-shot proactive recreate mode and the recreate-only knobs that used to live on `pocket-coffea run`: - `--recreate auto|0,1,3` (+ `--once`) — recreate/resubmit a chosen set of jobs and exit; unlike `--resubmit` it can act on running/idle jobs too. - `--use-redirector`, `--blocklist-sites` (unioned with the auto blacklist), `--recreate-queue`, `--skip-bad-files`. Shared/reused instead of duplicated: - new `utils/htcondor_queue.py` (QUEUES, bump_queue, set_queue) replaces check_jobs.bump_jobqueue and lxplus update_queue/set_queue + both `queues`. - `site_rewrite.find_other_file` extended (exclude_urls, blocklist-by-prefix, fallback_redirector=None) and now used by check-jobs in place of its private copy. - the submit-time inner_run_options helpers stay in executors_manual_jobs and are reused by check-jobs' `--skip-bad-files`. Removed: recreate_jobs from executors_lxplus/rubin, the ABC abstractmethod, the submit() dispatch and __init__ guard-skip, and the --blocklist-sites/--recreate-queue/--use-redirector options on the runner (--skip-bad-files stays; it also feeds the outer Coffea Runner). `run` now errors with a pointer to check-jobs if a moved flag is passed. Also fixes two latent bugs: the babysitter's undefined `log_file` NameError on the "failed-again after resubmit" path, and the old explicit-list recreate crashing on an undefined `runningjobs`. Docs updated; adds tests/test_check_jobs_recreate.py plus set_queue and find_other_file-extension cases. 39 offline unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # docs/running.md # pocket_coffea/executors/executors_lxplus.py # pocket_coffea/scripts/check_jobs.py # pocket_coffea/scripts/runner.py
Add per-dataset timeit files, event-targeted filtering, adaptive manual splitting, and automatic LXPLUS queue forecasts. Harden Condor/XRootD recovery and check-jobs state handling, with focused tests and documentation.
Prepare recreate replacements before removing active Condor jobs, refresh proxy credentials immediately before submission, and keep queue/resource escalation transactional. Persist replacement-attempt counts across restarts, align legacy timeout resource scaling with dynamic jobs, harden Rubin no-proxy submission, and document the resulting recovery contract.
Generate explicit LXPLUS and Rubin recovery metadata, use one authoritative job state, keep passive check-jobs read-only, and remove legacy migration paths.
mondalspandan
changed the title
timeit-driven Condor scheduling
Contributor
Author
|
Quick justification for the 
Only 4% jobs need to be resubmitted to a longer queue. And by resubmitting longer jobs with an ncpu bump (e.g. with |
Contributor
|
Hi @mondalspandan! I will have a look at your changes. Can we consider this PR to be reviewed? It's already quite huge :) |
Contributor
Author
|
Hi Davide, I did run full workflows with these changes a few times and everything appeared to be working. I agree it's huge, can you please also do a claude review? |
mondalspandan
marked this pull request as ready for review
September 2, 2026 08:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supersedes #541, retaining its user-facing consolidation of manual-job recovery while simplifying and hardening the underlying implementation. It also adds throughput-based job splitting, runtime forecasting, automatic queue selection, and worker-side recovery improvements.
1. Features already implemented in #541
Features from #541 kept unchanged in this PR
pocket-coffea check-jobs.pocket-coffea run --recreate-jobsworkflow.--recreate auto|<job list>and single-pass monitoring with--once.--use-redirector,--blocklist-sites,--recreate-queue,--skip-bad-files, and--remove-runningfromcheck-jobs.--resubmit.jobs_config.yamlas the source of the original per-job filesets during recreation.check-jobs.inner_run_options.yamlfor forwarding selected Coffea Runner options to Condor workers.Features from #541 modified code-wise
.subrecovery with a current-formatjob_state.json+resubmit.subrecovery model.check-jobsrestarts.condor_rmand handling jobs that finish while recreation is being prepared.check-jobs.--skip-bad-filesto update the shared current-formatinner_run_options.yamlwithout legacy wrapper/submit-file patching.2. New features implemented in this PR
.timeoutjob state and graceful worker process-group cleanup before Condor removes timed-out jobs.condor_submitfailures instead of leaving misleading.idlemarkers.--timeitfor per-dataset processing-throughput measurements.timeit/.--queue autoto select an LXPLUS queue per job from its estimated runtime.