Add Jobmachine delegation#128
Draft
SemMulder wants to merge 7 commits into
Draft
Conversation
- Add `paused INTEGER NOT NULL DEFAULT 0` column to `submissions` table
(migration 20260709170000_add_paused_to_submissions)
- Add `paused: bool` field to `Submission` struct; thread through
`insert_submission_raw`, `get_submission`, `submission_status`, and
`insert_submission_from_chunks` (new `paused` parameter)
- Add `#[serde(default)] paused: bool` to `InsertSubmission` so producers can
create submissions in a paused state; default is `false` for backward compat
- Filter paused submissions out of all chunk dispatch strategies: Oldest, Newest,
and both halves of Random use `WHERE submission_id NOT IN (SELECT id FROM
submissions WHERE paused)`; PreferDistinct inherits the filter via its CTE
- Update strategy query-plan snapshots to reflect the new subquery
- `job_delegate` now also clears `paused = 0` on the active submission and
calls `notify_on_insert` to wake consumers; add `notify_on_insert: Arc<Notify>`
to delegation `ServerState` and thread it from `server.rs`
- Implement `job_return`: for each task_id, find the still-paused submission,
clear `jm_task_id` (so the cancelled row is not picked up by the background
reporting loop), cancel the submission, then PUT the returned task IDs to
`{jm_master_url}/delegation/return`
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Re-pause the active submission (SET paused = 1, jm_task_id = NULL) rather than cancelling it. The submission stays in the queue so it can be re-delegated later; consumers simply stop picking up its chunks. If the submission is not active (already terminal or unknown), it is silently skipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…k_id Instead of calling /delegation/return directly from the handler, job_return now only re-pauses the submission (SET paused = 1) while leaving jm_task_id set. The background loop in report_delegated_submissions then picks up these paused-with-jm_task_id submissions, PUTs their task IDs to /delegation/return, and clears jm_task_id afterwards (same pattern as /delegation/complete). This keeps all outbound JM master communication in one place and makes job_return resilient to transient HTTP failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a keyword-only `paused: bool = False` parameter to all submission insertion methods in both the Rust bindings and the Python wrappers: - insert_submission_direct (Rust) - insert_submission_chunks (Rust + Python) - run_submission_chunks (Rust + Python) - insert_submission (Python) - run_submission / async_run_submission (Python) - async_run_submission_chunks (Python) When paused=True, the submission is held in the queue but chunks are not dispatched to consumers until the submission is explicitly unpaused (e.g. via the delegation job_delegate endpoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Discussed offline with @ReinierMaas: currently fetching chunks to work on is a bottleneck in their system. Adding paused the way we do here would make the issue worse. Therefore:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.