[Bug] Single-sbatch termination handling

[podkidyshev]

Summary

Single-sbatch cancellation didn't cancel the slurm job

Test Plan

• Automated CI
• Manual runs

Additional Notes

[coderabbitai]

📝 Walkthrough

Walkthrough

SingleSbatchRunner.run now tracks submitted jobs, kills active jobs during shutdown, and always clears job tracking. Tests cover shutdown during monitoring or submission and normal completion without killing completed jobs.

Changes

Slurm job lifecycle

Layer / File(s)	Summary
Track and terminate submitted jobs src/cloudai/systems/slurm/single_sbatch_runner.py	Submitted jobs are tracked, killed when shutdown occurs during monitoring, and removed from tracking in a finally block.
Validate lifecycle cleanup tests/test_single_sbatch_runner.py	Tests cover shutdown during monitoring or submission and confirm completed jobs are not killed while tracking is cleared.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jeffnvidia

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title matches the main change by describing single-sbatch termination handling, though it's a bit broader than the exact cancellation fix.
Description check	✅ Passed	The description is directly about single-sbatch cancellation not canceling the Slurm job, which matches the PR changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ipod/ssbatch-sigterm

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/cloudai/systems/slurm/single_sbatch_runner.py`:
- Around line 190-193: The job registration and shutdown check in
SingleSbatchRunner must prevent duplicate cancellation when shutdown occurs
between them. In src/cloudai/systems/slurm/single_sbatch_runner.py lines
190-193, synchronize registration with shutdown or make cancellation idempotent
so each job is killed exactly once. In tests/test_single_sbatch_runner.py lines
558-575, add coverage that triggers shutdown from jobs.append() and asserts
exactly one kill.

In `@tests/test_single_sbatch_runner.py`:
- Around line 558-575: Extend the shutdown regression coverage in
test_run_cancels_job_when_shutdown_occurs_during_submission by replacing or
augmenting the jobs collection with a custom append implementation that calls
runner.shutdown() after the submitted job is registered. Run the submission flow
and assert SlurmSystem.kill is called exactly once for the job, covering
shutdown cancellation after registration and preventing duplicate cancellation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 35b2fe29-bf0f-4f89-9990-4955a2d6b237

📥 Commits

Reviewing files that changed from the base of the PR and between 09f728c and 4061184.

📒 Files selected for processing (2)

• src/cloudai/systems/slurm/single_sbatch_runner.py
• tests/test_single_sbatch_runner.py

[jj10306]

lgtm 👍