Skip to content

fix: log exceptions instead of silently swallowing them in Job/JobGroup.status#569

Open
andrewwhitecdw wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
andrewwhitecdw:andrewwhitecdw/fix/job-status-exception-swallowing
Open

fix: log exceptions instead of silently swallowing them in Job/JobGroup.status#569
andrewwhitecdw wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
andrewwhitecdw:andrewwhitecdw/fix/job-status-exception-swallowing

Conversation

@andrewwhitecdw

Copy link
Copy Markdown

Bug

Job.status and JobGroup.status used except Exception: ... with a finally: return, which swallowed status-check failures silently and made debugging hard.

Fix

Preserve the existing fallback behavior (return last known state / UNKNOWN) but log the exception at ERROR level so operators can see when the runner is failing.

Test

Updated test_job_status_exception and test_job_group_status_exception to assert the error is logged using caplog.

Verification

uv run pytest test/run/test_job.py::test_job_status_exception test/run/test_job.py::test_job_group_status_exception -v passes. uv run --group lint ruff check ... and ruff format --check ... pass.

…up.status

Job.status and JobGroup.status used  with a

, which swallowed status-check failures silently and made

debugging hard. Preserve the existing fallback behavior (return last known

state / UNKNOWN) but log the exception at ERROR level so operators can see

when the runner is failing.

Updates existing exception tests to assert the error is logged.

Signed-off-by: Andrew White <andrewh@cdw.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant