Delete dull job and instance models created on no capacity retry#4059
Open
r4victor wants to merge 4 commits into
Open
Delete dull job and instance models created on no capacity retry#4059r4victor wants to merge 4 commits into
r4victor wants to merge 4 commits into
Conversation
r4victor
changed the title
jvstme
approved these changes
Jul 22, 2026
Comment on lines
+27
to
+41
| op.execute( | ||
| """ | ||
| UPDATE event_targets SET entity_run_id = entity_id | ||
| WHERE entity_type = 'run' AND entity_run_id IS NULL | ||
| """ | ||
| ) | ||
| op.execute( | ||
| """ | ||
| UPDATE event_targets SET entity_run_id = jobs.run_id | ||
| FROM jobs | ||
| WHERE jobs.id = event_targets.entity_id | ||
| AND event_targets.entity_type = 'job' | ||
| AND event_targets.entity_run_id IS NULL | ||
| """ | ||
| ) |
Collaborator
There was a problem hiding this comment.
Enums like entity_type hold uppercase values
$ sqlite3 ~/.dstack/server/data/sqlite.db "SELECT DISTINCT entity_type FROM event_targets;"
FLEET
GATEWAY
INSTANCE
JOB
PROJECT
RUN
SECRET
USER
VOLUME
Comment on lines
346
to
363
| if within_fleets is not None: | ||
| query = select(InstanceModel.id).where(InstanceModel.fleet_id.in_(within_fleets)) | ||
| res = await session.execute(query) | ||
| # In Postgres, fetching instance IDs separately is orders of magnitude faster | ||
| # than using a subquery. | ||
| instance_ids = list(res.unique().scalars().all()) | ||
| target_filters.append( | ||
| or_( | ||
| and_( | ||
| EventTargetModel.entity_type == EventTargetType.FLEET, | ||
| EventTargetModel.entity_id.in_(within_fleets), | ||
| ), | ||
| and_( | ||
| EventTargetModel.entity_type == EventTargetType.INSTANCE, | ||
| EventTargetModel.entity_id.in_(instance_ids), | ||
| ), | ||
| ) | ||
| ) |
Collaborator
There was a problem hiding this comment.
Event targets now store entity_run_id so that within_runs filter works even for job event when job models are deleted
Should we also introduce entity_fleet_id to be able to see deleted placeholder instance events when using the within_fleets filter?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4054
Prevent the accumulation of large number of job and instance models when run retries on no capacity:
Add runs to events about potentially never-provisioned jobs so that events are displayed with within_runs filter.Event targets now store entity_run_id so that within_runs filter works even for job event when job models are deleted. entity_run_id is backfilled for existing runs and jobs.Note: the cleanup is for FAILED_TO_START_DUE_TO_NO_CAPACITY jobs that a) carry no useful information and b) created in large quantities on retry. Job models failed due to other reasons are not cleaned up.