[Bug]: Slow `/api/runs/list` due to the large number of job submissions

[r4victor]

Steps to reproduce

1. Submit a run with a long retry (e.g. retry.duration: 7d) that never provisions.
2. The run accumulates lots of job submissions for every retry attempt (pending->submitted transitions).
3. Calling /api/runs/list (via the Python/HTTP API) is slow due to the large number of job submissions. Moreover, the entire server can become slow since the response serialization blocks the event loop.

Actual behaviour

Every run retry attempt creates a job submission. Retry is exponential but the backoff is capped at 10 minutes. So a run with retry configured for a long duration accumulates submissions at ~6/hour ≈ 144/day – a thousand of submissions over a week of retry.

The CLI / UI pass job_submissions_limit=1 to /api/runs/list. but users using API directly may request all job submissions.

Expected behaviour

Possible solutions:

• Enforce job_submissions_limit server-side, e.g. job_submissions_limit <=10. This fixes /api/runs/list issues, but runs can still have lots of job submissions produced and stored in the DB.
• Avoid creating JobModel for every retry attempt – need to figure out how to do it without losing the useful information about retry attempt time and reason.

dstack version

master

Server logs

Additional information

No response