[CELEBORN-2387] Add ReserveSlotsFailCount worker metric

[shellfish007]

What changes were proposed in this pull request?

Adds a new worker metric, ReserveSlotsFailCount, and increments it on every non-success path of Controller.handleReserveSlots.

Why are the changes needed?

handleReserveSlots already replies with distinct non-success status codes:

• WORKER_SHUTDOWN — worker is shutting down
• NO_AVAILABLE_WORKING_DIR — no healthy local dirs and no remote storage
• RESERVE_SLOTS_FAILED — not all primary partitions could be satisfied
• RESERVE_SLOTS_FAILED — not all replica partitions could be satisfied

...but none of these paths incremented a counter. Slot reservation was the only major worker operation without a *FailCount metric (push, replicate, fetch, commit, register all have one), so failed reservations were invisible to metrics/dashboards and only visible via logs.

WorkerSource.scala declares RESERVE_SLOTS_FAIL_COUNT = "ReserveSlotsFailCount" and registers it with addCounter(...), alongside SlotsAllocated. Controller.scala calls workerSource.incCounter(WorkerSource.RESERVE_SLOTS_FAIL_COUNT) before each of the four non-success context.reply(...) sites in handleReserveSlots.

One fail counter per failed reservation regardless of cause, matching the existing *FailCount convention (e.g. CommitFilesFailCount, RegisterWithMasterFailCount). This gives a single "reserve slots did not succeed" rate; the specific cause remains distinguishable via the status code in logs.

Does this PR introduce any user-facing change?

Yes — adds a new worker metric ReserveSlotsFailCount. No config or API changes.

How was this patch tested?

Follows the established counter pattern exactly (constant + addCounter registration + incCounter at the failure path), with no new imports or signatures.

[Copilot]

Pull request overview

This pull request adds a new worker-side metric counter to make slot reservation failures visible in metrics/dashboards by incrementing a ReserveSlotsFailCount counter on non-success reply paths in Controller.handleReserveSlots.

Changes:

• Register a new worker metric counter ReserveSlotsFailCount in WorkerSource.
• Increment ReserveSlotsFailCount on each non-success response path in Controller.handleReserveSlots.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/WorkerSource.scala	Declares and registers the new ReserveSlotsFailCount counter.
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala	Increments ReserveSlotsFailCount before returning non-success ReserveSlotsResponse results.

Comments suppressed due to low confidence (1)

worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala:248

• RESERVE_SLOTS_FAIL_COUNT is incremented after destroyWriters(...). If cleanup throws, the metric won't be recorded and the RPC reply may never be sent. Increment the fail counter before cleanup so reservation failures are still counted even when cleanup fails.

      destroyWriters(primaryLocs, shuffleKey)
      destroyWriters(replicaLocs, shuffleKey)
      workerSource.incCounter(WorkerSource.RESERVE_SLOTS_FAIL_COUNT)
      context.reply(ReserveSlotsResponse(StatusCode.RESERVE_SLOTS_FAILED, msg))
      return

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.