Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion src/base/compute/lium.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,12 +175,18 @@ async def provision(
)
template_id = await self._resolve_template(spec)

# InstanceSpec.gpu_count is a minimum-need filter for offer selection.
# Sending it as the rent gpu_count requests a *partial* split; Lium
# returns HTTP 400 "Provider doesn't allow GPU splitting" on most
# multi-GPU executors. Rent the full selected offer capacity instead
# (omit the field only when the offer itself has no gpu_count).
rent_body: dict[str, Any] = {
"pod_name": spec.name,
"user_public_key": list(spec.ssh_public_keys),
"termination_hours": int(lifetime),
"gpu_count": spec.gpu_count,
}
if selected.gpu_count and selected.gpu_count > 0:
rent_body["gpu_count"] = selected.gpu_count
Comment on lines +188 to +189

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject or filter undersized offers before renting.

_resolve_offer does not enforce spec.gpu_count: automatic selection chooses the cheapest offer regardless of GPU count, and explicit offers are returned without validation. This block can therefore rent fewer GPUs than requested instead of failing or selecting a compatible offer.

Proposed fix
+        if selected.gpu_count < spec.gpu_count:
+            raise LiumError(
+                f"offer {selected.id} provides {selected.gpu_count} GPUs, "
+                f"but {spec.gpu_count} are required"
+            )
         if selected.gpu_count and selected.gpu_count > 0:
             rent_body["gpu_count"] = selected.gpu_count

Also filter automatically selected offers by candidate.gpu_count >= spec.gpu_count.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if selected.gpu_count and selected.gpu_count > 0:
rent_body["gpu_count"] = selected.gpu_count
if selected.gpu_count < spec.gpu_count:
raise LiumError(
f"offer {selected.id} provides {selected.gpu_count} GPUs, "
f"but {spec.gpu_count} are required"
)
if selected.gpu_count and selected.gpu_count > 0:
rent_body["gpu_count"] = selected.gpu_count
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/base/compute/lium.py` around lines 188 - 189, Update _resolve_offer to
enforce spec.gpu_count for both selection paths: filter automatic candidates so
candidate.gpu_count is at least the requested count, and validate explicit
offers before returning them, rejecting undersized offers. Only include the
validated GPU count in rent_body.

if template_id is not None:
rent_body["template_id"] = template_id
if spec.dockerfile_content is not None:
Expand Down
24 changes: 24 additions & 0 deletions tests/unit/test_compute_lium_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,30 @@ async def test_provision_sends_termination_hours_and_ssh_key() -> None:
assert body["template_id"] == "tpl-1"


@respx.mock
async def test_provision_rents_full_offer_gpu_count_not_spec_minimum() -> None:
"""Lium rejects partial GPU rents on non-splittable executors.

Live API evidence (2026-07): POST .../rent with gpu_count=1 on an 8-GPU
machine returns HTTP 400 "Provider doesn't allow GPU splitting.".
``InstanceSpec.gpu_count`` is a minimum-need filter; the rent body must
request the selected offering's full gpu_count (or omit the field).
"""
routes = _mock_happy_path()
multi = Offer(
id="exec-1",
gpu_type="RTX A4000",
gpu_count=8,
price_per_hour=0.12,
)
await LiumClient("k").provision(_spec(gpu_count=1), offer=multi)
body = json.loads(routes["rent"].calls.last.request.content)
assert body.get("gpu_count") != 1
assert body.get("gpu_count") in (8, None)
if "gpu_count" in body:
assert body["gpu_count"] == multi.gpu_count
Comment on lines +218 to +222

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the full GPU count explicitly.

With multi.gpu_count == 8, accepting None allows an implementation that omits gpu_count entirely to pass. Assert the exact payload value for this positive offer.

Proposed fix
-    assert body.get("gpu_count") != 1
-    assert body.get("gpu_count") in (8, None)
-    if "gpu_count" in body:
-        assert body["gpu_count"] == multi.gpu_count
+    assert body["gpu_count"] == multi.gpu_count
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
body = json.loads(routes["rent"].calls.last.request.content)
assert body.get("gpu_count") != 1
assert body.get("gpu_count") in (8, None)
if "gpu_count" in body:
assert body["gpu_count"] == multi.gpu_count
assert body["gpu_count"] == multi.gpu_count
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_compute_lium_client.py` around lines 218 - 222, Update the
GPU payload assertions in the positive offer test to require body["gpu_count"]
== multi.gpu_count (8). Remove the permissive membership check that accepts None
or an omitted field, while retaining the existing rejection of the incorrect
value 1 if still needed.



@respx.mock
async def test_provision_rejects_overpriced_offer_without_rent() -> None:
rent = respx.post(f"{BASE}/executors/exec-1/rent")
Expand Down
8 changes: 6 additions & 2 deletions tests/unit/test_master_weights_sealer.py
Original file line number Diff line number Diff line change
Expand Up @@ -320,10 +320,14 @@ async def test_zero_miner_empty_slate_still_seals(
async def test_concurrent_get_seal_race_safe(
session_factory: async_sessionmaker,
) -> None:
# MasterWeightsResponse.expires_at is validated against wall-clock now()
# (not FakeClock). freshness_seconds=1 flakes under concurrent DB load on
# CI when seal latency > 1s. Keep TTL large vs wall time, expire via clock.
clock = FakeClock()
freshness = 120
service = _service(
session_factory,
freshness_seconds=1,
freshness_seconds=freshness,
epoch_interval_seconds=60,
clock=clock,
)
Expand All @@ -350,7 +354,7 @@ async def one_get() -> Any:
assert all(r.uids == [0] and r.weights == [1.0] for r in results)

# Expire + concurrent heal reseals once.
clock.advance(5)
clock.advance(freshness + 5)
healed = await asyncio.gather(*[one_get() for _ in range(6)])
healed_ids = {r.vector_id for r in healed}
assert len(healed_ids) == 1
Expand Down
Loading