Skip to content

[Bug]: LCB Service container is broken on mlcommon's container registry #467

Description

@zihaok

Bug Description

When I run this command, which is a srun ... --container-image=lcb_image_link, I couldn't get it to load properly

srun   --account=coreai_mlperf_inference   --partition=gb200   --job-name=coreai_mlperf_inference-lcb.enroot-zstd-repro   --nodes=1   --ntasks=1   --time=00:10:00   --segment=1   --container-image=ghcr.io/mlcommons/endpoints:598dbfde68ebfa42256762ebb60fd4da227b813c-livecodebench   bash -lc 'echo hello'
pyxis: importing docker image: ghcr.io/mlcommons/endpoints:598dbfde68ebfa42256762ebb60fd4da227b813c-livecodebench
[2026-08-24T23:56:56.408] error: pyxis: child 4023939 failed with error code: 4
[2026-08-24T23:56:56.408] error: pyxis: failed to import docker image: ghcr.io/mlcommons/endpoints:598dbfde68ebfa42256762ebb60fd4da227b813c-livecodebench
[2026-08-24T23:56:56.408] error: pyxis: printing enroot log file:
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Querying registry for permission grant
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Authenticating with user: <anonymous>
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Authentication succeeded
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Fetching image manifest list
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Fetching image manifest
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Downloading 14 missing layers...
[2026-08-24T23:56:56.408] error: pyxis:     [INFO] Extracting image layers...
[2026-08-24T23:56:56.408] error: pyxis:     tar: This does not look like a tar archive
[2026-08-24T23:56:56.408] error: pyxis:     tar: Skipping to next header
[2026-08-24T23:56:56.408] error: pyxis:     tar: Exiting with failure status due to previous errors
[2026-08-24T23:56:56.408] error: pyxis:     tar: This does not look like a tar archive
[2026-08-24T23:56:56.408] error: pyxis:     tar: Skipping to next header
[2026-08-24T23:56:56.408] error: pyxis:     tar: Exiting with failure status due to previous errors
[2026-08-24T23:56:56.408] error: pyxis:     tar: This does not look like a tar archive
[2026-08-24T23:56:56.408] error: pyxis:     tar: Skipping to next header
[2026-08-24T23:56:56.408] error: pyxis:     tar: Exiting with failure status due to previous errors
[2026-08-24T23:56:56.408] error: pyxis:     tar: This does not look like a tar archive
[2026-08-24T23:56:56.408] error: pyxis:     tar: Skipping to next header
[2026-08-24T23:56:56.408] error: pyxis:     tar: Exiting with failure status due to previous errors
[2026-08-24T23:56:56.408] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2026-08-24T23:56:56.408] error: Failed to invoke spank plugin stack
[2026-08-24T23:56:56.430] error: pyxis: child 4025632 failed with error code: 1
srun: error: lyris0162: task 0: Exited with exit code 1
srun: Terminating StepId=2782334.0

Steps to Reproduce

run this as minimum reproducer
srun --account=coreai_mlperf_inference --partition=gb200 --job-name=coreai_mlperf_inference-lcb.enroot-zstd-repro --nodes=1 --ntasks=1 --time=00:10:00 --segment=1 --container-image=ghcr.io/mlcommons/endpoints:598dbfde68ebfa42256762ebb60fd4da227b813c-livecodebench bash -lc 'echo hello'

Environment

N/A

Relevant Logs

N/A

Before submitting

  • I searched existing issues and found no duplicates

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions