Skip to content

test(harness): add BO767 VL library qualification sweep - #2401

Open
charlesbluca wants to merge 1 commit into
NVIDIA:mainfrom
charlesbluca:fix/harness-library-vl-sweep-2026-07-23
Open

test(harness): add BO767 VL library qualification sweep#2401
charlesbluca wants to merge 1 commit into
NVIDIA:mainfrom
charlesbluca:fix/harness-library-vl-sweep-2026-07-23

Conversation

@charlesbluca

Copy link
Copy Markdown
Collaborator

Description

Adds two checked-in BO767 library qualification runfiles for the nvidia/llama-nemotron-embed-vl-1b-v2 modality sweep. The runfiles cover text_image and image ingestion with element-level embeddings, hybrid LanceDB indexing, and automatic query retrieval.

Documents the isolated published-RC workflow, one-GPU co-scheduling profile, integrity gates, artifact handoff, and runtime-failure semantics. Dense retrieval remains available as a diagnostic control but is not part of this qualification sweep.

This is an opt-in harness configuration and documentation change. It does not change the library runtime or CLI API. Quality expectations remain unset until both modalities complete end to end.

Validation:

  • 38 passed, 6 deselected across focused harness and hybrid-retrieval tests
  • scoped Ruff check passed
  • uv lock --check passed
  • git diff --check passed
  • retriever harness run-files --help smoke test passed
  • partial RC runtime validation confirmed all three embed actors co-schedule on one visible GPU without DeepGEMM, engine-initialization, or OOM failures

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@charlesbluca
charlesbluca marked this pull request as ready for review July 29, 2026 15:01
@charlesbluca
charlesbluca requested review from a team as code owners July 29, 2026 15:01
@charlesbluca
charlesbluca requested a review from drobison00 July 29, 2026 15:01
@greptile-apps

greptile-apps Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds an opt-in BO767 vision-language qualification sweep.

  • Adds runfiles for element-level text_image and image embedding with hybrid LanceDB indexing and automatic retrieval.
  • Documents the published-RC workflow, single-GPU co-scheduling profile, integrity checks, artifact handling, and runtime-failure interpretation.
  • Adds unit coverage confirming both runfiles resolve to the intended benchmark configuration.

Confidence Score: 5/5

The pull request appears safe to merge with no actionable defects identified.

The new runfiles use supported override paths, inherit the required BO767 extraction defaults, and are accompanied by resolution tests and documentation that distinguishes automated integrity gates from manual qualification checks.

Important Files Changed

Filename Overview
nemo_retriever/harness/EXPECTED_RESULTS.md Documents the two BO767 VL sweep configurations, integrity expectations, and interpretation of failed quality runs.
nemo_retriever/harness/README.md Adds instructions for running the qualification sweep against an isolated published RC and reviewing its artifacts.
nemo_retriever/harness/runfiles/bo767_vl_image_hybrid_beir.json Defines the element-level image modality run with hybrid indexing and BO767 integrity gates.
nemo_retriever/harness/runfiles/bo767_vl_text_image_hybrid_beir.json Defines the element-level text-image modality run with hybrid indexing and BO767 integrity gates.
nemo_retriever/tests/test_harness_benchmark_registry.py Verifies that both new runfiles load and resolve to their documented model, modality, batching, indexing, retrieval, and gate settings.

Reviews (1): Last reviewed commit: "test(harness): add BO767 VL library qual..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant