feat: VDB operator entry-point registry for service-mode fluent ingestor

[jdye64]

Summary

Implements Option A: a setuptools entry-point registry so third-party packages can register custom vector database backends and select them from the fluent ingestor in service run mode.

Changes

• factory.py: get_vdb_op_cls() and list_vdb_ops() merge built-in lancedb with backends registered under the nemo_retriever.vdb_operators entry-point group. Adds extract_vdb_connection_uri() for backend-neutral URI validation.
• policy.py: New allowed_vdb_ops policy gate; generalized connection URI checks; registered_vdb_ops exposed via pipeline-config introspection.
• config.py / retriever-service.yaml: pipeline_overrides.allowed_vdb_ops (default: [lancedb]).
• ServiceIngestor.vdb_upload(): Accepts custom vdb_op values and generic connection URI fields (host, url, etc.).
• Tests: tests/test_vdb_factory.py (62 related tests passing locally).
• Docs: Updated common/vdb/README.md with registration and service-mode usage.

Register a custom backend

[project.entry-points."nemo_retriever.vdb_operators"]
opensearch = "my_company.vdb.opensearch:OpenSearchVDB"

Install the package on worker pods, then configure the service:

pipeline_overrides:
  allowed_vdb_ops: [lancedb, opensearch]
  extra_vdb_kwargs_keys: [host, index]
  sinks:
    vdb_uri_schemes: ["https://"]

Client usage:

create_ingestor(run_mode="service", base_url="http://retriever:7670")
  .vdb_upload(vdb_op="opensearch", vdb_kwargs={"host": "https://...", "index": "corpus"})
  .ingest()

Test plan

• pytest tests/test_vdb_factory.py tests/test_service_sinks.py tests/test_nv_ingest_vdb_operator.py

  

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.