Add model package support for sdkv2 - #981
Conversation
Add package-aware layout handling, loading, cache behavior, and catalog compatibility filtering.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Copilot resolve the merge conflicts in this pull request |
b75395a to
a6ceeca
Compare
a6ceeca to
88a77c5
Compare
# Conflicts: # sdk_v2/cpp/src/ep_detection/ep_detector.cc Co-authored-by: xiaoyu-work <85524621+xiaoyu-work@users.noreply.github.com>
Resolved in 2922026 — merged |
There was a problem hiding this comment.
Pull request overview
Adds C++ SDK v2 support for downloading, discovering, filtering, and loading ORT model packages.
Changes:
- Adds package layout detection and cache handling.
- Adds package metadata parsing and hardware compatibility filtering.
- Loads selected package variants through ORT GenAI and expands tests.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
sdk_v2/cpp/CMakeLists.txt |
Builds the layout classifier. |
sdk_v2/cpp/src/util/model_layout.h |
Declares model layout classification. |
sdk_v2/cpp/src/util/model_layout.cc |
Implements layout detection. |
sdk_v2/cpp/src/model.cc |
Passes catalog EP and task metadata when loading. |
sdk_v2/cpp/src/model_info.h |
Defines typed package metadata. |
sdk_v2/cpp/src/model_info.cc |
Serializes package metadata. |
sdk_v2/cpp/src/inferencing/model_load_manager.h |
Extends the loading interface. |
sdk_v2/cpp/src/inferencing/model_load_manager.cc |
Handles package runtime metadata. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h |
Tracks package and modality state. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc |
Loads packages through ORT GenAI. |
sdk_v2/cpp/src/ep_detection/ep_detector.h |
Adds compiled-model compatibility APIs. |
sdk_v2/cpp/src/ep_detection/ep_detector.cc |
Evaluates package compatibility. |
sdk_v2/cpp/src/download/inference_model_writer.h |
Documents package marker behavior. |
sdk_v2/cpp/src/download/inference_model_writer.cc |
Retains package root markers. |
sdk_v2/cpp/src/download/download_manager.h |
Updates download workflow documentation. |
sdk_v2/cpp/src/download/download_manager.cc |
Resolves package cache roots. |
sdk_v2/cpp/src/catalog/local_model_scanner.h |
Documents package discovery. |
sdk_v2/cpp/src/catalog/local_model_scanner.cc |
Discovers package roots without scanning variants. |
sdk_v2/cpp/src/catalog/azure_model_catalog.h |
Adds injectable catalog clients. |
sdk_v2/cpp/src/catalog/azure_model_catalog.cc |
Filters incompatible package entries. |
sdk_v2/cpp/src/catalog/azure_catalog_models.h |
Defines catalog package payloads. |
sdk_v2/cpp/src/catalog/azure_catalog_models.cc |
Parses package catalog metadata. |
sdk_v2/cpp/test/internal_api/model_load_manager_test.cc |
Tests package loading behavior. |
sdk_v2/cpp/test/internal_api/model_info_test.cc |
Tests metadata serialization. |
sdk_v2/cpp/test/internal_api/local_model_scanner_test.cc |
Tests package cache discovery. |
sdk_v2/cpp/test/internal_api/download_test.cc |
Tests package download and caching. |
sdk_v2/cpp/test/internal_api/catalog_cache_test.cc |
Tests cached package metadata. |
sdk_v2/cpp/test/internal_api/azure_catalog_test.cc |
Tests parsing and compatibility filtering. |
model-package-foundry-local-integration.md |
Documents the integration design and scope. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Restrict the catalog-task multimodal fallback to model packages so flat ASR models (e.g. nemotron_speech) are not routed through OgaMultiModalProcessor. Skip the provider override for kCPU, whose GenAI provider name is empty, so default-loaded CPU catalog models are not configured with an empty provider. Move the model package design doc under sdk_v2/cpp/docs and mark Stage 1 as implemented.
|
Copilot resolve the merge conflicts in this pull request |
# Conflicts: # sdk_v2/cpp/src/catalog/azure_model_catalog.cc # sdk_v2/cpp/src/catalog/azure_model_catalog.h Co-authored-by: xiaoyu-work <85524621+xiaoyu-work@users.noreply.github.com>
Resolved in 252ecef. Conflicts were in Verified with |
GetModelCompatibilityForEpDevices reset the device_type optional and then dereferenced it inside the device loop. GCC cannot correlate the has_value() guard with the deref across the loop and fails the Linux build under -Werror. Collapse the optional into plain locals before the loop. Behavior is unchanged: a disengaged optional and kNotSet both mean "no device filter".
cfbda91 to
d39c33c
Compare
The catalog EP is only needed to select a package variant via OgaConfig::CreateFromPackageEp. Applying it to flat models also converted kDefault to a concrete EP, which skipped the load manager's generic-gpu CUDA/WebGPU preference and forced explicit provider configuration on every flat GPU/NPU catalog model. Add ModelInfo::IsModelPackage and gate the override on it.
The scoping document is working material rather than shipped documentation. Kept locally, out of the repo.
|
Copilot resolve the merge conflicts in this pull request |
Co-authored-by: xiaoyu-work <85524621+xiaoyu-work@users.noreply.github.com>
Resolved in f079dbd. The only conflict was in Verified with |
Summary
Add support for ORT model packages in the C++ SDK — catalog, download, cache discovery, and loading — alongside the existing flat model directories.
A model package is a directory that bundles several compiled variants of the same model behind a top-level
manifest.json. ORT GenAI loads the directory directly and picks the variant matching the local hardware.Stage 1 scope: one package per execution provider, single component, multiple variants. Design notes:
sdk_v2/cpp/docs/ModelPackageIntegration.md.Changes
ClassifyModelLayout()insrc/util/model_layout.{h,cc}. A rootgenai_config.jsonmeans flat;manifest.jsonwithout it means package.model_load_managerparsesmanifest.jsonand merges the per-variantgenai_config.jsonfiles conservatively.GenAIModelInstanceusesOgaConfig::CreateFromPackageEp()so GenAI does the variant selection.variantMetadata.modelPackage, addModelInfo::IsModelPackage(), and hide entries whose package variants are all unsupported on the current machine (fails open when compatibility is unknown).EpDetector::GetModelCompatibilityForEpDevices()to evaluate a variant's compatibility string against registered EP devices.inference_model.jsonmarker at the package root, and treat a package as a leaf when scanning the cache so variants don't surface as separate models.Model::Load— for packages only, a default load adopts the catalog EP so the right variant is selected.