Reuse compatible GGUF weights as ONNX external data - #573
Open
justinchuby wants to merge 4 commits into
Open
Conversation
Add an explicit build-gguf opt-in that points compatible ONNX initializers at exact tensor ranges in the original GGUF while externalizing converted weights to a separate sidecar. Pin and verify source provenance, enforce safe flat packaging, and preserve mixed external references during save. Document the initial consumer and layout limitations, including deferred graph-expressible transforms and ORT optimization requirements, and cover mixed save/reload, exact ranges, unsafe paths, identity changes, and CPU inference. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
Reject ORT GenAI packaging when constant folding cannot be disabled, verify every external initializer and graph-level transform against the source manifest, and retain graph-expressible float transforms over GGUF-backed bytes. Stage, fsync, validate, and transactionally install the model, sidecar, and manifest with durable rollback journals, crash recovery, confined journal paths, obsolete artifact removal, and explicit filesystem capability checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Protect mixed-package recovery and replacement with a persistent advisory lock, publish transaction state atomically, and reject unsafe generated artifacts without following links. Validate converted sidecar references against their exact dtype-and-shape byte size and cover concurrent ownership, crash recovery, symlink mutation, and truncated or oversized ranges. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e99f9fcb-697e-4853-b34c-b854c3995535 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Remove filename-inferred stale staging cleanup so recovery deletes only artifacts named by a structurally validated transaction journal. Cover a source GGUF and unrelated user file whose names resemble generated staging files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e99f9fcb-697e-4853-b34c-b854c3995535 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #571 (
justinchuby-gguf-registry-foundation). Adds an explicit opt-in path that publishes ONNX beside the original GGUF without writing a second full copy of byte-compatible weights.Storage and transform behavior
location + offset + lengthranges.A_log/shape transforms keep the source bytes and run as ONNX graph operations. Opaque packed UINT8 is never generically transposed.model.onnx.data.Persistence and verification
A persistent same-directory advisory lock serializes writers, recovery, and shared verification. Mixed saving stages and fsyncs the sidecar, model, and manifest, validates the complete staged package, then installs it through an atomically published durable transaction journal with hard-linked rollback backups. The journal records replacing/committed state so crashes cannot produce mixed rollback or leak full backup copies. Interrupted reruns recover before save/verification; failed reruns restore prior artifacts; obsolete sidecars are removed transactionally.
Recovery deletes only staged and backup paths owned by a structurally and path-validated journal; generated-looking filenames are never treated as proof of ownership. Lock, journal, staged, backup, model, manifest, and sidecar paths reject symlinks and non-regular artifacts. Lock acquisition validates directory-entry/opened inode identity; journal files are created through exclusive no-follow temporary files and atomic replacement.
gguf-reuse.jsonpins source location, size, SHA-256, source tensor/qtype/range, initializer dtype/length, source/final shapes, transform name, and transform parameters. Verification enumerates every ONNX external initializer: each must be either an exact manifest-pinned GGUF range or a non-overlapping range in the one permitted sidecar. Sidecar ranges must exactly equal the byte count implied by concrete non-negative shape and supported dtype. Verification rejects unmanifested/unsafe locations, wrong dtype/qtype/shape/ranges, mixed reused/converted claims, and mutated transform wiring/constants/permutations.Scope and runtime requirements
This PR supports one flat text-model package. The real little-endian GGUF must already be in the output directory. It rejects nested/non-flat packaging, mmproj/MTP packages, symlinks, unsafe hardlinks, generated-name collisions, changed source identity, safetensors output, and sidecar sharding rather than copying or linking the GGUF.
Use direct ONNX Runtime with graph optimization disabled:
--reuse-gguf-weights --runtime ort-genaiis rejected because the currentgenai_config.jsonschema has no supported setting that requires disabled constant folding.Validation
lintrunner f --output oneline --all-filesandlintrunner -apass locally.Waiver
No multi-GB public GGUF was downloaded. Real-weight validation is deferred to avoid a large network/storage dependency; synthetic files use the real GGUF writer/reader, exact offsets, native block routes, mixed persistence, reload, and ORT execution.