fix(model): recognize symlink aliases as the loaded model during inline loading - #436
Open
Anai-Guo wants to merge 1 commit into
Open
fix(model): recognize symlink aliases as the loaded model during inline loading#436Anai-Guo wants to merge 1 commit into
Anai-Guo wants to merge 1 commit into
Conversation
…iases don't reload An inline model request identifies the loaded model by comparing the requested name against container.model_dir.name. Because the container is created from model_path.resolve(), model_dir.name is always the real directory name. When a model is addressed through a symlink alias (e.g. a 'coder' symlink to 'ArtusDev_Qwen_...-EXL3'), the names never match, so the already-loaded model is needlessly unloaded and reloaded on every request. Compare resolved paths instead of bare directory names in both the inline fast-path guard and load_model_gen, so a symlink that points at the loaded model is recognized as the same model. Fixes theroyallab#379
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #379.
With
inline_model_loading: true, a model can be addressed by directory name in the request'smodelfield. If that name is a symlink to the real model directory (the issue's example: acodersymlink pointing atArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3), the model is unloaded and reloaded on every request even though it is already loaded.Cause
The "is this already loaded?" check compares the requested name against
container.model_dir.name:But the container is created from
model_path.resolve()(common/model.py), somodel_dir.nameis always the real directory name. A symlink alias (coder) never equals the resolved name (ArtusDev_...-EXL3), so the guard fails and the model is reloaded. The same bare-name comparison sits in the inline fast-path guard inendpoints/OAI/utils/common_.py.Fix
Compare resolved paths instead of bare directory names in both places, so a symlink that points at the currently-loaded model is recognized as the same model. Non-symlink paths resolve to themselves, so existing behavior is unchanged.
common/model.py—load_model_gen:container.model_dir.resolve() == model_path.resolve()endpoints/OAI/utils/common_.py—load_inline_model: resolve the requestedmodel_dir / model_nameand compare against the loaded container's resolved dir.🤖 Generated with Claude Code