Skip to content

fix(model): recognize symlink aliases as the loaded model during inline loading - #436

Open
Anai-Guo wants to merge 1 commit into
theroyallab:mainfrom
Anai-Guo:fix/inline-model-symlink-reload
Open

fix(model): recognize symlink aliases as the loaded model during inline loading#436
Anai-Guo wants to merge 1 commit into
theroyallab:mainfrom
Anai-Guo:fix/inline-model-symlink-reload

Conversation

@Anai-Guo

Copy link
Copy Markdown

Problem

Fixes #379.

With inline_model_loading: true, a model can be addressed by directory name in the request's model field. If that name is a symlink to the real model directory (the issue's example: a coder symlink pointing at ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3), the model is unloaded and reloaded on every request even though it is already loaded.

Cause

The "is this already loaded?" check compares the requested name against container.model_dir.name:

loaded_model_name = container.model_dir.name
if loaded_model_name == model_path.name and container.loaded:
    ...  # already loaded, skip

But the container is created from model_path.resolve() (common/model.py), so model_dir.name is always the real directory name. A symlink alias (coder) never equals the resolved name (ArtusDev_...-EXL3), so the guard fails and the model is reloaded. The same bare-name comparison sits in the inline fast-path guard in endpoints/OAI/utils/common_.py.

Fix

Compare resolved paths instead of bare directory names in both places, so a symlink that points at the currently-loaded model is recognized as the same model. Non-symlink paths resolve to themselves, so existing behavior is unchanged.

  • common/model.pyload_model_gen: container.model_dir.resolve() == model_path.resolve()
  • endpoints/OAI/utils/common_.pyload_inline_model: resolve the requested model_dir / model_name and compare against the loaded container's resolved dir.

🤖 Generated with Claude Code

…iases don't reload

An inline model request identifies the loaded model by comparing the requested name against container.model_dir.name. Because the container is created from model_path.resolve(), model_dir.name is always the real directory name. When a model is addressed through a symlink alias (e.g. a 'coder' symlink to 'ArtusDev_Qwen_...-EXL3'), the names never match, so the already-loaded model is needlessly unloaded and reloaded on every request.

Compare resolved paths instead of bare directory names in both the inline fast-path guard and load_model_gen, so a symlink that points at the loaded model is recognized as the same model.

Fixes theroyallab#379
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Inline loading of models at Symlinks causes models to unload and reload

1 participant