Skip to content

Qwen model support - #150

Open
cv-chaitali wants to merge 5 commits into
AlignmentResearch:mainfrom
cv-chaitali:qwen-model-support
Open

Qwen model support#150
cv-chaitali wants to merge 5 commits into
AlignmentResearch:mainfrom
cv-chaitali:qwen-model-support

Conversation

@cv-chaitali

Copy link
Copy Markdown

Summary

Adds support for Qwen-family decoder models in tuned-lens.

Changes

  • Supports Qwen decoder models that expose base_model.norm and base_model.layers
  • Adds test coverage for Qwen2, Qwen2-MoE, Qwen3, Qwen3-MoE, and Qwen3-Next
  • Adds a small Qwen support note with a real Qwen/Qwen3-0.6B LogitLens result

Related issue: #143

Copilot AI review requested due to automatic review settings July 18, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for Qwen-family decoder models in tuned_lens.model_surgery by recognizing Qwen base-model structures that expose a decoder stack (layers) and final normalization (norm), unblocking LogitLens / PredictionTrajectory usage for Qwen models (as reported in issue #143).

Changes:

  • Add a Qwen-family decoder detection helper and use it to route get_final_norm() and get_transformer_layers() to base_model.norm / base_model.layers.
  • Extend the test model fixture to include tiny Qwen2/Qwen3 (dense + MoE + Next) configs instantiated via AutoModelForCausalLM.from_config.
  • Add a short documentation note with an example LogitLens result for Qwen/Qwen3-0.6B.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.

File Description
tuned_lens/model_surgery.py Adds Qwen decoder recognition and maps Qwen final norm/layers to base_model.norm and base_model.layers.
tests/conftest.py Adds tiny Qwen config helpers and includes Qwen2/Qwen3 variants in the randomized small-model test fixture.
docs/qwen-support/README.md Documents the motivating error/issue and summarizes the Qwen support changes and a sample result.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants