Reject swift://<repo-id>, add --model-name, guard zero-drift in analyze - #70
Open
alexkroman wants to merge 3 commits into
Open
Reject swift://<repo-id>, add --model-name, guard zero-drift in analyze#70alexkroman wants to merge 3 commits into
alexkroman wants to merge 3 commits into
Conversation
…zero-drift
- scripts/eval/cli.py:
- Reject `swift://<hf-repo-id>` invocations with a clear error pointing
at the build-bundle workflow. The Swift binary loads its SDK-pinned
bundle and ignores `repo_id`, so this form previously produced output
directories labeled with a model that was never actually evaluated —
e.g. `swift://mazesmazes/tiny-audio-next-multiasr` wrote
`*_tiny-audio-next-multiasr_<dataset>/` dirs whose contents were
byte-identical to the default Swift bundle eval. `ta analysis compare`
against such dirs surfaces matching rows that look like a bug in
`compare` but are actually identical underlying data.
- Add `--model-name` to override the auto-derived label used in output
dir names and downstream `ta analysis` matching. Routed through
`model_name or model_id` at all 5 save_*_results sites (ASR,
diarization, alignment, MCQ, classification).
- scripts/debug/analyze_weights.py:
- Guard the embed_tokens summary against zero `mean_drift`, which is
now the default whenever a checkpoint is trained with
`freeze_text_embed_tokens=true`. The previous code path divided
`max_drift / mean_drift` and crashed with ZeroDivisionError before
printing the RMSNorm section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Model + training plumbing: - Add `freeze_audio_encoder` config flag (default true). When false, the encoder is trainable and `ASRTrainer.create_optimizer` routes encoder params through `encoder_learning_rate` / `encoder_weight_decay` groups. Gradient checkpointing extends to the encoder only when trainable. - Add SpecAugment config family (`apply_spec_augment`, `mask_time_*`, `mask_feature_*`) on log-mel input — high-leverage when the encoder is trainable, no-op when frozen. - New `configs/experiments/encoder_train.yaml`: Whisper-Medium.en (308M, trainable) + MLP projector + Qwen3-0.6B (frozen). Sibling to stage_1's opposite-freeze recipe. - Production + main config: switch default data to multiasr, add projector_dropout 0.1, freeze_text_embed_tokens true, asymmetric LRs (1e-3 projector / 2e-5 decoder), projector_weight_decay 0, max_grad_norm 2.5 (grounded against observed grad-norm distribution). - MPS smoke simplified to mirror production shape. Eval confidence metrics: - `ASRPipeline` accepts `output_scores=True` and returns per-step top-1 and top-2 log-probabilities alongside generated IDs (back-compatible: off by default). - `LocalEvaluator.transcribe` returns an optional confidence dict; `EvalResult` carries `mean_top1_logprob` / `mean_margin` / `num_tokens` per sample; `Evaluator.compute_metrics` aggregates to corpus level (token-weighted); `scripts/analysis.py` surfaces `corpus_mean_top1_logprob` / `corpus_mean_margin`. Data mix (configs/data/multiasr.yaml): - Add MrDragonFox/EN_Emilia_Yodas_616h (228K / 616hr, Parquet, CC-BY-4.0). YouTube monologue re-transcribed with ElevenLabs Scribe v1 (`text_scribe`, cased + punctuated). Provenance + label-quality rationale in per-source comment. Caveat: single-speaker filter selects for monologue, not multi-speaker conversational. - Step Peoples `clean_sa` (311K) → `clean` capped at 1M. Closes the noisy-label dominance gap that motivated the prior rollback by keeping Peoples below the 1.5M threshold; relies on stage_1's wd=0.01 for label-noise damping. Recipe tuning (configs/experiments/stage_1.yaml): - `weight_decay`: 0 → 0.01. Closes the ~0.17 eval-train gap observed in run 4j97atjp (aug-off baseline). Prior wd=0 rationale block retained for context with a REVISED note inline. - Kept at 1 epoch: data mix already grew (Peoples 311K → 1M plus Emilia-YODAS 228K), so 1 epoch over the larger mix already exceeds the prior 1-epoch step count. Cleanup: - Remove unused `configs/data/loquacious*.yaml`, `multitask.yaml`, `demo/agent_demo.py`, `demo/streaming_demo.py`. - Add `scripts/debug/bench_specaugment.py` + `bench_specaugment_breakdown.py` for SpecAugment performance profiling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion WHAM (nguyenvulebinh/wham, 25K clips of pure background noise — cafes, restaurants, parks) is added at ~4.8% mix share via `target_samples: 265000` (10.6× upsample over the 25K natural clips). Every row trains the model on an empty assistant transcript via a new `text_override` config field, teaching "non-speech audio → emit nothing." Targets the classic Whisper-style hallucination where ambient noise produces "Thanks for watching" or other high-prior continuations. Plumbing in `scripts/train.py`: - `_prepare_split` honors `text_override` in dataset configs: replaces any existing text column with a constant string and adds the per-row `_allow_empty_label: True` marker. Marker is included in `keep_cols` so it survives the extra-column pruning. - `DataCollator._extract_audio_arrays` skips the empty-label filter for rows where `_allow_empty_label` is set. Other rows still get the filter, so Gigaspeech `<NOISE>`-only segments etc. continue to be dropped. - `DataCollator._build_sample` skips `_normalize_label` for override rows so the literal `""` reaches the chat template unmodified. multiasr.yaml: new WHAM entry at the end of the dataset list with provenance + upsampling rationale; header table updated to reflect the new mix shape (~5.57M effective total, WHAM at 265K = 4.8%). Verified with `pytest tests/test_data_collator.py tests/test_dataset_loader.py` — 24/24 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ta evalfootgun fix:-m swift://<hf-repo-id>is now rejected with an actionable error. The Swift binary loads its SDK-pinned bundle and ignoresrepo_id, so this form previously evaluated the default bundle while labeling output dirs with the passed repo id. Result: identical-contentresults.txtunder different model labels, which surfaces as "ta analysis compareshows identical WER for two different models" — looks like a bug incomparebut is actually identical underlying data.--model-nameflag: overrides the auto-derived label used in output dir names and downstreamta analysismatching. Useful when the path-derived label is unhelpful (e.g.swift-local-Modelwhen bundles live atswift/Sources/TinyAudio/Resources/Model/). Routed throughmodel_name or model_idat all 5 save_*_results sites (ASR, diarization, alignment, MCQ, classification).analyze_weights.pyzero-division fix: the embed_tokens drift summary dividedmax_drift / mean_driftwithout guarding against both being zero. That branch fires for every checkpoint trained withfreeze_text_embed_tokens=true(which is now the recipe default), crashing the decoder analysis before the RMSNorm section.Test plan
ta eval --helpshows--model-nameta eval -m swift://mazesmazes/foo …now errors with the build-bundle hint instead of silently evaluating the default bundleta eval -m swift(empty suffix) still works → labels outputsswift-default-bundleta eval -m swift://~/path/to/bundlestill works → labels outputsswift-local-<dirname>scripts/debug/analyze_weights.py … --component decodercompletes the embed_tokens section instead of crashing on a frozen-embed checkpoint (verified againstmazesmazes/tiny-audio-next-multiasrstep ~20k)--model-name foowrites*_foo_<dataset>/dirs across diarization/alignment/MCQ/classification (only ASR path manually tested)🤖 Generated with Claude Code