Summary
pocketsphinx_lm_convert aborts (and, in a release build, can corrupt memory) when writing an ARPA file from a trie language model whose stored header n-gram counts disagree with the model's actual trie content. The shipped model/en-us/en-us.lm.bin triggers it.
Mechanism (v5.1.1, commit 511126b)
ngram_model_trie_write_arpa (src/lm/ngram_model_trie.c) writes each n-gram section's count from the stored header (base->n_counts), allocates a raw_ngrams array of that size, fills it by walking the trie with lm_trie_fill_raw_ngram, and asserts the number of n-grams recovered equals the header count (line 288):
assert(raw_ngram_idx == base->n_counts[i - 1]);
When the header and the trie content disagree:
- With assertions enabled: the program aborts.
- With assertions disabled (a
-DNDEBUG / release build): if the trie holds fewer n-grams than the header claims, the subsequent qsort and write loop index raw_ngrams entries that were never filled and dereference their NULL words pointer; if the trie holds more, lm_trie_fill_raw_ngram writes past the end of the allocation. Both are memory-safety faults driven by the model file.
Reproduction
The shipped English model is inconsistent: its header declares 2051547 bigrams while the trie holds 2051541 (the six extra header slots are zero-padding). Converting it to ARPA aborts:
pocketsphinx_lm_convert -i model/en-us/en-us.lm.bin -o /tmp/en-us.arpa
# Assertion failed: (raw_ngram_idx == base->n_counts[i - 1]),
# function ngram_model_trie_write_arpa, file ngram_model_trie.c, line 288.
What this establishes
The abort and the count disagreement on the shipped model are directly reproducible. The header of en-us.lm.bin over-counts bigrams by six relative to its own trie content; the model still decodes correctly because scoring reads the bigram ranges from the trie body, not the header. The inconsistency appears to originate at model-build time (the count reserved by lm_trie_fix_counts and the count populated by recursive_insert can disagree); the current lm_convert build path does not reproduce it, so a consistent model round-trips cleanly.
Suggested handling
Discover the true per-order counts by walking the trie before writing, emit those counts so the ARPA is self-consistent, and bound the fill so it cannot overflow. On a header/content disagreement, report it and fail by default, with an option to write the actual counts.
Summary
pocketsphinx_lm_convertaborts (and, in a release build, can corrupt memory) when writing an ARPA file from a trie language model whose stored header n-gram counts disagree with the model's actual trie content. The shippedmodel/en-us/en-us.lm.bintriggers it.Mechanism (v5.1.1, commit 511126b)
ngram_model_trie_write_arpa(src/lm/ngram_model_trie.c) writes each n-gram section's count from the stored header (base->n_counts), allocates araw_ngramsarray of that size, fills it by walking the trie withlm_trie_fill_raw_ngram, and asserts the number of n-grams recovered equals the header count (line 288):When the header and the trie content disagree:
-DNDEBUG/ release build): if the trie holds fewer n-grams than the header claims, the subsequentqsortand write loop indexraw_ngramsentries that were never filled and dereference their NULLwordspointer; if the trie holds more,lm_trie_fill_raw_ngramwrites past the end of the allocation. Both are memory-safety faults driven by the model file.Reproduction
The shipped English model is inconsistent: its header declares 2051547 bigrams while the trie holds 2051541 (the six extra header slots are zero-padding). Converting it to ARPA aborts:
What this establishes
The abort and the count disagreement on the shipped model are directly reproducible. The header of
en-us.lm.binover-counts bigrams by six relative to its own trie content; the model still decodes correctly because scoring reads the bigram ranges from the trie body, not the header. The inconsistency appears to originate at model-build time (the count reserved bylm_trie_fix_countsand the count populated byrecursive_insertcan disagree); the currentlm_convertbuild path does not reproduce it, so a consistent model round-trips cleanly.Suggested handling
Discover the true per-order counts by walking the trie before writing, emit those counts so the ARPA is self-consistent, and bound the fill so it cannot overflow. On a header/content disagreement, report it and fail by default, with an option to write the actual counts.