feat(internal): dictionary-free entropy coding, RLE blocks, and multi-block frames

AGENTS.md

@@ -36,10 +36,22 @@ would be over-engineering here.
 
 - **One-shot only (no incremental streaming API yet).** Every frame is independently
   decodable; there is no cross-call state.
-- **One block per frame (current limit).** The encoder emits a single block, so
-  `compress` rejects inputs > 128 KiB (zstd's `Block_Maximum_Size`) with a
-  `ZstdException`. Keep that guard until multi-block encoding lands — without it a
-  large input silently produces a frame neither libzstd nor kzstd can decode.
+- **Blocks are independent, and per-frame state is threaded through them.** The
+  encoder cuts input into 128 KiB (`Block_Maximum_Size`) chunks and emits a
+  multi-block frame; a chunk is matched only against itself and the dictionary,
+  never against an earlier block's output. The three repeat offsets and the tables
+  the "Repeat" / "Treeless" modes name are FRAME state — `PureZstdEncoder`'s
+  `FrameEntropy` must keep mirroring `PureZstdDecoder`'s `DecodeState` exactly, or
+  a mode names one table on each side and the frame decodes to garbage that only
+  a real-libzstd oracle catches.
+- **Total history (dictionary content + input) is capped at 128 MiB.** The
+  frame header always declares a window covering the full history; beyond
+  128 MiB that window exceeds libzstd's default decompression limit
+  (`ZSTD_WINDOWLOG_LIMIT_DEFAULT` = windowLog 27), so `encode()` rejects it
+  with a `ZstdException` up front rather than emit a frame most real-world
+  libzstd consumers refuse. Keep this guard — it's what "frames stay
+  libzstd-interoperable in both directions" actually requires now that
+  multi-block encoding has no other size limit.
 - **No shared mutable state, no lock.** A `ZstdDictionary` digests its dictionary
   once in its constructor and is immutable thereafter; the engine objects keep all
   per-call state in locals. Do not reintroduce global caches. The encoder's

CHANGELOG.md

@@ -7,9 +7,10 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [Unreleased]
 
 Encoder-side parity work closing several gaps against the libzstd/RFC 8878
-spec — real ratio improvements for dictionary-compressed frames, no wire
-format or public-API changes (except where noted below, for the
-Content_Checksum fix).
+spec — the 128 KiB single-block input limit lifted, real ratio improvements
+for both dictionary-compressed and dictionary-free frames, plus
+dictionary-ID and content-checksum validation. No wire format or
+public-API changes, except where noted below.
 
 ### Fixed
 
@@ -19,13 +20,28 @@ Content_Checksum fix).
   content, for ANY conformant frame — not just kzstd's own — so a real
   `libzstd`-produced frame (checksums are on by default in the `zstd` CLI)
   is no longer accepted with silently-corrupted content.
+- The decoder now validates a frame's declared Dictionary_ID against the
+  supplied dictionary's own embedded ID (when both are present), throwing a
+  clear `ZstdException` on mismatch instead of a generic corruption error.
 
 ### Added
 
 - `Zstd.compress` takes an opt-in `checksum: Boolean = false` parameter; when
   true, the encoder sets `Content_Checksum_Flag` and appends the XXH64
   checksum of the input. Defaults to false, so every existing call site's
   frame bytes are unchanged.
+- `Zstd.compress` now accepts input up to 128 MiB (dictionary content + data
+  combined), up from 128 KiB. It cuts the input into 128 KiB
+  (`Block_Maximum_Size`) chunks and emits them as one multi-block frame,
+  `Last_Block` set on the final block only; anything larger than 128 KiB used
+  to be rejected with a `ZstdException`. The new 128 MiB ceiling replaces that
+  guard: beyond it, the frame's required window would exceed libzstd's default
+  decompression limit (`ZSTD_WINDOWLOG_LIMIT_DEFAULT`), so it's still rejected
+  rather than emitting a frame most real-world libzstd consumers would refuse.
+  `Zstd.decompress` has always read multi-block frames from any encoder, and
+  real libzstd reads these. 3 MB of synthetic JSON telemetry compresses to
+  556,657 bytes across 24 blocks — between libzstd's level 3 (579,374) and
+  level 19 (362,967).
 
 ### Changed
 
@@ -38,11 +54,39 @@ Content_Checksum fix).
   needs and doing so is smaller than the previous fallback (predefined FSE
   tables, raw literals) — a real, measurable size reduction for
   dictionary-compressed frames, not just a wire-format curiosity (#50, #51).
+- Without a dictionary — the plain `Zstd.compress(data)` call — the encoder now
+  entropy-codes each block from the block's OWN data, where before it could
+  only emit raw literals and the spec's predefined FSE distributions: Huffman
+  literals built from the block's byte histogram (`Literals_Block_Type` 2), and
+  FSE tables for the literal-length / offset / match-length streams normalized
+  from the block's own code counts (`Symbol_Compression_Mode` 2). Measured:
+  ~7.8 KB of synthetic JSON telemetry records 2007 → 1521 bytes, 887 bytes of
+  concatenated structured records 487 → 425, a 208-byte prose sample 213 → 190.
+- The encoder now also emits the RLE forms the decoder has always read:
+  `RLE_Block` for a constant input (a 1500-byte run of one byte, 17 → 10
+  bytes), RLE literals when every literal is the same byte, and RLE sequence
+  tables when a stream's every code is the same (a sample of 26 byte-runs,
+  85 → 42 bytes).
+- Every per-block encoding choice — the literals section and each of the three
+  sequence streams independently — is now made by measuring every valid
+  alternative and taking the smallest, so a form is used only when it actually
+  wins. Ties keep the previous behaviour, and dictionary-compressed frames come
+  out the same size as before.
+- Entropy tables and the three repeat offsets are now carried from block to
+  block within a frame, which is what the format means by them: "Repeat"
+  sequence tables and "Treeless" literals name the PREVIOUS block's tables, and
+  the dictionary's only for a frame's first block. A later block therefore
+  reuses a table for nothing instead of describing its own, and a `Raw` or
+  `RLE` block describes nothing and so leaves the state untouched — the block
+  after a stretch of incompressible data still reaches the dictionary's own
+  tables. A 128 KiB noise block followed by a dictionary-trained sample
+  compresses that sample's block to 46 bytes rather than 54.
 - `level` (1–22) now governs match-finding search depth: higher levels search
   more candidate matches per position, which can shrink output at the cost of
-  more work. Level 19 (`Zstd.DEFAULT_LEVEL`) is byte-identical to every
-  earlier release; the encoder still uses one fixed strategy at every level,
-  not zstd's other per-level parameters (#52).
+  more work. Level 19 (`Zstd.DEFAULT_LEVEL`) maps to exactly the search depth
+  the encoder always used, so the mapping itself changes no output; the encoder
+  still uses one fixed strategy at every level, not zstd's other per-level
+  parameters (#52).
 
 ### Fixed
 
@@ -58,6 +102,18 @@ Content_Checksum fix).
 
 ### Notes
 
+- Blocks are compressed independently: a match never reaches back into an
+  earlier block's output, only into this block and the dictionary. Large
+  inputs therefore compress less well than a windowed encoder would manage —
+  and combined with the 1023-byte literals cap below, a full 128 KiB block
+  keeps raw literals and takes its ratio from the sequence tables alone. A
+  windowed, cross-block matcher is the follow-up.
+- Huffman-coded literals stay single-stream, so they apply to at most 1023
+  bytes of literals per block, and their tree description uses the direct
+  4-bit weight form, so a block containing a literal byte above 128 falls back
+  to raw literals. FSE-compressed weight descriptions and the 4-stream literals
+  layout would lift those limits and are not implemented; neither affects
+  decoding, which reads both.
 - A dictionary-compressed frame's correctness now depends on the dictionary's
   entropy tables matching what the decoder is seeded with, not just its
   content — decoding with the wrong dictionary was already silently wrong

README.md

@@ -65,22 +65,41 @@ val back = Zstd.decompress(small, dict, maxSize = 64 * 1024)
   single fixed greedy/lazy strategy at every level — it does not implement zstd's
   other per-level parameters (window log, target length, etc.) — but a higher
   level does search more candidate matches per position, which can shrink output
-  at the cost of more work. Level 19 (`Zstd.DEFAULT_LEVEL`) is unchanged from
-  every earlier release. Frames remain fully libzstd-compatible at every level.
-- **Single block per frame (≤ 128 KiB input).** `Zstd.compress` emits one zstd block,
-  so its input is bounded by zstd's 128 KiB `Block_Maximum_Size`; a larger input
-  throws `ZstdException`. (`Zstd.decompress` reads multi-block frames from any
-  encoder.) Multi-block encoding to lift the cap is planned.
+  at the cost of more work. Frames remain fully libzstd-compatible at every level.
+- **Blocks are compressed independently (no cross-block matching).** `Zstd.compress`
+  cuts input into zstd's 128 KiB `Block_Maximum_Size` chunks and emits one
+  multi-block frame. Each chunk is matched only against itself and the
+  dictionary, never against an earlier block's output, so a large input
+  compresses less well than a windowed encoder manages — 3 MB of synthetic
+  JSON telemetry lands between libzstd's levels 3 and 19. Entropy tables and the
+  repeat offsets ARE carried across blocks. A windowed matcher is a planned
+  improvement.
+- **Total input (dictionary content + data) is capped at 128 MiB.** Beyond that,
+  the window a frame must declare exceeds libzstd's default decompression limit
+  (`ZSTD_WINDOWLOG_LIMIT_DEFAULT`), so `Zstd.compress` throws `ZstdException`
+  rather than emit a frame most real-world libzstd consumers would refuse to
+  decode.
+- **Huffman-coded literals are single-stream and directly described.** The encoder
+  builds a Huffman table from a block's own literals, but writes only the
+  single-stream layout (so it applies to at most 1023 bytes of literals per block)
+  and only the direct 4-bit weight tree description (so a block containing a literal
+  byte above 128 falls back to raw literals). Both limits are encoder-side only —
+  `Zstd.decompress` reads the 4-stream layout and FSE-compressed weight descriptions
+  that libzstd emits. The 1023-byte cap is why a full 128 KiB block keeps raw
+  literals: on large inputs the ratio comes from the sequence tables alone.
 
 ## Interoperability
 
 kzstd reads frames produced by libzstd (including dictionary-compressed frames
 that use the dictionary's Huffman/FSE entropy tables), and libzstd reads frames
-produced by kzstd — including, now, dictionary-compressed frames kzstd itself
-produces using the dictionary's trained entropy tables and repeat-offset codes,
-when doing so is smaller than the fallback. The test suite cross-checks both
-directions against [zstd-jni](https://github.com/luben/zstd-jni) (a
-JVM-test-only oracle, never a runtime dependency).
+produced by kzstd — including frames kzstd itself entropy-codes: dictionary
+frames using the dictionary's trained tables and repeat-offset codes, and
+dictionary-free frames using Huffman/FSE tables built from the block's own data
+(or the RLE forms, when a block, its literals or a symbol stream is constant).
+Each of those forms is picked only when it is the smallest valid encoding. The
+test suite cross-checks both directions against
+[zstd-jni](https://github.com/luben/zstd-jni) (a JVM-test-only oracle, never a
+runtime dependency).
 
 ## Building & testing
 

src/commonMain/kotlin/org/meshtastic/kzstd/Zstd.kt

@@ -13,10 +13,12 @@ import org.meshtastic.kzstd.internal.PureZstdEncoder
  * no cross-call state, so every frame is independently decodable (what packet /
  * mesh transports need).
  *
- * [compress] emits a single zstd block per frame, so its input is bounded by zstd's
- * 128 KiB `Block_Maximum_Size`; a larger input throws [ZstdException] (multi-block
- * encoding is a planned addition). [decompress] reads any conformant frame,
- * including multi-block frames produced by other encoders.
+ * [compress] takes an input of any size: it cuts the input into zstd's 128 KiB
+ * `Block_Maximum_Size` chunks and emits them as one multi-block frame. The chunks
+ * are compressed independently — a match never reaches back into an earlier
+ * block — so a large input compresses somewhat less well than a windowed encoder
+ * would manage. [decompress] reads any conformant frame, including multi-block
+ * frames produced by other encoders.
  *
  * Pass a [ZstdDictionary] for dictionary compression; the dictionary-less
  * overloads operate on plain frames.

src/commonMain/kotlin/org/meshtastic/kzstd/internal/Fse.kt

@@ -107,6 +107,20 @@ internal class FseTable(
             return FseTable(tableLog, symbolTable, nbBitsTable, newStateTable)
         }
 
+        /**
+         * The degenerate one-symbol table used by Symbol_Compression_Mode 1
+         * (RLE): [symbol] has probability 1, so decoding stays in state 0
+         * forever and consumes no bits. Shared by the decoder (which reads the
+         * mode-1 description byte) and the encoder (which emits it when a
+         * stream's every code is the same).
+         */
+        fun rle(symbol: Int): FseTable = FseTable(
+            tableLog = 0,
+            symbol = intArrayOf(symbol),
+            nbBits = intArrayOf(0),
+            newState = intArrayOf(0),
+        )
+
         /** Floor(log2(v)) for v >= 1. */
         private fun highBit(v: Int): Int {
             var n = 0

src/commonMain/kotlin/org/meshtastic/kzstd/internal/FseEncoder.kt

@@ -53,25 +53,53 @@ internal class FseEncTable private constructor(
      * initial state via [initialState].
      */
     fun encode(bw: ReverseBitWriter, state: Int, symbol: Int): Int {
-        val states = symbolStates[symbol]
-        // Find the decode-state `ds` emitting `symbol` whose output range
-        // [base, base + 2^nb) contains the target `state`. Ranges partition
-        // [0,tableSize), so exactly one matches. The encoder emits `state - base`
-        // in `nb` bits and moves to `ds`.
-        for (ds in states) {
-            val nb = nbBits[ds]
+        val ds = transition(state, symbol)
+        bw.writeBits(state - newStateBase[ds], nbBits[ds])
+        return ds
+    }
+
+    /**
+     * The decode-state `ds` emitting [symbol] whose output range
+     * `[base, base + 2^nb)` contains [state]. Ranges partition `[0,tableSize)`,
+     * so exactly one matches; the caller emits `state - base` in `nb` bits and
+     * moves to `ds`.
+     */
+    private fun transition(state: Int, symbol: Int): Int {
+        for (ds in symbolStates[symbol]) {
             val base = newStateBase[ds]
-            val hi = base + (1 shl nb)
-            if (state in base until hi) {
-                bw.writeBits(state - base, nb)
-                return ds
-            }
+            if (state >= base && state < base + (1 shl nbBits[ds])) return ds
         }
         // The FSE invariant guarantees a match; reaching here means a corrupt
         // table or an out-of-range symbol the caller failed to bound.
         throw ZstdException("FSE encode: no transition for symbol $symbol from state $state")
     }
 
+    /**
+     * Exactly how many bits encoding [codes] (chronological order) with this
+     * table would cost — every transition plus the flushed initial state — or
+     * null when some code has no code point here ([isCovered]).
+     *
+     * This walks the SAME path [encode] does (backwards from the last code,
+     * starting at [initialState]), so it is an exact count and not an entropy
+     * estimate: the cost model that chooses between Predefined / RLE /
+     * FSE_Compressed / Repeat can therefore never pick a mode that turns out
+     * bigger than it predicted. Bits from the three sequence streams simply
+     * add, so streams can be costed independently even though they interleave
+     * in the final bitstream.
+     */
+    fun streamBitCost(codes: IntArray): Long? {
+        if (codes.isEmpty()) return null
+        for (c in codes) if (!isCovered(c)) return null
+        var state = initialState(codes[codes.size - 1])
+        var bits = 0L
+        for (i in codes.size - 2 downTo 0) {
+            val ds = transition(state, codes[i])
+            bits += nbBits[ds]
+            state = ds
+        }
+        return bits + tableLog
+    }
+
     /**
      * Pick the initial encoder state for [symbol] (the LAST output symbol, which
      * the encoder processes first). Any decode-state that emits [symbol] is a