loxc

Train a compression model on your text. Ship it. Compress and decompress with table-driven decode.

A trainable, table-driven text codec in pure C99. LOXC is intended for many small, structurally similar text messages where a trained table can be shared by sender and receiver. It trades a training step and table-distribution cost for a simple C99 runtime path. It is not a general-purpose archival codec and does not aim to beat zstd, Brotli, or LZ4 on arbitrary or repetitive data.

At a glance

Format generations:  v2 + matrix-based v3 (shared or split dictionary matrix)
V3 objectives:       payload / embedded / amortized
Intended deployment: one trained table reused across many related messages
Self-contained mode: includes the table per object and can be much larger
Runtime dependencies: standard C library only
Host (held-out JSON): v3 payload 6.3% (781 B / 12.1 KiB), 100.0 MB/s encode, 339.3 MB/s decode
Host deployment cost: JSON payload table 654 B; self-contained object 3,056 B
MCU capture (separate): ESP32-S3 @ 160 MHz, 32 B -> 60 B; 1,606 enc/s, 1,584 dec/s
Unfavorable case: held-out paths are 16.8% with LOXC v3 payload vs 3.3% Brotli CLI reference

See BENCHMARKS.md for the reproducible corpus and reporting rules. A table's bytes are part of the deployment cost: payload-only, self-contained, and shared-table/amortized results answer different questions. The ESP32-S3 benchmark was built, flashed, and serially captured on real hardware. Its raw JSON Lines, board/toolchain metadata, and ESP-IDF size report are retained in benchmarks/esp32s3/captures/. Those MCU results remain separate from host benchmarks; host results must never be presented as MCU measurements.

Host validation

Run the host-only checks with:

make clean && make && make embedded && make examples && make test
make fuzz-smoke
make sanitize
make analyze

fuzz-smoke generates deterministic mutations from valid v2/v3 tables and a framed container, then executes both parsers over valid, truncated, malformed, and invalid-embedded-table inputs. Full fuzzing uses the same targets with a libFuzzer-capable compiler and the sanitizer flags configured by the build environment:

make fuzz-full FUZZ_SECONDS=300

Language agnostic

loxc has no built-in language assumptions. The included demo module is trained on a public-domain text sample (Pride and Prejudice) for testing purposes only. For your own data:

• Slovak, Czech, Polish, and other natural language corpora: train on your corpus
• JSON, XML, and log lines: train on representative samples of your format
• URLs and file paths: train on a representative set
• Source code: train on files from your codebase

The codec works on bytes. Any text-like data with repeated domain-specific patterns can benefit when its table is trained on representative data.

Three lines to compress text

loxc_ctx_t *ctx = loxc_open("modules/loxc_demo.loxctab");
if (ctx != NULL) {
    loxc_buffer_t out = loxc_compress_buffer(ctx, "Hello world!", 12, 0);
    if (out.error == LOXC_OK) {
        /* use out.data / out.size */
        loxc_buffer_free(&out);
    }
    loxc_close(ctx);
}

That's it. out.data now holds compressed bytes.

See full examples -> | 5-minute tutorial ->

Why loxc?

Built for

• Domain-specific text: JSON APIs, log lines, URL paths, localization files
• Constrained C99 integrations: a table-driven runtime path and standard-library dependencies
• Repeated payloads: train once and reuse a domain table many times
• Explicit decode structure: matrix traversal and explicit table identity
• Deployment choice: external tables, self-contained embedded tables, or amortized table cost

Not for

• Archival compression, where zstd, brotli, xz, or similar codecs will usually give better ratios
• One-shot compression of unknown text, where a general-purpose compressor is simpler
• Encryption or authenticated storage; LOXC is a codec, not a cryptographic primitive

How it works

TRAINING (offline, once per corpus)

your_corpus.txt  -->  loxc_train  -->  mytable.loxctab
    |
    +-- Counts byte frequencies
    +-- Extracts deterministic dictionary candidates and corpus fragments
    +-- V3 refines the dictionary and evaluates shared/split layouts against the selected objective
    +-- Builds/scans canonical HIER4 and HIER8 matrix layouts
    `-- Emits .loxctab plus optional generated/static C module

RUNTIME (online, many times)

input text  -->  [encode via lookup tables]  -->  .loxc file
.loxc file  -->  [decode via lookup tables]  -->  output text

V3 training objectives and dictionary layouts

V3 separates the compression objective from the deployment model:

payload
    minimize payload bits; table size is only a deterministic tie-break

embedded
    minimize payload bits + serialized table bits for one self-contained payload

amortized --messages N
    minimize serialized table bits + N * payload bits

Dictionary candidates are ordered deterministically by isolated gain, then candidate length, then original index. The trainer scores the empty dictionary and every prefix of that order using the complete v3 layout scorer. It chooses the best prefix with deterministic tie-breaks on objective cost, payload cost, table size, dictionary count, and matrix dimension.

This is exact over the deterministic prefix search space. It is deliberately not a combinatorial search over every possible dictionary subset.

After this initial selection, v3 performs deterministic corpus-backed fragment refinement using exact longest-match usage accounting. It then evaluates both table architectures when a dictionary is present: shared, where bytes and dictionary entries occupy one matrix, and split, where MAIN dispatches a dictionary match to an independently optimized HIER4/HIER8 dictionary matrix. The selected table is whichever has the lower requested objective; this is automatic and has no separate command-line switch.

The v3 matrix idea

V3 uses real nested 4x4 or 8x8 matrix nodes. Diagonal positions save one coordinate:

HIER8 normal cell:   3-bit X + 3-bit Y = 6 bits
HIER8 diagonal cell: 3-bit X             = 3 bits

HIER4 normal cell:   2-bit X + 2-bit Y = 4 bits
HIER4 diagonal cell: 2-bit X             = 2 bits

Cells may hold a direct symbol, RAW fallback, or a child matrix. The trainer places weighted symbols and child transitions deterministically and compares HIER4/HIER8 using the selected objective.

See docs/FORMAT_V3.md for the exact grammar and on-disk ABI.

Benchmarks

The current make bench-full report intentionally uses objective-matched v3 tables:

• loxc-ext(v3_*_payload) uses --objective payload
• loxc-emb(v3_*_embedded) uses a separate --objective embedded table

That avoids charging self-contained mode for a table deliberately optimized only for repeated external payloads.

Current v0.4.3 measurements

The current checked-in report uses deterministic held-out JSON, structured-log, path, and key/value messages. It reports payload-only, self-contained, shared, and amortized deployment cost separately; see BENCHMARKS.md and the generated DEPLOYMENT_MODELS.md from make bench-full for exact values and break-even cases.

For an intentionally unfavorable device case, the captured 32-byte ESP32-S3 message expands to 60 bytes with the functional static table. The same capture shows a 223020-byte firmware image and stable 323764-byte free 8-bit heap; these are validation facts, not a claim that LOXC beats general-purpose codecs.

Baseline tools in the host report are CLI measurements and include process startup. LOXC host measurements are in-process, so the report labels those categories separately and does not use them for direct speed claims.

Full benchmark details ->

Quick start

Build

git clone https://github.com/Vanderhell/loxc
cd loxc && make

Use the CLI

./tools/loxc_cli compress \
    --table modules/loxc_demo.loxctab --embed \
    your_file.txt your_file.loxc

./tools/loxc_cli decompress your_file.loxc restored.txt

Use as a library

#include "loxc_simple.h"
#include <stdio.h>
#include <string.h>

int main(void) {
    loxc_ctx_t *ctx = loxc_open("modules/loxc_demo.loxctab");
    const char *text = "compress me";
    loxc_buffer_t out;

    if (ctx == NULL)
        return 1;

    out = loxc_compress_buffer(ctx, text, strlen(text), 0);
    if (out.error != LOXC_OK) {
        loxc_close(ctx);
        return 1;
    }

    printf("Original: %zu bytes, Compressed: %zu bytes\n",
           strlen(text), out.size);

    loxc_buffer_free(&out);
    loxc_close(ctx);
    return 0;
}

cc -Iinclude -Imodules myapp.c libloxc.a -o myapp && ./myapp

Train on your own data

For repeated external payloads:

./tools/loxc_train \
    --input your_data.txt \
    --output modules/loxc_mytable \
    --module-name mytable --module-id 50 \
    --format v3 --objective payload

For self-contained output where the table is embedded with each deployment:

./tools/loxc_train \
    --input your_data.txt \
    --output modules/loxc_mytable_embedded \
    --module-name mytable_embedded --module-id 51 \
    --format v3 --objective embedded

For an expected number of payloads sharing one table:

./tools/loxc_train \
    --input your_data.txt \
    --output modules/loxc_mytable_amortized \
    --module-name mytable_amortized --module-id 52 \
    --format v3 --objective amortized --messages 100

V2 training remains available for compatibility.

Full tutorial -> | Cookbook ->

Examples

Working code in examples/:

#	File	Shows
1	01_hello_world.c	Smallest possible usage
2	02_compress_file.c	File operations with timing
3	03_embedded_mode.c	Self-contained .loxc files
4	04_error_handling.c	Error handling paths
5	05_training_pipeline.c	Train and use a custom module
6	06_compare_modes.c	External vs embedded size tradeoff
7	07_streaming_chunks.c	Bounded framed v3 file streaming

Run them with make examples && ./examples/01_hello_world.

API

Simple API (recommended)

loxc_ctx_t *loxc_open(const char *table_path);
void        loxc_close(loxc_ctx_t *ctx);

int loxc_compress_file(loxc_ctx_t *ctx, const char *in_path,
                       const char *out_path, int embed_table);
int loxc_decompress_file(loxc_ctx_t *ctx, const char *in_path,
                         const char *out_path);

loxc_buffer_t loxc_compress_buffer(loxc_ctx_t *ctx,
                                   const void *data, size_t len,
                                   int embed_table);
loxc_buffer_t loxc_decompress_buffer(loxc_ctx_t *ctx,
                                     const void *data, size_t len);
void          loxc_buffer_free(loxc_buffer_t *buf);

const char   *loxc_strerror(int code);

Full API reference ->

Advanced API

For direct registry and buffer control, see docs/API.md#advanced-api.

Architecture

+-----------------------------------------------------------+
| Application                                               |
|  +-----------------------------------------------+        |
|  | loxc_simple.h  (recommended)                  |        |
|  | loxc.h         (low-level)                    |        |
|  +-----------------------------------------------+        |
+--------------------------+--------------------------------+
                           |
                           v
+-----------------------------------------------------------+
| libloxc.a                                                 |
|  +--------------+  +---------------+  +----------------+ |
|  | V2 strategy  |  | V3 matrix     |  | Stream Reader/ | |
|  | paths        |  | codec         |  | Writer         | |
|  +--------------+  +---------------+  +----------------+ |
|  +--------------+  +---------------+                     |
|  | Dictionary   |  | Module /      |                     |
|  | matching     |  | table loader  |                     |
|  +--------------+  +---------------+                     |
+--------------------------+--------------------------------+
                           |
                           v
+-----------------------------------------------------------+
| Module tables (.loxctab files)                            |
| Generated/static C modules or runtime-loaded tables       |
+-----------------------------------------------------------+

Full architecture ->

File structure

include/    public headers
src/        library implementation
tools/      loxc_train, loxc_cli, loxc_bench
tests/      unit tests
modules/    generated modules
benchmarks/ benchmark inputs and reports
trainings/  training data
examples/   runnable example programs
docs/       implementation documentation

Roadmap

• v0.1.0 - Initial release
• v0.2.0 - Benchmark suite + documentation overhaul
• v0.2.4 - Release workflow + documentation fixes
• v0.3.0 - Multi-module support, streaming API
• v0.4.1 - Matrix codec v3, framed streaming, deterministic training, generated/static v3 modules
• v0.4.2 - Objective-aware v3 dictionary training, runtime/codegen fixes, objective-matched benchmarks
• v0.4.3 - Corpus-backed dictionary fragment refinement and objective-selected shared/split v3 dictionary matrices
• v1.0.0 - Production-stable release target

Related work

Detailed comparison with Dense Codes, FSST, zstd dictionary mode, and Shared Brotli ->

Briefly, loxc is not a new compression principle. It is a practical recombination of:

• Dense-code-like prefix structure
• Learned per-corpus symbol tables
• Trained dictionary deployment
• External or embedded packaging

License

MIT - see LICENSE

Contributing

PRs welcome. See CONTRIBUTING.md.

Questions or bug reports: open an issue.