sist2 user script that embeds the text of documents for semantic search. Long documents are split into overlapping chunks, each with its own embedding, so a search matches the passage that answers it rather than the document as a whole. sist2 shows that passage as the result excerpt.
Needs sist2 4.1 or newer, and the SQLite search backend (sist2 sqlite-index + sist2 web --search-index).
Usage:
python run.py --help
| Option | Default | |
|---|---|---|
--model-name |
sentence-transformers/all-MiniLM-L6-v2 |
Any sentence-transformers model. Its embedding size must be one sist2 maps: 384, 512, 768 or 1024. |
--name |
MiniLM |
Name of the model in the index, at most 15 characters. |
--model-url |
the published MiniLM encoder | URL of an .onnx text encoder for the web UI, with its tokenizer.json next to it. Defaults to the published one only for the default --model-name; another model needs its own, or its embeddings cannot be searched from the browser. |
--model-id |
2 |
Model id in the index. The CLIP script uses 1, so both can run on the same index. |
--chunk-size |
1200 |
Chunk length in bytes. Roughly 250 English tokens, which is what MiniLM takes. |
--chunk-overlap |
200 |
Bytes each chunk repeats from the one before, so a passage split across a boundary is still whole in one of them. |
--max-chunks |
0 |
Chunks per document, 0 for no limit. What is dropped is logged. |
--batch-size |
32 |
Chunks per inference batch. |
--force |
Discard the existing embeddings and process every document again. Implied when --model-name changes. |
Only documents with extracted text are processed, and only those modified since the last run. Progress is committed between documents, so an interrupted run resumes where it stopped and never leaves a document half embedded.
Every chunk stores 4 × dimensions bytes: 1536 for MiniLM.