FastFileContentIndex 0.1.3 [ALPHA] — Ultra-Fast In-File Text Search & 3-Gram Bloom Index for Java

---

⚡ High-performance 3-Gram Bloom Filter indexer and SIMD substring scanner for ultra-fast full-text search across documents, code, PDFs, and OCR image content in sub-millisecond speeds.

FastFileContentIndex is the third pillar of the FastJava search ecosystem (alongside FastFileIndex and FastFileSearch). It provides a highly optimized, 3-gram bitmask index designed specifically for real-time universal search ("Raycast / Spotlight for Documents, Code, PDFs, and OCR Screenshots").

Unlike heavy solutions (Elasticsearch, Lucene) that heavily tokenize and parse text, FastFileContentIndex uses a lightweight bitmask of 3-grams to quickly filter out non-matching files, completing substring searches in sub-millisecond speeds.

---

Quick Start

import fastfilecontentindex.FastFileContentIndex;
import fastfilecontentindex.ContentMatchResult;
import fastfileindex.FastFileIndex;
import fastansi.FastANSI;

import java.io.File;
import java.util.List;

public class FastContentIndexDemo {
    public static void main(String[] args) throws Exception {
        File targetDir = new File(".");

        // STEP 1: FastFileIndex — Instant Memory-Mapped Directory Tree Discovery
        System.out.println("--- Step 1: FastFileIndex Directory Tree Discovery ---");
        FastFileIndex.build(new String[]{targetDir.getAbsolutePath()});
        System.out.printf("Scanned %d file entries.%n%n", FastFileIndex.getEntryCount());

        // STEP 2: FastFileContentIndex — FastIO Direct Streaming & 64-Bit Bloom Indexing
        System.out.println("--- Step 2: 3-Gram Bloom Filter Chunk Indexing ---");
        FastFileContentIndex index = new FastFileContentIndex();
        index.indexDirectory(targetDir);
        System.out.printf("Indexed %d files (%d chunks of 64 KiB).%n%n",
            index.getIndexedFileCount(), index.getIndexedChunkCount());

        // STEP 3: Sub-Millisecond SIMD Candidate Search & FastANSI Highlighting
        System.out.println("--- Step 3: Sub-Millisecond SIMD Content Search ---");
        List<ContentMatchResult> results = index.search("TrigramBloomFilter");

        for (ContentMatchResult r : results) {
            double ms = r.searchTimeNs() / 1_000_000.0;
            System.out.printf("[%s%5.2f ms%s] %s:%d:%d -> %s%n",
                FastANSI.fg(0x9E, 0xCE, 0x6A), ms, FastANSI.RESET,
                r.filePath(), r.lineNumber(), r.charOffset(), r.lineSnippet().trim());
        }
    }
}

---

Table of Contents

• Why FastFileContentIndex?
• Key Features
• Real-World Use Cases
• Technical Architecture
• Installation
• Documentation
• Platform Support
• License
• Related Projects

---

Why FastFileContentIndex?

Traditional full-text search engines (Lucene, Elasticsearch) rely on heavy inverted indexes and lexical tokenization pipelines that consume huge amounts of memory and CPU during indexing. FastFileContentIndex provides:

• Fast Bitmask Rejection — Evaluates 24-bit 3-gram bitmasks to reject non-matching files without reading disk contents.
• Zero-Allocation Result Streaming — Low-overhead result models returning exact line numbers, char offsets, and line snippets.
• Lightweight Memory Footprint — Requires only a fraction of the RAM used by traditional text search engines.
• Zero Dependencies — Standalone, lightweight JAR (< 50 KB).

---

Key Features

• ⚡ 3-Gram Bloom Filter Rejection — Fast 64-bit bitmask rejection per 64 KiB chunk without touching disk contents.
• 🚀 FastIO Native JNI Direct I/O — Leverages FastIO JNI unbuffered native file reading with allocateAlignedBuffer() for direct, zero-copy sector streaming.
• 🔍 Sub-Millisecond SIMD Search — Blazing fast full-text substring queries using 256-bit / 32-byte AVX2 vector loads (FastSIMD & FastBytes).
• 🎯 O(log N) Zero-Alloc Result Extraction — Binary-searchable pre-indexed line/char offsets with zero temporary String or getBytes() allocations during scan loops.
• 🎨 FastANSI Integration — Native support for 24-bit TrueColor terminal output formatting and match highlighting.
• 🧱 FastJava Stack Compatibility — Integrates seamlessly with FastFileIndex, FastIO, FastContentParse, FastBytes, and FastSIMD.

---

Core Engineering Pillars

FastFileContentIndex achieves its extreme performance by combining 5 complementary low-level technologies:

1. 🚀 FastIO Native JNI Unbuffered Streaming (FastIO): Reads raw file chunks using native Windows direct I/O with sector-aligned allocateAlignedBuffer() memory blocks, bypassing Java IO buffering overhead.
2. 🛡️ 64-Bit 3-Gram Bloom Filter Rejection (TrigramBloomFilter): Generates compact 64-bit 3-gram bitmask signatures directly from raw byte streams (buildFromBytes), rejecting non-matching 64 KiB chunks instantly.
3. ⚡ SIMD AVX2 Substring Candidate Scan (FastBytes & FastContentScanner): Executes 256-bit / 32-byte AVX2 vector sweeps on candidate byte buffers.
4. 📏 UTF-8 Boundary-Aligned Chunking & Overlap Support: Splits large documents into 64 KiB chunks aligned strictly to UTF-8 continuation-byte boundaries, with 256-byte cross-chunk overlaps so matches across boundaries are never lost.
5. 🎯 O(log N) Zero-Allocation Line/Char Mapping: Uses pre-indexed int[] newline byte/char offsets with Arrays.binarySearch() for instant line/col/snippet extraction without allocating temporary String or byte[] objects.

---

Real-World Use Cases

• 🧭 Spotlight / Raycast Desktop & CLI Search: Power instant universal search ("Find in Documents, Code, PDFs & Screenshots") across local storage drives.
• 📄 FastContentParse & PDF Document Indexing: Index normalized text extractions from PDFs, Office documents, and Markdown notes for sub-millisecond retrieval.
• 🖼️ FastOCR Screenshot & Image Search: Index text extracted from screen captures and images via FastOCR so users can instantly find screenshots by typing any text present in the image.
• 🗣️ FastSTT Audio & Meeting Transcripts: Index spoken-text transcripts generated by FastSTT / Whisper for instant voice-memo search.
• 🤖 FastAI & RAG Document Pre-Filtering: Pre-filter gigabytes of enterprise documents and codebase repositories in $&lt; 1 \text{ ms}$ before feeding candidates to LLM context engines (FastContentChunk, FastAIRag).

---

Performance Benchmarks

FastFileContentIndex is engineered for ultra-fast full-text indexing and sub-millisecond query evaluation. In the official JMH Benchmark, the system measured query throughput across indexed codebases:

Benchmark                                             Mode  Cnt       Score        Error  Units
IndexerBenchmark.benchmark3GramBloomQuery            thrpt    3  151327.851 ±  94216.118  ops/s
IndexerBenchmark.benchmarkFastFileContentIndexQuery  thrpt    3  139860.251 ± 659822.168  ops/s

151,000 Queries per Second: FastFileContentIndex evaluates 3-gram Bloom filters and SIMD substring candidate verification in ~6.6 microseconds per query.

---

FastJava Native Memory & Hardware Substrate

FastFileContentIndex is part of the FastJava Low-Level Native Memory Substrate — a suite of modules designed to give Java applications raw C++ speed and direct hardware access:

Substrate Module	Role & Key Capability
FastSharedMemory	Zero-Copy IPC Substrate — Ultra-fast inter-process shared memory buffers (< 78 ns latency) between Java processes and native C++ services.
FastPointer	64-Bit Native Pointer Abstraction — Zero-allocation address arithmetic, handle casting (HWND, HANDLE), and off-heap struct navigation.
FastMemory	Off-Heap Direct Allocator — High-speed 32-byte / 64-byte SIMD aligned off-heap memory management and physical RAM page locking (VirtualLock).
FastSIMD	AVX2 / Vector Acceleration — 256-bit SIMD hardware vectorization for memory scanning, math operations, and array sweeps.
FastBytes	Native Byte Buffer Engine — Off-heap byte arrays with zero-copy slicing, bulk copy, and direct native memory I/O.

---

Technical Architecture — The FastJava Pipeline Chain

FastFileContentIndex operates as the second high-speed filtering layer in the unified FastJava Search & AI Infrastructure:

┌──────────────────┐       ┌────────────────────────┐       ┌────────────────────────┐       ┌────────────────────┐
│   FastFileIndex  │ ────► │         FastIO         │ ────► │  FastFileContentIndex  │ ────► │    FastTokenize    │
│  (Tree / mmap)   │       │ (JNI Direct Aligned I/O)│       │  (3-Gram Bloom < 1µs)  │       │ (Single-Pass O(n)) │
└──────────────────┘       └────────────────────────┘       └────────────────────────┘       └────────────────────┘
                                                                                                        │
                                                                                                        ▼
┌──────────────────┐       ┌────────────────────────┐       ┌────────────────────────┐       ┌────────────────────┐
│    FastAIRag     │ ◄──── │     FastAIVectorDB     │ ◄──── │    FastContentParse    │ ◄──── │  FastContentChunk  │
│  (LLM Context)   │       │  (SIMD Vector Match)   │       │   (PDF/Doc Extract)    │       │ (Syntax Chunking)  │
└──────────────────┘       └────────────────────────┘       └────────────────────────┘       └────────────────────┘

---

Installation

Option 1: Maven (via JitPack)

Add the JitPack repository and dependency to your pom.xml:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastFileContentIndex</artifactId>
        <version>0.1.2</version>
    </dependency>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>fastio</artifactId>
        <version>0.1.1</version>
    </dependency>
</dependencies>

Option 2: Gradle (via JitPack)

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    implementation 'com.github.andrestubbe:FastFileContentIndex:0.1.2'
    implementation 'com.github.andrestubbe:fastio:0.1.1'
}

---

Documentation

• DESCRIPTION.md — Architectural design blueprint and sub-millisecond search strategy.
• PHILOSOPHY.md — Engineering rationale for 3-gram bitmask filtering.
• ROADMAP.md — Future milestones and SIMD/AVX2 native acceleration.

---

Platform Support

Platform	Status
Windows 10/11	🚀 Fully Supported
Linux	🚀 Fully Supported
macOS	🚀 Fully Supported

---

License

MIT License — see LICENSE for details.

---

Related Projects

• FastFileIndex — Native mmap file indexing engine.
• FastFileSearch — High-speed trie-based filename search engine.
• FastTokenize — Zero-allocation multi-language lexer.
• FastANSI — Zero-allocation 24-bit TrueColor ANSI formatter.

---

Part of the FastJava Ecosystem
Making the JVM faster. Small package. Maximum speed. Zero bloat. 🚀