Humble Knowledge Builder

HKB builds traceable question-and-answer datasets from repository documentation. The current MVP discovers Markdown files, splits them into stable chunks, generates grounded questions and answers with an LLM, validates and deduplicates the results, and exports extended Alpaca JSONL plus a build manifest.

Build

cargo build

Ollama

Start Ollama with the model you want to use, then run:

cargo run -- build \
  --repo /path/to/repository \
  --out dataset.jsonl \
  --provider ollama \
  --model llama3.2

The default Ollama endpoint is http://localhost:11434.

During a build, HKB reports every Markdown file it chunks and shows a live per-chunk progress bar. The active status distinguishes LLM requests from cache hits and includes the source file and line range currently being processed.

Project-Specific Prompts

The default Q&A prompt lives in prompts/qa-v1.md. To customize generation for a project, create a prompt file in that repository and pass its repository-relative path:

cargo run -- build \
  --repo /path/to/repository \
  --prompt-file hkb-prompt.md

Custom templates must contain {{chunk_text}} so generated answers remain grounded in the source. They may also use these placeholders:

• {{questions_per_chunk}}
• {{path}}
• {{start_line}}
• {{end_line}}

Absolute prompt paths are also accepted. The resolved prompt text is recorded in manifest.json and included in the cache identity, so changing the prompt invalidates only the affected cached generations.

Parallel Generation

Use --concurrency to allow multiple chunks to be sent to the LLM at the same time:

cargo run -- build \
  --repo /path/to/repository \
  --model llama3.2 \
  --concurrency 2

The default is 1, preserving sequential behavior. Completed chunks are cached immediately, even while other requests are still running. HKB restores the original chunk order before validation and export, so changing concurrency does not reorder the dataset. Retryable failures use exponential backoff; configure the retry count with --max-retries (default: 2).

Ollama also limits parallel work on the server. To serve two requests concurrently, start it with a matching limit:

OLLAMA_NUM_PARALLEL=2 ollama serve

Higher concurrency can increase throughput, but Ollama's memory use grows with the number of parallel requests and their context lengths. Increase this setting gradually for the available hardware.

Additional Ignore Rules

Use --ignore-file to apply repository-relative gitignore-style exclusions without modifying the target repository:

path/to/module/
path/to/another/**test.md

cargo run -- build \
  --repo /path/to/repository \
  --ignore-file ~/.config/hkb/java.ignore

The option may be repeated. Custom rules apply even with --no-gitignore and take final exclusion precedence over repository rules. Relative ignore-file paths are resolved from the directory where HKB is launched; patterns inside each file are rooted at --repo.

OpenAI-Compatible APIs

Set an API key when the server requires one:

export OPENAI_API_KEY="..."

To use a different environment variable:

cargo run -- build \
  --provider openai-compatible \
  --model your-model \
  --api-key-env HKB_API_KEY

Alternatively, read the key from a file:

cargo run -- build \
  --provider openai-compatible \
  --model your-model \
  --api-key-file ~/.config/hkb/api-key

Then provide the model and, for non-default servers, the endpoint:

cargo run -- build \
  --repo /path/to/repository \
  --out dataset.jsonl \
  --provider openai-compatible \
  --model your-model \
  --endpoint https://example.com/v1

--api-key-file takes precedence over --api-key-env. The key is never stored in the manifest, cache, or progress output. HKB deliberately does not accept a literal --api-key value because command-line arguments can be exposed through shell history and process inspection.

Output

dataset.jsonl contains one extended Alpaca record per line:

{
  "instruction": "What does HKB build?",
  "input": "",
  "output": "HKB builds question-and-answer datasets.",
  "metadata": {
    "id": "...",
    "source": {
      "path": "README.md",
      "start_line": 1,
      "end_line": 4
    },
    "chunk_id": "...",
    "model": "llama3.2",
    "generated_at": "2026-07-30T12:00:00Z"
  }
}

manifest.json records the repository, public build configuration, prompt version and template, Git commit when available, and processing statistics.

Cached generations are stored under .hkb by default. A cache entry is invalidated when the chunk, provider, endpoint, model, prompt contents, temperature, or requested question count changes.

Quality Checks

cargo fmt --check
cargo test
cargo clippy --all-targets --all-features -- -D warnings