EvalForge is offline-first: by default it runs against the deterministic mock backend with no network access and no secrets. Real LLM providers and the optional history API are opt-in. This guide covers the trust boundaries, secret handling, the plugin sandbox considerations, and the operational checklist.

flowchart LR
    subgraph Trusted["Trusted (local, no secrets)"]
        CLI[evalforge CLI]
        Mock[Mock backend]
        Judges[Judges + compliance engine]
        SQLite[(SQLite history)]
    end
    subgraph SemiTrusted["Operator-supplied"]
        Suite[YAML suites]
        Plugin[Custom judge plugins]
        Rules[Rule packs]
    end
    subgraph External["External / untrusted"]
        LLM[OpenAI / Anthropic / LiteLLM]
        HF[HuggingFace datasets]
        Webhook[Slack / Discord webhooks]
    end

    Suite --> CLI
    Plugin --> CLI
    Rules --> Judges
    CLI --> Mock
    CLI -. opt-in + API key .-> LLM
    CLI -. opt-in .-> HF
    Judges --> SQLite
    CLI -. opt-in .-> Webhook

The dashed edges are the only paths that leave the local trust boundary, and each is opt-in: a backend other than mock, the --from-hf flag, or a configured webhook URL.

API keys are read from the environment via the EVALFORGE_ prefix (see evalforge/config.py). They are never written to reports or the history database.

export EVALFORGE_OPENAI_API_KEY=sk-...
evalforge eval suite.yaml --backend openai

evalforge eval suite.yaml            # --backend mock is the default

No API keys are required. The mock backend matches prompts against a local JSON map and returns a deterministic default otherwise, so CI on PRs needs no secrets.

The plugin loader (evalforge/plugins.py) executes a user-supplied Python file that defines a judge(test_case, response) function. A plugin runs with the same privileges as the CLI — treat plugin files as trusted code, the same way you treat a test in your repo. Mitigations built in:

• validate_plugin() checks the file exists, is a .py, imports cleanly, and exposes a 2-arg judge before it is used — run evalforge plugins validate --path <file> in review.
• discover_plugins() skips files that fail to import rather than crashing the run.
• A plugin raising at judge time is caught and recorded as a failed JudgeResult (score 0.0) — a malicious or buggy plugin cannot abort the whole suite.
• Plugins are applied as scoped runner overrides (RAGRunner(judge_overrides=...)), never mutating the global registry, so one suite's plugin cannot leak into another.

Do not load plugin files from untrusted sources. If you accept third-party suites, review their plugin files exactly as you would review a pull request.

The FastAPI history API (evalforge serve) binds to 127.0.0.1 by default and is meant for local dashboard use. It uses the vendored evalforge.shared_core request-logging middleware and application error handler (so internal errors are not leaked verbatim). If you expose it beyond localhost, put it behind an authenticating reverse proxy — it has no built-in auth.

Slack/Discord notifiers are no-ops unless a webhook URL is configured, preserving offline-first behavior. Webhook URLs are secrets — store them in the environment, not in suite files.

evalforge eval --evidence-dir copies configuration into manifest.json only after recursively redacting secret-shaped keys such as API keys, tokens, authorization headers, passwords, cookies, and webhook URLs. Review the bundle before sharing it: custom backends may attach provider metadata, and redaction is key-based rather than a guarantee that arbitrary response text contains no secrets. Calibration reasoning and agent trace content are also operator-supplied report data; inspect them for prompt or customer information before publication. The verifier checks schema-v1/v2 SHA-256 integrity and report consistency; it does not provide cryptographic signing or authenticity for an untrusted Git checkout.

The canonical make evidence path uses the mock backend, no credentials, and no network. Keep real-provider evidence opt-in and treat its reports as sensitive operational data.

• CI uses the mock backend for PRs (no secrets in untrusted contexts)
• Real API keys live only in protected-branch / environment secrets, never in suites
• No keys committed in example_suites/ or rule packs
• Custom judge plugins are reviewed and evalforge plugins validate-ed before use
• The history API is bound to localhost or fronted by auth if exposed
• Webhook URLs come from the environment, not from files in the repo
• EVALFORGE_OPENAI_BASE_URL points only at endpoints you trust
• Evidence bundles are verified before review or publication, and custom backend metadata has been checked for sensitive values