🦉🫥 PDF Anonymizer

This tool hides personal details in documents and can put them back. It is a reversible document pseudonymizer, not a legal certificate.

Feature	Meaning
Reversible pseudonymization	Personal values become typed placeholders (PERSON_1) plus a mapping file. This is not irreversible anonymization.
Hybrid named-entity recognition	RE2 regular expressions find structured identifiers; a language model finds names. Optional local span NER (GLiNER).
Checksums	A failed Luhn or IBAN check is still hidden as IBAN_LIKE, so leftover digits do not stay visible.
Identity clues	The careful profile also hides phrases that pick out one person without a name (quasi-identifiers, type INDIRECT).
Leftover measurement	Residual scan after masking. Gold-corpus leftover rate and recall in CI, split like TAB into direct vs quasi identifiers.
Anonymization statistics	After a run, data/stats/ records leftovers (*.residual_pii.json) and linkage-risk clumps (*.risk.json). Deanonymize writes unused and missing mapping counts.
Span-based replacement	Replacement is by character interval. The longer span wins when two hits overlap.
File types	PDF, Markdown, plain text, CSV, Excel, and Word (.docx).
Country-specific PII	National-ID regexes for 30+ countries (US, CA, GB, ES, IT, FR, IN, CN, and others). --countries US,GB keeps only those national IDs. Email, IBAN, and cards always stay.
HIPAA Safe Harbor aid	--entity-profile hipaa-safe-harbor covers the 18 identifier classes (year-only dates, ZIP3, age 90+). It is a coverage aid, not a compliance certificate.
OCR	--ocr runs Tesseract when a PDF has pages but no text layer.
Local or remote	Ollama on this machine, or Gemini, OpenAI, Anthropic, Hugging Face, OpenRouter.

Operators, locked maps, HTTP, and the rest live in the docs.

📖 Documentation

A comprehensive documentation site is available at leo-gan.github.io/anonymizer/.

The documentation includes:

• Anonymization 101: A guide on data anonymization and deanonymization techniques.
• Installation Guide: System requirements, package extras, and setup.
• CLI Usage: Reference for run, verify, report, and deanonymize, plus a History of what landed.
• API Usage: Programmatic usage guide for the core SDK.
• HTTP service & Docker: POST /anonymize and the image in packages/pdf-anonymizer-api. No authentication.
• API Reference (auto): Auto-generated function signatures and details.
• Recipes & Common Workflows: Practical patterns (local Ollama, external LLM round-trips, batching, profiles, caching, debugging, etc.).
• Troubleshooting: Common issues and solutions.
• Architecture: Understanding how the anonymizer operates.

Project Structure

This project is a monorepo containing three packages:

• packages/pdf-anonymizer-core: The core library containing the anonymization and deanonymization logic. See the core README for more details.
• packages/pdf-anonymizer-cli: A command-line interface for using the anonymizer. See the CLI README for detailed usage instructions.
• packages/pdf-anonymizer-api: Optional local HTTP service. Depends on core only, not the CLI.

Development Installation

1. Install uv: This project uses uv for package management. Follow the official installation instructions.

2. Clone the repository:

   git clone https://github.com/leo-gan/anonymizer.git
   cd anonymizer

3. Install dependencies:

   uv sync --group dev

4. Install Ollama (optional): If you want to use a local model for anonymization, install Ollama.

5. Set up environment variables: Create a .env file in the root directory of the repository (or in the packages/pdf-anonymizer-cli directory) and add the necessary API keys for the providers you want to use. For example:

   # For Google models
   GOOGLE_API_KEY="YOUR_GOOGLE_API_KEY"
   
   # For OpenAI models
   OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
   
   # For Anthropic models
   ANTHROPIC_API_KEY="YOUR_ANTHROPIC_API_KEY"
   
   # For Hugging Face models
   HUGGING_FACE_TOKEN="YOUR_HF_TOKEN"
   
   # For OpenRouter models
   OPENROUTER_API_KEY="YOUR_OPENROUTER_KEY"
   
   # Optional: lock mapping files / seed fake names
   ANONYMIZER_MAPPING_KEY="a long secret"
   ANONYMIZER_FAKE_SECRET="another long secret"

Quick Start

To anonymize a file, run the pdf-anonymizer CLI tool using uv run:

uv run pdf-anonymizer run document.pdf

Commonly you pick a configuration profile with -p / --config-profile (or let it default to best-speed):

uv run pdf-anonymizer run contract.pdf -p best-quality
uv run pdf-anonymizer run large-notes.md -p best-cost --model-name "ollama/phi4-mini"

best-quality uses the careful instructions. Those also hide identity clues — phrases that pick out one person even when no name is written (for example "the CEO of Tesla"). The default best-speed profile skips that extra hunt so it stays fast.

To deanonymize the file later:

uv run pdf-anonymizer deanonymize document.anonymized.md document.mapping.json

For detailed command-line options and examples (including all three profiles and overrides), please refer to the CLI README, the CLI Usage Docs, or the Recipes & Common Workflows page.

Demo: Anonymization & Deanonymization Example

To demonstrate the hybrid NER (RE2 regex pre-filter + LLM) and the complete round-trip process on a real document, we have provided a demo script:

1. Prepare the Demo Document: This script downloads an open-access arXiv research paper PDF and injects synthetic PII (name, email, phone, IP, SSN) onto the first page:

   uv run python scripts/prepare_demo_pdf.py

2. Run the Demo: This script runs the hybrid NER (fast RE2-based regex for structured PII + LLM semantic stage) on the PDF, anonymizes the PII to structured placeholders, saves the mapping vocabulary, and performs a complete round-trip deanonymization:

   uv run python scripts/demo_anonymize.py

You will see colorized console logs showing the exact matched entities (including the rich set of regex-detected PII types), the anonymized text, and the fully-reverted deanonymized output.

The built-in regex stage now automatically detects a wide range of additional structured PII (credit cards, IBANs, crypto addresses, VINs, MAC addresses, many country-specific national/tax/driver IDs, etc.) on top of classic items like email/phone/IP/SSN.

Testing

To run the test suite:

uv run pytest

---

See Also

• Documentation Site — Full guides, 101 course, and API reference.
• Core Package README — SDK details and installation.
• CLI Package README — Command-line specifics.
• Recipes & Common Workflows — Practical usage examples.
• Troubleshooting — Solutions to common problems.