TIJP — The Intelligence Journal Protocol

A Personal Geopolitical Intelligence System

Professional-grade intelligence analysis, automated daily, for the solo operator.

Features • Quick Start • Architecture • Documentation • Status

📖 Overview

TIJP is a fully automated geopolitical intelligence system that ingests raw data from 50+ primary sources daily, processes it through structured AI synthesis, maintains persistent long-term memory, and generates comprehensive intelligence reports with tracked predictions and self-improving calibration.

What makes TIJP different:

🎯 Signal, not noise — Primary sources (think tanks, data APIs, OSINT) instead of social media
🧠 Persistent memory — 365-day rolling intelligence context via vector embeddings
📊 Prediction tracking — Every prediction is tracked, verified, and scored
🔄 Self-calibrating — System learns from prediction accuracy and adjusts reasoning
⚡ Zero maintenance — Runs automatically, delivers daily, requires no intervention

Core Differentiator

TIJP is not a news aggregator. It's an analytical system that:

Accumulates intelligence over time
Connects signals across domains and timeframes
Tracks the accuracy of its own predictions
Recalibrates reasoning based on empirical performance

✨ Features

🔍 Intelligence Collection

20+ RSS sources from top think tanks (RAND, Crisis Group, CSIS, ECFR, etc.)
Data APIs for conflict events (ACLED), energy flows (EIA), trade data (UN COMTRADE)
OSINT feeds for vessel tracking (MarineTraffic), flight monitoring (ADS-B), satellite imagery
Paywall bypass for academic journals and premium publications

🧮 Processing Pipeline

3-level deduplication — URL normalization, content hashing, fuzzy title matching
Relevance filtering — Keyword groups, geography priority, source reliability weighting
Semantic clustering — Groups related articles into single narratives
Event linking — Tracks story continuity across days and weeks

🤖 AI Analysis

Pattern detection — Escalation, persistence, convergence patterns
Prediction engine — Forward-looking intelligence with confidence scoring
Risk prioritization — HIGH/MEDIUM/LOW classification with alert generation
ACH framework — Analysis of Competing Hypotheses methodology

📬 Delivery

Telegram — Executive summary with critical alerts
Email — Full intelligence report with analysis
Web archive — Searchable historical reports
API — JSON endpoints for custom integrations

🚀 Quick Start

Prerequisites

Python 3.11+
PostgreSQL database (Supabase recommended)
API keys (see Configuration)

Installation

# Clone the repository
git clone https://github.com/yourusername/tijp.git
cd tijp

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Set up environment variables
cp .env.example .env
# Edit .env with your API keys

# Initialize database
python scripts/setup_db.py

# Seed sources
python scripts/seed_sources.py

Run Your First Pipeline

# Run the full intelligence pipeline
python run_pipeline.py

# Output will be saved to reports/YYYY-MM-DD.txt

Start the API Server

# Start FastAPI server
cd api
python server.py

# API available at http://localhost:8000
# Docs at http://localhost:8000/docs

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        LAYER 1: INGEST                          │
│  RSS Feeds → OSINT APIs → Data APIs → Email Parser → Scheduler  │
└────────────────────────────┬────────────────────────────────────┘
                             │ raw content + metadata
┌────────────────────────────▼────────────────────────────────────┐
│                       LAYER 2: PROCESS                          │
│  Deduplicator → Relevance Filter → Chunker → Embedder           │
└────────────────────────────┬────────────────────────────────────┘
                             │ embeddings + filtered chunks
┌────────────────────────────▼────────────────────────────────────┐
│                       LAYER 3: MEMORY                           │
│  ChromaDB (vector store) → Signal Index → Report Archive        │
└────────────────────────────┬────────────────────────────────────┘
                             │ semantic retrieval (top-K)
┌────────────────────────────▼────────────────────────────────────┐
│                      LAYER 4: GENERATE                          │
│  Context Retriever → AI Synthesis Engine → Prediction Engine    │
└────────────────────────────┬────────────────────────────────────┘
                             │ structured JSON report
┌────────────────────────────▼────────────────────────────────────┐
│                      LAYER 5: DELIVER                           │
│  Telegram Bot → Email → Web Archive → API                       │
└─────────────────────────────────────────────────────────────────┘

Technology Stack

Component	Technology	Why
Language	Python 3.11+	Data science ecosystem, async support
Database	PostgreSQL (Supabase)	Free tier, vector extension, hosted
Vector DB	ChromaDB	Zero cost, local, Python-native
AI Synthesis	Claude Sonnet 4	Best long-context reasoning, structured output
Embeddings	OpenAI text-embedding-3-small	Best quality/cost ratio
RSS Parsing	feedparser + httpx	Lightweight, handles malformed feeds
Clustering	scikit-learn	TF-IDF + cosine similarity
API	FastAPI	Modern, fast, auto-docs

📊 System Status

Current Implementation

✅ Operational Modules:

RSS ingestion (20 sources)
3-level deduplication
Relevance filtering
Semantic clustering
Event linking
Pattern detection
Prediction generation
Risk prioritization
Report formatting
API server

⚠️ In Development:

AI synthesis engine (Claude API)
Vector memory (ChromaDB)
Telegram/Email delivery
Error handling & resilience (Phase 14)

Latest Pipeline Run

Execution Time: 11.9 seconds
├─ RSS Fetch:        209 items from 20 sources
├─ Deduplication:    209 → 201 items (3.8% removed)
├─ Relevance Filter: 201 → 101 items (avg score 0.565)
├─ Clustering:       101 items → 85 clusters
├─ Summarization:    85 summaries generated
├─ Event Linking:    85 updates, 0 new events
├─ Pattern Detection: 1 escalation pattern
├─ Predictions:      1 prediction generated
└─ Alerts:           1 HIGH risk alert

Test Coverage

171 tests, all passing ✅

# Run tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=pipeline --cov-report=html

⚙️ Configuration

Environment Variables

Create a .env file with the following:

# AI APIs
ANTHROPIC_API_KEY=sk-ant-...          # Claude API
OPENAI_API_KEY=sk-...                 # Embeddings

# Intelligence Data APIs
ACLED_API_KEY=...                     # Conflict events (free)
EIA_API_KEY=...                       # Energy data (free)
MARINE_TRAFFIC_API_KEY=...            # Vessel tracking ($9/mo)

# Delivery
TELEGRAM_BOT_TOKEN=...                # From @BotFather
TELEGRAM_CHAT_ID=...                  # Your chat ID
RESEND_API_KEY=...                    # Email delivery
EMAIL_FROM=tijp@yourdomain.com
EMAIL_TO=you@youremail.com

# Database
SUPABASE_URL=https://....supabase.co
SUPABASE_KEY=eyJ...

Source Configuration

Edit config/sources.yaml to add/remove sources:

sources:
  - id: "SRC-001"
    name: "RAND Corporation"
    category: "think_tank"
    tier: 1
    rss_url: "https://www.rand.org/blog/feed.xml"
    reliability_default: "A"
    active: true

📚 Documentation

Core Documents

PRD — Complete product requirements (2351 lines)
System State — Current implementation status
Phase 14 Readiness — Next implementation phase
Database Schema — PostgreSQL schema

Module Documentation

Each module has inline documentation:

# Example: pipeline/process/deduplicator.py
"""
Three-level deduplication applied in strict order:
  1. URL normalization match (primary)
  2. Content hash match (secondary)
  3. Title similarity match (fuzzy, ≥90%, safe mode)
"""

API Documentation

Start the server and visit:

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Endpoints:

GET /events — All prioritized events
GET /alerts — HIGH risk events only
GET /patterns — Detected patterns
GET /predictions — Forward-looking predictions

🔧 Development

Project Structure

tijp/
├── api/                    # FastAPI server
│   ├── server.py          # Entry point
│   └── routes.py          # API endpoints
├── config/                # Configuration files
│   ├── sources.yaml       # Source registry
│   ├── categories.yaml    # Interest categories
│   └── prompts/           # AI prompts
├── pipeline/              # Core intelligence pipeline
│   ├── ingest/           # Data collection
│   ├── process/          # Dedup, filter, cluster
│   ├── memory/           # Storage & linking
│   ├── generate/         # Summarization
│   ├── analysis/         # Patterns & predictions
│   ├── delivery/         # Report formatting
│   └── db/               # Database interface
├── scripts/              # Setup & maintenance
├── tests/                # 171 test files
├── web/                  # Web archive (optional)
└── run_pipeline.py       # Main entry point

Running Tests

# All tests
pytest tests/ -v

# Specific module
pytest tests/test_ingest.py -v

# With coverage
pytest tests/ --cov=pipeline --cov-report=html

# Watch mode (requires pytest-watch)
ptw tests/

Adding a New Source

Edit config/sources.yaml:

- id: "SRC-XXX"
  name: "New Source"
  rss_url: "https://example.com/feed.xml"
  active: true

Run seed script:

python scripts/seed_sources.py

Test:

python run_pipeline.py

💰 Cost Estimate

Monthly operational cost: ₹1,040–₹1,290 (~$12–15 USD)

Component	Cost
Claude API (30 daily reports)	~₹900
OpenAI Embeddings	~₹40
MarineTraffic API	~₹750
Residential Proxy	~₹250
Database (Supabase free tier)	₹0
Email (Resend free tier)	₹0
Telegram	₹0
Total	₹1,040–₹1,290

🛣️ Roadmap

Phase 14 (Current) — Error Handling & Resilience

Phase 15 — AI Synthesis

Claude API integration
ACH framework implementation
Hypothesis tracking
Prediction accuracy calibration

Phase 16 — Vector Memory

ChromaDB integration
OpenAI embeddings
Semantic similarity search
365-day rolling memory

Phase 17 — Full Source Stack

Data API clients (ACLED, EIA, etc.)
OSINT feeds (MarineTraffic, ADS-B)
Paywall bypass (12ft.io, Unpaywall)
Email newsletter parsing

🤝 Contributing

This is a personal intelligence system, but contributions are welcome!

Development Setup

# Fork and clone
git clone https://github.com/yourusername/tijp.git
cd tijp

# Create feature branch
git checkout -b feature/your-feature

# Make changes and test
pytest tests/ -v

# Commit and push
git commit -m "Add: your feature"
git push origin feature/your-feature

Code Style

Follow PEP 8
Add docstrings to all functions
Write tests for new features
Keep functions small and focused
Use type hints where possible

📄 License

MIT License — see LICENSE file for details.

🙏 Acknowledgments

Data Sources:

RAND Corporation, Crisis Group, CSIS, ECFR, Brookings, Chatham House
ACLED, EIA, UN COMTRADE, World Bank, IMF
MarineTraffic, ADS-B Exchange, NASA FIRMS

Technology:

Anthropic (Claude API)
OpenAI (Embeddings)
Supabase (Database)
FastAPI, scikit-learn, feedparser

📞 Support

Documentation: See docs/ folder
Issues: GitHub Issues
Discussions: GitHub Discussions

Built with ❤️ for intelligence professionals and geopolitical analysts

"A personal intelligence system that thinks like a senior analyst, remembers everything, and gets measurably smarter every month."

⭐ Star this repo if you find it useful!

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
api		api
assets		assets
config		config
data		data
docs/assets/diagrams		docs/assets/diagrams
pipeline		pipeline
reports		reports
scripts		scripts
tests		tests
web		web
.env.example		.env.example
.gitignore		.gitignore
COMMANDS.md		COMMANDS.md
README.md		README.md
requirements.txt		requirements.txt
run_pipeline.py		run_pipeline.py
sources_list.txt		sources_list.txt

Folders and files

Latest commit

History

Repository files navigation