4scan

4scan is a research webapp for political-science researchers studying the discursive dynamics of imageboards (4chan's /pol/). It collects, stores and visualizes discussion threads through concrete, observable metrics only — volumes, activity over time, thread size and term frequencies — with ethical anonymization of every poster identifier. It deliberately does not assign political, ideological or moral categories to posts.

⚠️ Scope of use. This tool is built for academic research on extreme online discourse. It systematically anonymizes identifiers and keeps no personal data in clear text. Respect 4chan's ToS, the GDPR, and your institution's ethics rules (ethics board, data-management plan).

🔓 Security — read before deploying. 4scan ships in open access (PUBLIC_MODE=true): the UI and API require no login, and the Admin page (which can trigger collection) is reachable by anyone who can reach the host. This is fine on localhost or a trusted LAN. Before exposing it to the internet, set PUBLIC_MODE=false (JWT login) and change JWT_SECRET, HASH_SALT and ADMIN_PASSWORD in your .env.

The project runs entirely locally with Docker, with no cloud dependency, but its architecture is designed to migrate easily to the cloud (managed database, managed Redis) — see § Going to the cloud.

---

Table of contents

• Requirements
• Quick start
• Environment variables
• Architecture
• Repository layout
• Researcher guide
• What 4scan measures
• Retention
• Ethical anonymization
• Going to the cloud
• Troubleshooting
• License

---

Requirements

Docker only. No Node.js or Python needed on the host machine.

• Docker ≥ 24
• Docker Compose v2 (docker compose)
• ~2 GB of free RAM (Postgres + Redis + Python + Vite). No GPU, no PyTorch.

---

Quick start

# 1. Copy and edit secrets
cp .env.example .env
#    -> change JWT_SECRET, HASH_SALT and ADMIN_PASSWORD

# 2. Build and start the whole stack
docker compose up --build

On first boot, the backend:

1. waits for Postgres,
2. applies the Alembic migrations (schema + GIN full-text index),
3. creates the admin user and the target board.

Then open:

Service	URL
Frontend	http://localhost:3000
API (Swagger docs)	http://localhost:8000/docs
Postgres	localhost:5432

By default the app runs in open access (PUBLIC_MODE=true): no login is required — anyone who can reach the machine sees the Dashboard. Set PUBLIC_MODE=false to require a JWT login (ADMIN_USERNAME / ADMIN_PASSWORD, defaults admin / changeme).

🔓 Only run in open access on a machine or LAN you trust. Do not expose a PUBLIC_MODE=true instance to the public internet — the Admin page (which can trigger collection) is reachable without authentication too.

The Celery worker (with Beat) automatically runs a collection cycle every SCRAPE_INTERVAL_SECONDS, analyzes pending posts, and purges data older than the retention window. You can also trigger a manual collection from the Admin page.

---

Environment variables

All defined in .env (see .env.example). The most important ones:

Variable	Purpose
JWT_SECRET	JWT signing key. Change it.
HASH_SALT	Rotating salt for hashing poster identifiers. Change it.
ADMIN_USERNAME / ADMIN_PASSWORD	Admin account (used only when PUBLIC_MODE=false).
PUBLIC_MODE	true = open access, no login (default); false = JWT required.
TARGET_BOARDS	Boards to collect, comma-separated (defaults to pol).
SCRAPE_RATE_LIMIT_SECONDS	Minimum delay between two 4chan API requests (≥ 1, politeness).
SCRAPE_INTERVAL_SECONDS	Periodic collection frequency (Celery Beat).
RETENTION_HOURS	Keep only the last N hours of data (default 24).
PURGE_INTERVAL_SECONDS	How often the purge job runs (default 43200 = 12h).
DATABASE_URL / DATABASE_URL_SYNC	Async (API) and sync (Celery/Alembic) connections.

---

Architecture

                    ┌──────────────┐   HTTP/JWT    ┌───────────────────┐
                    │  Frontend    │ ────────────▶ │  Backend FastAPI  │
                    │ React + Vite │ ◀──────────── │  (async, uvicorn) │
                    └──────────────┘    JSON       └─────────┬─────────┘
                                                             │ ORM (async)
                                                             ▼
   ┌────────────┐   tasks   ┌────────────────┐        ┌────────────┐
   │  Celery    │◀──────────│     Redis      │        │ PostgreSQL │
   │  worker    │  broker   │  broker+result │        │ + GIN index│
   │  + Beat    │──────────▶│                │        └─────┬──────┘
   └─────┬──────┘  results  └────────────────┘              │
         │ scraping / text cleaning / purge (sync ORM)      │
         └───────────────▶ 4chan API (aiohttp, rate-limited)┘

Chosen stack

• Backend: Python 3.11, FastAPI (async), SQLAlchemy 2 (async + sync engines), Alembic, Celery + Redis.
• Database: PostgreSQL 16. Full-text search via a generated tsvector column + GIN index (no Elasticsearch locally).
• No ML/NLP: text is normalized with the standard library (regex/html) and aggregated in SQL. No classification, no scikit-learn/torch — the app records only observable data.
• Frontend: React + TypeScript + Vite, Tailwind (retro dark theme), Recharts (time series).

Backend layout (backend/app/)

Folder	Purpose
api/	FastAPI routes (auth, posts, dashboard, admin) + Pydantic schemas.
tasks/	Celery app + tasks (scraping, text processing, purge).
models/	SQLAlchemy ORM.
services/	Business logic (scraper, analytics, reply edges, stopwords).
ml/	chan-speak text cleaning (mechanical normalization only).
core/	config, DB, security (JWT + anonymization), bootstrap.

---

Repository layout

4scan/
├── docker-compose.yml          # db, redis, backend, worker, frontend
├── .env.example
├── .gitignore  LICENSE  README.md
├── backend/
│   ├── Dockerfile  requirements.txt  entrypoint.sh  alembic.ini
│   ├── alembic/                # env.py + versions/ (schema + GIN, migrations)
│   └── app/
│       ├── main.py             # FastAPI app
│       ├── core/               # config, database, security, bootstrap
│       ├── models/orm.py       # User, Board, Thread, Post, ScrapeLog
│       ├── ml/                 # cleaning.py (text normalization only)
│       ├── services/           # scraper.py, analytics.py, network.py, stopwords.py
│       ├── tasks/              # celery_app.py, jobs.py
│       └── api/                # routes_*.py, schemas.py, deps.py
├── frontend/
│   ├── Dockerfile  package.json  vite/tailwind/postcss configs
│   └── src/
│       ├── App.tsx  main.tsx  index.css
│       ├── lib/ (api.ts, types.ts)
│       ├── components/ (Layout, Controls)
│       └── pages/ (Dashboard, Admin, Methodology)
└── docker/

---

Researcher guide

1. Open the app

Go to http://localhost:3000. In open access (default) you land straight on the Dashboard — no login.

2. The Dashboard

• Selector top-right: time period (6h / 12h / 24h — see Retention). The app targets a single board, /pol/.
• Stat cards: number of posts, active threads, window, board.
• Hourly activity: a single curve of posts per hour.
• Top 5 most active threads: ranked by post count in the window; the ↗ arrow opens the live source thread on 4chan.
• Word cloud: most frequent terms (over cleaned text, with stopword filtering so only topical terms surface).

3. Methodology

The Methodology page documents, in the app itself, how every metric is derived: collection, anonymization, mechanical text cleaning, metric definitions, and retention — plus an explicit statement of what 4scan does not do (no categories, labels or interpretation).

4. Export (API)

The collected data (cleaned text, poster hash, reply target, timestamp) is available through the REST API — no clear-text personal data is ever exported. In open access (PUBLIC_MODE=true) no token is needed:

curl -s "http://localhost:8000/api/posts/export/csv"  -o 4scan.csv
curl -s "http://localhost:8000/api/posts/export/json" -o 4scan.json

If PUBLIC_MODE=false, first obtain a token from /api/auth/login and pass it as Authorization: Bearer <token>. Full API docs are at http://localhost:8000/docs.

---

What 4scan measures

4scan is descriptive by design. It records and aggregates only concrete, observable facts, and draws no interpretive conclusions:

Metric	Definition
Posts / Active threads	Direct counts within the time window.
Hourly activity	Number of posts per hour.
Top active threads	Threads ranked by post count (ties: most recent first).
Word cloud	Most frequent terms in cleaned text, minus stopwords/numbers.

It does not classify posts into political, ideological or moral categories, and computes no irony/"bait" score — those would be interpretive, not observable, and are deliberately out of scope. Interpretation is left to the researcher, working from the exported data.

---

Retention

To keep storage bounded, 4scan retains only the last RETENTION_HOURS of data (default 24h). A Celery Beat job (purge_old_data) runs every PURGE_INTERVAL_SECONDS (default 12h) and deletes posts whose timestamp is older than the window, then removes empty threads and stale scrape logs.

The dashboard period selector is capped at 24h accordingly.

---

Ethical anonymization

No poster identifier is stored in clear text. Before insertion, tripcodes, id, and country are passed through an HMAC-SHA256 with HASH_SALT (app/core/security.py::anonymize). Properties:

• Stable pseudonymization: the same identifier → the same hash while the salt is unchanged (enables co-participation analysis).
• Non-reversible: the original identifier cannot be recovered.
• Rotating salt: changing HASH_SALT breaks linkage across "epochs", limiting longitudinal re-identification.

No IP address is ever requested from the 4chan API or stored.

---

Going to the cloud

The local architecture migrates without a rewrite:

1. Managed database (RDS/Cloud SQL): point DATABASE_URL / DATABASE_URL_SYNC at the remote instance. Run alembic upgrade head.
2. Managed Redis (ElastiCache/MemoryStore): adjust CELERY_BROKER_URL / CELERY_RESULT_BACKEND.
3. Containers: the backend/worker/frontend images deploy as-is to ECS/Fargate, Cloud Run, or Kubernetes.

This is a text-only build — no object storage is needed (no files are written). The only stateful stores are Postgres and Redis.

---

Troubleshooting

Symptom	Hint
backend restart loop	Postgres not ready: the entrypoint waits, but check the db logs.
No data on the dashboard	Let the worker run ≥ 1 cycle, or trigger a collection from Admin.
Text not cleaned	Check the worker logs; process_pending runs every 30s.
First build slow	node_modules install; later builds are cached.
401 from the API	Auth is on (PUBLIC_MODE=false) and the token expired — sign in again.

Useful logs:

docker compose logs -f backend
docker compose logs -f worker

---

License

MIT © 2026 4scan contributors.