🚀 PayhipScraper — Enterprise Product Intelligence for Arena.ai

Scrape Payhip & marketplaces → Analyze trends → Generate Arena.ai prompts to create better versions.

Vision

92% of Payhip/Gumroad sellers make <$100/mo because they guess products. PayhipScraper reverses this:

1. Scrapes the public Payhip marketplace via its own search endpoint
2. Analyzes demand, competition, pricing and quality → Opportunity Score 0-100
3. Generates production Arena.ai prompts (Cloner, Differentiator, Bundle, Niche Report) that create superior versions in minutes

What the data actually is

Payhip does not publish sales counts, so nothing here estimates them. Demand is derived from review volume, quality from star rating weighted by review count, and competition from the marketplace-wide result count for the keyword plus category saturation within the sample. Scoring is deterministic: scan the same keyword twice and you get the same scores.

Payhip's marketplace search matches any term in the phrase, so a search for "notion budget template" returns everything containing "budget". The scraper works around this: it probes each term, narrows on the most selective one, then filters locally for products matching the whole phrase. Each scan reports whether its results were exact or partial matches.

Reduce research from 10 hours to 7 minutes.

Screenshots (Concept)

• Dashboard: search bar, stats cards, product grid with score circles, right rail prompts
• Drawer: detailed breakdown, tags, prompt actions
• Dark/light theme, responsive

Quick Start

git clone ...
cd payhipscraper
npm install
cp .env.example .env
# Edit .env: set JWT_SECRET >=32 chars
npm start
# Open http://localhost:3000

Default login: create new account in UI (any email). No external email verification required for MVP.

Architecture (Monolith)

• Backend: Node.js 20 + Express 4, SQLite (better-sqlite3, WAL), JWT httpOnly cookies, Zod validation, Helmet, RateLimit, Pino logs
• Scraper: axios + cheerio against marketplace_aux/get_search_results_ajax (the endpoint the marketplace grid itself calls), UA rotation, 1.5s delay, DNS-level SSRF guard, cache 6h in-memory + SQLite
• Analyzer: Opportunity Score = 0.40demand + 0.25competition + 0.20priceFit + 0.15quality — every term from an observed signal, no randomness
• Prompter: 4 template types, gap analysis, SEO suggestions, Arena deep-link https://arena.ai/agent?prompt=...
• Frontend: Vanilla ES Modules, no build step, CSS variables design system, Chart.js, hash router, PubSub store

Folder Tree: see ../docs/ARCHITECTURE.md

API (Base /api)

• POST /auth/signup, /login, /me, /refresh, /logout
• POST /scans {keyword, platforms} → 202 scanId
• GET /scans list, GET /scans/:id, GET /scans/:id/status, GET /scans/:id/products, GET /scans/:id/prompts
• GET /products/trending, GET /products/:id
• GET /prompts library, POST /prompts, PUT, DELETE, GET /prompts/:id/arena-url
• GET /user/profile, PUT, PUT /user/password, GET /user/stats, /api-keys, /export?format=csv|json
• GET /health, GET /platforms

See ../docs/API_SPEC.md for full spec.

Core Features Implemented

• ✅ Onboarding & JWT auth (bcrypt 12, access 15m, refresh 7d)
• ✅ Primary dashboard: search, stats, filters, grid/table, charts, polling job progress
• ✅ Scraping engine: Payhip live via its search API, relevance filtering, cached; Gumroad/Etsy are sample data only and always labelled as such
• ✅ Trend analyzer: deterministic scores, keyword extraction, price/category stats
• ✅ Prompt generator for Arena.ai with copy & open link, library, starring
• ✅ Settings: profile, security, API keys, export, GDPR delete
• ✅ Security: helmet, CORS, rateLimit, SSRF check, prepared statements, sanitize, auth middleware

Environment Variables

See .env.example. Two things to know:

• JWT_SECRET and JWT_REFRESH_SECRET are required (≥32 chars, must differ). In production the app exits rather than start without them.
• ENABLE_MOCK_FALLBACK defaults to false. When enabled, a failed scrape returns sample products instead of an error — useful for a demo, misleading anywhere else.

Deployment

See DEPLOY.md. Short version: this is a stateful monolith (SQLite on disk + an in-process job queue), so it needs a long-lived container with a volume — Fly.io, Render, or any Docker host. It cannot run on serverless without swapping the database and the queue.

docker compose up -d --build      # local / self-hosted
fly deploy                        # Fly.io, config in fly.toml

Tests

npm test

Parser tests run against a captured copy of Payhip's real search markup (test/fixtures/), so a layout change upstream fails loudly instead of degrading silently to sample data.

Testing the Flow

1. Sign up → Onboarding
2. Search "notion budget template" → Scan → wait ~20-40s
3. See products sorted by opportunity score
4. Right rail prompts → Copy → Open in Arena.ai
5. Check Trending tab, Prompt Library, Settings Export

Legal & Ethics

• Only public metadata scraped, max 500 chars snippet, link back to original, attribution
• Respects robots.txt spirit: delay, UA rotate, cache
• Not affiliated with Payhip. Sellers can request delist (future endpoint).
• Disclaimer in footer

Roadmap

• v1.1: Stripe, Team workspaces, BullMQ+Redis queue, Puppeteer fallback, Chrome Extension, Webhooks
• v1.2: Affiliate auto-inject, Scheduled reports, API rate higher, Etsy/Gumroad real scraping upgrade

Docs

All enterprise docs in /docs:

• PRD.md – personas, journeys, functional specs
• ARCHITECTURE.md – tech stack, folder tree, data flow
• SCHEMA.md – SQL schema, models
• API_SPEC.md – endpoint specs, UI hierarchy
• SECURITY.md – threat model, mitigations, quality
• IMPLEMENTATION_PLAN.md – 3 phases

License

MIT – Build something profitable.

---

Built according to 2026 production standards: monolith first, modular, secure, fallback-safe, one-command deploy.