LLM Analytics Apps

Internal tooling for the PostHog LLM Analytics team. Contains demo data generators, trace generators, test scripts, and a runner for the SDK examples that live in posthog-python and posthog-js.

For copy-paste-able provider integration examples, see the examples/example-ai-* directories in each SDK repo.

Setup

Requires:

• uv
• pnpm
• phrocs

cp .env.example .env
# Fill in your API keys in .env
make setup

Tools

SDK Example Runner

Discovers and runs all example-ai-* examples from sibling posthog-python and posthog-js repos.

# List all available examples (with cache status)
./run-examples.sh --list

# Run a specific example or group by name
./run-examples.sh anthropic        # all anthropic examples
./run-examples.sh python/openai    # python openai examples only

# Run all examples in parallel via phrocs
./run-examples.sh --parallel

# Run all sequentially
./run-examples.sh --all

# Force re-run (ignore cache)
./run-examples.sh --rerun --all

# Install dependencies for all examples
./run-examples.sh --install

Demo Data Generator

Generates realistic multi-turn conversations across all supported providers using the PostHog AI SDKs directly. Useful for populating a PostHog instance with representative LLM analytics data.

make demo-data                # 5 conversations, random providers
make demo-data-quick          # 3 short conversations, single provider
make demo-data-tools          # tool-heavy conversations
make demo-data-negative       # negative sentiment for testing

Trace Generator

Creates complex nested LLM trace data (traces, spans, generations) without making real LLM calls. Useful for testing PostHog's trace visualization.

make run-trace-generator

Test Scripts

Various scripts for testing specific SDK integrations:

# Python
uv run scripts/test_litellm.py
uv run scripts/test_langchain_otel.py
uv run scripts/test_pydantic_ai_otel.py

# Node — @posthog/ai
npx tsx scripts/test_posthog_ai_sdk.ts   # subpath clients + captureAiGeneration
npx tsx scripts/test_vercel_ai_otel.ts   # OTel via PostHogSpanProcessor
npx tsx scripts/test_vercel_anthropic.ts # withTracing (Vercel AI SDK)

The Node scripts read POSTHOG_API_KEY / POSTHOG_HOST from the environment. For a local PostHog, fetch the project key first and pass it through op run (which resolves the provider keys in .env):

KEY=$(uv run scripts/get_localhost_api_key.py -q)
POSTHOG_API_KEY="$KEY" POSTHOG_HOST=http://localhost:8010 \
  op run --env-file=.env -- pnpm exec tsx scripts/test_posthog_ai_sdk.ts

Local SDK Development

To develop against local SDK checkouts, set these in .env:

POSTHOG_PYTHON_PATH=../../posthog-python
POSTHOG_JS_PATH=../../posthog-js

Testing unreleased @posthog/ai

By default package.json pins the published @posthog/ai, so make setup works with no posthog-js checkout. To run the Node scripts against a local SDK build instead (e.g. to validate an unreleased change), link it in:

make link-local-sdk                                       # uses ../posthog-js
POSTHOG_JS_PATH=/path/to/posthog-js make link-local-sdk   # or any location

link-local-sdk builds @posthog/ai (and its workspace deps, in topo order) and rewrites package.json to point @posthog/ai + posthog-node at your checkout. That edit is local-only — don't commit it. To restore the published SDK: git checkout -- package.json pnpm-lock.yaml && pnpm install.

License

MIT License - see LICENSE file for details.