AutoPrompt — Automated Prompt Optimization & Benchmarking

A structured-extraction task — pull product, sentiment, and reason out of a product review — needs a good prompt. Instead of hand-tuning one, AutoPrompt has a model draft several candidate prompts, scores each against labeled ground truth on a training split, locks in the best one (optionally injecting a training example it got exactly right as a few-shot), then benchmarks it against a hand-written baseline on a held-out test split — both fired concurrently through Groq.

This is the project behind an accepted, co-authored paper on prompt optimization. See technical_deep_dive.md for the full architecture writeup and code walkthrough.

Results

Optimized prompt vs. the hand-written baseline, on the held-out test split:

Metric	Result
Extraction accuracy vs. baseline	+13%
Wall-clock time vs. a sequential run	-90%, from concurrent execution via asyncio.gather + semaphore pacing

How it works

data/reviews_30.csv + data/ground_truth_30.json
        ↓
Phase 1 — Optimization (src/autoprompt.py)
  Meta-LLM writes 4 candidate prompts → each tested against the training split
  → scored against ground truth → best candidate locked in as the production prompt
  → a training example it got exactly right is kept and injected back in as a few-shot
        ↓
Phase 2 — Execution (main.py, src/baseline.py)
  Baseline (static, hand-written) and optimized prompts both run against the test split,
  concurrently via asyncio.gather, paced by a semaphore so the free-tier rate limit holds
        ↓
Phase 3 — Evaluation (src/evaluator.py)
  Per-field accuracy (product, sentiment, reason) + failure rate, baseline vs. optimized
        ↓
results/benchmark_report.json

Every call goes through Groq's native JSON mode (response_format={"type": "json_object"}), so a malformed response is something the API itself guards against, not something the evaluator has to parse defensively.

Tech stack

Layer	Tech
LLM	Groq API — llama-3.1-8b-instant by default (config/prompt_config.yaml)
Concurrency	asyncio.gather, semaphore-paced requests
Reliability	tenacity retries on API/network errors
Structured output	Groq JSON mode
Logging	loguru → logs/run.log
Data	Pandas

Setup

git clone https://github.com/Devasurya05/autoprompt
cd autoprompt

python -m venv venv
.\venv\Scripts\Activate.ps1   # Windows
# source venv/bin/activate    # Mac/Linux

pip install -r requirements.txt

Create a .env file in the repo root:

GROQ_API_KEY=your_groq_api_key

Check API connectivity:

python check_model.py

Run the optimization + benchmark pipeline:

python main.py --train-limit 5 --test-limit 30

Flag	Meaning	Default
--train-limit	Reviews used to generate and score candidate prompts	5
--test-limit	Reviews used for the final benchmark (0 = every remaining row)	0
--data	Path to the reviews CSV	data/reviews_30.csv
--truth	Path to the matching ground-truth JSON	data/ground_truth_30.json
--model	Override the Groq model for both generation and scoring	config default

Output

• logs/run.log — full run audit trail
• results/benchmark_report.json — baseline vs. optimized accuracy and failure rate, per field

Limitations

• Built around Groq's request/response shape — adding OpenAI or Gemini support means a small mapping layer next to src/baseline.py, not a rewrite.
• Scoring in src/evaluator.py is a full/partial string match against ground truth, not a semantic check — a correct answer worded differently than the label can under-score.
• Very long reviews (order of 10k+ words) can hit context limits, same as any single-prompt approach.