CSV Insight Generator

[ms-shashank]

Current file: app/src/lib/tools/csv-insight-generator.ts
Current model: deepseek-r1-0528
Current approach: Single prompt. Pastes raw CSV into the LLM and asks for statistics, trends, anomalies. No actual computation -- all numbers are hallucinated by the model.

Problems with current approach:

• The LLM does not actually compute statistics. Mean, median, min, max values are frequently wrong.
• No programmatic CSV parsing, so malformed rows or encoding issues cause garbage output.
• No ability to handle large datasets (context window limits).
• Trend detection and anomaly flagging are speculative, not data-driven.

Upgrade plan:

Step	Agent	Action
1	CSV Parser	Programmatic: Parse CSV using Python pandas. Validate structure, detect column types, handle encoding issues. Compute actual statistics (mean, median, mode, min, max, std dev, null counts, unique counts).
2	Statistical Analyzer	Programmatic: Run correlation analysis for numeric columns. Detect time-series patterns if date columns exist. Flag statistical outliers using IQR method.
3	Insight Agent	Receive the computed statistics and detected patterns as structured input. Generate human-readable narrative insights, trend explanations, and actionable recommendations.
4	Formatter	Programmatic: Assemble final report with computed tables, statistics, and LLM narrative.

• You are free to enhance the agents stacks in the above plan layout, the above one is just for reference. You can enhance more if needed.

Model suggestions to start with:

• Step 3: Try deepseek-r1-0528 for analytical reasoning. Also try kimi-k2.6 or glm-5 for complex pattern interpretation.
• Since the heavy lifting is programmatic (pandas), the LLM only needs to narrate -- so even llama-3.3-70b or qwen-3-32b may work well here.

Model Selection Guidance

• You are free to pick any model from the Oxlo catalog based on your own testing and evaluation.
• The Models suggestions above, not mandates. Try them first, and if they do not meet the accuracy target, experiment with alternatives.

Compare against: Claude Sonnet 4.6 Thinking & ChatGPT 5.5 Thinking.

Acceptance criteria:

• All reported statistics (mean, median, min, max) must be mathematically correct (programmatic computation, not LLM-generated).
• CSV parsing handles at least: UTF-8/Latin-1 encoding, quoted fields, missing values, mixed types.
• Outlier detection uses a documented statistical method.
• Narrative quality matches or exceeds Claude Sonnet 4.6 Thinking & ChatGPT 5.3 Thinking on test datasets.
• Overall accuracy at 80%+.