InsightForge is a production-grade, highly optimized full-stack application designed to transform complex, wordy web pages into clean, actionable, high-level learning insights in under 30 seconds.
By leveraging a robust microservices architecture, InsightForge distributes scraping and heavy AI inferences asynchronously, ensuring zero blockage and instant responsiveness.
- ⚡ Sub-30s Asynchronous Summarization: Instantly distills web pages into core summaries, targeted audiences, and key topics.
- 🧠 Intelligent Content Extraction: Employs
Readabilityalgorithms andBeautifulSoupto scrape clean article bodies, stripping away ads, banners, scripts, and trackers. - 🤖 State-of-the-Art LLM Processing: Powered by Groq's high-speed inference engine running Llama 3.1 (8B) for human-like reading and extreme compression.
- 🗄️ Multi-Layer Caching & Storage:
- Redis Cache: Caches URL hashes and active job details for 1 hour to prevent redundant scraping and cut AI costs.
- MongoDB: Acts as a permanent warehouse for generated summaries and historical insights.
- ⛓️ Background Worker Model: Uses Celery task queues to handle scraping and AI jobs asynchronously, protecting the Node.js API gateway from heavy tasks.
- 🎨 Premium UI Dashboard: Beautiful responsive frontend presenting clean summaries, interactive topic badges, and a history log.
InsightForge operates on a decoupled distributed system:
graph TD
A[Frontend Dashboard] -- 1. Submit URL (POST /submit) --> B(Node.js / Express Gateway)
B -- 2. Push Asynchronous Job --> C[Redis Message Broker]
C -- 3. Fetch Job --> D[Celery Python Worker]
D -- 4. Raw HTML Scraping --> E[Target Website]
D -- 5. Clean Page Text --> F[Readability/BS4 Parser]
D -- 6. High-Speed Inference --> G[Groq Cloud / Llama 3.1]
D -- 7. Store Result --> H[(MongoDB Permanent Store)]
D -- 8. Update Cache (1 Hour) --> I[(Redis Cache)]
B -- 9. Poll Status (GET /status) --> I
A -- 10. Load Complete Summary --> A
- Frontend: HTML5, Vanilla CSS3 (modern glassmorphic design), JavaScript ES6.
- API Gateway: Node.js, Express.js, MongoDB (Mongoose), Redis.
- Worker & Backend API: Python 3.11+, FastAPI, Celery, Redis, PyMongo, Groq, BeautifulSoup4, Readability-lxml.
- Databases: MongoDB (Permanent data storage), Redis (In-memory cache & Celery broker).
Follow these steps to run InsightForge locally:
Ensure you have the following installed on your machine:
- Node.js (v18+)
- Python (v3.10+)
- Redis Server (Running on
localhost:6379) - MongoDB Server (Running on
localhost:27017)
Create a .env file inside the worker-python/ directory:
GROQ_API_KEY=your_groq_api_key_here
HF_TOKEN=your_huggingface_token_here (optional)Open three separate terminals in your workspace:
Start the Celery asynchronous worker to process incoming scraping requests:
cd worker-python
python -m celery -A celery_app:app worker --loglevel=info --pool=soloStart the Express API gateway to coordinate requests and database entries:
cd backend-node
npm install
node index.jsIf using the FastAPI service:
cd worker-python
pip install -r requirements.txt
python -m uvicorn api:fastapi_app --port 8000 --reloadSimply open frontend/index.html directly in your browser or run it via a local live server to interact with the dashboard!
- Zero hardcoded secrets: All API keys, database credentials, and Hugging Face/Groq secrets are loaded securely using
.envconfigurations. - Clean Repository Boundaries: Built-in
.gitignoreprevents virtual environments (.venv/), Node dependencies (node_modules/), local config folders (.vscode/), and.envfiles from ever leaking onto GitHub.
This project is licensed under the MIT License. See LICENSE for details.
Developed with ❤️ by VisionStack-404. The first pull request