CogniSight is an autonomous web agent designed to understand and execute complex, high-level goals within any web environment. It represents a paradigm shift from traditional, brittle automation frameworks to intelligent, agentic interaction powered by multimodal AI and computer vision.
The core vision for CogniSight is to create a "co-pilot for the web" that perceives and interacts with user interfaces visually, just as a human does. This "vision-first" approach, powered by multimodal AI and YOLO object detection, allows it to understand the semantic meaning of a UI, not just its structural code.
The project follows a strict monorepo structure managed by Turborepo with clear separation of concerns:
apps/cli: The user-facing command-line entry pointpackages/agent-core: The "brain" of the agent (reasoning, planning, orchestration)packages/tools: The "hands" of the agent (browser interaction tools)packages/vision: The "eyes" of the agent (visual analysis)packages/shared: Shared utilities and services (logging, storage, utils)packages/types: Shared TypeScript interfaces and contractspackages/typescript-config: Shared TypeScript configurationpackages/vision-detector: Python vision detection service (YOLO + WebSocket)
- Runtime: Node.js (LTS) + Python 3.13+
- Package Manager: pnpm (Node.js) + uv (Python)
- Build System: Turborepo + tsup
- Language: TypeScript (Strict Mode) + Python
- Configuration: Environment variables
- Logging: Pino (structured JSON)
- CLI Framework: Commander.js
- Browser Automation: Playwright with human emulation
- Vision: YOLOv8n + Multimodal LLM (GPT-4o/Gemini)
- Python Environment: uv workspace with unified virtual environment
- Node.js 18+ (LTS)
- pnpm 9.0.0+
- Python 3.8+ (for vision services)
- uv (Python package manager)
# Clone the repository
git clone <repository-url>
cd cognisight
# Install Node.js dependencies
pnpm install
# Setup Python environment (uv will be installed automatically)
pnpm setup:all
# Build all packages
pnpm build-
Copy the example environment file:
cp .env.example .env
-
Configure your environment variables in
.env:# Required for AI features OPENAI_API_KEY=sk-your-openai-api-key-here # Optional: Override defaults BROWSER_HEADLESS=false LOG_LEVEL=info STORAGE_ENDPOINT=http://localhost:9000
Note: Environment variables are automatically loaded from the root .env file across all packages in the monorepo. No need to create separate .env files in individual packages.
The project uses uv for Python dependency management with a unified workspace:
- Single virtual environment:
.venv/at root - Unified lockfile:
uv.lockfor all Python packages - Zero IDE configuration: VS Code/Cursor auto-detects Python interpreter
- Scalable: Easy to add new Python services
# Verify Python environment setup
pnpm verify:python
# Run Python tests
pnpm py:test
# Run Python linting
pnpm py:lint# Show help
node apps/cli/dist/index.js --help
# Run the agent with a goal
node apps/cli/dist/index.js run "your goal here"cognisight/
├── apps/
│ └── cli/ # Command-line interface
├── packages/
│ ├── agent-core/ # Main agent logic
│ ├── tools/ # Browser interaction tools
│ ├── vision/ # Visual analysis module
│ ├── shared/ # Shared utilities and services
│ ├── types/ # Shared TypeScript interfaces
│ ├── typescript-config/ # Shared TypeScript configuration
│ └── vision-detector/ # Python vision detection service
├── data/ # Data directory
│ ├── assets/ # Static assets (screenshots, models)
│ │ └── samples/
│ │ └── screenshots/ # Test screenshots
│ └── runs/ # Agent execution logs and artifacts
├── docs/ # Project documentation
│ ├── guides/ # Development guides
│ ├── vision/ # Vision module documentation
│ └── python-environment/ # Python environment documentation
├── scripts/ # Development scripts
│ └── verify-python-setup.sh # Python environment verification
├── .venv/ # Python virtual environment (uv)
├── uv.lock # Python dependency lockfile
├── pyproject.toml # Root Python workspace config
└── pnpm-workspace.yaml # Node.js workspace config
# Development mode (watch for changes)
pnpm dev
# Build specific package
pnpm build --filter=@cognisight/cli
# Python development
pnpm py:dev # Start Python development servers
pnpm py:test # Run Python tests
pnpm py:lint # Run Python linting
# TypeScript development
pnpm lint # Lint all TypeScript packages
pnpm test # Test all packages
pnpm clean # Clean all builds
# Verification
pnpm verify:python # Verify Python environment setuppnpm verify:pythonComprehensive verification of:
- ✅ uv installation and configuration
- ✅ Python dependencies (ultralytics, PIL, websockets)
- ✅ Turborepo integration
- ✅ IDE auto-detection
- ✅ TypeScript integration
- ✅ Performance benchmarks
# Test vision detection
pnpm --filter @cognisight/py-vision-detector test:detector
# Test performance
pnpm --filter @cognisight/py-vision-detector test:performance
# Test WebSocket server
pnpm --filter @cognisight/py-vision-detector test:websocket- Technical Planning & Architectural Blueprint
- Vision Module Progress - Detailed implementation status
- Python Development Guide - Python environment setup
- Development Checklists - Progress tracking for all phases
- Comprehensive Implementation Summary - Complete project overview
This project is currently in active development. The architecture is designed to support future contributions and extensions.
- Follow TypeScript strict mode
- Use shared interfaces from
@cognisight/types - Implement comprehensive testing
- Follow monorepo package structure
- Use
uvfor Python dependencies
MIT License - see LICENSE file for details.