🚀 LinkedIn Target Lead Scraper & Curator

An open-source, full-stack lead generation and profile curation tool. It allows you to search for target personas by Role, Location, and Designation, automatically discover public profile URLs via search engine queries without requiring paid API keys, scrape detailed profile fields using Playwright browser automation, filter out low-follower/spam accounts, and curate leads in a dark glassmorphic dashboard.

---

✨ Features

• Keyless Search Discovery: Search target leads by Role, Location, and Designation. Uses search engine dorking (ddgs) without requiring any paid search API keys.
• Headless & Headed Browser Engine: Powered by Playwright and an extensible linkedin_scraper core to log into LinkedIn sessions and extract 18 distinct profile attributes:
  • Full Name, LinkedIn URL, Current Title, Company, Company LinkedIn URL, Location, About, Experience Timeline, Education History, Skills, Contact Email, Contact Phone, Headline, Industry, Personal Website, Seniority Level, Followers/Connections Count, Last Scraped Timestamp.
• Anti-Spam Filtering: Automatically ignores profiles with fewer than 500 followers/connections to keep your lead lists high-quality.
• Interactive Glassmorphic UI: React + TypeScript frontend with live status updates, search targeting controls, lead queue manager, curation drawer, and CSV exporter.
• Zero API Keys Needed: Operates completely self-hosted with local SQLite storage (leads.db) and local session cookie persistence.

---

🏗️ Architecture & Project Structure

ScraperS/
├── backend/                  # FastAPI Backend Server
│   ├── main.py               # API Endpoints & Scraping Queue Worker Loop
│   ├── scraper.py            # Custom PersonScraper & DDGS Target Discovery Engine
│   ├── database.py           # SQLite Models (Profile & QueueItem)
│   └── requirements.txt      # Python Dependencies
├── frontend/                 # Vite + React + TypeScript Dashboard
│   ├── src/
│   │   ├── App.tsx           # Lead Curator & Queue Management Dashboard
│   │   └── index.css         # Cyber Glassmorphism Styling System
│   └── package.json
├── linkedin_scraper_repo/    # Core Playwright LinkedIn Scraper Package
├── .gitignore                # Git exclusion rules for secrets, DBs & venvs
├── LICENSE                   # MIT Open Source License
└── README.md                 # Setup & Usage Instructions

---

⚡ Prerequisites

• Python: 3.10 or higher
• Node.js: 18.x or higher (with npm)

---

🛠️ Quickstart Installation

1. Clone the Repository

git clone https://github.com/YOUR_USERNAME/YOUR_REPO_NAME.git
cd YOUR_REPO_NAME

2. Backend Setup

cd backend

# Create Python virtual environment
python3 -m venv venv
source venv/bin/activate    # On Windows: venv\Scripts\activate

# Install requirements
pip install -r requirements.txt

# Install local linkedin_scraper package
pip install -e ../linkedin_scraper_repo

# Install Playwright browser binaries
playwright install chromium

3. Frontend Setup

cd ../frontend

# Install dependencies
npm install

---

🚦 Running the Application

Start the Backend API

In your terminal (with the backend venv activated):

cd backend
source venv/bin/activate
uvicorn main:app --reload --port 8000

Backend runs on http://localhost:8000

Start the Frontend Dashboard

In a second terminal window:

cd frontend
npm run dev

Frontend runs on http://localhost:5173

---

📖 How to Use

1. Authenticate LinkedIn Account:

   • Open http://localhost:5173.
   • Under Authentication Setup, click Authenticate LinkedIn Account.
   • A browser window will open. Log into your LinkedIn account and solve any captcha challenges if prompted.
   • Once logged in, close or return to the app; your session cookies will be saved securely to linkedin_session.json locally.

2. Discover Leads by Target Persona:

   • Navigate to the Target Search tab in the sidebar.
   • Enter your target criteria (e.g., Role: Software Engineer, Location: San Francisco, Designation: Senior).
   • Select the target limit and click Find & Queue Leads.
   • Profile URLs will automatically populate the queue.

3. Scrape & Curate Leads:

   • Click Start Scraper. The backend will sequentially log into profiles, extract data, and filter out low-follower spam profiles (<500 followers).
   • View, filter, edit, write custom notes, and update lead statuses (Pending, Contacted, Qualified, Rejected) directly in the table drawer.
   • Export curated leads anytime via Export CSV.

---

🛡️ Privacy & Safety Note

• This project is built for educational and research purposes.
• Your session cookies (linkedin_session.json) and database (leads.db) are stored only on your local machine and are listed in .gitignore so they will never be pushed to GitHub.

---

📜 License

This project is open-source and available under the MIT License.