Facial Recognition System

[notzabir]

🎯 Top Resources for Facial Recognition

1. DeepFace (BEST for your setup)
   GitHub: https://github.com/serengil/deepface
   Documentation: https://github.com/serengil/deepface#usage
   Why: Already in your requirements.txt (line 103), supports multiple backends (FaceNet, ArcFace, VGGFace2), real-time performance
   Quick Start:
   Python
   from deepface import DeepFace
   result = DeepFace.find(img_path, db_path, model_name="FaceNet512")

2. FaceNet (Research Foundation)
   Paper: https://arxiv.org/abs/1503.03832
   GitHub: https://github.com/davidsandberg/facenet
   Why: Foundational model, 128D embeddings, proven for person identification
   Use: Works through DeepFace wrapper

3. ArcFace
   Paper: https://arxiv.org/abs/1801.07698
   GitHub Implementations:
   https://github.com/ronghuaiyang/arcface-pytorch
   https://github.com/deepinsight/insightface
   Why: Best accuracy (99.83%), supports InsightFace library

4. InsightFace (Production-Ready)
   GitHub: https://github.com/deepinsight/insightface
   Documentation: https://insightface.ai/
   Why: Optimized for speed + accuracy, integrates ArcFace/CosFace
   Perfect for: Real-time deployment on Jetson
   Quick Install: pip install insightface onnxruntime

⚡ Implementation Plan for Your Setup
Architecture (Leveraging Your Current System)
Code
Your Current System:
┌─────────────────────────────────────────────┐
│ Camera → YOLOv8 Face Detection → Emotion │
│ Classification │
└─────────────────────────────────────────────┘

Enhanced With Facial Recognition:
┌────────────────────────────────────────────────────────────┐
│ Camera → YOLOv8 Face Detection → Face Embedding Extraction │
│ (NEW: FaceNet/ArcFace) │
│ ↓ │
│ Database Lookup │
│ ↓ │
│ Identify Person + Confidence │
│ ↓ │
│ Emotion Classification (Existing) │
│ ↓ │
│ Log: Person + Emotion + Context │
└────────────────────────────────────────────────────────────┘
📋 3-Week Implementation Plan
Week 1: Setup & Prototyping
Task 1.1: Install Dependencies

bash

DeepFace (already in requirements.txt)

pip install deepface

InsightFace (alternative/optional)

pip install insightface onnxruntime

Database & utilities

pip install sqlite3 annoy numpy scikit-learn
Task 1.2: Create cv/facial_recognition.py

Python
from deepface import DeepFace
import numpy as np
import sqlite3

class FacialRecognitionSystem:
def init(self, db_path="faces.db"):
self.db_path = db_path
self.init_database()

def init_database(self):
    """Create SQLite DB for storing embeddings"""
    conn = sqlite3.connect(self.db_path)
    cursor = conn.cursor()
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS people (
            person_id TEXT PRIMARY KEY,
            name TEXT NOT NULL,
            created_at TIMESTAMP
        )
    """)
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS face_embeddings (
            id INTEGER PRIMARY KEY,
            person_id TEXT REFERENCES people(person_id),
            embedding BLOB,
            confidence FLOAT
        )
    """)
    conn.commit()
    conn.close()

def extract_embedding(self, face_image):
    """Extract face embedding using DeepFace"""
    try:
        embedding = DeepFace.represent(
            img_path=face_image,
            model_name="FaceNet512",  # or ArcFace
            enforce_detection=False
        )
        return embedding[0]['embedding']
    except Exception as e:
        print(f"Error extracting embedding: {e}")
        return None

def register_person(self, name, face_image):
    """Register new person into database"""
    embedding = self.extract_embedding(face_image)
    if embedding is None:
        return False
    
    conn = sqlite3.connect(self.db_path)
    cursor = conn.cursor()
    person_id = f"{name}_{int(time.time())}"
    
    cursor.execute(
        "INSERT INTO people VALUES (?, ?, datetime('now'))",
        (person_id, name)
    )
    cursor.execute(
        "INSERT INTO face_embeddings VALUES (NULL, ?, ?, ?)",
        (person_id, np.array(embedding).tobytes(), 1.0)
    )
    conn.commit()
    conn.close()
    return True

def recognize_face(self, face_image, threshold=0.6):
    """Recognize person from face image"""
    embedding = self.extract_embedding(face_image)
    if embedding is None:
        return None
    
    # Compare with all stored embeddings
    conn = sqlite3.connect(self.db_path)
    cursor = conn.cursor()
    cursor.execute("SELECT * FROM face_embeddings JOIN people USING(person_id)")
    
    best_match = None
    best_distance = float('inf')
    
    for row in cursor.fetchall():
        stored_embedding = np.frombuffer(row[2], dtype=np.float32)
        distance = np.linalg.norm(
            np.array(embedding) - stored_embedding
        )
        if distance < best_distance:
            best_distance = distance
            best_match = row
    
    conn.close()
    
    if best_distance < threshold:
        return {
            "person_id": best_match[6],
            "name": best_match[7],
            "confidence": 1 - (best_distance / threshold),
            "distance": best_distance
        }
    return None

Task 1.3: Test with Sample Images

Download 5 images of 2 different people from LFW dataset
Test registration & recognition
Validate accuracy
Week 2: Integration with Current System
Task 2.1: Modify cv/picture.py

Python

Add to CVPipeline class

def execute_with_recognition(self):
"""Enhanced execute() with facial recognition"""
emotions = []
person_info = None

if not self.camera:
    return {"person": None, "emotions": emotions}

success, image = self.camera.read()
if not success:
    return {"person": None, "emotions": emotions}

# Detect faces (existing code)
analysis = self.face_detector(image)[0].boxes

if analysis.xyxy.cpu().numpy() is not None:
    # Get face region
    x1, y1, x2, y2 = map(int, analysis.xyxy.cpu().numpy()[0])
    x1, y1, x2, y2 = self._expand_face(w, h, x1, y1, x2, y2)
    face_region = image[y1:y2, x1:x2]
    
    # NEW: Recognize person
    person_info = self.face_recognizer.recognize_face(face_region)
    
    # Existing: Classify emotion
    emotions = self._classify_emotion(face_region)
    
    # Draw both on image
    label = f"{person_info['name']}" if person_info else "Unknown"
    cv2.putText(image, label, (x1, y1-10), cv2.FONT_HERSHEY_SIMPLEX, 
                1, (0, 255, 0), 2)

return {
    "person": person_info,
    "emotions": emotions,
    "image": image
}

Task 2.2: Update main.py

Python

In continuous_conversation with recognition

result = cv_pipeline.execute_with_recognition()
person = result["person"]
emotions = result["emotions"]

if person:
print(f"Recognized: {person['name']} (confidence: {person['confidence']:.2f})")
# Load person-specific context
user_context = f"This is {person['name']}"
else:
print("Unknown person")
user_context = "I don't recognize you, but I'm happy to chat"
Week 3: Optimization & Deployment
Task 3.1: Performance Optimization

Python

Use caching for faster recognition

from functools import lru_cache
import annoy

class OptimizedFacialRecognition:
def init(self):
self.embeddings_index = annoy.AnnoyIndex(512, metric='euclidean')
self.person_map = {}
self.load_cached_embeddings()

def load_cached_embeddings(self):
    """Pre-load all embeddings into memory for O(log n) lookup"""
    # Faster than database queries for each frame
    pass

Task 3.2: Enrollment Tool

Python

Create enrollment script

def enroll_caregiver(name, image_folder):
"""
Enroll new caregiver with multiple photos
images/
└── caregiver_name/
├── angle1.jpg
├── angle2.jpg
└── angle3.jpg
"""
fr_system = FacialRecognitionSystem()

for image_file in os.listdir(image_folder):
    image_path = os.path.join(image_folder, image_file)
    fr_system.register_person(name, image_path)

print(f"Enrolled {name} successfully")

Task 3.3: Deployment Checklist

Test on Jetson with live camera
Measure inference time (target: <100ms per face)
Test with glasses, masks, different lighting
Validate accuracy on 5+ test people
Optimize CUDA usage for Jetson
📦 File Structure After Implementation
Code
cv/
├── picture.py (MODIFY - add execute_with_recognition)
├── cv_model.py (no change)
├── fer2013.py (no change)
├── facial_recognition.py (NEW)
├── emotions_model.pt (existing)
├── faces.db (NEW - created at runtime)
└── README.md (UPDATE)

scripts/
├── enroll_caregiver.py (NEW)
└── test_recognition.py (NEW)

main.py (MODIFY - add recognition flow)
🚀 Quick Start Commands
bash

1. Install

pip install deepface insightface onnxruntime

2. Enroll a person

python scripts/enroll_caregiver.py --name "Alice" --folder ./images/alice/

3. Test recognition

python scripts/test_recognition.py --image ./test_face.jpg

4. Run full system

python main.py --enable-recognition