VTON (Virtual Try-On) Application

An intelligent, microservice-based Virtual Try-On application. It combines semantic search with generative AI to provide a seamless product discovery and try-on experience.

Features

• AI Stylist: Conversational outfit recommendations ("I have a job interview tomorrow") over your existing garment catalog — complete top/bottom/shoes/accessories picks with confidence scores and explanations, before you ever generate a try-on image. See AI Stylist Architecture.
• Virtual Try-On: Generate realistic try-on images using a dual generative pipeline — a Stable Diffusion route and a toggleable Flux-2 route.
• Semantic Search: Text-to-image and image-to-image search capabilities powered by sentence-transformers and pgvector.
• Microservice Architecture:
  • Backend: FastAPI (Async)
  • Frontend: Next.js
  • Workers: Celery + RabbitMQ (Scalable & GPU-accelerated)
  • Database: PostgreSQL (pgvector)

Generative Pipelines

The system supports two interchangeable generation routes, selectable at inference time:

1. Stable Diffusion route

The original try-on pipeline, built on Stable Diffusion for garment-swap inpainting.

2. Flux-2 route (toggleable)

An alternative pipeline built around a Quantized Rectified Flow Transformer (Flux-2 Klein) with a custom LoRA adapter fine-tuned for cloth-swap tasks. Highlights, based on our internal research writeup (see Research below):

• Runs at Q2 quantization, fitting fully in VRAM on consumer 12 GB GPUs.
• Uses fixed-step flow matching at 4 denoising steps, bringing inference down to roughly 8–10 seconds per image on 12 GB VRAM hardware (vs. 25–60s for typical full diffusion pipelines).
• LoRA adapter (default merge strength 0.85) preserves high-frequency garment detail — logos, embroidery, knit texture — that standard diffusion pipelines tend to blur.
• Automated preprocessing: Rembg for garment background/foreground extraction, Gemini Flash API for garment captioning, Qwen_4b (quantized) as the text/image conditioning encoder.

Both routes share the same FastAPI → RabbitMQ → Celery → Redis backend, so switching routes doesn't change the deployment topology — only which model the worker loads.

Documentation

• Master Plan: Project roadmap and high-level goals.
• Architecture Overview: System design and component interaction.
• Backend Architecture: Detailed API and worker design.
• Frontend Architecture: Frontend components and state management.
• Generation Backends: The ImageGenerationBackend interface behind both routes, and how to plug in a new one (SDXL, Krea, or anything else).
• AI Stylist Architecture: The conversational outfit-recommendation engine — NLU, retrieval, composition, confidence scoring, data model.
• LLM Backends: The LanguageModelBackend interface behind the stylist — local Gemma 4 E4B via llama-server, with a Gemini API fallback for VRAM-constrained machines.
• AI Stylist Architecture: Conversational outfit recommendation — data flow, confidence scoring, data model, API.
• LLM Backends: The LanguageModelBackend interface behind the stylist — local Gemma 4 E4B vs. the Gemini fallback.

Research

docs/journal.docx is our project writeup covering the Flux-2 route in depth: motivation, related work (CP-VTON+, VITON-HD, GP-VTON, TryOnDiffusion, StableVITON, OOTDiffusion, IDM-VTON, CAT-DM), the quantized rectified flow + LoRA architecture, dataset preparation, and evaluation.

Summary of what's in there:

• Problem: existing GAN- and diffusion-based VTON methods either distort textures/warp garments poorly, or are too slow (25–60s) for real-time use.
• Approach: a Flux-2 Klein backbone (rectified flow transformer) distilled and quantized to Q2, paired with a LoRA adapter trained on a ~90-sample custom dataset (garment images + human images + ComfyUI-generated targets), running fixed-step flow matching at 4 steps.
• Result: ~6-8s generation on 12 GB VRAM hardware in the optimized configuration.
• Scaling test: near-linear throughput scaling across Celery workers (~30 req/min single worker → ~60 req/min with two workers).
• Search eval: 0.78 top-5 precision on text-to-image garment retrieval over a 500-item catalog.
• Future work: broader LoRA training data (body types, skin tones, poses), single-pass full-outfit generation, ControlNet/DensePose conditioning for non-frontal poses, video VTON, and a lighter mobile-oriented model variant.

Quick Start (Docker Compose)

Best for local testing with GPU support.

Prerequisites: an NVIDIA GPU with nvidia-container-toolkit installed on the host if you're using either generation profile — worker-inpainting and comfyui request a GPU via Docker Compose's deploy.resources.reservations.devices. docker compose (Compose V2) is required (see the flag-ordering note in step 3).

1. Clone & Setup:

   git clone <repository-url>
   cd vton
   

2. Configure environment (optional but recommended):

   Create backend/.env with at least:

   GEMINI_API_KEY=your-key-here
   

   Without it, uploads still work and garments still get background-removed, but automatic metadata tagging (category/color/pattern) is skipped — which also means the AI Stylist's search/matching quality drops, since it ranks garments using those tags.

   The AI Stylist also needs an LLM backend for its NLU/explanation steps: GEMINI_API_KEY above doubles as this (it's the default fallback), so no extra setup is needed to try the stylist. If you'd rather run its LLM fully locally instead of calling Gemini, add --profile stylist-local to the commands in step 3 below — see LLM Backends for the VRAM tradeoff.

   If Rupz25/vton-cloth-swap-lora (used by the transformer profile) is ever made gated/private, launch_comfy.py's huggingface-cli download step needs an HF_TOKEN in the comfyui container's own environment (it's a plain shell command, not routed through the app's .env loading) — add environment: - HF_TOKEN=${HF_TOKEN:-} to the comfyui service in docker-compose.yml and set HF_TOKEN in a .env at the repo root (next to docker-compose.yml, not backend/.env). Not needed for a public repo.

3. Run Backend & Services — pick a generation profile. Uploading, background removal, and metadata tagging now work regardless of which profile you choose (or neither) — they run on an always-on worker-preprocessing service, decoupled from both generation routes. --profile is a global flag and must come before up, not after:

   # Stable Diffusion route only:
   docker compose --profile inpainting up --build
   
   # Flux-2 route only — auto-downloads model weights on first run, see below:
   docker compose --profile transformer up --build
   
   # Both (needs enough VRAM for both to coexist, or run them on separate machines):
   docker compose --profile inpainting --profile transformer up --build
   

   Note: this is docker compose (space — the current Compose V2 plugin), not the older standalone docker-compose (hyphen) binary. If docker compose (space) isn't recognized on your machine, install/update via Docker Desktop or the compose-plugin package.

   --profile inpainting builds worker-inpainting from Dockerfile.inpainting, which clones and builds Detectron2 from source — this first build can take a long time. --profile transformer starts comfyui, which runs app/core/scripts/launch_comfy.py on boot — a single script that installs ComfyUI itself via comfy-cli (comfy install), installs the ComfyUI-GGUF custom node (required for the GGUF-quantized Flux files), downloads any missing model weights from Rupz25/vton-cloth-swap-lora via huggingface-cli, then launches the server. Each step is guarded (checks whether ComfyUI/the node/each weight file already exists before doing anything), so a first run takes a while — several GB of weights plus the ComfyUI install itself — but restarting the stack afterward doesn't redo any of it. Two things make that persistence work: the comfy_site_packages named volume (holds everything comfy install/comfy node install pulled in) and backend/app/core/weights/ being a regular bind-mounted directory (holds the downloaded model files). Run the script manually anytime with docker compose run --rm comfyui python -m app.core.scripts.launch_comfy, or tail its output live with docker compose logs -f comfyui.

   Model/package caching: huggingface_hub's cache (used by the SD1.5 pipeline and the search embedding model) and rembg's background-removal weights are now stored in named Docker volumes (huggingface_cache, u2net_cache) instead of each container's ephemeral writable layer — they survive docker compose down and container recreation, so restarting the stack shouldn't redownload multi-GB files anymore. If you want to reclaim that disk space entirely, docker compose down -v removes the volumes too (next run starts fresh).

4. Run Frontend: Open a new terminal:

   cd frontend
   npm install
   npm run dev
   

5. Use:

   • Frontend: http://localhost:3000
   • API Docs: http://localhost:8000/docs
   • Toggle the generation route (Stable Diffusion / Flux-2) from the try-on studio in the frontend, or via the relevant backend config — see Backend Architecture for the flag/setting name.

Troubleshooting — uploads stay as raw, unprocessed images: worker-preprocessing should be running regardless of which profile(s) you started — check with docker compose ps. If it's not listed at all, you're on an old checkout before background removal was decoupled from the inpainting profile; pull latest. If it's listed but keeps restarting, check docker compose logs -f worker-preprocessing — the usual cause is rembg's model downloading its weights from GitHub on first use, which needs outbound network access. You can also confirm a specific upload's status directly: GET /api/v1/ingestion/status/{task_id} (the task_id is returned by the upload response) will show "status": "failed" with an error message if the task ran but errored, versus "status": "PENDING" forever if nothing is consuming the queue at all.

Troubleshooting — comfyui fails with a missing-node error (e.g. CLIPLoaderGGUF / UnetLoaderGGUF not found), or model weights are missing: check docker compose logs comfyui first — launch_comfy.py logs every step (ComfyUI install, node install, each weight file) live, so this is usually the fastest way to see exactly which step failed and why (network issue, disk space, a renamed file upstream). If ComfyUI itself seems stuck mid-install, docker compose exec comfyui bash and check whether ~/comfy/ComfyUI exists — the script only reruns its setup steps when that directory (or the GGUF node's directory) is missing, so a partial/interrupted install can leave it in a state where the guard thinks setup already finished. Deleting the comfy_site_packages volume (docker volume rm virtual-try-on_comfy_site_packages) forces everything to reinstall from scratch on the next up.

Quick Start (Kubernetes)

(Coming Soon)