Visual Search

Upload a fashion product image and find the most visually similar items in a 170,000-image gallery, powered by CLIP embeddings and cosine similarity search.

How it works

1. The gallery is a balanced sample of ~34,000 images from each of the 5 categories in fashion200k (dresses, tops, skirts, pants, jackets), targeting 170,000 images total — see build_gallery.py. An earlier version took an unshuffled slice of the source dataset, which happened to land entirely inside "dresses", so pants/tops/skirts/jackets queries had nothing relevant to match against.
2. Each gallery image is embedded with patrickjohncyh/fashion-clip (CLIP fine-tuned on fashion product images) and stored as a normalized vector in 170k_image_embeddings.npy, alongside its item ID in 170k_image_IDs.npy and its original row position in 170k_gallery_indices.npy. A prebuilt FAISS index (gallery.faiss) is used for fast nearest-neighbor search; a Qdrant vector DB backend is also available (see Configuration).
3. When you upload an image, the app embeds it with the same model.
4. The query embedding is compared against the gallery via cosine similarity, and the top TOP_K matches are shown.

The source dataset (Marqo/fashion200k on the Hugging Face Hub) is loaded from a local cache at data/fashion200k if present, otherwise streamed from the Hub; 170k_gallery_indices.npy re-selects the exact sampled rows in the same order as the saved embeddings.

To regenerate the gallery (e.g. after changing the model or sample size):

uv run python build_gallery.py

Running locally

uv sync
uv run streamlit run app.py

Or with plain pip:

pip install -r requirements.txt
streamlit run app.py

The app will download the CLIP model weights and the gallery dataset on first run if they aren't already present locally.

FastAPI alternative

A FastAPI + Jinja2 version of the same app lives alongside the Streamlit one, and is what the Docker image runs:

uv run uvicorn fastapi_app:app --reload

It exposes:

• GET / — web UI
• POST / — web UI image search
• GET /health — health check
• POST /api/search — search via file upload (multipart form, field file)
• POST /api/search-base64 — search via base64-encoded image (JSON body, field image_base64)
• GET /docs — interactive API docs (Swagger UI)

Example API responses:

{
  "success": true,
  "results": [
    {"rank": 1, "item_id": "123456", "score": 0.9234, "image_url": "data:image/jpeg;base64,..."}
  ],
  "warning": null,
  "error": null,
  "query_image_url": "data:image/jpeg;base64,..."
}

To get base64 from an image file for the /api/search-base64 endpoint:

# macOS/Linux
base64 -i image.jpg | tr -d '\n'
# Windows (PowerShell)
[Convert]::ToBase64String([IO.File]::ReadAllBytes("image.jpg"))

Running with Docker

docker build -t visual-search:latest .
docker run -p 8000:8000 visual-search:latest
# or: docker compose up

Verify it's running:

curl http://localhost:8000/health

The image bundles gallery.faiss, the 170k_* embedding/ID/index files, and the local data/fashion200k dataset cache via COPY . ., so the container starts with no runtime network dependency on the Hugging Face Hub. torch is pinned to the CPU-only build on Linux ([tool.uv.sources] in pyproject.toml) since the app only ever runs on CPU/MPS, keeping the image several GB smaller than the default CUDA-enabled wheel.

Troubleshooting

• Port 8000 already in use: lsof -i :8000 then kill -9 <PID>, or run with a different host port, e.g. docker run -p 8001:8000 ....
• Container fails to start: check docker logs <container_id>, or run interactively with docker run -it -p 8000:8000 visual-search:latest.
• "No close matches" on every query: the similarity score is below SIMILARITY_THRESHOLD (default 0.50) — try a different image or adjust the threshold in visual_search/config.py.

Configuration

Key settings in visual_search/config.py (most overridable via environment variables):

• TOP_K — number of results returned (default: 10)
• SIMILARITY_THRESHOLD — minimum similarity score (default: 0.50)
• MODEL_NAME — embedding model (default: patrickjohncyh/fashion-clip)
• EMBEDDINGS_PATH, ITEM_IDS_PATH, GALLERY_INDICES_PATH, FAISS_INDEX_PATH, DATASET_REPO — override to point at custom asset locations instead of the project-root defaults
• USE_VECTOR_DB / VECTOR_DB_URL / VECTOR_DB_COLLECTION — switch the gallery index from local FAISS to a Qdrant instance

Notebook / exploration dependencies

ipykernel, ipywidgets, and matplotlib are only needed for local notebook exploration, not for running the app. They live in the dev dependency group and are excluded from the Docker image and from requirements.txt:

uv sync                 # includes dev group (default)
uv sync --no-dev         # app-only, matches what Docker installs

Tech stack

• Streamlit — UI
• FastAPI — alternative HTTP UI, used in Docker
• sentence-transformers (fashion-clip) — image embeddings
• FAISS — similarity search index
• 🤗 Datasets — gallery image hosting/loading
• Qdrant — optional vector DB backend