A Go vector database focused on fast in-memory vector storage, RESP-compatible commands, optional HNSW search, and snapshot persistence. It includes SIMD-accelerated dot products on supported AMD64 CPUs, with pure Go fallback elsewhere.
- SIMD-Optimized Dot Products: AVX2/AVX512 assembly on supported AMD64 CPUs, pure Go fallback otherwise
- Optional HNSW Index: Hierarchical Navigable Small World search via
-index=hnsw;-index=autokeeps small searches on sharded full scan - Snapshot Persistence: RDB-style persistence with compression and automatic scheduling
- High-Performance Storage: 32-way sharded in-memory storage with lock-free metrics
- RESP Protocol: Compatible with Redis protocol for easy integration
- Observability: Built-in metrics, structured logging, and request tracing
- Operational Features: Graceful shutdown, automatic snapshots, structured logging, and error handling
- SIMD Assembly Optimizations: AVX2/AVX512 instructions for dot products on supported AMD64 CPUs
- HNSW Algorithm: Optional multi-layer graph index for approximate search
- Snapshot Persistence: Atomic RDB-style snapshots with Snappy compression
- 32-way Sharding: Reduces lock contention with CPU cache-line padding
- Optimized Vector Search: Normalized vectors enable dot-product computation for cosine similarity
- Operational Features: Graceful shutdown, automatic snapshots, memory monitoring
- Go 1.22+ - Download
- Git - For cloning and version info
- Make - For build commands (optional)
# Build everything
make build
# Run the server
make run
# Run with HNSW search index enabled
go run ./cmd/vex-server/main.go -index=hnsw
# Run with automatic search selection
go run ./cmd/vex-server/main.go -index=auto
# Run with JSON logging
make run-json
# Run with debug logging
make run-debug# Build the image
docker build -t vex .
# Run the container
docker run -p 6379:6379 vex# Install vex-server binary to $GOPATH/bin
go install github.com/uzqw/vex/cmd/vex-server@latestConnect using any RESP protocol client (like redis-cli) or netcat:
# Using netcat
nc localhost 6379
# Using redis-cli
redis-cli -p 6379If you don't have redis-cli installed, you can install it with:
Ubuntu/Debian:
sudo apt update && sudo apt install redis-toolsmacOS (Homebrew):
brew install redisArch Linux:
sudo pacman -S redisPING [message]- Test connectionECHO message- Echo back a messageSTATS/INFO- Get vex statisticsQUIT- Close connection
VSET key "[0.1, 0.2, 0.3, ...]"
Example:
VSET vec:1 "[0.12, 0.33, 0.95]"
+OK
VGET key
Example:
VGET vec:1
$26
[0.120000, 0.330000, 0.950000]
VDEL key
Returns :1 if deleted, :0 if key didn't exist.
VSEARCH "[0.1, 0.2, 0.3, ...]" k
Example (find top 5 similar vectors):
VSEARCH "[0.12, 0.33, 0.95]" 5
*2
$5
vec:9
$5
vec:3
CLEAR
+OK
BGSAVE
+Background saving started
Creates a snapshot asynchronously without blocking operations.
SAVE
+OK
Creates a snapshot synchronously (blocks until complete).
INFO persistence
${JSON}
{
"last_snapshot_time": "2026-01-30T16:00:00Z",
"snapshot_in_progress": false,
"total_snapshots": 42,
"snapshot_errors": 0
}
Get real-time vex metrics:
STATS
${JSON}
{
"goroutines": 54,
"total_commands": 125000,
"active_connections": 12,
"total_keys": 50000,
"memory_usage_mb": 245.3,
"uptime": "1h20m15s",
"qps": 12500.5
}
Vex includes comprehensive performance benchmarks at both the unit and integration levels.
Test individual components in isolation:
# Run all unit benchmarks
make bench-unit
# Or directly with Go
go test -bench=. -benchmem ./benchmarks/...Test the complete system with realistic workloads. Requires a running server:
# Terminal 1: Start the server
make run
# Terminal 2: Run benchmarks
make bench-integrationRun both unit and integration benchmarks:
# Run all benchmarks
./benchmarks/run-all.sh
# Or using make
make benchFor comprehensive documentation on benchmarking, see benchmarks/README.md.
# Run storage benchmarks
go test -bench=BenchmarkStorage -benchmem ./benchmarks/storage/
# Run with CPU profiling
go test -bench=. -benchmem -cpuprofile=cpu.prof ./benchmarks/storage/
go tool pprof -http=:8080 cpu.prof
# Compare with previous results
go install golang.org/x/perf/cmd/benchstat@latest
go test -bench=. -benchmem ./benchmarks/storage/ > new.txt
benchstat old.txt new.txt# Custom insert benchmark
make benchmark-custom ARGS="-mode=insert -concurrency=100 -n=200000 -dim=256"
# Custom search benchmark
make benchmark-custom ARGS="-mode=search -concurrency=50 -n=100000 -prepare-n=50000 -warmup=5000 -k=10"
# Or run directly
go run cmd/vex-benchmark/main.go -mode=insert -concurrency=50 -n=100000 -warmup=5000=== Vex Benchmark ===
Mode: insert
Concurrency: 50
Total Ops: 100000
---
Total Time: 1.25s
QPS: 80000 ops/sec
Success: 100000
Errors: 0
Latency Statistics:
Min: 245µs
Avg: 625µs
P50: 580µs
P95: 1.2ms
P99: 2.1ms
Max: 5.3ms
make bench # Run all benchmarks (unit + integration)
make bench-all # Run all unit benchmarks
make bench-unit # Run unit benchmarks only
make bench-storage # Run storage layer benchmarks
make bench-integration # Run integration benchmarks (requires running server)
make benchmark # Legacy: insert benchmark
make benchmark-search # Legacy: search benchmark
make benchmark-custom # Custom benchmark with parametersvex/
├── benchmarks/ # Performance benchmarks
│ ├── storage/ # Storage layer benchmarks (HNSW, brute force)
│ ├── README.md # Comprehensive benchmark guide
│ └── run-all.sh # Run all benchmarks script
├── cmd/
│ ├── vex-server/ # Main server entry point
│ └── vex-benchmark/ # Integration benchmark tool
├── internal/
│ ├── protocol/ # RESP protocol parsing
│ ├── storage/ # Sharded vector storage
│ │ ├── index.go # Index interface
│ │ ├── index_hnsw.go # HNSW implementation
│ │ └── persistence/ # Snapshot persistence layer
│ ├── vector/ # Vector computation
│ │ ├── asm/ # AVX2/AVX512 assembly (goat-generated)
│ │ └── c/ # C source for SIMD (from Weaviate)
│ └── metrics/ # Performance metrics
├── pkg/
│ └── logger/ # Structured logging
├── docs/
│ ├── README.md # Documentation index
│ ├── PERFORMANCE.md # SIMD benchmarks and results
│ ├── BUILD.md # Assembly compilation guide
│ ├── PERSISTENCE.md # Persistence design
│ └── LICENSES.md # License compliance
├── Dockerfile.goat # Docker image for goat tool
└── README.md # This file
make help # Show all available commands
make build # Build binaries
make run # Run server
make test # Run tests
make test-coverage # Run tests with coverage
make fmt # Format code
make vet # Run go vet
make lint # Run golangci-lint
make test-race # Run tests with race detector
make install-tools # Install dev tools
make clean # Clean build artifacts# Run all tests
make test
# Run with coverage report
make test-coverage-host- Host to bind to (default: "0.0.0.0")-port- Port to listen on (default: "6379")-index- Search index: "none", "bruteforce", "hnsw", or "auto" (default: "none")-auto-index-min-vectors- Minimum vector count before auto mode uses HNSW (default: 10000)-hnsw-m- HNSW max neighbors per upper layer (default: 16)-hnsw-ef- HNSW search beam width (default: 600)-hnsw-ef-construction- HNSW construction beam width (default: 600)-hnsw-seed- HNSW random seed; 0 uses a random seed (default: 0)-log-format- Log format: "text" or "json" (default: "text")-log-level- Log level: "debug", "info", "warn", "error" (default: "info")
Use -index=hnsw to route VSEARCH through the HNSW index. Use -index=auto
to keep small datasets on sharded full-scan search and switch to HNSW above
-auto-index-min-vectors. The default none keeps the original sharded
full-scan search path.
Persistence can be configured via environment variables:
VEX_PERSISTENCE_ENABLED- Enable persistence (default: false)VEX_DATA_DIR- Data directory for snapshots (default: "./data")VEX_SNAPSHOT_SECONDS- Auto-snapshot interval in seconds (default: 300)VEX_KEEP_SNAPSHOTS- Number of snapshots to retain (default: 3)
Example:
VEX_PERSISTENCE_ENABLED=true VEX_SNAPSHOT_SECONDS=600 ./vex-server-host- Server host (default: "localhost")-port- Server port (default: "6379")-concurrency- Number of concurrent connections (default: 50)-n- Total number of measured operations (default: 100000)-mode- Benchmark mode: "insert" or "search" (default: "insert")-dim- Vector dimension (default: 128)-prepare-n- Number of vectors to load before search benchmarks (default: 1000)-warmup- Number of warmup operations to run before measuring (default: 0)-k- Top-k value forVSEARCH(default: 10)-seed- Random seed for deterministic vectors (default: 42)-key-prefix- Key prefix used for generated vectors (default: "vec")
vector.DotProduct selects the best available implementation at runtime:
AVX512, AVX2, then pure Go fallback. The speedup depends heavily on CPU
features, vector dimension, and frequency scaling, so benchmark on the target
machine before quoting performance:
go test -run='^$' -bench=BenchmarkDotProductComparison -benchmem ./internal/vector
go test -tags=noasm -run='^$' -bench=BenchmarkDotProduct -benchmem ./internal/vectorSee docs/PERFORMANCE.md for the SIMD implementation notes.
End-to-end QPS and latency depend on CPU, dimension, dataset size, concurrency, index mode, and snapshot settings. Use the bundled benchmark tool to measure your target workload instead of relying on fixed headline numbers:
go run ./cmd/vex-benchmark/main.go \
-mode=search -concurrency=50 -n=100000 -dim=768 \
-prepare-n=50000 -warmup=5000 -k=10- Search modes: default sharded full scan, optional brute-force index, optional HNSW index
- Concurrency: 32-way sharded storage reduces lock contention for writes and direct storage reads
- Memory: vectors are stored as float32 values (~4 bytes per dimension before Go/map/index overhead)
- SIMD: dot-product calls use the selected AVX2/AVX512 implementation when available
- Semantic search applications
- Recommendation systems
- Similarity detection
- Embedding storage and retrieval
- Development and testing of vector-based systems
- In-Memory First: Primary storage is in-memory; persistence uses periodic snapshots (no WAL yet)
- Single Node: No clustering or replication support
- No Authentication: No built-in auth mechanism (use network isolation)
- Cosine Similarity Only: Only cosine similarity distance metric is supported
- AMD64 Only: SIMD optimizations require AMD64 with AVX2 (ARM64 support planned)
Comprehensive documentation is available in the docs/ directory:
- docs/README.md - Documentation index and quick reference
- docs/PERFORMANCE.md - SIMD optimization benchmarks and analysis
- docs/BUILD.md - Guide for compiling C to assembly with goat
- docs/PERSISTENCE.md - Persistence design and implementation
- docs/LICENSES.md - License compliance (BSD-3 + Apache 2.0)
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Vex includes SIMD optimization code derived from Weaviate (BSD-3-Clause), which is fully compatible with Apache 2.0. See THIRD_PARTY_LICENSES.md and NOTICE for details.
Copyright © 2025 uzqw
- You can use this software for commercial and non-commercial purposes
- You must include a copy of the license and copyright notice
- You must document significant changes made to the code
- The software is provided "as is" without warranties or liabilities
For full details, please refer to the Apache 2.0 License.
Key implementation principles:
- Performance First: SIMD optimizations, efficient algorithms (HNSW), lock-free operations
- Production-Ready: Persistence, graceful shutdown, comprehensive error handling
- Observability: Structured logging, atomic metrics, detailed benchmarks
- Code Quality: Comprehensive tests, race detection, proper documentation
When contributing:
- Follow existing code style and patterns
- Add tests for new functionality
- Update documentation (especially performance benchmarks)
- Run
make testandmake test-racebefore submitting - For SIMD changes, see docs/BUILD.md
- Weaviate: SIMD optimization C code (BSD-3-Clause)
- goat: C to Go assembly conversion tool
- hnswlib: HNSW algorithm reference implementation
Built following production-grade practices for high-performance Go services.