Vector Index Benchmark¶
Recall@10–QPS was measured on GIST1M and SIFT1M. The index configurations were AkasicDB Hierarchical Navigable Small World (HNSW), Vamana, and DiskANN; pgvector HNSW; and Milvus HNSW. AkasicDB quantization and DiskANN are Enterprise Edition features. For index creation and query examples, see Compare Index Families.
Methodology¶
All products used the same query-vector order and k=10 for each dataset. Products ran one at a
time on the same host with identical server and client CPU and memory limits and CPU frequency
policy. Each product used its native protocol. Every HNSW index used M=24 and
ef_construction=200.
| Item | Condition |
|---|---|
| Dataset | GIST1M: 1,000,000 960-dimensional float32 vectors and 1,000 official queries. SIFT1M: 1,000,000 128-dimensional float32 vectors and 10,000 official queries. |
| Recall@10 | Mean tie-aware Recall@10 over a common 1,000-query sample, calculated from exact squared Euclidean distance (L2) on the original vectors. Duplicate IDs were removed; results tied at the 10th ground-truth distance were accepted. |
| QPS | One client submitted 10,000 top-10 queries sequentially. QPS is the completed query count divided by measurement time and includes serialization, transport, server decode, ANN search, and result receipt. |
| Repetition | The same index was measured repeatedly with different search-width orders. Points show median QPS; error bars show the measured range. |
| Segments | GIST1M: one per AkasicDB HNSW or Vamana index, four for Milvus Plain, and five each for Milvus SQ and PQ. SIFT1M: one per index configuration for both AkasicDB and Milvus. |
| In-memory indexes | Loaded into memory without ANN warm-up queries. Index loading was outside the measurement interval. |
| DiskANN | Measured cold after clearing the host page cache before each point and reading the graph through O_DIRECT. Cache initialization was outside the measurement interval. |
Each product used the segment layout produced by its build. Milvus searches loaded segments in parallel within a query. GIST1M Recall@10 and one-client QPS include this multi-segment execution. The reported values are not a comparison between single-segment implementations.
HNSW search width ef_search and Vamana or DiskANN search width L do not perform equivalent
work at the same value. QPS ratios across products or index families were calculated only when
Recall@10 differed by no more than 0.5 percentage points (pp). DiskANN and in-memory indexes are
shown in the same charts, but their storage states differ and no QPS ratios were calculated
between them.
Plain denotes an index without quantization. SQ and PQ comparisons cover complete index
configurations, not isolated quantizers; compression formats and storage sizes differ by product.
The AkasicDB series use the HNSW and Vamana recommended profiles in
Quantization.
GIST1M¶
Plain Index¶
Figure 1. GIST1M Plain index. Labels use search width | QPS.
Plain comparisons within 0.5 pp Recall@10:
| Product / index | Search setting | Recall@10 | QPS | Comparison |
|---|---|---|---|---|
| pgvector HNSW Plain | ef_search=200 |
97.11% | 131.848 | Baseline |
| AkasicDB HNSW Plain | ef_search=200 |
97.12% | 323.850 | 2.456× pgvector; Recall difference 0.01 pp |
| AkasicDB Vamana Plain | L=200 |
97.37% | 328.126 | 2.489× pgvector; Recall difference 0.26 pp |
Scalar Quantization¶
Figure 2. GIST1M scalar quantization. AkasicDB HNSW used SQ4, AkasicDB Vamana used ProjectedMinMax8, and Milvus HNSW used SQ4U with 32-bit floating-point (FP32) refinement.
| Product / index | Search setting | Recall@10 | QPS | Comparison |
|---|---|---|---|---|
| Milvus HNSW SQ4U | ef_search=200 |
97.08% | 496.138 | Baseline |
| AkasicDB Vamana ProjectedMinMax8 | L=200 |
97.31% | 761.938 | 1.536× Milvus; Recall difference 0.23 pp |
Product Quantization and DiskANN¶
Figure 3. GIST1M product quantization and DiskANN. AkasicDB HNSW used PQ8 ratio=4, and
AkasicDB Vamana used PQ8 ratio=6. DiskANN was measured under the cold condition.
Relative to Plain at the same width, PQ QPS was 23.71–35.67% higher for AkasicDB HNSW and 35.07–49.28% higher for AkasicDB Vamana. Recall changed by +0.03 to +0.50 pp and −1.49 to +0.22 pp, respectively.
| DiskANN setting | L=50 Recall@10 / QPS |
L=100 Recall@10 / QPS |
L=200 Recall@10 / QPS |
|---|---|---|---|
| No internal cache | 91.64% / 289.569 | 95.58% / 237.070 | 98.23% / 167.432 |
cache_nodes=500000 |
91.73% / 399.337 | 95.54% / 325.872 | 98.22% / 229.786 |
SIFT1M¶
Plain Index¶
Figure 4. SIFT1M Plain index. Labels use search width | QPS.
| Product / index | Search setting | Recall@10 | QPS | Comparison |
|---|---|---|---|---|
| pgvector HNSW Plain | ef_search=100 |
99.47% | 396.701 | Baseline |
| AkasicDB HNSW Plain | ef_search=100 |
99.47% | 1214.514 | 3.062× pgvector; Recall difference 0.00 pp |
| Milvus HNSW Plain | ef_search=200 |
99.87% | 635.153 | Baseline |
| AkasicDB HNSW Plain | ef_search=200 |
99.91% | 952.188 | 1.499× Milvus; Recall difference 0.04 pp |
| AkasicDB Vamana Plain | L=100 |
99.83% | 1201.449 | 1.892× Milvus; Recall difference 0.04 pp |
Scalar Quantization¶
Figure 5. SIFT1M scalar quantization. AkasicDB HNSW SQ4 QPS was 1.15–2.77% higher than Plain at the same width, with a Recall difference of −0.05 to +0.24 pp. ProjectedMinMax8 requires at least 161 dimensions and was not applied to 128-dimensional SIFT1M.
Product Quantization and DiskANN¶
Figure 6. SIFT1M product quantization and DiskANN. AkasicDB HNSW PQ8 ratio=4 QPS was
3.65–9.29% higher than Plain at the same width, with a Recall difference of −0.29 to +0.09 pp.
AkasicDB Vamana PQ8 ratio=6 had lower Recall and QPS than Plain. DiskANN was measured under the
cold condition.
| DiskANN setting | L=50 Recall@10 / QPS |
L=100 Recall@10 / QPS |
L=200 Recall@10 / QPS |
|---|---|---|---|
| No internal cache | 98.67% / 527.020 | 99.74% / 410.937 | 99.94% / 289.283 |
cache_nodes=500000 |
98.79% / 728.548 | 99.71% / 553.036 | 99.94% / 367.385 |