Qdrant, the high-performance, open-source vector search engine, today released Qdrant-FineWeb-10B, a public dataset for vector retrieval benchmarking built on 10 billion documents and 120k ground truth queries, together with Supernova, the toolkit that computed Qdrant-FineWeb-10B’s ground truth.
Engineers Benchmark Vector Search to Simulate Production
As part of vector search evaluation and testing, engineers often benchmark vector search for speed and accuracy. Three properties help determine whether a benchmark is representative of a production environment.
- The vectors should come from real data. Embeddings of real documents are distributed unevenly, with dense regions, sparse regions, and near-duplicates throughout. Approximate indexes are built to exploit exactly that structure, so numbers measured on randomly generated vectors or on a small curated set are not representative of a real corpus.
- The corpus has to be large enough that size-dependent behavior shows up. Memory footprint, index build time, and shard layout all change as a corpus grows, and recall can change with them. A configuration tuned on a few million vectors does not easily extrapolate at scale.
- The correct answers have to be known exactly and in depth. To know whether an approximate query returned the correct results, you need a list of what the correct results actually are. Without that list there is nothing to measure against. This helps give confidence about the index quality and robustness.
What Public Datasets Are Missing
Large real-world embedding sets exist, but they do not come with a deep ground truth query set. That list of right answers is the ground truth, and computing it exactly means comparing every query against every vector in the corpus, with no index and no approximation, which is why it is expensive.
Along with the 10 billion documents, Qdrant-FineWeb-10B’s ground truth covers 100,000 queries drawn from MS MARCO, a public set of real search queries, plus 10,000 sparse and 10,000 filtered queries. Running 120,000 queries against 10 billion documents is over a quadrillion distance computations.
With Qdrant-FineWeb-10B, you can measure recall, the share of the true nearest results a system actually returned, for any k up to 1,000 on public data. That works on public data at 10 billion documents, across dense vectors, which encode meaning, sparse vectors, which encode terms, and metadata filters.
“Accurate benchmarking is one of the hardest things we get asked about, both in our open-source community and in customer discussions,” said André Zayarni, CEO and Co-Founder of Qdrant. “Teams need real data at the size they actually run, and they need to know what the correct answers are. Assembling both yourself is expensive. We did it once and made it public.”
Qdrant Partnered with Vultr and SkyPilot to Build Qdrant-FineWeb-10B
The source corpus is FineWeb, published by Hugging Face and derived from Common Crawl. Qdrant generated the embeddings on infrastructure provided by Vultr, running every embedding through gte-multilingual-base, an open embedding model from Alibaba, to produce one dense and one sparse vector.
Qdrant then computed the ground truth queries with Supernova, using SkyPilot to distribute the exhaustive search across a fleet by rank. Qdrant-FineWeb-10B ships embeddings, payload fields, and ground truth, not the source text.
|
Property |
Value |
|
Documents |
10 billion, from FineWeb |
|
Vectors per record |
One dense (768 dimensions), one sparse |
|
Dense ground truth |
100,000 MS MARCO queries, exact, depth 1,000 |
|
Sparse ground truth |
10,000 queries, exact, depth 1,000 |
|
Filtered ground truth |
10,000 queries with metadata predicates, exact, depth 1,000 |
Because the ground truth runs to depth 1,000, you can measure recall@k for any k up to 1,000 by truncating the list, with no recomputation.
Qdrant-FineWeb-10B is Publicly Available on Hugging Face; Supernova on Github
Qdrant-FineWeb-10B is on Hugging Face. Supernova is at github.com/qdrant-labs/supernova. It generates embeddings, computes exact ground truth, and loads and queries a cluster, so you can build the same kind of benchmark on your own corpus.
About Qdrant
Qdrant is a high-performance, composable vector search engine built in Rust for production-grade semantic, hybrid, and agentic workloads. Engineers combine retrieval primitives (dense and sparse vectors, metadata filters, multi-vector representations, and custom scoring) with explicit control over ranking, indexing, latency, and relevance trade-offs. Qdrant delivers predictable, low-tail latency across cloud, hybrid, on-premises, and edge deployments.
Qdrant has more than 34,000 GitHub stars, more than 250 million downloads across all packages, and a Discord community of >8,000 members. Customers include Canva, HubSpot, Tripadvisor, Bosch, Roche, and Deutsche Telekom. The company raised a $50M Series B in March 2026, led by AVP.
View source version on businesswire.com: https://www.businesswire.com/news/home/20260901335719/en/
Media gallery
