1. What Is Quantization
When ES vector search scales to hundreds of millions or even billions of vectors, two challenges need to be addressed: high memory usage and slow computation speed. ES's vector quantization is an important optimization policy at this stage. It effectively combines the speed advantage of quantization with the precision advantage of raw vectors, striking a balance between performance and accuracy. Quantization is essentially a lossy compression technique. The core idea is to convert high-precision vector data (typically 32-bit floating point, float) into a low-precision representation (such as 8-bit integer, int8). It can significantly reduce memory usage and improve query speed with minimal recall loss.
2. Changes in Memory and Storage Usage with Quantization
If you specify quantization in the index mapping settings, ES generates both the original precision data storage (assuming float32) and an additional quantized precision data storage for the float32 data during index creation. Therefore, for float32 vectors, int8, int4, and bbq quantization can reduce memory usage by 4x, 8x, and 32x respectively, but will also reduce vector precision and increase disk space usage (by 25%, 12.5%, and 3.125%, respectively). For example, when int8 quantization is used on 40 GB of float vectors, the quantized vectors will consume an additional 10 GB of disk space, bringing the total disk usage to 50 GB, but the memory usage will be reduced to 10 GB.
3. Coarse Ranking and Oversampling with Quantization
To address the precision loss caused by quantization, ES vector search is divided into two phases: approximate coarse ranking and exact fine ranking:
Coarse ranking phase: ES uses the quantized query vector to search on the Hierarchical Navigable Small World (HNSW) graph index built on quantized data. After completing the search on each shard, it obtains a list of num_candidates document IDs with approximate scores.
Fine ranking phase: By configuring rescore_vector to specify an oversample factor, ES reads the raw float32 vectors from disk for the top k*oversample documents out of the num_candidates documents returned by each shard. It then performs high-precision similarity computation between the original query vector and these original document vectors, selecting the top k results.
Clearly, the oversampling mechanism combines the performance and memory advantages of approximate search using quantized vectors with the accuracy of rescoring the best candidates using raw vectors. In practice, int8 typically does not require oversampling; int4 can achieve higher precision and recall with 1.5x to 2x oversampling; bbq typically requires oversampling, and 3x to 5x oversampling is usually sufficient.
4. Recall Assessment with Quantization
Based on our practical experience, if the recall without quantization is 99%, int8 quantization yields a recall of about 96–97%, and bbq quantization yields about 90–94%. You should be aware that the recall with quantization has a high correlation with data scale. The larger the dataset, the smaller the recall loss from quantization, and vice versa. Therefore, you should validate the final recall results on as large a dataset as possible.
5. Example
PUT /my-quantized-index
{
"mappings": {
"properties": {
"title": {
"type": "text"
},
"title_vector": {
"type": "dense_vector",
"dims": 768,
"index": true,
"similarity": "cosine",
"index_options": {
"type": "int8_hnsw",
"m": 16,
"ef_construction": 100
}
}
}
}
}
GET my-quantized-index/_search
{
"knn": {
"field": "title_vector",
"query_vector": [0.15, 0.50, ..., 0.05],
"k": 10,
"num_candidates": 100,
"rescore_vector": {
"oversample": 2.0
}
}
}