Introduction to Vector Search
1. What Is Vector Search
Vector search represents a paradigm shift in search technology. It converts unstructured data, such as text and images, into high-dimensional vectors using deep learning models. This process enables semantically similar content to be positioned close to each other in the vector space. When a user searches for "smartphone", the system matches not only literal keywords but also semantically relevant content such as "iPhone" and "Android phones".
2. Vector Search Scenarios
Vector search is not just a technological upgrade but also a cornerstone for building intelligent applications. Its primary scenarios include:
Semantic search: It overcomes the limitations of traditional keyword-based search and searches for semantically similar content.
Accurate recommendation: It recommends similar products or articles based on user preferences or the semantics of the currently viewed content.
Cross-modal search: It enables cross-modal search experiences, such as text-to-image search, image-to-image search, and audio/video search.
Intelligent Q&A: As the core of the Retrieval-Augmented Generation (RAG) architecture, it provides accurate and relevant knowledge sources for large language models.
With these capabilities, vector search in Tencent Cloud Elasticsearch Service (ES) delivers a set of core tools that transform data into deep semantic insights, helping you build more intelligent, user-intent-aware modern AI-powered search applications.
Introduction to ES Vector Search
As a leader in the search domain, ES has evolved from full-text search into an AI-powered search engine that supports hybrid text-vector search, and natively supports vector search technology at the Lucene and ES kernel levels.
1. Vector Search Algorithms
ES uses k-nearest neighbor (kNN) search to implement vector search. It searches for results based on semantics rather than exact keyword matching. The fundamental principle is to compare the query vector from the search request with vectors in the target vector collection. It evaluates the distance between them using similarity measurement methods such as cosine similarity or L2 norm, where a shorter distance indicates higher similarity. This process identifies the k vectors closest to the query vector. ES supports two kNN search methods:
|
Exact brute-force kNN | Small datasets or scenarios requiring precise scoring, typically with a vector scale under 100,000. | This is a brute-force scanning method that compares the query vector against every vector in the target vector collection and identifies all eligible nearest neighbor vectors. Although it provides high accuracy, it is inefficient and consumes significant resources and time. It uses script_score to execute queries with vector functions. |
Approximate nearest neighbor (ANN) | Most production workloads, supporting billions to tens of billions of vectors. | Commonly referred to as ANN. To address the limitations of brute-force kNN, the ANN algorithm requires the target vector collection to be pre-organized through indexing. The algorithm focuses on finding the top k similarity results with higher efficiency. The throughput, memory usage, and search accuracy of ANN search may vary depending on the selected index type. You need to strike a balance between search performance and accuracy. |
2. Vector Indexing Algorithms
ES primarily supports the Hierarchical Navigable Small World (HNSW) algorithm. This is an efficient graph indexing algorithm for high-dimensional vector similarity search, renowned for its excellent search speed and recall. HNSW combines the hierarchical concept of skip lists with the "shortcut" characteristic of small-world networks. It constructs a multi-layer graph structure. The top layer contains a small number of nodes with long-distance connections for fast navigation, while the bottom layer contains all nodes with short-distance connections for fine-grained search. This design achieves search complexity at an approximately logarithmic level. In the HNSW search process, a search starts from the top layer, greedily moves to the nearest neighbor node, and then descends to the next layer to continue the search after finding a local minimum value. This process repeats until the bottom layer is reached. This method effectively mitigates the local optimum trap common in traditional graph search.
ES added support for the DiskBBQ algorithm in version 9.2. We will continue rolling out support for additional indexing algorithms in upcoming releases. Stay tuned.
3. Vector Similarity Algorithms
ES supports the following similarity algorithms, which you can specify during field configuration:
|
l2_norm (Euclidean distance (L2 distance)) | Calculates the absolute spatial distance, where a smaller value indicates greater similarity. |
cosine (cosine similarity) | Focuses on vector direction and is suitable for semantic search. |
dot_product (vector dot product) | Considers both direction and magnitude. Vectors should first be normalized to unit vectors (with a length of 1). |
max_inner_product (maximum inner product) | Applies to unnormalized vectors where length carries meaningful information. |
4. Vector Search Methods
ES supports both text and vector search and seamlessly supports flexible combinations of the two methods:
Vector search: It uses a dedicated kNN query API to directly perform efficient approximate nearest neighbor search.
Hybrid search: It enables you to easily apply business filter conditions (such as category and price range) before/after vector search, seamlessly integrating vector search with text search.
Intelligent fusion: It enables you to intelligently rerank the results from vector search and full-text search (BM25) using algorithms such as Reciprocal Rank Fusion (RRF), balancing semantic relevance and keyword precision.
5. Seamless Integration with AI Applications
Tencent Cloud ES provides comprehensive capabilities to integrate with AI applications, enabling enterprises to efficiently implement semantic search, multimodal search, and RAG applications:
AI application end-to-end model inference capabilities
Out-of-the-box atomic services: It provides online services such as document parsing, chunking, embedding, reranking, and large language model (LLM).
Self-deployment using machine learning nodes: You can add machine learning nodes to your ES cluster, which are dedicated to model inference. You can deploy built-in models on these nodes, or upload custom models to them.
GPU-accelerated inference performance: Through in-house development, Tencent Cloud ES is the first in the world to support GPU-based inference, including NVIDIA GPUs and domestic GPUs such as Zixiao.
One-stop RAG construction experience: The Tencent Cloud ES console provides a one-stop RAG construction playground, which includes:
Document upload: It enables you to upload documents online, which are automatically parsed, chunked, embedded, and stored.
Online Q&A: It offers visualized online Q&A experience. You can flexibly optimize Q&A effectiveness by adjusting configurations. It supports testing and comparison of multiple RAG configurations.
Effectiveness assessment: It can generate a comprehensive assessment report for RAG, facilitating problem identification and targeted optimization.
Application deployment: It supports code export and the online deployment of RAG services (coming soon).
6. Advantages of Tencent Cloud ES Based on Self-Developed Optimizations
Tencent Cloud has implemented in-depth self-developed optimizations on the open-source ES, covering aspects such as text search, vector search, hybrid search, fusion ranking, inference performance, and cost optimization. This makes Tencent Cloud ES a reliable, enterprise-grade technology foundation for the AI-powered search era.
Text search: Traditional vector databases only support simple scalar filtering, while ES supports capabilities such as full-text search, keyword matching, phrase query, fuzzy match, and aggregation analysis. It can also effectively improve recall through nested fields, function_score, and other methods. In the field of text search, Tencent Cloud ES has accumulated long-term self-developed technologies, such as query pruning, adaptive routing, and core operator optimization. These technologies improve write performance by 100% and query performance by 200% to 1000%.
Vector search: It supports HNSW graph indexing and features self-developed Multi-Path query parallelization. Compared to the traditional HNSW algorithm, it achieves up to four times the acceleration while maintaining the same recall.
Hybrid search: It supports intelligent path analysis based on the cost-based optimizer (CBO), improving hybrid search query performance by 30% to 80%. The optimized built-in RRF reciprocal ranking algorithm brings a 10%–20% increase in recall.
Inference service: Building upon productized machine learning nodes, it is compatible with GPUs such as NVIDIA and Zixiao, and supports the deployment of models such as Embedding, Rerank, NER, and Text Classification.
Model capabilities: It provides the WeChat-developed image-text search model WeClip as well as text embedding models such as KaLM, Conan, and BGE series. Users can also leverage their own enterprise models through inference APIs.