Tencent Cloud Elasticsearch Service (ES) is a distributed multi-node cluster, where each node consists of computing and storage resources. Based on our practical operational experience, we provide configuration recommendations for common ES usage scenarios to help you choose the appropriate configuration according to your business needs. These recommendations can serve as a useful reference, but the best approach is to gradually explore and optimize based on your actual business usage. Leveraging the auto scaling mechanism provided by Tencent Cloud ES, you can scale out the cluster and adjust its specifications at any time as your business grows or when performance bottlenecks arise.
I. Computing/Storage Resource Assessment
Given the differences in configuration estimation between the traditional scenario (such as text search and log analysis) and the vector search scenario, the two scenarios are described separately as follows:
Scenario 1: Text Search and Log Analysis
1. Storage Capacity Assessment
Key factors affecting the storage capacity of Tencent Cloud ES are as follows:
Number of replicas: Replicas can improve data reliability but also increase storage costs. The default and recommended number of replicas is 1. For certain scenarios where the data loss caused by exceptions is tolerable, you can consider setting the number of replicas to 0.
Data expansion: In addition to the original data, ES needs to store indexes, columnar data, and so on. The data size generally expands by 10% after technologies such as encoding and compression are applied.
Internal task overhead: ES occupies approximately 20% of the disk space for segment merging, ES Translog, logs, and other purposes.
OS reserve: Linux reserves 5% of the disk space for root users by default for handling key processes, restoring the system, preventing disk defragmentation disk fragmentation issues, and so on.
Therefore, the actual space occupied by data in ES can be estimated with the following formula:
Actual space = Source data x (1 + Number of replicas) x (1 + Data expansion) / (1 – Internal task overhead) / (1 – OS reserve)
≈ Source data x (1 + Number of replicas) x 1.45
To ensure stable service operation, it is recommended to reserve at least 15% of storage space. Therefore, the recommended storage capacity to apply for is:
Storage capacity = Source data x (1 + Number of replicas) x 1.45 x (1 + Reserve space)
≈ Source Data × (1 + Number of Replicas) × 1.67
2. Computing Resource Assessment
ES computing resources are primarily consumed during write and query processes. The complexity of writes and queries and their proportion vary by business scenario, making computing resources more difficult to assess compared to storage resources. In most cases, however, storage resources become the bottleneck first. Therefore, it is recommended that you assess the storage resource requirements first. Then, you can refer to Tencent Cloud ES Node Types to initially select computing resources, and confirm whether the computing resources are sufficient during the testing process. The following introduces some experience in computing resource assessment for several common usage scenarios.
Log scenario: It is a typical write-heavy, read-light scenario where computing resources are primarily consumed during the write process. Our experience in this scenario is that resources with 2 cores and 8 GB of memory can support a maximum write capacity of 5,000 writes/s. Note that this may vary across different business scenarios. Since instance performance scales linearly with the total amount of computing resources, you can estimate the write capacity based on the total instance resources. For example, resources with 8 cores and 32 GB of memory can support a write capacity of 20,000 writes/s.
Structured data scenarios such as metrics and APM: They are also write-heavy, read-light scenarios, but they consume fewer computing resources compared to the log scenario. Resources with 2 cores and 8 GB of memory can generally support a write capacity of 10,000 writes/s. You can refer to the linear scaling method used in the log scenario to assess the actual write capacity of instances with different specifications.
Text search scenario: This is a read-heavy, write-light scenario where computing resources are primarily consumed during the query process. As the query complexity varies significantly by usage scenario, computing resource requirements are the most difficult to assess. It is recommended that you initially select computing resources based on storage resources, and then verify and adjust them during the testing process.
Scenario 2: Vector Search
1. Computing Resource Assessment
The ES vector scenario supports hybrid vector-text search (for Tencent Cloud ES, we recommend using version 8.16 or later). Since the vector scenario requires loading a large number of vectors into memory to ensure search performance, the core of vector scenario assessment is memory. ES primarily recommends the Hierarchical Navigable Small World (HNSW) graph index. This algorithm is widely used and offers fast query speed and high search accuracy. The following introduces the memory assessment method for the HNSW index:
Step 1: Assessing Vector Memory (Off-Heap Memory)
Vector memory = (Size per vector x Vector dimensions) x Number of vectors x (1 + Number of replicas) x 1.1
Note:
The coefficient 1.1 accounts for reserving 10% as temporary space for the graph index.
If quantization (typically considered when the number of vectors exceeds 100 million) is adopted in vector search, such as int8_hnsw and bbq_hnsw, memory usage can be reduced proportionally as follows:
Vector memory = (Size per vector x Quantization compression ratio x Vector dimensions + Additional memory for quantization) x Number of vectors x (1 + Number of replicas) x 1.1
Note:
Quantization compression ratio: Memory size is proportionally reduced, calculated as "Size per vector x Quantization compression ratio". For example, if the vector type is float, the compression ratio is 1/4 for int8 quantization and 1/32 for bbq quantization.
Additional memory for quantization: When the vector type is float or bfloat16, set the value to 14 for bbq_hnsw. This overhead can be ignored for other quantization methods.
Step 2: Assessing Total Memory (Off-Heap Memory + Heap Memory)
Total memory = Vector memory + Heap memory
Note:
For heap memory, estimate it at a 1:2 ratio when the single-node memory is less than or equal to 32 GB. A fixed value of 32 GB can be adopted for single-node memory exceeding 32 GB.
Example (without quantization): Assume there are 100 million 1,024-dimension float vectors (without quantization), with an index configuration of one primary shard and one replica shard:
Vector memory = (4 bytes x 1,024 dimensions) x 100,000,000 vectors x (1 + 1) x 1.1 = 880 GB
Total memory = 880 GB + 32 GB x Number of data nodes ≤ 128 GB x Number of data nodes.
(Note: High-memory nodes such as those with 32 cores and 128 GB are preferred, with 32 GB allocated for heap memory per node.)
Calculation result: Number of data nodes ≥ 9.2
Recommended configuration: 32 cores and 128 GB x 10 data nodes
Example (with quantization): Using the preceding example, assume that int8 quantization is adopted:
Vector memory = (4 bytes x 1/4 x 1,024 dimensions) x 100,000,000 vectors x (1 + 1) x 1.1 = 220 GB
Total memory = 220 GB + 32 GB x Number of data nodes ≤ 128 GB x Number of data nodes
(Note: High-memory nodes such as those with 32 cores and 128 GB are preferred, with 32 GB allocated for heap memory per node.)
Calculation result: Number of data nodes ≥ 2.3
Recommended configuration: 32 cores and 128 GB x 3 data nodes
2. Storage Capacity Assessment
Vector storage can be estimated using the same method applied for text search, with a space reservation factor of 1.67.
Vector storage space = (Size per vector x Vector dimensions) x Number of vectors x (1 + Number of replicas) x 1.67
If quantization is adopted in vector search, such as int8_hnsw and bbq_hnsw, the vector storage will increase by "Size per vector x Quantization compression ratio". This is because quantization reduces memory usage but generates an additional copy of the quantized data for storage. Therefore, the storage space for this portion of data should also be considered as follows:
Vector storage space = (Size per vector x (1 + Quantization compression ratio) x Vector dimensions) x Number of vectors x (1 + Number of replicas) x 1.67
For hybrid vector-text search:
Total storage space = Vector storage space + Text storage space
Note: The text storage space is calculated using the method described in the preceding text search scenario.
If row storage pruning is enabled, it will save 70% of storage:
Total storage space = (Vector storage space + Text storage space) x (1 – 70%).
II. Instance Type Selection and Testing
After you complete the storage and computing resource assessment, you can refer to Tencent Cloud ES Node Types to determine node specifications and the number of nodes. Common recommendations are as follows: 1. Data Nodes
We recommend that you select at least three nodes to prevent split-brain issues in your ES instance and ensure its high node failure tolerance.
Note
Split-brain: Two nodes simultaneously believe they are the only active server, resulting in resource contention.
Prioritize high-specification nodes instead of deploying a large number of low-specification nodes. This approach provides better performance and stability for large instances. For example, if your workload requires 40 cores, 160 GB of memory, and 5 TB of storage, it is recommended that you select an instance with five nodes, each with 8 cores, 32 GB of memory, and 1 TB of storage. Similarly, when you need to scale up your instance, it is recommended that you prioritize vertical scaling by upgrading node specifications to 8 cores and 32 GB or 16 cores and 64 GB. After that, you can consider horizontal scaling by increasing the number of nodes.
After the initial instance type selection, you can conduct tests with real data and observe monitoring information such as CPU utilization, write metrics (performance and rejection rate), and query metrics (QPS and rejection rate) to further verify that the instance type is appropriate. In addition, it is recommended that you configure alarms for the above monitoring information to promptly identify issues such as resource insufficiency in actual production environments.
2. Dedicated Primary Nodes
If the number of data nodes in an instance exceeds a certain threshold, it is recommended that you enable dedicated primary nodes. The recommended configurations are as follows:
More than 10 data nodes: 4 cores and 16 GB of memory.
More than 30 data nodes: 8 cores and 32 GB of memory.
More than 50 data nodes: 16 cores and 64 GB of memory.
Note: For scenarios where a cluster has a large number of indexes or shards, or where data changes frequently, you should appropriately upgrade the specifications of dedicated primary nodes.
3. Coordinating Nodes
For scenarios with heavy write or query loads, it is recommended that you enable coordinating nodes to offload pressure from data nodes. The recommendations are as follows:
Deploy at least two coordinating nodes. A CPU-to-memory ratio of 1:4 or 1:8 is recommended.
The recommended ratio of coordinating nodes to data nodes is approximately 1:5. Coordinating node specifications should be equal to or higher than those of data nodes, or adjusted based on actual business performance.
For example, for a cluster with 10 data nodes, each with 8 cores and 32 GB of memory, it is recommended that you configure two coordinating nodes, each with 8 cores and 32 GB of memory.
III. Assessment of Number of Shards
Each ES index is split into multiple shards, and data is distributed among different shards using a hash algorithm. The number of shards per index affects read/write performance and failure recovery speed, and it is usually difficult to change later. Therefore, you should plan your index shard settings in advance based on future business development (such as growth over the next one to two years). The following are common recommendations:
The recommended size for a single shard is 20 GB to 50 GB, which can be used to initially determine the number of shards for your index. Shards should neither be too large nor too small. Excessively large shards may slow down ES failure recovery; excessively small shards will result in a massive number of shards. Since each shard consumes a certain amount of CPU and memory resources, this can lead to issues such as degraded read/write performance and insufficient memory.
The number of shards should be as close as possible to the number of data nodes. If you have a large number of shards, it is recommended that you set the number of shards to an integer multiple of the number of data nodes to facilitate even distribution of shards across the data nodes.
The total number of shards for all indexes on a single node should not exceed 1,000. The total number of shards in the cluster should be kept below 30,000.
During the testing phase, you can appropriately adjust the number of shards based on the actual size of each index and expected future growth.
For log and metric scenarios, it is recommended that you leverage ES's built-in Rollover Index feature. This feature continuously creates new indexes. When you find that the shard size is inappropriate, you can use this feature to promptly adjust the number of shards. For example, assume an instance has five data nodes, and the current index size is 150 GB, which is expected to grow by 50% over the next year. If each shard is set to 30 GB, approximately seven shards are required according to the following formula: 150 GB x (1 + 50%)/30. However, with seven shards distributed across five data nodes, two nodes will each host 2/7 of the data, resulting in a less balanced workload across the nodes. For this reason, the number of shards is adjusted to 10.