By running inference services on machine learning nodes, you can gain high-performance, low-latency model inference capabilities and benefit from full integration with the Elasticsearch ecosystem.
Key Strengths
GPU acceleration: Tencent Cloud Elasticsearch Service (ES) has independently developed support for NVIDIA and domestic GPUs (for example, support for GPU acceleration), improving inference efficiency by 30 times.
Ecosystem integration: It seamlessly integrates with the Elasticsearch ecosystem for data collection and vector search.
Steps
1. Creating a Machine Learning Node
If you have not created an ES cluster, you can enable Machine Learning Node when creating a cluster. If you have already created an ES cluster, you can enable Machine Learning Node through configuration adjustment.
Note that you need to reserve sufficient memory space for different models. For example:
The .multilingual-e5-small model requires at least 8 GB of memory.
The bge-base-zh-v1.5 model requires at least 16 GB of memory.
The bge-m3 model requires at least 32 GB of memory.
Selecting a GPU delivers higher inference efficiency and lower inference costs.
2. Deploying a Model Inference Service
Go to the cluster details page and switch to Model Management:
Click Add Model to select one from the preset models:
You can also upload a custom model:
The status after installation is as follows:
After the model is installed, click Deploy to deploy the model:
Switch to Deployment List to view the model deployment status:
3. Testing the Inference Service
You can test the service in Kibana Dev Tools as follows:
POST _ml/trained_models/bge-base-zh/_infer
{
"docs": [{ "text_field": "Hello" }]
}
4. Creating an Inference Endpoint (Optional)
If you call the model service through an ingest pipeline within ES, you do not need to create an inference endpoint. If you want to call the inference service (for example, semantic search) through the inference API, you can refer to the following method to create an inference endpoint.
PUT _inference/text_embedding/bge-base-zh-service
{
"service": "elasticsearch",
"service_settings": {
"model_id": "bge-base-zh",
"num_allocations": 1,
"num_threads": 1
}
}
After creating an inference endpoint, you can call _inference to test it:
POST _inference/text_embedding/bge-base-zh-service/_infer
{
"input": "Hello"
}
Monitoring and Ops of Machine Learning Nodes
You can view the metrics of machine learning nodes through node monitoring and perform machine learning node scaling promptly based on the load.