This document introduces basic implementation methods for pagination search. For more details on pagination search, you can refer to the related document. Pagination Search (Real-Time)
search_after is an efficient, real-time deep pagination solution provided by ES. Unlike traditional from+size pagination, whose performance drops sharply when processing large amounts of data, and unlike the Scroll API primarily used for offline exporting, search_after uses the sort values of the last result from the previous page as a "cursor" to fetch the next page, enabling high-performance deep pagination.
Principle:
On the first query, you should specify one or more unique or high-cardinality fields for sorting (such as _id).
From the return results, capture the sort values of the last document.
On the next query, pass these sort values as the search_after parameter, and ES will start searching directly after that "cursor."
Strengths:
High performance: Avoids re-sorting and re-traversing the first N results as in from-based pagination. Query time remains essentially constant regardless of page depth.
Real-time: Queries are always based on the latest index data, without the need to maintain a data snapshot context as in scroll.
Resource-friendly: No long-term resource usage on the server side. Resources are released after the query completes.
Suitable for deep pagination: Standard method for searching data beyond tens of thousands of pages.
In addition, the pagination results in the following example may be unstable. To ensure consistency and stability throughout the pagination process, search_after should be used together with the Point-in-Time (PIT) API. PIT creates a snapshot of a data view, keeping the index status unchanged during pagination and preventing disordered or missing results caused by data changes. For details, you can refer to the related document.
GET /book-index/_search
{
"knn": {
"field": "title_vector",
"query_vector": [0.1, 0.2, 0.3 ...],
"k": 2,
"num_candidates": 2
},
"sort": [
{"id": "asc"},
{"price": "asc"}
]
}
GET /book-index/_search
{
"knn": {
"field": "title_vector",
"query_vector": [0.1, 0.2, 0.3],
"k": 2,
"num_candidates": 2
},
"search_after": ["1002","20"],
"sort": [
{"id": "asc"},
{"price": "desc"}
]
}
Pagination Search (Offline)
The Scroll API is a deep pagination mechanism designed by ES for searching large volumes of data in a single pass (such as full exporting and batch offline processing). Its core concept is to create a data snapshot and then fetch data in batches using a scrollable "cursor" (scroll_id) until all results are traversed.
Core features and scenarios:
Non-real-time: Scroll creates a snapshot of the index on the first query, and all subsequent pages are based on this snapshot. Data changes during this period do not affect the scroll results.
Sequential traversal: Primarily used for sequential traversal from start to end. Jumping to arbitrary pages is not supported.
Resource usage: The search context needs to be maintained in the cluster until the timeout, consuming memory and compute resources.
Typical scenarios: Data migration, full backups, offline analysis, and tasks that need to process all or a large number of matching documents in an index.
The following is a complete example of the scroll pagination process.
Step 1: Initializing the Scroll Query
POST book-index/_search?scroll=1m
{
"size": 20,
"knn": {
"field": "title_vector",
"query_vector": [0.1, 0.2, 0.3],
"k": 100,
"num_candidates": 500
}
}
scroll=5m: Sets the search context's time to live to 5 minutes. This time should be sufficient for you to process one batch of data and fetch the next.
size: Specifies the number of documents returned per scroll (the batch size).
Response: In addition to returning the first batch of hits, the response body also contains a _scroll_id, which is key for subsequent pagination.
Step 2: Using the scroll_id to Fetch Subsequent Batches
Use the _scroll_id returned from the previous step to fetch the next batch of results.
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "XXXXXXXXXX"
}
Each call returns the next batch of size documents and a new _scroll_id (use the latest one even if it appears identical).
Repeat this step until the returned hits array is empty.
Step 3: Clearing the Scroll Context (Very Important)
After processing is complete, you should proactively release resources.
DELETE /_search/scroll
{
"scroll_id": "XXXXXXXXXX"
}