Scenarios
In simple Ops scenarios, logs are usually output directly to local server files and then searched using the commonly used grep command in Linux. This method becomes difficult when business systems are complex due to log dispersion across different servers, non-intuitive command line operations, and server permission management restrictions, severely affecting Ops efficiency. It's even more difficult if you need to do some statistical analysis or monitor alarms based on logs.
This article describes how to quickly migrate local logs to CLS (Cloud Log Service) using the grep command to obtain the following advantages:
Data is stored and searched centrally, and you do not need to log in to multiple servers to query it individually, which is critical in Load Balancer, microservice, and other architectures.
With a simple click, you can quickly search logs, eliminating the need for command line operations and cumbersome server permission management.
Based on logs, perform statistical analysis to obtain key business metrics such as PV, API response time, and API error rate.
Real-time detection of abnormal logs and obtain notifications through multiple channels such as SMS, email, and WeChat.
Note:
If your logs have been collected to CLS, you can skip the log collection and index configuration steps and directly proceed to Step 3: Searching for Logs. Operation Steps
Step 1: Collecting Logs
If your server is a Tencent Cloud CVM (Cloud Virtual Machine, CVM), you can also automatically install LogListener through the console. For details, see CVM Batch Deployment of LogListener. Unlike server local logs, to facilitate subsequent log search and statistical analysis, unstructured raw logs can be converted into formatted data during collection. For example, the raw log is:
10.20.20.10 ::: [Tue Jan 22 14:49:45 CST 2019 +0800] ::: GET /online/sample HTTP/1.1 ::: 127.0.0.1 ::: 200 ::: 647 ::: 35 ::: http://127.0.0.1/
You can use the delimiter ::: to split it into eight fields and define names for each field:
IP: 10.20.20.10
bytes: 35
host: 127.0.0.1
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
For operation details, see Separator Format. In addition to using separators to split logs, CLS also supports various log splitting methods such as regular expression, JSON, and full text. For more details, see Collecting Text Logs. Step 2: Configuring Indexes
The purpose of configuring an index is to define which fields need to be searched and their data types, to facilitate subsequent log search. For most usage scenarios, you can use the auto-indexing feature to complete the configuration with one click. For details, see Configuring Indexes. Step 3: Searching for Logs
This document uses the commonly used grep command as an example to introduce how to achieve similar log search effects through CLS. For CLS search method operation steps, see Syntax Rules > Operation Steps. Assume that the raw log is as follows:
10.20.20.10 ::: [Tue Jan 22 14:49:45 CST 2019 +0800] ::: GET /online/sample HTTP/1.1 ::: 127.0.0.1 ::: 200 ::: 647 ::: 35 ::: http://127.0.0.1/
The formatted log in CLS is:
IP: 10.20.20.10
bytes: 35
host: 127.0.0.1
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
Case 1
Search for logs where the request is /online/sample.
Use the grep command:
grep "/online/sample" test.log
Use the CLS search method:
Case 2
Search for logs where the status code is not 200.
Use the grep command:
In fact, this method may exclude some logs (where 200 appears but the status is not 200). For an accurate search, you need to write a regular expression.
Use the CLS search method:
CLS also supports more flexible search methods, such as searching for logs with status codes greater than or equal to 500.
Case 3
Count the number of log entries where the status is not 200.
Use the grep command:
grep -c -v "200" test.log
Use the CLS search method:
NOT status:200 | select count(*) as errorLogCounts
Case 4
Search for logs where the status code is 200 and the request is /online/sample.
Use the grep command:
grep "200" test.log | grep "/online/sample"
Use the CLS search method:
status:200 AND request:"/online/sample"
Case 5
Search for logs where the request is /online/sample or /offline/sample.
Use the grep command:
grep -E "/online/sample|/offline/sample" test.log
Use the CLS search method:
request:"/online/sample" OR request:"/offline/sample"
Case 6
Search for logs where the request is /online/sample, but the log file is not test.log.
Use the grep command:
grep -rn "/online/sample" --exclude=test.log
Use the CLS search method:
request:"/online/sample" AND NOT __FILENAME__:"test.log"
Case 7
Search for the first 10 lines of logs where the time is [Tue Jan 22 14:49:45 CST 2019 +0800].
Use the grep command:
grep "[Tue Jan 22 14:49:45 CST 2019 +0800]" -B 10 test.log
Use the CLS search method:
time:"[Tue Jan 22 14:49:45 CST 2019 +0800]"
After searching for matching logs, use the context search feature to view surrounding log entries.