Item | Capability and Limit |
File encoding | UTF8 and GBK encoding formats are supported. Note: The GBK encoding format requires LogListener 2.6.2 or later. |
Soft link | Supported. |
Size of a single log | The size of a single log line is limited to 512 KB. If a log exceeds 512 KB, it will be truncated to retain only the first 512 KB. The maximum size for each log entry in multi-line logs is restricted to 512 KB following division using the line-start regular expression. Up to 1 MB of data is read when the first line is determined. If the size of a log entry exceeds 512 KB, it will be truncated to retain only the first 512 KB. |
Regular expression | Perl-compatible regular expressions are supported. |
First log collection behavior | LogListener supports full and incremental collection policies: Full collection: Upon initial installation and startup, LogListener collects all logs that meet the specified conditions, including any unwritten files. Incremental collection: Upon initial installation and startup, LogListener collects existing files starting from the most recent position. Note: Both incremental collection and full collection require LogListener 2.6.2 or later. |
Log file rotation | Supported. (It is recommended that the file name after rotation should not be overwritten by the collection wildcard path.) Note: If no new log entries are written to a rotated log file within 60 seconds after rotation, it will not be collected again. Otherwise, the collection of the rotated log file will continue. |
Collection behavior when log parsing fails | We recommend you enable the feature of uploading parsing-failure logs. If the feature is enabled, parsing-failure logs will be uploaded to the preset index in the format of full text in a single line. Otherwise, the logs will be discarded. |
File opening | LogListener opens files for reading when collection begins and closes them immediately after collection. |
Filter rule. | Only one filter rule can be configured for both single-line and multi-line full-text collection operations. |
| Up to five filter rules can be configured for delimiter, regular expression, and JSON parsing modes. |
Key field length | The maximum length for a Key field is 128 bytes for both delimiter and regular expression parsing modes. |
Number of key-value pairs | In all parsing modes, the maximum number of key-value pairs that can be extracted is 1,000. Any excess will be discarded. |
LogListener maximum connections | It is limited to 1,024. |
Memory usage | The memory usage of LogListener can be up to 50 MB under normal conditions. |
CPU usage | Under a log throughput of 5 MB/s, the total CPU usage for all LogListener service processes should not exceed 20% per core. |
Collection configuration activation delay | There will be a 1-minute delay before the collection configuration takes effect after it is updated. |
Log file | It is recommended that the size of the log file targeted by LogListener for collection be less than 500 MB. |
Item | Capability and Limit |
Checkpoint storage location | The default storage path is located in the data directory within the LogListener installation directory. To change this location, you can modify checkpoint_cache_file in the LogListener configuration file with reference to Configuring LogListener. |
Checkpoint storage policy | LogListener stores two copies of the checkpoint metadata: One copy records the information only of checkpoints where upload is completed, and the information is persisted to a disk in real time. The other copy records the information of checkpoints where upload is completed and checkpoints where upload is not completed, and the information is periodically persisted to a disk. Persistence takes precedence when the program exits. |
Item | Capability and Limit |
Default CPU core limit | LogListener is designed to operate on a single core only. |
Default CPU resource limits | LogListener does not impose CPU resource limits by default. To limit it, refer to Configuring LogListener and modify the cpu_usage_thres parameter in the LogListener configuration file to set the maximum CPU utilization. Under the current LogListener implementation architecture, if no CPU resource limit is applied, the maximum single-core CPU utilization it can achieve is approximately 110% (with business threads consuming up to 100% and management threads around 10%). |
Default thread count limit | LogListener 2.x.x supports single-threaded operation only. LogListener 3.x.x supports multi-threading capability. |
Default memory resource limits | LogListener has a default memory threshold of 2 GB. To limit it, refer to Configuring LogListener and modify the
max_mem parameter in the LogListener configuration file to set the maximum memory usage. It is recommended to set it to no less than 300 MB. |
Default bandwidth resource limits | LogListener does not impose bandwidth limits by default. To limit it, you can restrict the network bandwidth used by the program by modifying max_send_rate in the LogListener configuration file with reference to Configuring LogListener. |
Resource overrun handling policy | If LogListener consumes CPU and memory resources beyond the maximum limit for more than five minutes, the collection program will automatically restart. |
Log compression | The collected logs are uploaded in a compressed format by default. If you do not require compression, see Configuring LogListener to modify request_compression in the LogListener configuration file to disable compressed uploads. |
Number of monitored directories | The default maximum number of monitored directories is 5,000. Exceeding this limit may lead to collection failures. |
Number of monitored files | The recommended maximum number of monitored files is 10,000. If this limit is exceeded, log collection may fail. |
Maximum number of monitored events | In polling mode, the recommended maximum number of monitored events is 10,000. Exceeding this limit may lead to collection failures. |
Item | Capability and Limit |
Network error handling | Except exceptions requiring special handling (such as log topic deletion), all other errors (such as network exceptions, timeout, frequency control, and overdue payment) will be retried. |
Maximum retry timeout duration | If data fails to be sent after retrying for more than 1 hour, the system discards the data. The default behavior is to retry at an interval, and the retry interval becomes longer and longer until the maximum retry timeout duration is exceeded. |
Maximum number of retries | The maximum number of retries can be set in the If the maximum number of retries is set, the system retries until the maximum number of retries is exceeded, and then discards the data. |
Line break missing | During collection, if a single log entry lacks a line break at its end, the collector waits for five minutes. If no line break is detected within five minutes, it automatically appends a line break and continues collecting. |
Item | Capability and Limit |
Log upload policy | LogListener automatically aggregates and uploads logs from the same file when any of the following conditions are met: 10,000 log entries, a total logset size of 1 MB, or a log collection time exceeding three seconds. Once any of these conditions is fulfilled, the upload process is initiated. |
File collection handling policy | A single target file (any file matching a collection path) can only be uploaded to one log topic. Multiple collection paths cannot point to the same file. To upload a single file to multiple log topics, create multiple soft links to that file, and configure each log topic to collect from its own soft link. |
Log collection delay | In the case of real-time collection, data collection, transmission, and storage to disks will be completed within 1 minute, and the logs are retrievable in the console. If the log volume is large, or LogListener can use only a limited amount of resources, there will be some collection delay. |
Item | Capability and Limit |
Maximum number of collection configurations | The maximum number of collection configurations that can be associated with one log topic is 100. |
Item | Capability and Limit |
Machine group logic | At present, machine groups are divided into the following two categories, and their usage methods are independent of and incompatible with each other. If the two usage methods are used together, LogListener will not be able to pull the correct collection configuration, resulting in no collection. IP machine group: a machine IP must be manually added to the machine group in the console, and the Label machine group: the machine group label is set in the console, and the |
Relationships between machine groups and log topics | A single log topic can be bound to multiple machine groups. A single machine group can be bound to multiple log topics. |
Relationships between machine groups and machines | A single machine can be added to multiple machine groups. For an IP machine group, there is no limit to the number of machine groups that a machine can join. For a label machine group, the maximum number of machine groups that a machine can join is 20. |
Limits on the labels of label machine groups | The label for a Tag machine group is currently limited to 32 characters. Up to 20 labels can be configured for a single label machine group. |
Item | Capability and Limit |
Collection blocklist | This feature is used to specify the content to ignore in the collection path. Currently, collection blocklists support two modes: FILE mode: Specify the files to ignore in the collection path. Wildcard filtering is supported. PATH mode: Specify the subdirectories to ignore in the collection path. Wildcard filtering is supported. Note: The FILE and PATH modes can be used together. The collection blocklist excludes paths under the collection path. Therefore, in both file name mode and directory mode, the specified path should be a subset of the collection path. |
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback