Overview
The system supports customizing the output format of log data. This includes selecting different logging styles, such as JSON Lines, CSV, or custom formats, and defining prefixes, suffixes for batches or individual logs, and delimiters between log entries or fields. The default format for both real-time and offline logs is JSON Lines. Configuration Items
The log output format for real-time logs falls into two categories: preset format and custom format.
Preset formats (JSON Lines, CSV): These are pre-packaged standard formats with all parameters locked, ready for use out of the box.
Custom format: This option allows you to explicitly select a basic format (Key-Value, Value-Only, or Template-Defined) and exercise fine-grained control over the data format for a single log push request. The control includes settings at the batch dimension (batch prefix/suffix, log record delimiter) and the single log dimension (record prefix/suffix, field delimiter).
Log Output Format
Log Output Format: The type of output format for log delivery.
JSON Lines: This format specification, also known as NDJSON, defines that each line is a valid JSON object. Within a single log entry, fields are presented as key-value pairs according to JSON syntax. CSV: This format specification, compliant with RFC 4180, defines that fields are separated by commas, each record occupies a single line, field values are arranged in a fixed order, and field names are not included. Custom: The presentation of individual log entries is determined by the basic format. After selecting Custom, you must further select a basic format.
Basic Format (Configurable only when the log output format is set to Custom): The presentation style of fields within a single log entry.
Key-Value: This format presents individual log entries as key-value pairs.
Value Only: This format presents only field values and does not include field names.
Define by Template: This option allows you to define the content of individual log entries using a custom template. The template supports referencing ${field_name} to include either preset fields or custom fields. Sub-Parameters
The configurable granular parameters under the custom output format are as follows.
Batch Dimension: This level controls the data format for a single log push request (that is, a batch).
Batch Prefix and Suffix: Users can define a prefix and a suffix for log batches. A "batch" refers to a single log push request, and each log delivery batch may contain multiple log records.
Prefix: A string added before each log delivery batch.
Suffix: A string appended after each log delivery batch.
Log Record Delimiter: A string inserted between log records as a separator.
Single Log Dimension: This level controls the data format for individual log records within a batch.
Record Prefix and Suffix: This option adds a string before and after each log record.
Prefix: A string added before each log record.
Suffix: A string appended after each log record.
Field Delimiter: A string inserted between fields within a single log record as a separator.
Note:
1. When log sampling or filtering is not enabled, one HTTP request corresponds to one log record for domain name services. 2. Custom expression fields are output in logs from top to bottom according to their defined order. To customize the field output order, switch the log output format to "Custom → Define by Template" and specify the field arrangement within the template.
3. Output Behavior for Duplicate Field Names: When multiple fields define the same keyname:
KV Format (JSON Lines, Custom Key-Value Format): Only the field with the highest priority is output.
Value-Only Format (CSV, Custom Value-Only Format): The field is output repeatedly according to the defined number of times, and its Value uniformly follows the highest priority.
Field value override priority: expression-based custom fields > existing four-category custom fields > preset fields.
Sample Configuration
The following are log samples corresponding to different log output formats.
JSON Lines
Configuration Samples
|
Log Output Format | JSON Lines |
Single Log Record Prefix | { |
Single Log Record Suffix | } |
Log Record Delimiter | \\n |
Field Delimiter | , |
Log Output Samples
{"SecurityAction":"Deny","RequestID":"14941044941971548881","RequestTime":"2024-08-12T08:12:15Z","ClientIP":"1.1.1.1"}
{"SecurityAction":"Deny","RequestID":"14941045941971548882","RequestTime":"2024-08-12T08:12:30Z","ClientIP":"2.2.2.2"}
CSV
Configuration Samples
|
Log Output Format | csv |
Log Record Delimiter | \\n |
Field Delimiter | , |
Log Output Samples
Deny,14941044941971548881,2024-08-12T08:12:15Z,1.1.1.1
Deny,14941045941971548882,2024-08-12T08:12:30Z,2.2.2.2
Custom
Key-Value Sample
Configuration Sample for JSON Array Format
|
Log Output Format | Custom |
Basic Format | Key-Value |
Batch Prefix | [ |
Batch Suffix | ] |
Single Log Record Prefix | { |
Single Log Record Suffix | } |
Log Record Delimiter | , |
Field Delimiter | , |
Log Output Sample for JSON Array Format
[
{"SecurityAction":"Deny","RequestID":"14941044941971548881","RequestTime":"2024-08-12T08:12:15Z","ClientIP":"1.1.1.1"},
{"SecurityAction":"Deny","RequestID":"14941045941971548882","RequestTime":"2024-08-12T08:12:30Z","ClientIP":"2.2.2.2"},
{"SecurityAction":"Allow","RequestID":"14941046941971548883","RequestTime":"2024-08-12T08:12:45Z","ClientIP":"3.3.3.3"}
]
Configuration Sample for Embedded JSON Object Format
|
Log Output Format | Custom |
Basic Format | Key-Value |
Batch Prefix | {"events":[ |
Batch Suffix | ]} |
Single Log Record Prefix | {"info":{ |
Single Log Record Suffix | }} |
Log Record Delimiter | , |
Field Delimiter | , |
Log Output Sample for Embedded JSON Object Format
{"events": [
{"info": {"SecurityAction":"Deny","RequestID":"14941044941971548881","RequestTime":"2024-08-12T08:12:15Z","ClientIP":"1.1.1.1"}},
{"info": {"SecurityAction":"Deny","RequestID":"14941045941971548882","RequestTime":"2024-08-12T08:12:30Z","ClientIP":"2.2.2.2"}}
]}
Value-Only Sample
Configuration Sample for CSV with Header Format
|
Log Output Format | Custom |
Basic Format | Value Only |
Batch Prefix | SecurityAction,RequestID,RequestTime,ClientIP\\n |
Log Record Delimiter | \\n |
Field Delimiter | , |
Log Output Sample for CSV with Header Format
SecurityAction,RequestID,RequestTime,ClientIP
Deny,14941044941971548881,2024-08-12T08:12:15Z,1.1.1.1
Deny,14941045941971548882,2024-08-12T08:12:30Z,2.2.2.2
Allow,14941046941971548883,2024-08-12T08:12:45Z,3.3.3.3
Configuration Sample for TSV Format
|
Log Output Format | Custom |
Basic Format | Value Only |
Log Record Delimiter | \\n |
Field Delimiter | \\t |
Log Output Sample for TSV Format
Deny 14941044941971548881 2024-08-12T08:12:15Z 1.1.1.1
Deny 14941045941971548882 2024-08-12T08:12:30Z 2.2.2.2
Allow 14941046941971548883 2024-08-12T08:12:45Z 3.3.3.3
Template Definition Example
Configuration Example
|
Log Output Format | Custom |
Basic Format | Define by template |
Log Record Delimiter | \\n |
Single Log Content Template | {"SecurityAction":"${SecurityAction}","ClientIP":"${ClientIP}","RequestID":"${RequestID}","RequestTime":"${RequestTime}","CustomField":"${custom_field_name}","Timestamp":"${timestamp}","Source":"EO"} |
Log Output Samples
{ "SecurityAction":"Deny", "ClientIP":"1.1.1.1", "RequestID":"14941044941971548881", "RequestTime":"2024-08-12T08:12:15Z", "CustomField":"my_value", "Timestamp":"1723450335", "Source":"EO" }
{ "SecurityAction":"Deny", "ClientIP":"2.2.2.2", "RequestID":"14941045941971548882", "RequestTime":"2024-08-12T08:12:30Z", "CustomField":"my_value", "Timestamp":"1723450350", "Source":"EO" }
Use Limits
The feature of log output format is constrained by the subscription plan, log type, and delivery destination, as detailed below:
Plan Edition
|
Preset Formats (JSON Lines, CSV) | ✓ | ✓ | ✓ | ✓ | ✓ |
Custom Formats (Key-Value, Value-Only) | ✓ | ✓ | ✓ | ✓ | ✓ |
Custom Format (Define by Template) | ✕ | ✕ | ✕ | ✕ | ✓ |
Log Type
L7 Access Logs and L4 Proxy Logs: Support the full capability of log output formats, including preset formats (JSON Lines, CSV) and custom formats (Key-Value, Value-Only, Define by Template).
Edge Function Execution Logs: Only the preset JSON Lines format and the custom Key-Value format are supported.
Managed Rule Logs: Configuring the log output format is not currently supported.
Log Delivery Destination
CLS and EO log analysis is only supported in the JSON Lines format and cannot be switched to CSV or custom formats.