Overview
DLC Native Table is a user-friendly table format with high performance based on the Iceberg lake format. It simplifies operations, making it easy for users to perform comprehensive data exploration and build applications like Lakehouse. When using the DLC Native Table for the first time, users need to perform the following 5 main steps:
1. Enable DLC managed storage.
2. Purchase an engine.
3. Create a database and table. Select to create a native table according to the usage scenario and carry optimization parameters.
4. Configure data optimization. Select an independent optimization engine and configure optimization options according to the table type.
5. Import data into the DLC Native Table. Data Lake Compute (DLC) supports various data write modes, including insert into/merge into/upsert, and supports multiple import methods, including spark/presto/flink/inlong/oceanus.
Native Table Types
DLC Native Table is the recommended format of DLC Lakehouse. It provides two types of table formats: Iceberg and TC-Iceberg. Their characteristics and usage scenarios are as follows:
|
Iceberg | Uses the Apache Iceberg table format, including ACID transactions, hidden partitions, data version control, and other features, and provides two table version selections: V1/V2. | 1. Traditional batch processing scenario (Append): The Iceberg V1 table version is used. In this scenario, tables only support Append, Overwrite, and Merge into write modes. 2. Real-time writing scenario (Upsert): The Iceberg V2 table version is used. Compared with the Append scenario, this scenario additionally supports the Upsert write mode and supports real-time writing modes such as InLong, Oceanus, or self-managed Flink. |
TC-Iceberg | Tencent Cloud's batch-stream integrated table format, expanded based on Iceberg. It is compatible with and includes all the strengths of Apache Iceberg, and provides performance enhancement and near-real-time lakehouse construction capabilities. | The current version only supports primary key update scenarios. It can meet customer business requirements in scenarios with real-time writing (Upsert) and near-real-time lakehouse construction (CDC streaming consumption). |
Advantages of Native Tables
DLC Native Table adopts the managed data storage mode. After users use an Iceberg table, DLC will uniformly manage the metadata and data files of the Iceberg table. This brings the following benefits to users:
Higher Data Security
Iceberg table data is divided into two parts: metadata and data. Once any of these files are damaged, it can cause query exceptions for the entire table (compared with Hive, it might be that damaged file data cannot be queried). Storing data in DLC can reduce the possibility of users damaging certain files without understanding Iceberg.
Enhanced Storage Performance
DLC managed storage uses chdfs as storage by default. Compared with ordinary Cloud Object Storage (COS), the performance is greatly enhanced. Based on DLC's Smart Optimizer service for managed tables, regularly merging small files and purging orphan snapshots can further improve query performance.
Reduced Ops Costs
The COS service is enabled automatically. Storage resource assignment is automatically performed. Built-in table data optimization and lifecycle management features are provided, significantly reducing Ops costs.