Real-time Data Capture from TDSQL Boundless via DTS
Download
Modo Foco
Tamanho da Fonte
TDSQL Boundless supports real-time monitoring and capture of data changes in instances through DTS, converting them into message objects and pushing them to Kafka, facilitating downstream businesses to subscribe, obtain, and consume.
Feature Limits
When using Binlog to subscribe to TDSQL Boundless data changes, note the following relevant restrictions and currently unsupported scenarios.
Note:
Enabling or disabling Binlog is performed smoothly, requiring no instance restart and causing no impact on business read/write operations. The process is typically completed within 1 to 2 minutes.
Enabling or disabling Binlog is supported only for Provisioned resource instances, not for Serverless instances.
The Binlog feature is officially supported starting from kernel version V21.2.0. After Binlog is enabled, DDL operations on FUNCTIONs, PROCEDUREs, and TRIGGERs are not currently recorded to Binlog. If changes are made to these objects, you must manually synchronize them to downstream through other means. The feature limitations for historical kernel versions are as follows:
Kernel version earlier than V21.0.0: DDL operations on VIEWs are not recorded to Binlog.
Kernel version earlier than V20.0.0: After Binlog is enabled, the
CREATE TABLE ... AS SELECT ... syntax is automatically disabled. To restore it, contact technical support to disable the synchronization feature and restart the instance.HBase compatibility mode is not supported; only MySQL compatibility mode is supported.
Enabling Binlog is not currently supported for instances with cold data archiving enabled. Furthermore, the Binlog service and cold data archiving are mutually exclusive, and only one of them can be enabled.
Enabling Binlog is not currently supported for instances with database table recycle bin enabled. Furthermore, the Binlog service and the table recycle bin are mutually exclusive, and only one of them can be enabled.
Standalone download of Binlog files is not currently supported. You can subscribe to logs via the MySQL Binlog Dump protocol or CKafka. For details, refer to Real-time Data Capture from TDSQL Boundless via DTS.
Binlog logs are retained for 7 days by default. Binlog logs that exceed the retention period are automatically cleaned up and cannot be recovered.
Enabling or disabling Binlog is supported only for instances that are running.
Step 1: Enable the Binlog of the TDSQL Boundless Instance
1. Follow the procedures described in Managing Binlog to enable Binlog for the instance.
After successful activation, you will receive the IP address and port for Binlog CDC. Please keep it secure as it will be used to create data subscription tasks later.
2. Create a user for data subscription and grant the
REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT privileges on all objects.2.1 Log in to the Console, navigate to the Instance List, and select Instance ID for which Binlog is enabled.
2.2 Go to the Account Management page, click Create Account, and create a user for data synchronization.
2.3 In the row of the target account, click Modify Permissions in the Operation column to grant the
REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT permissions on all objects.
Step 2: Prepare the Network
To use DTS for database subscription, you must establish network connectivity between the DTS service and your source/target databases.
DTS supports accessing source/target databases via the "Cloud Database". Follow the instructions in Granting DTS Access IP for a Single Task to configure network access.
Step 3: Create a Data Subscription Task in DTS
1. Log in to the DTS Console, select the Data Subscription page in the left navigation, and click Create Subscription.
2. On the Create Subscription page, select the appropriate configuration and click Buy Now.
Configuration Item | Description |
Service Type | Data Subscription |
Database | Select TDSQL Boundless. |
Region | Must be the same region as your source database instance.Version |
Edition | Select Kafka Edition, which supports direct consumption via Kafka clients. |
Subscribed Instance Name | Provide a name for your data subscription instance. |
3. After the purchase is success, return to the Data Subscription list. click Configure Subscription in the Operation column, configure the newly purchased subscription; it can be used after configuration is completed.
4. On the Configure data subscription page, set up the source database connection. Click Test Connectivity, and after the test passes, click Save and Go Next.
Configuration Item | Description |
Instance Name | Select the instance where Binlog has been enabled in the first step. |
Account/Password | The account created in the first step that has the REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT permissions on all objects. |
5. On the Subscription Type and Object page, select the subscription type, then click Save and Go Next.
Configuration Item | Description |
Data Subscription Type | Data Update: Data Updates: Subscribes to data changes ( INSERT, UPDATE, DELETE) for all objects.Structure Update: Subscribes to schema changes (e.g., creating, modifying, or deleting objects). Full Instance: Subscribes to both data and structure updates for the entire instance. |
Format of Subscribed Data | Supports ProtoBuf, Avro, and JSON. ProtoBuf and Avro are binary formats offering higher consumption efficiency. JSON is a lightweight text format that is simpler to use. |
Number of Topic Partition | Sets the number of partitions for the topic in the built-in Kafka. Increasing partitions can improve write and consumption throughput. A single partition guarantees message order, while multiple partitions do not. If strict ordering is required, select 1. |
Topic Partitioning Policy | Required when the number of partitions is 4 or 8. By table name: Routes data for the same table to the same partition. By table name + primary key: Routes data with the same primary key to the same partition. Recommended for tables with hot spots to improve concurrent consumption. |
Custom Partitioning Policy | When using 4 or 8 partitions, you can define a custom strategy. Data is first matched against regular expressions for database/table names and partitioned accordingly (by table, table + PK, or column). Unmatched data is routed using the default strategy. For details, refer to Setting Partition Policy. |
6. On the Pre-verification page, the validation task will run for 2-3 minutes. After the pre-check passes, click Start to complete the configuration of the data subscription task.
7. After clicking Start, the subscription task will initialize, which takes approximately 3-4 minutes. Once initialization is successful, the task status will change to Running.
Ajuda e Suporte
Esta página foi útil?
Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.
comentários