msp-agent is a tool for migrating data to COS. You can deploy it on a server in your data center or in the cloud. It will execute the semi-managed migration tasks created in the Cloud Migration console, facilitating the easy migration of cloud storage data to COS.
Support Features
Migration from all the source vendors supported in the console.
Easy-to-deploy distributed master-worker architecture for efficient migration of a high volume of data.
Supports Breakpoint Resume.
Supports Traffic Control.
Supports all features that can be entered in the console, in line with fully managed service features.
Push mode: Deploy MSP Agent on a host near the data source to push data to the destination COS bucket.
Pull mode: Deploy MSP Agent on a host near the Tencent Cloud COS destination bucket to pull data from the data source and write it to the destination COS bucket.
Supports both Dedicated Line and Public Internet networking modes, see Network Traffic.
For files with an average file size greater than 1M, a memory ratio of -1:4 is recommended for the number of server cores. For example, select 16C for cores and recommend choose 64G or more for memory. For files with an average file size less than 1M, a memory ratio of 1:2 is recommended.
It is advisable to deploy master and worker separately. Open an additional machine with a configuration similar to or slightly smaller than the worker as the master in a cluster. This way, the stability of the entire cluster is more guaranteed.
Running environment
Suitable for Linux systems.
System Deployment Method
Installation
Download the msp-agent installation package from msp-agent, then decompress it. The directory structure after decompression is as follows:
Note:
msp-agent adopts the distributed master-worker architecture, where one master can connect to one or multiple workers.
The 'master' directory is for the master, and the 'worker' directory is for the workers.
If you need to deploy multiple workers, copy the entire 'worker' directory, then modify the relevant parameters before starting them using the method below.
You can start multiple worker processes on a single server. However, you need to modify relevant parameters as detailed below to avoid port conflicts.
Configuration Parameter Description
Both the master and worker directories contain the same configs structure:
Among them, pl_config.yaml configures the main parameters for process execution; app_logger_config.yaml configures the application runtime log format; query_logger_config.yaml configures the master-slave RPC communication log format.
Log configuration
You can basically use the default log parameter settings.
Note:
Log rotation configuration: If disk space is limited and the task scale is huge, adjust the log configuration to save disk space. During migration, only log storage consumes disk space, while actual file migration does not.
Master configuration
Parameter
Definition
Description
"....gRPCPort"
The master listens on the gRpc port.
Used to receive information reported by workers. This port on the master server must be opened to worker servers.
failFilePartSize
The part size for recording failed files
This parameter specifies the part size of a failed file, which is 10,485,760 bytes by default. The product of this value multiplied by 10,000 is the maximum size of failed files, i.e., 100 GB.
If there are both high numbers of files to be migrated and failed files, for example, more than 200 million files, you can increase this value appropriately.
fragMaxSize
The size of files contained in a single fragment of a distribution task
To reduce the pressure on master-worker communication, the master packages multiple file paths into a fragment and assigns it to a worker as a subtask. fragMaxSize is the maximum size of files in each fragment in bytes.
If the value is too small, the pressure on master-worker communication will get higher, and server resources will be wasted; if it is too large, workers will become too slow to report the completion of fragment migration and have imbalanced loads.
The default value is 10737418240, i.e., 100 GB.
When fragMaxSize or fragMaxNum reaches the limit during packaging, the system will stop adding more files.
fragMaxNum
The maximum number of files contained in a single fragment of a distribution task
To reduce the pressure on master-worker communication, the master packages multiple file paths into a fragment and assigns it to a worker as a subtask. fragMaxNum is the maximum number of files in each fragment.
If the value is too small, the pressure on master-worker communication will get higher, and server resources will be wasted; if it is too large, workers will become too slow to report the completion of fragment migration and have imbalanced loads.
The default value is -1,000.
When fragMaxSize or fragMaxNum reaches the limit during packaging, the system will stop adding more files.
secretId
SecretId of the key used to call the TencentCloud API for Cloud Migration
The Master process needs to call the Cloud Migration TencentCloud API to obtain tasks created by users in the Console. Therefore, you need to enter your key here, specifically the SecretId in the key.
Note that the key here refers to your key used to create Cloud Migration tasks, not the keys of source and target buckets.
secretKey
SecretKey of the key used to call the TencentCloud API for Cloud Migration
The Master process needs to call the Cloud Migration TencentCloud API to obtain tasks created by users in the Console. Therefore, you need to enter your key here, specifically the SecretKey in the key.
Note that the key here refers to your key used to create Cloud Migration tasks, not the keys of source and target buckets.
listerIp
The private IP address of the server where the Master process is deployed
You may create multiple tasks, and want them to run on different clusters. Therefore, you need to enter the Private IP Address of the server where the Master process is deployed. This way, the Master will only run tasks that are assigned to this server's IP during task creation in the Console.
When creating tasks in the Console, enter the same IP in the form for the Master Node Private IP.
useAccelerateDomain
Access the internal COS bucket holding the Failed File List using the Global Acceleration Domain
The failed file list of a task is uniformly stored in a bucket in a Chinese mainland region to prevent retry failures caused by failure to save the failed file list. If MSP Agent is deployed outside the Chinese mainland, enable this configuration in the configuration file to ensure task success.
Worker configuration
Parameter
Definition
Description
"....gRPCPort"
GRPC port listened on by the Worker
This parameter specifies the port used to receive the scheduling information from the master. This port must be opened to the master server on the worker server.
If multiple worker processes run on a single server, you need to change this parameter to a value different from that of other workers, so as to avoid process startup failure due to port conflicts.
fileMigrateTryTimes
Number of Retries
The number of retries after a single file fails.
goroutineConcurrentNum
Number of concurrent coroutines
This parameter specifies the number of concurrent coroutines, i.e., the number of files to be migrated concurrently.
The value is subject to two factors: server configuration and average file size.
The more the server CPU cores, the greater the value can be set.
The larger the average file size, the smaller the value can be set. As a small number of concurrent coroutines can lead to a high bandwidth usage, if the average file size is small, you should add more concurrent coroutines to increase the overall bandwidth.
baseWorkerMaxConcurrentFileNum
The cached queue of files to be migrated
To improve the assignment efficiency of distributed tasks, each worker caches a certain number of file fragments to be migrated. If the files to be migrated are small (that is, the migration qps is high), you can increase this value to make workers less hungry during consumption.
However, the larger the cache, the more the status data that the master needs to store, which will increase the master load and instability. Therefore, you should choose an appropriate value.
partSize
Part size
The default part size for multipart upload when large files are migrated.
downloadPartTimeout
Download timeout
The timeout for downloading files, in seconds.
uploadPartTimeout
Upload timeout
The timeout for uploading files, in seconds.
perHostMaxIdle
http client concurrency settings
This parameter specifies the connection pool size per host and generally should be set to the same value as
addr
Private network communication address of the master
This parameter specifies the master communication address of workers, so that workers will register with the master to form a cluster.
For example, if the listerIp of the master is 10.0.0.1 and the GRPC port listened on by the master is 22011, set addr to 10.0.0.1:22011.
sample
Whether to perform spot check
The overview section describes the cases of data consistency check after migration. If source files don't have a Content-MD5 or CRC-64 checksum, they cannot be directly checked for data consistency. In this case, you can only perform spot check to reduce their probability of inconsistency.
If this parameter is set to true, spot check will be performed on such files.
sampleTimes
The number of sampled fragments per file
The number of sampled fragments per file. Each sampled fragment incurs an additional download request, which also means additional download traffic.
sampleByte
The number of bytes in a sampled fragment
This parameter specifies the size of each sampled fragment. The greater the value, the higher the sampling bandwidth.
useInternalAccDomain
Using the COS private network global acceleration domain
If this parameter is set to true, the private network global acceleration domain is used when files are uploaded to the destination COS bucket, which improves the performance of data upload to COS. Before enabling this parameter, make sure that global acceleration is enabled for the destination COS bucket.
Note:
Enabling this configuration will incur COS global acceleration traffic fees, which are charged by COS. For details, see Global Acceleration Overview.
The billing policy of the source side (such as a third party) prevails, and no fees are charged on the target side (Tencent Cloud).
Public network
Migration performance is limited by public network bandwidth.
The billing policy of the source side (such as a third party) prevails, and no fees are charged on the target side (Tencent Cloud).
If the data source is Tencent Cloud COS, migrating buckets within the same region uses the private network; migrating buckets across regions uses the public network.
