You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The split column selection order is:
1、primary key and primary key type is tinyint
2、unique key and unique key type is tinyint
3、no tinyint key , compare primary key and unique key type,the type order is: SMALLINT、INT、BIGINT、DECIMAL、STRING
This is a design issue,and I think can add user defined split column,if need can assigned to me
Search before asking
What happened
使用CDC同步MySQL一张表到doris,表数据大概有6千万,在进行数据分割时seatunnel同步任务并没有使用主键(id)进行分割,而是使用唯一索引(ad_type)进行分割,数据同步非常慢。使用show full processlist查看MySQL正在执行的SQL如下:SELECT MIN(
ad_type
), MAX(ad_type
) FROMtest
.abroad_day_recharge
,同步任务一直卡在这个SQL处, 表结构如下:CREATE TABLE
abroad_day_recharge
(id
int unsigned NOT NULL AUTO_INCREMENT,union_link_id
int unsigned DEFAULT '0',book_id
bigint unsigned DEFAULT '0',system
tinyint unsigned DEFAULT '1',user_date
int unsigned DEFAULT '0',order_date
int unsigned DEFAULT '0',platform
int unsigned DEFAULT '0',ad_type
tinyint DEFAULT '1',money
int unsigned DEFAULT '0',created_at
timestamp NULL DEFAULT CURRENT_TIMESTAMP,updated_at
timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,PRIMARY KEY (
id
),UNIQUE KEY
uniq_key_word
(union_link_id
,user_date
,order_date
,ad_type
,system
) USING BTREE,KEY
idx_user_date
(user_date
),KEY
idx_order_date
(order_date
),KEY
idx_platform
(platform
) USING BTREE) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci
SeaTunnel Version
2.3.5
SeaTunnel Config
Running Command
../bin/seatunnel.sh -c mysql-to-doris.conf -e local
Error Exception
Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: