-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Connector-V2]Jdbc chunk split add “snapshot.split.column” params #7794 #7840
base: dev
Are you sure you want to change the base?
Changes from 16 commits
fc75772
1576162
1a636d3
87d749a
69bf03c
1165447
0c2676b
7165bad
f5c5ee8
8a0b6ba
401723e
f9c973a
83020fd
9a87c8b
7be5d6d
afd9cd9
2cae910
edc6f70
275520d
fe8f7d3
768ca66
1973c18
08bd831
047665f
733d823
1388c01
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -39,6 +39,7 @@ | |
import java.util.List; | ||
import java.util.Objects; | ||
import java.util.Optional; | ||
import java.util.Properties; | ||
|
||
import static java.math.BigDecimal.ROUND_CEILING; | ||
import static org.apache.seatunnel.connectors.cdc.base.utils.ObjectUtils.doubleCompare; | ||
|
@@ -379,12 +380,38 @@ protected SnapshotSplit createSnapshotSplit( | |
protected Column getSplitColumn( | ||
JdbcConnection jdbc, JdbcDataSourceDialect dialect, TableId tableId) | ||
throws SQLException { | ||
Optional<PrimaryKey> primaryKey = dialect.getPrimaryKey(jdbc, tableId); | ||
Column splitColumn = null; | ||
Table table = dialect.queryTableSchema(jdbc, tableId).getTable(); | ||
|
||
// first , compare user defined split column is in the primary key or unique key | ||
Properties splitColumnProperties = new Properties(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why not just use Map? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
org.apache.seatunnel.connectors.cdc.base.config.BaseSourceConfig , I see dbzProperties use Properties, so do the same There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ping @XenosK There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please use Map, properties are used because they interact with debezium, I think we should use Map in here just like other connectors. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
ok |
||
try { | ||
splitColumnProperties = sourceConfig.getSplitColumn(); | ||
} catch (Exception e) { | ||
log.error("Config snapshot.split.column get exception in {}:{}", tableId, e); | ||
} | ||
String tableSc = | ||
(String) splitColumnProperties.get(tableId.catalog() + "." + tableId.table()); | ||
Boolean isUniqueKey = dialect.isUniqueKey(jdbc, tableId, tableSc); | ||
if (isUniqueKey) { | ||
Column column = table.columnWithName(tableSc); | ||
if (isEvenlySplitColumn(column)) { | ||
return column; | ||
} else { | ||
log.warn( | ||
"Config snapshot.split.column type in {} is not TINYINT、SMALLINT、INT、BIGINT、DECIMAL、STRING", | ||
tableId); | ||
} | ||
} else { | ||
log.warn( | ||
XenosK marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"Config snapshot.split.column not exists or not unique key for table {}", | ||
tableId); | ||
} | ||
|
||
Optional<PrimaryKey> primaryKey = dialect.getPrimaryKey(jdbc, tableId); | ||
if (primaryKey.isPresent()) { | ||
List<String> pkColumns = primaryKey.get().getColumnNames(); | ||
|
||
Table table = dialect.queryTableSchema(jdbc, tableId).getTable(); | ||
for (String pkColumn : pkColumns) { | ||
Column column = table.columnWithName(pkColumn); | ||
if (isEvenlySplitColumn(column)) { | ||
|
@@ -400,7 +427,6 @@ protected Column getSplitColumn( | |
|
||
List<ConstraintKey> uniqueKeys = dialect.getUniqueKeys(jdbc, tableId); | ||
if (!uniqueKeys.isEmpty()) { | ||
Table table = dialect.queryTableSchema(jdbc, tableId).getTable(); | ||
for (ConstraintKey uniqueKey : uniqueKeys) { | ||
List<ConstraintKey.ConstraintKeyColumn> uniqueKeyColumns = | ||
uniqueKey.getColumnNames(); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
move to this location
seatunnel/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java
Line 155 in 1399193
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
link
#6106
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code or doc need to move? snapshot.split.size in this location
https://github.com/apache/seatunnel/blob/139919334df43b8c320cb1139700a05124f55358/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes,
snapshot.split.column
requires binding to a specific tableThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now snapshot.split.column config like this, has binded to a specific table, you mean snapshot.split.column into table-name-config?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
table-names-config must have primaryKeys , otherwise, an error will be reported, now I only want to config table and snapshot.split.column