-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Connector-V2]Jdbc chunk split add “snapshot.split.column” params #7794 #7840
base: dev
Are you sure you want to change the base?
Conversation
# Conflicts: # plugin-mapping.properties
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add test case.
… update test case
Test case have been added, take a look. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @hailin0
...se/src/main/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java
Outdated
Show resolved
Hide resolved
...eatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java
Outdated
Show resolved
Hide resolved
...se/src/main/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java
Outdated
Show resolved
Hide resolved
Table table = dialect.queryTableSchema(jdbc, tableId).getTable(); | ||
|
||
// first , compare user defined split column is in the primary key or unique key | ||
Properties splitColumnProperties = new Properties(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just use Map?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just use Map?
org.apache.seatunnel.connectors.cdc.base.config.BaseSourceConfig , I see dbzProperties use Properties, so do the same
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ping @XenosK
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use Map, properties are used because they interact with debezium, I think we should use Map in here just like other connectors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use Map, properties are used because they interact with debezium, I think we should use Map in here just like other connectors.
ok
…ain/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java Co-authored-by: Jia Fan <[email protected]>
…ain/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java Co-authored-by: Jia Fan <[email protected]>
…ain/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java Co-authored-by: Jia Fan <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @hailin0 take a look.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
@@ -184,6 +184,7 @@ When an initial consistent snapshot is made for large databases, your establishe | |||
| stop.specific-offset.file | String | No | - | Stop from the specified binlog file name. **Note, This option is required when the `stop.mode` option used `specific`.** | | |||
| stop.specific-offset.pos | Long | No | - | Stop from the specified binlog file position. **Note, This option is required when the `stop.mode` option used `specific`.** | | |||
| snapshot.split.size | Integer | No | 8096 | The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | | |||
| snapshot.split.column | Config | No | - | The table name and split column (must be unique key) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
move to this location
Line 155 in 1399193
+ " \"primaryKeys\": [\"key1\",\"key2\"]" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
link
#6106
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
move to this location
Line 155 in 1399193
+ " \"primaryKeys\": [\"key1\",\"key2\"]"
code or doc need to move? snapshot.split.size in this location
https://github.com/apache/seatunnel/blob/139919334df43b8c320cb1139700a05124f55358/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, snapshot.split.column
requires binding to a specific table
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
table-names-config must have primaryKeys , otherwise, an error will be reported, now I only want to config table and snapshot.split.column
Purpose of this pull request
fix: #7794
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note
.