[Feature][Connector-V2]Jdbc chunk split add “snapshot.split.column” params #7794 #7840
Conversation
# Conflicts: # plugin-mapping.properties
… update test case
Test case have been added, take a look. |
...se/src/main/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java
Outdated
Show resolved
Hide resolved
...eatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java
Outdated
Show resolved
Hide resolved
...se/src/main/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java
Outdated
Show resolved
Hide resolved
| Table table = dialect.queryTableSchema(jdbc, tableId).getTable(); | ||
|
|
||
| // first , compare user defined split column is in the primary key or unique key | ||
| Properties splitColumnProperties = new Properties(); |
There was a problem hiding this comment.
Why not just use Map?
org.apache.seatunnel.connectors.cdc.base.config.BaseSourceConfig , I see dbzProperties use Properties, so do the same
There was a problem hiding this comment.
Please use Map, properties are used because they interact with debezium, I think we should use Map in here just like other connectors.
There was a problem hiding this comment.
Please use Map, properties are used because they interact with debezium, I think we should use Map in here just like other connectors.
ok
…ain/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java Co-authored-by: Jia Fan <[email protected]>
…ain/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java Co-authored-by: Jia Fan <[email protected]>
…ain/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java Co-authored-by: Jia Fan <[email protected]>
| @@ -184,6 +184,7 @@ When an initial consistent snapshot is made for large databases, your establishe | |||
| | stop.specific-offset.file | String | No | - | Stop from the specified binlog file name. **Note, This option is required when the `stop.mode` option used `specific`.** | | |||
| | stop.specific-offset.pos | Long | No | - | Stop from the specified binlog file position. **Note, This option is required when the `stop.mode` option used `specific`.** | | |||
| | snapshot.split.size | Integer | No | 8096 | The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | | |||
| | snapshot.split.column | Config | No | - | The table name and split column (must be unique key) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | | |||
There was a problem hiding this comment.
There was a problem hiding this comment.
move to this location
code or doc need to move? snapshot.split.size in this location
https://github.com/apache/seatunnel/blob/139919334df43b8c320cb1139700a05124f55358/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java
There was a problem hiding this comment.
Yes, snapshot.split.column requires binding to a specific table
There was a problem hiding this comment.
Yes,
snapshot.split.columnrequires binding to a specific table
Now snapshot.split.column config like this, has binded to a specific table, you mean snapshot.split.column into table-name-config?
There was a problem hiding this comment.
Yes
table-names-config must have primaryKeys , otherwise, an error will be reported, now I only want to config table and snapshot.split.column
Purpose of this pull request
fix: #7794
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note.