Initial addition of common functionality between S3 sink and source connectors #312
Conversation
aindriu-aiven
commented
Oct 10, 2024
•
aindriu-aiven
commented
- Addition of a new module/project to allow the sharing of common code between S3 Sink and Source
- Handles Creation of S3 Aws client
- Refactors commons AivenCommonConfig and splits into a common config and a Sink and Source config, deprecating the AivenCommonConfig
- Has common Configuration in S3SinkBaseConfig and S3SourceBaseConfig
- Includes some config loader methods that are common to both for AWS STS and S3 Retry Policies
- Updates the S3 Sink Connector to use the s3-common lib
- Added deprecated config & a separate method to load into config so S3 Sink can decide when to remove deprecated config
…nnectors Signed-off-by: Aindriu Lavelle <[email protected]>
aindriu-aiven
changed the title
aindriu-aiven
force-pushed
the
aindriu-aiven/split-common-s3-connector-config
branch
4 times, most recently
from
d41fe78
to
2df807f
Compare
s3-connectors-common/src/main/java/io/aiven/kafka/connect/iam/AwsAccessSecret.java
Outdated
Show resolved
Hide resolved
s3-connectors-common/src/main/java/io/aiven/kafka/connect/s3/S3BaseConfig.java
Outdated
Show resolved
Hide resolved
s3-connectors-common/src/main/java/io/aiven/kafka/connect/iam/AwsCredentialProviderFactory.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Like AwsAccessSecret this too can be removed.
| final AwsStsEndpointConfig endpointConfig = config.getStsEndpointConfig(); | ||
| final AwsClientBuilder.EndpointConfiguration stsConfig = new AwsClientBuilder.EndpointConfiguration( | ||
| endpointConfig.getServiceEndpoint(), endpointConfig.getSigningRegion()); | ||
| final AWSSecurityTokenServiceClientBuilder stsBuilder = AWSSecurityTokenServiceClientBuilder.standard(); | ||
| stsBuilder.setEndpointConfiguration(stsConfig); | ||
| return stsBuilder.build(); |
There was a problem hiding this comment.
Config should return the AwcClientBuilder.EndpointConfiguration object if the config has the properties. No point in making a bean to carry values when the actual object can be created in the config.
Config could have hasX() methods for the 3 ways AWS lets you build AWSCredential providers.
There was a problem hiding this comment.
Another candidate for removal.
s3-connectors-common/src/main/java/io/aiven/kafka/connect/s3/S3BaseConfig.java
Outdated
Show resolved
Hide resolved
| ConfigDef.Width.NONE, AWS_S3_REGION_CONFIG); | ||
| } | ||
|
|
||
| protected static void addS3SinkConfig(final ConfigDef configDef) { |
There was a problem hiding this comment.
Perhaps this one should remain in the S3SinkConfig class?
There was a problem hiding this comment.
Yeah, I split it out here on purpose so it wouldn't be forgotten about as it is contained in the sink class'
addAwsConfigGroup but not in the source addAwsConfigGroup and I didn't want it forgotten about, by someone doing a straight swap,
Would adding a TODO addAwsConfigGroup on this suffice?
WDYT?
There was a problem hiding this comment.
Sounds good, ad the TODO plz
There was a problem hiding this comment.
Oh actually @AnatolyPopov asked me to make the changes to the S3 sink connector in this PR as well, I just need to make time for that at some point today as I am switching between multiple things, however, I will try to include in this PR as well.
s3-connectors-common/src/main/java/io/aiven/kafka/connect/s3/S3ClientUtils.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Aindriu Lavelle <[email protected]>
aindriu-aiven
force-pushed
the
aindriu-aiven/split-common-s3-connector-config
branch
from
2dbcfb1
to
21d5761
Compare
aindriu-aiven
force-pushed
the
aindriu-aiven/split-common-s3-connector-config
branch
from
21d5761
to
7a3073f
Compare
Actually copied from Sink as the source of truth as Source is a moving target at the moment. |
Signed-off-by: Aindriu Lavelle <[email protected]>
…ector use the sink source for configuration Signed-off-by: Aindriu Lavelle <[email protected]>
aindriu-aiven
force-pushed
the
aindriu-aiven/split-common-s3-connector-config
branch
from
8b1b349
to
c6638b7
Compare
