-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial addition of common functionality between S3 sink and source connectors #312
base: main
Are you sure you want to change the base?
Initial addition of common functionality between S3 sink and source connectors #312
Conversation
aindriu-aiven
commented
Oct 10, 2024
•
edited
Loading
edited
- Addition of a new module/project to allow the sharing of common code between S3 Sink and Source
- Handles Creation of S3 Aws client
- Refactors commons AivenCommonConfig and splits into a common config and a Sink and Source config, deprecating the AivenCommonConfig
- Has common Configuration in S3SinkBaseConfig and S3SourceBaseConfig
- Includes some config loader methods that are common to both for AWS STS and S3 Retry Policies
- Updates the S3 Sink Connector to use the s3-common lib
- Added deprecated config & a separate method to load into config so S3 Sink can decide when to remove deprecated config
…nnectors Signed-off-by: Aindriu Lavelle <[email protected]>
d41fe78
to
2df807f
Compare
s3-connectors-common/src/main/java/io/aiven/kafka/connect/iam/AwsAccessSecret.java
Outdated
Show resolved
Hide resolved
s3-connectors-common/src/main/java/io/aiven/kafka/connect/s3/S3BaseConfig.java
Outdated
Show resolved
Hide resolved
s3-connectors-common/src/main/java/io/aiven/kafka/connect/iam/AwsCredentialProviderFactory.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Like AwsAccessSecret this too can be removed.
final AwsStsEndpointConfig endpointConfig = config.getStsEndpointConfig(); | ||
final AwsClientBuilder.EndpointConfiguration stsConfig = new AwsClientBuilder.EndpointConfiguration( | ||
endpointConfig.getServiceEndpoint(), endpointConfig.getSigningRegion()); | ||
final AWSSecurityTokenServiceClientBuilder stsBuilder = AWSSecurityTokenServiceClientBuilder.standard(); | ||
stsBuilder.setEndpointConfiguration(stsConfig); | ||
return stsBuilder.build(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Config should return the AwcClientBuilder.EndpointConfiguration object if the config has the properties. No point in making a bean to carry values when the actual object can be created in the config.
Config could have hasX() methods for the 3 ways AWS lets you build AWSCredential providers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another candidate for removal.
s3-connectors-common/src/main/java/io/aiven/kafka/connect/s3/S3BaseConfig.java
Outdated
Show resolved
Hide resolved
ConfigDef.Width.NONE, AWS_S3_REGION_CONFIG); | ||
} | ||
|
||
protected static void addS3SinkConfig(final ConfigDef configDef) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps this one should remain in the S3SinkConfig class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I split it out here on purpose so it wouldn't be forgotten about as it is contained in the sink class'
addAwsConfigGroup but not in the source addAwsConfigGroup and I didn't want it forgotten about, by someone doing a straight swap,
Would adding a TODO addAwsConfigGroup on this suffice?
WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good, ad the TODO plz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh actually @AnatolyPopov asked me to make the changes to the S3 sink connector in this PR as well, I just need to make time for that at some point today as I am switching between multiple things, however, I will try to include in this PR as well.
s3-connectors-common/src/main/java/io/aiven/kafka/connect/s3/S3ClientUtils.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Aindriu Lavelle <[email protected]>
2dbcfb1
to
21d5761
Compare
21d5761
to
7a3073f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code in this change is copied/merged from the source and sink projects or just from source?
Actually copied from Sink as the source of truth as Source is a moving target at the moment. |
Signed-off-by: Aindriu Lavelle <[email protected]>
…ector use the sink source for configuration Signed-off-by: Aindriu Lavelle <[email protected]>
8b1b349
to
c6638b7
Compare