Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve initial connection timeouts with Amazon MSK clusters #275

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

antcalvente
Copy link

This PR solves the issue of the control topics not being able to join successfully on a cloud environment as the current timeout is set to 1 second and it's not enough.

Before this PR: cg-control-XXX consumers are not always joining on time and the connector fails
After this PR: new timeout allows consumers join the consumer group and sink connectors work without issues

How to reproduce the issue?

  1. Create an AWS MSK cluster with 1 topic (a local cluster is not valid as it's local network and it will surely work)
  2. Install kafka-ui or any tool to check consumers of the topic
  3. Create an Iceberg connector through MSK Connect
  4. Check logs in Cloudwatch or the enabled logs tool when creating the connector
  5. Post a message after initialisation
  6. Check if the message is consumed (logs) and saved in your target system

Note: The connector will be marked as Running on MSK even if it couldn't start properly with the result of not consuming messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant