[Fix][Rocketmq] Rocketmq sink error when use multi-table synchronization #7771

[YalikWang]

Purpose of this pull request

Fix Rocketmq sink error when use multi-table synchronization.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Use existed test cases

Check list

• If any new Jar binary package adding in your PR, please add License Notice according
  New License Guide
• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update plugin-mapping.properties and add new connector information in it
  2. Update the pom file of seatunnel-dist
  3. Add ci label in label-scope-conf
  4. Add e2e testcase in seatunnel-e2e
  5. Update connector plugin_config
• Update the release-note.

[Hisoka-X]

Could you add a e2e test case for rocketmq with multi table? Thanks

[Hisoka-X]

Hi @YalikWang , please do not close/reopen PR, it will lost ci status.
CI : https://github.com/YalikWang/seatunnel/actions/runs/11233083288/job/31226282144

[YalikWang]

Hi @YalikWang , please do not close/reopen PR, it will lost ci status. CI : https://github.com/YalikWang/seatunnel/actions/runs/11233083288/job/31226282144

How can I manually trigger E2E tests?

[Hisoka-X]

You can tirgger ci again by yourself.

[YalikWang]

@Hisoka-X E2E test failed with Spark Engine because of time-out. After submitting to Spark Engine, it seems that the job did not run. Could you please help me to analysis the reason of it ? Or should I disable spark engine container?

e2e test console log :
63392 [docker-java-stream-1626631782] INFO 🐳 [tyrantlucifer/spark:3.3.0] - STDOUT: 24/10/12 09:25:23 INFO Utils: Successfully started service 'MasterUI' on port 8080.
63454 [main] INFO org.apache.seatunnel.e2e.common.container.AbstractTestContainer - Execute command in container[tyrantlucifer/spark:3.3.0]
==================== Shell Command start ====================
/tmp/seatunnel/bin/start-seatunnel-spark-3-connector-v2.sh --config /tmp/rocketmq-sink_fake_to_rocketmq_multi_tables.conf --name rocketmq-sink_fake_to_rocketmq_multi_tables.conf --master local --deploy-mode client
==================== Shell Command end ====================
63485 [docker-java-stream-1626631782] INFO 🐳 [tyrantlucifer/spark:3.3.0] - STDOUT: 24/10/12 09:25:23 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://39bd998c4ce0:8080
63793 [docker-java-stream-1626631782] INFO 🐳 [tyrantlucifer/spark:3.3.0] - STDOUT: 24/10/12 09:25:24 INFO Master: I have been elected leader! New state: ALIVE

spark docker logs:
09:25:19.92
09:25:19.93 Welcome to the Bitnami spark container
09:25:19.93 Subscribe to project updates by watching https://github.com/bitnami/containers
09:25:19.93 Submit issues and feature requests at https://github.com/bitnami/containers/issues
09:25:19.93
09:25:19.94 INFO ==> ** Starting Spark setup **
realpath: /bitnami/spark/conf: No such file or directory
09:25:20.22 INFO ==> Generating Spark configuration file...
find: '/docker-entrypoint-initdb.d/': No such file or directory
09:25:20.23 INFO ==> No custom scripts in /docker-entrypoint-initdb.d
09:25:20.24 INFO ==> ** Spark setup finished! **

09:25:20.26 INFO ==> ** Starting Spark in master mode **
starting org.apache.spark.deploy.master.Master, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.master.Master-1-39bd998c4ce0.out
Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 39bd998c4ce0 --port 7077 --webui-port 8080

Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
24/10/12 09:25:22 INFO Master: Started daemon with process name: 66@39bd998c4ce0
24/10/12 09:25:22 INFO SignalUtils: Registering signal handler for TERM
24/10/12 09:25:22 INFO SignalUtils: Registering signal handler for HUP
24/10/12 09:25:22 INFO SignalUtils: Registering signal handler for INT
24/10/12 09:25:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/10/12 09:25:22 INFO SecurityManager: Changing view acls to: spark
24/10/12 09:25:22 INFO SecurityManager: Changing modify acls to: spark
24/10/12 09:25:22 INFO SecurityManager: Changing view acls groups to:
24/10/12 09:25:22 INFO SecurityManager: Changing modify acls groups to:
24/10/12 09:25:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set()
24/10/12 09:25:23 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
24/10/12 09:25:23 INFO Master: Starting Spark master at spark://39bd998c4ce0:7077
24/10/12 09:25:23 INFO Master: Running Spark version 3.3.0
24/10/12 09:25:23 INFO Utils: Successfully started service 'MasterUI' on port 8080.
24/10/12 09:25:23 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://39bd998c4ce0:8080
24/10/12 09:25:24 INFO Master: I have been elected leader! New state: ALIVE