Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [SftpFile FtpFile S3File] Bug binary sync abnormal #7737

Closed
2 of 3 tasks
taohaozhi1129 opened this issue Sep 24, 2024 · 2 comments · Fixed by #7942
Closed
2 of 3 tasks

[Bug] [SftpFile FtpFile S3File] Bug binary sync abnormal #7737

taohaozhi1129 opened this issue Sep 24, 2024 · 2 comments · Fixed by #7942
Assignees
@zhangshenghang
Labels
bug help wanted

Comments

@taohaozhi1129
Copy link
Contributor

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

"When I use the SFTP FTP and MinIO connector to synchronize files, the file_format_type is set to binary. The example runs normally on my local Windows machine, successfully syncing the files to the specified sink path. However, when I submit the task on the Linux server, it directly overwrites the sink path with a file."

SeaTunnel Version

2.3.7

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
 SFtpFile {
    host= "192.168.137.22"
    port= 22
    user= "sftpuser"
    password= "sftpuser"
    path= "/uploads/ssss/code.txt"
    file_format_type= "binary"
    encoding = "UTF-8"
}
}


sink {
    SftpFile {
      host= "192.168.137.22"
      port= 22
      user= "sftpuser"
      password= "sftpuser"
      tmp_path = "/uploads/tmp/seatunnel"
      path= "/uploads/20240924/cc"
      file_format_type= "binary"
      encoding="UTF-8"
  }
}

Running Command

./bin/seatunnel.sh --config config/sftp-sftp.tmp.cpnfig.template

Error Exception

client.log
[root@node3 apache-seatunnel-2.3.7]# ./bin/seatunnel.sh --config config/sftp-sftp.tmp.cpnfig.template 
2024-09-24 20:34:15,559 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/bigdata/apache-seatunnel-2.3.7/config/seatunnel.yaml' from System property 'seatunnel.config'
2024-09-24 20:34:15,563 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/bigdata/apache-seatunnel-2.3.7/config/seatunnel.yaml
2024-09-24 20:34:15,565 INFO  [o.a.s.e.c.c.SeaTunnelConfig   ] [main] - seatunnel.home is /opt/bigdata/apache-seatunnel-2.3.7
2024-09-24 20:34:15,638 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/bigdata/apache-seatunnel-2.3.7/config/hazelcast.yaml' from System property 'hazelcast.config'
2024-09-24 20:34:15,638 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/bigdata/apache-seatunnel-2.3.7/config/hazelcast.yaml
2024-09-24 20:34:15,891 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/bigdata/apache-seatunnel-2.3.7/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
2024-09-24 20:34:15,891 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/bigdata/apache-seatunnel-2.3.7/config/hazelcast-client.yaml
2024-09-24 20:34:16,118 INFO  [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2024-09-24 20:34:16,164 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING2024-09-24 20:34:16,164 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2024-09-24 20:34:16,181 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2024-09-24 20:34:16,184 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to [node1]:5801
2024-09-24 20:34:16,193 WARN  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Exception during initial connection to [node1]:5801: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused to address node1/192.168.137.20:5801
2024-09-24 20:34:16,193 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to [node3]:5801
2024-09-24 20:34:16,214 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2024-09-24 20:34:16,214 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [node3]:5801:70e60315-e588-40ae-8202-1d11169bfdce, server version: 5.1, local address: /192.168.137.22:41535
2024-09-24 20:34:16,215 INFO  [c.h.i.d.Diagnostics           ] [main] - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-09-24 20:34:16,223 INFO  [c.h.c.i.s.ClientClusterService] [hz.client_1.event-4] - hz.client_1 [seatunnel] [5.1] 

Members [1] {
        Member [node3]:5801 - 70e60315-e588-40ae-8202-1d11169bfdce
}

2024-09-24 20:34:16,250 INFO  [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
2024-09-24 20:34:16,436 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Loading config file from path: config/sftp-sftp.tmp.cpnfig.template
2024-09-24 20:34:16,474 INFO  [o.a.s.c.s.u.ConfigShadeUtils  ] [main] - Load config shade spi: [base64]
2024-09-24 20:34:16,534 INFO  [p.MultipleTableJobConfigParser] [main] - add common jar in plugins :[]
2024-09-24 20:34:16,544 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/bigdata/apache-seatunnel-2.3.7/connectors
2024-09-24 20:34:16,548 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='SFtpFile'} at: file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar
2024-09-24 20:34:16,548 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='SftpFile'} at: file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar
2024-09-24 20:34:16,550 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sources.
2024-09-24 20:34:16,559 WARN  [p.MultipleTableJobConfigParser] [main] - The Factory has not been implemented and the deprecated Plugin will be used.
2024-09-24 20:34:16,560 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSource Plugin from /opt/bigdata/apache-seatunnel-2.3.7/connectors
2024-09-24 20:34:16,565 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='SFtpFile'} at: file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar
2024-09-24 20:34:16,567 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='SFtpFile'} from classpath
2024-09-24 20:34:16,727 WARN  [o.a.h.u.NativeCodeLoader      ] [main] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-09-24 20:34:17,616 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all transforms.
2024-09-24 20:34:17,616 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sinks.
2024-09-24 20:34:17,617 WARN  [p.MultipleTableJobConfigParser] [main] - The Factory has not been implemented and the deprecated Plugin will be used.
2024-09-24 20:34:17,620 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/bigdata/apache-seatunnel-2.3.7/connectors
2024-09-24 20:34:17,621 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='SftpFile'} at: file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar
2024-09-24 20:34:17,621 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='SftpFile'} from classpath
2024-09-24 20:34:17,655 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Start submit job, job id: 890939192865980417, with plugin jar [file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar]
2024-09-24 20:34:17,681 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Submit job finished, job id: 890939192865980417, job name: SeaTunnel_Job
2024-09-24 20:34:17,690 WARN  [o.a.s.e.c.j.JobMetricsRunner  ] [job-metrics-runner-890939192865980417] - Failed to get job metrics summary, it maybe first-run
2024-09-24 20:34:21,130 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (890939192865980417) end with state FINISHED
2024-09-24 20:34:21,135 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - 
***********************************************
           Job Statistic Information
***********************************************
Start Time                : 2024-09-24 20:34:16
End Time                  : 2024-09-24 20:34:21
Total Time(s)             :                   4
Total Read Count          :                   5
Total Write Count         :                   5
Total Failed Count        :                   0
***********************************************

2024-09-24 20:34:21,135 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-09-24 20:34:21,138 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node3]:5801:70e60315-e588-40ae-8202-1d11169bfdce, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/192.168.137.22:41535->node3/192.168.137.22:5801}, remoteAddress=[node3]:5801, lastReadTime=2024-09-24 20:34:21.132, lastWriteTime=2024-09-24 20:34:21.130, closedTime=2024-09-24 20:34:21.137, connected server version=5.1}
2024-09-24 20:34:21,138 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2024-09-24 20:34:21,140 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN2024-09-24 20:34:21,140 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2024-09-24 20:34:21,140 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2024-09-24 20:34:21,141 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-2] - run shutdown hook because get close signal
---------------------------------------
server.log
2024-09-24 20:34:16,210 INFO  [.p.t.AuthenticationMessageTask] [hz.main.priority-generic-operation.thread-0] - [node3]:5801 [seatunnel] [5.1] Received auth from Connection[id=34, /192.168.137.22:5801->/192.168.137.22:41535, qualifier=null, endpoint=[192.168.137.22]:41535, remoteUuid=0d6771a5-e176-4844-8d3c-99c0e4ae1022, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 0d6771a5-e176-4844-8d3c-99c0e4ae1022, client name: hz.client_1, client version: 5.1
2024-09-24 20:34:17,658 INFO  [o.a.s.e.s.m.JobMaster         ] [seatunnel-coordinator-service-20] - in batch mode, the 'checkpoint.interval' configuration of env is missing, so checkpoint will be disabled
2024-09-24 20:34:17,658 INFO  [o.a.s.e.s.m.JobMaster         ] [seatunnel-coordinator-service-20] - Init JobMaster for Job SeaTunnel_Job (890939192865980417) 
2024-09-24 20:34:17,658 INFO  [o.a.s.e.s.m.JobMaster         ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417) needed jar urls [file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar]
2024-09-24 20:34:17,658 INFO  [.c.c.DefaultClassLoaderService] [seatunnel-coordinator-service-20] - Create classloader for job 890939192865980417 with jars [file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar]
2024-09-24 20:34:17,675 INFO  [.c.c.DefaultClassLoaderService] [seatunnel-coordinator-service-20] - Release classloader for job 890939192865980417 with jars [file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar]
2024-09-24 20:34:17,679 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-20] - Create CheckpointCoordinator for job(1@890939192865980417) with plan(CheckpointPlan(pipelineId=1, pipelineSubtasks=[TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}, taskID=20000, index=0}, TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, taskID=60000, index=0}, TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, taskID=70000, index=0}, TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}, taskID=40000, index=0}], startingSubtasks=[TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}, taskID=20000, index=0}], pipelineActions={ActionStateKey(name=ActionStateKey - pipeline-1 [Sink[0]-SftpFile])=1, ActionStateKey(name=ActionStateKey - pipeline-1 [Source[0]-SFtpFile])=1}, subtaskActions={TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}, taskID=20000, index=0}=[(ActionStateKey(name=ActionStateKey - pipeline-1 [Source[0]-SFtpFile]), -1)], TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, taskID=60000, index=0}=[(ActionStateKey(name=ActionStateKey - pipeline-1 [Source[0]-SFtpFile]), 0)], TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, taskID=70000, index=0}=[(ActionStateKey(name=ActionStateKey - pipeline-1 [Sink[0]-SftpFile]), 0)], TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}, taskID=40000, index=0}=[(ActionStateKey(name=ActionStateKey - pipeline-1 [Sink[0]-SftpFile]), -1)]}))
2024-09-24 20:34:17,679 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - The task Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] is in state CREATED when init state future
2024-09-24 20:34:17,679 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - The task Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] is in state CREATED when init state future
2024-09-24 20:34:17,679 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - The task Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] is in state CREATED when init state future
2024-09-24 20:34:17,680 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417) state process is start
2024-09-24 20:34:17,680 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417) turned from state CREATED to SCHEDULED.
2024-09-24 20:34:17,680 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] state process is start
2024-09-24 20:34:17,680 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] turned from state CREATED to SCHEDULED.
2024-09-24 20:34:17,681 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] turned from state SCHEDULED to DEPLOYING.
2024-09-24 20:34:17,682 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] state process is start
2024-09-24 20:34:17,682 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] turned from state CREATED to DEPLOYING.
2024-09-24 20:34:17,684 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] received deploying task executionId [890939198121443329]
2024-09-24 20:34:17,684 INFO  [.c.c.DefaultClassLoaderService] [seatunnel-coordinator-service-20] - Create classloader for job 890939192865980417 with jars [file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar]
2024-09-24 20:34:17,691 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying task TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}, executionId [890939198121443329]
2024-09-24 20:34:17,691 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1} init success
2024-09-24 20:34:17,692 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1} success
2024-09-24 20:34:17,692 INFO  [.s.t.SourceSplitEnumeratorTask] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}] - starting seatunnel source split enumerator task, source name: pipeline-1 [Source[0]-SFtpFile]
2024-09-24 20:34:17,692 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] turned from state DEPLOYING to RUNNING.
2024-09-24 20:34:17,692 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] current state equals target state: RUNNING, skip
2024-09-24 20:34:17,692 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] state process is start
2024-09-24 20:34:17,693 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] turned from state CREATED to DEPLOYING.
2024-09-24 20:34:17,696 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] received deploying task executionId [890939198121508865]
2024-09-24 20:34:17,703 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying task TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}, executionId [890939198121508865]
2024-09-24 20:34:17,703 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3} init success
2024-09-24 20:34:17,703 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3} success
2024-09-24 20:34:17,704 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] turned from state DEPLOYING to RUNNING.
2024-09-24 20:34:17,704 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] current state equals target state: RUNNING, skip
2024-09-24 20:34:17,704 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] state process is start
2024-09-24 20:34:17,704 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] turned from state CREATED to DEPLOYING.
2024-09-24 20:34:17,705 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] received deploying task executionId [890939198121574401]
2024-09-24 20:34:17,706 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying task TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, executionId [890939198121574401]
2024-09-24 20:34:17,706 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000} init success
2024-09-24 20:34:17,707 INFO  [s.e.s.t.TransformSeaTunnelTask] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - starting seatunnel transform task, index 0
2024-09-24 20:34:17,707 INFO  [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-20] - [node3]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000} success
2024-09-24 20:34:17,707 INFO  [.a.s.e.s.t.SourceSeaTunnelTask] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - starting seatunnel source task, index 0
2024-09-24 20:34:17,708 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] turned from state DEPLOYING to RUNNING.
2024-09-24 20:34:17,708 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] current state equals target state: RUNNING, skip
2024-09-24 20:34:17,710 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] turned from state DEPLOYING to RUNNING.
2024-09-24 20:34:17,710 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-20] - Job SeaTunnel_Job (890939192865980417) turned from state SCHEDULED to RUNNING.
2024-09-24 20:34:17,808 INFO  [.s.t.SourceSplitEnumeratorTask] [hz.main.seaTunnel.task.thread-35] - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, taskID=60000, index=0}
2024-09-24 20:34:17,809 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-22] - checkpoint is disabled, because in batch mode and 'checkpoint.interval' of env is missing.
2024-09-24 20:34:17,894 INFO  [.s.t.SourceSplitEnumeratorTask] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}] - received enough reader, starting enumerator...
2024-09-24 20:34:17,894 INFO  [.s.s.FileSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}] - Assigned splits to reader [0]
2024-09-24 20:34:17,896 INFO  [.s.s.FileSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}] - SubTask 0 is assigned to [sftp://192.168.137.22:22/uploads/ssss/code.txt]
2024-09-24 20:34:18,388 INFO  [s.c.s.f.s.BaseFileSourceReader] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - Closed the bounded File source
2024-09-24 20:34:18,397 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-22] - wait checkpoint completed: 9223372036854775807
2024-09-24 20:34:19,209 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-22] - pending checkpoint(9223372036854775807/1@890939192865980417) notify finished!
2024-09-24 20:34:19,209 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-22] - start notify checkpoint completed, job id: 890939192865980417, pipeline id: 1, checkpoint id:9223372036854775807
2024-09-24 20:34:19,290 INFO  [o.a.s.c.s.f.s.s.SFTPFileSystem] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - Inside shutdown, con2infoMap size=1
2024-09-24 20:34:19,290 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - [node3]:5801 [seatunnel] [5.1] taskDone, taskId = 60000, taskGroup = TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}
2024-09-24 20:34:19,298 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}] - [node3]:5801 [seatunnel] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}
2024-09-24 20:34:19,299 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}] - [node3]:5801 [seatunnel] [5.1] taskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1} complete with FINISHED
2024-09-24 20:34:19,299 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-33] - [node3]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1} complete with state FINISHED
2024-09-24 20:34:19,299 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-33] - [node3]:5801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=1}, state FINISHED
2024-09-24 20:34:19,300 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-33] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] turned from state RUNNING to FINISHED.
2024-09-24 20:34:19,300 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-33] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] state process is stopped
2024-09-24 20:34:19,300 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-23] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SplitEnumerator (1/1)] future complete with state FINISHED
2024-09-24 20:34:19,309 INFO  [o.a.s.c.s.f.s.s.SFTPFileSystem] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - Inside shutdown, con2infoMap size=0
2024-09-24 20:34:19,309 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - [node3]:5801 [seatunnel] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}] - [node3]:5801 [seatunnel] [5.1] taskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000} complete with FINISHED
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-33] - [node3]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000} complete with state FINISHED
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-33] - [node3]:5801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}, state FINISHED
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-33] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] turned from state RUNNING to FINISHED.
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-33] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] state process is stopped
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-23] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-SFtpFile]-SourceTask (1/1)] future complete with state FINISHED
2024-09-24 20:34:19,310 INFO  [o.a.s.e.s.m.JobMaster         ] [hz.main.seaTunnel.task.thread-33] - release the task group resource TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=50000}
2024-09-24 20:34:19,310 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-23] - received slot release request, jobID: 890939192865980417, slot: SlotProfile{worker=[node3]:5801, slotID=14, ownerJobID=890939192865980417, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='4a3d2e5a-29a8-406b-8f0f-2493284a2a5d'}
2024-09-24 20:34:20,784 INFO  [.c.s.f.h.HadoopFileSystemProxy] [hz.main.generic-operation.thread-21] - rename file :[/uploads/tmp/seatunnel/seatunnel/890939192865980417/9f3595fa4c/T_890939192865980417_9f3595fa4c_0_1] to [/uploads/20240924/cc] finish
2024-09-24 20:34:21,035 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-22] - start clean pending checkpoint cause CheckpointCoordinator completed.
2024-09-24 20:34:21,037 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-22] - Turn checkpoint_state_890939192865980417_1 state from null to FINISHED
2024-09-24 20:34:21,107 INFO  [o.a.s.c.s.f.s.s.SFTPFileSystem] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}] - Inside shutdown, con2infoMap size=0
2024-09-24 20:34:21,107 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}] - [node3]:5801 [seatunnel] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}
2024-09-24 20:34:21,107 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}] - Release classloader for job 890939192865980417 with jars [file:/opt/bigdata/apache-seatunnel-2.3.7/connectors/connector-file-sftp-2.3.8-SNAPSHOT-2.12.15.jar]
2024-09-24 20:34:21,110 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}
2024-09-24 20:34:21,111 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}] - [node3]:5801 [seatunnel] [5.1] taskGroup TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3} complete with FINISHED
2024-09-24 20:34:21,111 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-33] - [node3]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3} complete with state FINISHED
2024-09-24 20:34:21,111 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-33] - [node3]:5801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=890939192865980417, pipelineId=1, taskGroupId=3}, state FINISHED
2024-09-24 20:34:21,112 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-33] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] turned from state RUNNING to FINISHED.
2024-09-24 20:34:21,112 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-33] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] state process is stopped
2024-09-24 20:34:21,112 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-22] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-SftpFile]-AggregatedCommitterTask (1/1)] future complete with state FINISHED
2024-09-24 20:34:21,112 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-22] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] will end with state FINISHED
2024-09-24 20:34:21,112 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-22] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] turned from state RUNNING to FINISHED.
2024-09-24 20:34:21,119 INFO  [o.a.s.e.s.m.JobMaster         ] [seatunnel-coordinator-service-22] - release the pipeline Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] resource
2024-09-24 20:34:21,119 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-29] - received slot release request, jobID: 890939192865980417, slot: SlotProfile{worker=[node3]:5801, slotID=12, ownerJobID=890939192865980417, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='4a3d2e5a-29a8-406b-8f0f-2493284a2a5d'}
2024-09-24 20:34:21,119 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-30] - received slot release request, jobID: 890939192865980417, slot: SlotProfile{worker=[node3]:5801, slotID=13, ownerJobID=890939192865980417, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='4a3d2e5a-29a8-406b-8f0f-2493284a2a5d'}
2024-09-24 20:34:21,119 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-22] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] state process is stop
2024-09-24 20:34:21,119 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-23] - Job SeaTunnel_Job (890939192865980417), Pipeline: [(1/1)] future complete with state FINISHED
2024-09-24 20:34:21,120 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-23] - Job SeaTunnel_Job (890939192865980417) turned from state RUNNING to FINISHED.
2024-09-24 20:34:21,120 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-23] - Job SeaTunnel_Job (890939192865980417) state process is stop
2024-09-24 20:34:21,138 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-0] - [node3]:5801 [seatunnel] [5.1] Connection[id=34, /192.168.137.22:5801->/192.168.137.22:41535, qualifier=null, endpoint=[192.168.137.22]:41535, remoteUuid=0d6771a5-e176-4844-8d3c-99c0e4ae1022, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-09-24 20:34:21,138 INFO  [c.h.c.i.ClientEndpointManager ] [hz.main.event-3] - [node3]:5801 [seatunnel] [5.1] Destroying ClientEndpoint{connection=Connection[id=34, /192.168.137.22:5801->/192.168.137.22:41535, qualifier=null, endpoint=[192.168.137.22]:41535, remoteUuid=0d6771a5-e176-4844-8d3c-99c0e4ae1022, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=0d6771a5-e176-4844-8d3c-99c0e4ae1022, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1727181256209, latest clientAttributes=lastStatisticsCollectionTime=1727181256250,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1727181256194,clientAddress=192.168.137.22,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=4445507584,os.freePhysicalMemorySize=18620403712,os.freeSwapSpaceSize=1879044096,os.maxFileDescriptorCount=4096,os.openFileDescriptorCount=21,os.processCpuTime=4150000000,os.systemLoadAverage=0.03,os.totalPhysicalMemorySize=33529778176,os.totalSwapSpaceSize=1879044096,runtime.availableProcessors=8,runtime.freeMemory=289417384,runtime.maxMemory=477626368,runtime.totalMemory=324534272,runtime.uptime=1180,runtime.usedMemory=35116888, labels=[]}

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

080edeb4a577990c0ba96ba7b49e4ab

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@zhangshenghang
Copy link
Member

Please assign it to me. If this issue really exists, I will fix it

@zhangshenghang
Copy link
Member

@Hisoka-X Please assign it to me, thank you

Please assign it to me. If this issue really exists, I will fix it

@zhangshenghang zhangshenghang mentioned this issue Oct 29, 2024
Merged
4 tasks
@Hisoka-X Hisoka-X linked a pull request Oct 30, 2024 that will close this issue
[Fix][Connector-V2] Fix file binary format sync convert directory to file #7942
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhangshenghang zhangshenghang

Labels
bug help wanted
Projects
None yet
Milestone
No milestone
3 participants
@hailin0 @zhangshenghang @taohaozhi1129