[Feature][Zeta] Support delete logs regularly #7787

corgy-w · 2024-10-07T06:46:28Z

Purpose of this pull request

Delete logs regularly

For details, see logging.md

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

...rver/src/main/java/org/apache/seatunnel/engine/server/telemetry/log/TaskLogCleanService.java

config/seatunnel.yaml

docs/en/seatunnel-engine/logging.md

docs/zh/seatunnel-engine/logging.md

...nector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/joblog/JobLogIT.java

...nnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/seatunnel.yaml

...src/main/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelDomConfigProcessor.java

...mmon/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java

...nector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/joblog/JobLogIT.java

...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java

...rver/src/main/java/org/apache/seatunnel/engine/server/telemetry/log/TaskLogCleanService.java

docs/en/seatunnel-engine/logging.md

hailin0 · 2024-10-11T14:44:50Z

link #7712

dailai · 2024-10-14T00:45:20Z

Thinks @corgy-w. Please fix the ci.

corgy-w · 2024-10-14T03:16:30Z

Thinks @corgy-w. Please fix the ci.

In progress, but I don't know what went wrong. I've been looking for it.

...er/src/main/java/org/apache/seatunnel/engine/server/telemetry/log/TaskLogManagerService.java

docs/en/seatunnel-engine/logging.md

corgy-w · 2024-10-23T10:48:08Z

@dailai @hailin0 @Hisoka-X PTAL

config/seatunnel.yaml

...engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobHistoryService.java

seatunnel-engine/seatunnel-engine-common/pom.xml

...src/main/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelDomConfigProcessor.java

This reverts commit de2bc94.

# Conflicts: # seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java # seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/BaseServlet.java # seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/LogBaseServlet.java

...nnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/seatunnel.yaml

...e/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/DAGUtils.java

Hisoka-X · 2024-10-29T08:18:02Z

...atunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java

+            for (Map.Entry<TaskGroupLocation, CompletableFuture<SlotProfile>> entry :
+                    preApplyResourceFutures.entrySet()) {
+                try {
+                    Address worker = entry.getValue().get().getWorker();
+                    historyExecutionPlan.add(
+                            new ExecutionAddress(worker.getHost(), worker.getPort()));
+
+                } catch (Exception e) {
+                    LOGGER.warning("history execution plan add worker failed", e);
+                }
+            }


cc @zhangshenghang for a look.

Hisoka-X · 2024-10-29T08:18:29Z

...ine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/BaseLogService.java

-                log.warn(String.format("Log file path is empty, get logRef : %s", logRef));
-                return null;
-            }
+            return LogUtil.getLogPath();


cc @zhangshenghang

docs/en/seatunnel-engine/logging.md

Hisoka-X

LGTM if ci passes. Thanks @corgy-w !

hailin0 · 2024-10-30T15:21:25Z

docs/en/seatunnel-engine/logging.md

+    history-job-expire-minutes: 1440
+    telemetry:
+      logs:
+         scheduled-deletion-enable: false


Should this configuration be deleted? Keeping it as part of the job history information may cause disk accumulation.

cc @Hisoka-X

github-actions bot added document Zeta e2e dependencies Pull requests that update a dependency file labels Oct 7, 2024

hailin0 reviewed Oct 8, 2024

View reviewed changes

...rver/src/main/java/org/apache/seatunnel/engine/server/telemetry/log/TaskLogCleanService.java Outdated Show resolved Hide resolved