[#5071] Improvement (test): Add integration tests for Trino cascading queries #5073

diqiu50 · 2024-10-09T06:47:16Z

What changes were proposed in this pull request?

Add integration tests for Trino cascading queries

Why are the changes needed?

Fix: #5071

Does this PR introduce any user-facing change?

NO

How was this patch tested?

New integration testers

yuqi1129 · 2024-10-11T07:43:11Z

.github/workflows/trino-integration-test.yml

@@ -76,7 +76,7 @@ jobs:

      - name: Package Gravitino
        run: |
-          ./gradlew compileDistribution -x test -PjdkVersion=${{ matrix.java-version }}
+          ./gradlew build compileDistribution -x test -PjdkVersion=${{ matrix.java-version }}


Why do we need to add the build task? I believe the build task is included in compileDistribution

the compileDistribution task does not build the Trino connector

yuqi1129 · 2024-10-11T07:53:56Z

trino-connector/integration-test/trino-test-tools/trino-cascading-env/init/trino-local/init.sh

+#create test metalake
+counter=0
+while [ $counter -le 30 ]; do
+  sleep 1


Better move this line to L53.

We need to pause for a moment because the Gravitino server has not completed startup.

yuqi1129 · 2024-10-11T08:05:06Z

trino-connector/integration-test/trino-test-tools/trino-cascading-env/init/trino-local/init.sh

+cp -r /opt/gravitino-server /tmp/gravitino-server
+rm -fr /tmp/gravitino-server/logs
+rm -fr /tmp/gravitino-server/data
+/tmp/gravitino-server/bin/gravitino.sh start


When will you stop the Gravitino server? I haven't noticed such logic in the PR.

The Gravitino server was stopped by the container shutdown.

yuqi1129 · 2024-10-11T08:07:38Z

...n-test/trino-test-tools/trino-cascading-env/init/trino-local/config/catalog/trino.properties

+# under the License.
+
+connector.name=trino
+connection-url=jdbc:trino://trino-remote:8080/tpcds


${trino-remote-address}:${trino-remote-port}

That URL is a valid connection-url used by the Docker Compose environment.

yuqi1129 · 2024-10-22T02:29:10Z

.github/workflows/trino-integration-test.yml

@@ -76,7 +76,7 @@ jobs:

      - name: Package Gravitino
        run: |
-          ./gradlew compileDistribution -x test -PjdkVersion=${{ matrix.java-version }}
+          ./gradlew compileDistribution compileTrinoConnector -x test -PjdkVersion=${{ matrix.java-version }}


Is the task compileTrinoConnector NOT included in compileDistribution?

Yes, the compileTrinoConnector task does not include compileDistribution

.github/workflows/trino-integration-test.yml

mchades · 2024-10-28T06:23:40Z

.github/workflows/trino-integration-test.yml

@@ -87,6 +87,7 @@ jobs:
        run: |
          ./gradlew -PskipTests -PtestMode=embedded -PjdkVersion=${{ matrix.java-version }} -PskipDockerTests=false :trino-connector:integration-test:test
          ./gradlew -PskipTests -PtestMode=deploy -PjdkVersion=${{ matrix.java-version }} -PskipDockerTests=false :trino-connector:integration-test:test
+          trino-connector/integration-test/trino-test-tools/run_test.sh


What is the difference between this test script and the above?

The script uses a different test set and testing environment to test the Trino connector. It can also specify a custom test set and testing environment for any test cases. Additionally, the above command can be modified to use the script.

"a different test set" means tpcds and tpch, right? If so, plz add comment on it otherwise we don't know why run the script individually

"a different test set" means the test set for Trino cascading queries. Now the test set only includes TPCH and TCPDS queries.

trino-connector/integration-test/trino-test-tools/run_test.sh

mchades · 2024-10-28T14:13:21Z

trino-connector/integration-test/trino-test-tools/download_jar.sh

+}
+
+# Specific functions for each driver or bundle
+download_mysql_jar() {


Where was this function called?

mchades · 2024-10-28T14:15:29Z

trino-connector/integration-test/trino-test-tools/download_jar.sh

+# Specific functions for each driver or bundle
+download_mysql_jar() {
+  download_jar "mysql-connector-java-8.0.26.jar" \
+  "https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.26/mysql-connector-java-8.0.26.jar" \


Once the version is rectified, we can validate its integrity by verifying the MD5 checksum.

There generally shouldn’t be any issues here. If an issue does occur, the download script will raise an error, exit execution, and output the error message.
Let’s fix it later if any issues arise.

mchades · 2024-10-28T14:19:28Z

trino-connector/integration-test/trino-test-tools/trino-cascading-env/docker-compose.yaml

why not reuse the docker-compose in integration-test-common

integration-test-common is a single Trino cluster environment; it does not test Trino cascading queries. The new Docker Compose setup includes two Trino clusters to test cascading queries in Trino.

… queries (#5073) ### What changes were proposed in this pull request? Add integration tests for Trino cascading queries ### Why are the changes needed? Fix: #5071 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? New integration testers

diqiu50 self-assigned this Oct 9, 2024

diqiu50 changed the title ~~[TEST] Add integration tests for Trino cascading queries~~ [#5071] Improvement (test): Add integration tests for Trino cascading queries Oct 9, 2024

yuqi1129 reviewed Oct 11, 2024

View reviewed changes

diqiu50 requested a review from yuqi1129 October 12, 2024 07:43

yuqi1129 reviewed Oct 22, 2024

View reviewed changes

diqiu50 added 18 commits October 22, 2024 17:06

Add integration test for Trino cascading query

1060a7f

Add a pipeline to trigger the Trino cascading query integration test

b7b0ce1

Fix error for the trino_test.sh

2b70537

Add logic to try create metalake

834bd45

Fix path miss match on the dowload trino cascading connector

47c5b44

Fix error of checking gravitino status

92f1182

Fix error of trino-cascading-env/inspect_ip.sh

eacb79b

Add container logs for Trino cascading query testers

e9a781c

fix ci error

87c5b28

Fix ci error

c46c152

fix ci failed

137480d

Fix ci error

eb5720d

fix ci

b3acc8f

Fix ci error

abbe5a8

Fix ci error

9af0ce4

Fix ci error

445194a

Update trino cascading connector download url

1739e6c

Update for review

4b068d9

diqiu50 force-pushed the t2t-connector-tests branch from 5fe7502 to 4b068d9 Compare October 22, 2024 09:08

diqiu50 added 5 commits October 22, 2024 20:59

fix ci error

636e912

Fix ci error

79f5bcd

Fix ci error

7eba430

fix ci error

d28256d

fix ci error

f194abb

diqiu50 force-pushed the t2t-connector-tests branch from 397a16a to f194abb Compare October 23, 2024 06:45

diqiu50 added 9 commits October 23, 2024 19:57

fix ci error

d76cca2

Add tester trigger

5a27e85

Fix ci error

0cb677c

Fix ci error

93c3fa3

Fix ci error

bc25f0e

Fix ci error

1ea9c15

Fix ci error

8b181ee

Fix ci error

23cf698

Fix ci error

f3428e3

diqiu50 closed this Oct 24, 2024

diqiu50 reopened this Oct 24, 2024

diqiu50 closed this Oct 24, 2024

diqiu50 reopened this Oct 24, 2024

Fix ci error

30de930

mchades reviewed Oct 25, 2024

View reviewed changes

.github/workflows/trino-integration-test.yml Outdated Show resolved Hide resolved

Update for review

1ee9781

mchades reviewed Oct 28, 2024

View reviewed changes

trino-connector/integration-test/trino-test-tools/run_test.sh Show resolved Hide resolved

diqiu50 added 3 commits October 28, 2024 15:42

Add comments for the run_test script

5eece2e

increase timeout of trino integration test

1d6a3c0

Update for review

68b6b7b

mchades reviewed Oct 28, 2024

View reviewed changes

diqiu50 requested a review from mchades October 29, 2024 02:27

mchades approved these changes Oct 29, 2024

View reviewed changes

mchades added the branch-0.7 Automatically cherry-pick commit to branch-0.7 label Oct 29, 2024

mchades merged commit a384cd8 into apache:main Oct 29, 2024
26 checks passed

diqiu50 deleted the t2t-connector-tests branch October 30, 2024 03:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#5071] Improvement (test): Add integration tests for Trino cascading queries #5073

[#5071] Improvement (test): Add integration tests for Trino cascading queries #5073

diqiu50 commented Oct 9, 2024 •

edited

Loading

yuqi1129 Oct 11, 2024

diqiu50 Oct 12, 2024

yuqi1129 Oct 11, 2024

diqiu50 Oct 12, 2024

yuqi1129 Oct 11, 2024

diqiu50 Oct 12, 2024

yuqi1129 Oct 11, 2024

diqiu50 Oct 12, 2024 •

edited

Loading

yuqi1129 Oct 22, 2024

diqiu50 Oct 22, 2024

mchades Oct 28, 2024

diqiu50 Oct 28, 2024

mchades Oct 28, 2024

diqiu50 Oct 28, 2024 •

edited

Loading

mchades Oct 28, 2024

diqiu50 Oct 29, 2024 •

edited

Loading

mchades Oct 28, 2024

diqiu50 Oct 29, 2024

mchades Oct 28, 2024

diqiu50 Oct 29, 2024

[#5071] Improvement (test): Add integration tests for Trino cascading queries #5073

[#5071] Improvement (test): Add integration tests for Trino cascading queries #5073

Conversation

diqiu50 commented Oct 9, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diqiu50 Oct 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diqiu50 Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diqiu50 Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diqiu50 commented Oct 9, 2024 •

edited

Loading

diqiu50 Oct 12, 2024 •

edited

Loading

diqiu50 Oct 28, 2024 •

edited

Loading

diqiu50 Oct 29, 2024 •

edited

Loading