Custom Tests #148

Luc45 · 2024-03-19T00:13:29Z

Custom Tests

This PR allows Partner developers of the Woo marketplace to submit their own E2E tests to QIT, and to run the E2E tests that we make available to them.

Testing instructions

Spin up the Manager on the custom tests branch
Go to _tests/custom_tests and run composer install
Copy _tests/custom_tests/.env.sample to _tests/custom_tests/.env
Fill out the .env with the necessary values
Run ./vendor/bin/paratest, it should take around 5 minutes and pass
This provides the coverage outlined here

Manual testing instructions

Explore the new commands added, which are:

qit scaffold:e2e
qit run:e2e
qit tag:upload
qit tag:delete
qit tag:list

Scaffold a test, upload it, list it and run it:

qit scaffold:e2e /tmp/foo
qit tag:upload automatewoo /tmp/foo
qit tag:list
qit run:e2e automatewoo --plugin woocommerce:activate

Publish another test tag

Now scaffold another test and publish it for automatewoo-birthdays
Run it with qit run:e2e automatewoo-birthdays --plugin woocommerce:activate --plugin automatewoo:test
You can also do this using a qit-env.yml or qit-env.json files

There are many options, try out with --php_version 8.3, different WP and Woo versions, object cache, etc.

Also try out --codegen and --ui.

The self-tests is pretty self-explanatory too, if you're interested. Check this out for instance:

	public function test_runs_scaffolded_e2e() {
		$output = qit( [
				'run:e2e',
				'automatewoo',
				$this->scaffold_test(),
				'--plugin',
				'woocommerce:activate',
			]
		);

		$output = $this->normalize_scaffolded_test_run_output( $output );

		$this->assertMatchesNormalizedSnapshot( $output );
	}

Documentation:

The documentation contains example commands that we can use to try more scenarios: https://qit.woo.com/docs/custom-tests/introduction

This reverts commit 8a1c83f.

rcstr · 2024-05-06T20:36:43Z

Hi @Luc45 - I have tested both PRs and noticed some issues while running the different commands:

`./vendor/bin/paratest` 🔴

ParaTest v7.4.3 upon PHPUnit 11.1.3 by Sebastian Bergmann and contributors.

Processes:     12
Runtime:       PHP 8.3.0
Configuration: /Users/rommel/Projects/qit-cli/_tests/custom_tests/phpunit.xml.dist

..F..FFFFFFFFFF............S..                                    30 / 30 (100%)

Time: 06:09.393, Memory: 10.00 MB

There were 2 PHPUnit test runner warnings:

1) Exception in third-party event subscriber: Failed to remove directory "/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/tmp_qit_config-qit_custom_tests_66393a99f0ab5": rmdir(/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/._0ED): Directory not empty
#0 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/symfony/filesystem/Filesystem.php(153): Symfony\Component\Filesystem\Filesystem::doRemove(Array, false)
#1 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(274): Symfony\Component\Filesystem\Filesystem->remove(Array)
#2 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(237): QITTestFinish::delete_temp_environment()
#3 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DirectDispatcher.php(102): QITTestFinish->notify(Object(PHPUnit\Event\TestRunner\ExecutionFinished))
#4 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DeferringDispatcher.php(45): PHPUnit\Event\DirectDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished))
#5 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Emitter/DispatchingEmitter.php(1143): PHPUnit\Event\DeferringDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished))
#6 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/src/WrapperRunner/ApplicationForWrapperWorker.php(224): PHPUnit\Event\DispatchingEmitter->testRunnerExecutionFinished()
#7 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(70): ParaTest\WrapperRunner\ApplicationForWrapperWorker->end()
#8 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(80): {closure}()
#9 {main}

2) Exception in third-party event subscriber: Failed to remove directory "/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/tmp_qit_config-qit_custom_tests_66393a99f183f": rmdir(/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/._YUT): Directory not empty
#0 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/symfony/filesystem/Filesystem.php(153): Symfony\Component\Filesystem\Filesystem::doRemove(Array, false)
#1 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(274): Symfony\Component\Filesystem\Filesystem->remove(Array)
#2 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(237): QITTestFinish::delete_temp_environment()
#3 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DirectDispatcher.php(102): QITTestFinish->notify(Object(PHPUnit\Event\TestRunner\ExecutionFinished))
#4 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DeferringDispatcher.php(45): PHPUnit\Event\DirectDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished))
#5 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Emitter/DispatchingEmitter.php(1143): PHPUnit\Event\DeferringDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished))
#6 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/src/WrapperRunner/ApplicationForWrapperWorker.php(224): PHPUnit\Event\DispatchingEmitter->testRunnerExecutionFinished()
#7 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(70): ParaTest\WrapperRunner\ApplicationForWrapperWorker->end()
#8 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(80): {closure}()
#9 {main}

--

There were 11 failures:

1) CompatibilityTest::test_sut_and_activate_additional
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 1 test using 1 worker\n
 [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n
 1 passed (TIME)\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:22

2) CompatibilityTest::test_sut_and_bootstrap_additional
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 1 test using 1 worker\n
 [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n
 1 passed (TIME)\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:49

3) CompatibilityTest::test_sut_and_test_additional
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n
 Exporting DB\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 3 tests using 1 worker\n
 [1/3] [woocommerce-self-test-sut-and-test-additional] › woocommerce/self-test-sut-and-test-additional/example.spec.js:9:5 › I can see my plugin menu\n
 [2/3] [db-import] › db-import.js:5:5 › Database Import\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:76

4) CompatibilityTest::test_multiple_tags_and_multiple_plugins_with_multiple_tags
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-multiple-test-tags-another/bootstrap/bootstrap.sh\n
 Exporting DB\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 7 tests using 1 worker\n
 [1/7] [woocommerce-self-test-multiple-test-tags] › woocommerce/self-test-multiple-test-tags/example.spec.js:9:5 › I can see my plugin menu\n
 [2/7] [db-import] › db-import.js:5:5 › Database Import\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:118

5) RunE2ETest::test_runs_scaffolded_e2e
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 1 test using 1 worker\n
 [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n
 1 passed (TIME)\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:37

6) RunE2ETest::test_tag_and_run_test
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-tag-and-run/bootstrap/bootstrap.php\n
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-tag-and-run/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 1 test using 1 worker\n
 [1/1] [automatewoo-self-test-tag-and-run] › automatewoo/self-test-tag-and-run/example.spec.js:9:5 › I can see my plugin menu\n
 1 passed (TIME)\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:58

7) RunE2ETest::test_multiple_tags_and_run_tests
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-multiple-test-tags-another/bootstrap/bootstrap.sh\n
 Exporting DB\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 3 tests using 1 worker\n
 [1/3] [automatewoo-self-test-multiple-test-tags] › automatewoo/self-test-multiple-test-tags/example.spec.js:9:5 › I can see my plugin menu\n
 [2/3] [db-import] › db-import.js:5:5 › Database Import\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:86

8) RunE2ETest::test_theme_as_sut
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.php\n
 Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 2 tests using 1 worker\n
 [1/2] [deli-local] › deli/local/activate-theme.spec.js:4:5 › I can activate Deli\n
 [2/2] [deli-local] › deli/local/example.spec.js:9:5 › I can see my plugin menu\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:124

9) RunE2ETest::test_run_with_snapshot
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.php\n
 Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 2 tests using 1 worker\n
 [1/2] [deli-local] › deli/local/activate-theme.spec.js:4:5 › I can activate Deli\n
 [deli-local] › deli/local/activate-theme.spec.js:4:5 › I can activate Deli\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:164

10) RunE2ETest::test_playwright_config_override
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n
 Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n
 Running E2E Tests\n
+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n
 Running 1 test using 1 worker\n
 [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n
 Slow test file: [automatewoo-local] › automatewoo/local/example.spec.js (<time>s)\n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:205

11) ScaffoldTest::test_scaffold_e2e
Failed asserting that two strings are equal.

Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments.
--- Expected
+++ Actual
@@ @@
-'/tmp-normalized/qit_scaffolded_e2e-NORMALIZED_ID/bootstrap/bootstrap.php: |\n
+'/tmp-normalized/qit_scaffolded_e2e-66393ac232e17/bootstrap/bootstrap.php: |\n
     <?php\n
     /*\n
      * Bootstrap File (Optional)\n
@@ @@
      *\n
      * Documentation: For detailed usage instructions, visit https://qit.woo.com/docs/\n
      */\n
-/tmp-normalized/qit_scaffolded_e2e-NORMALIZED_ID/bootstrap/bootstrap.sh: |-\n
+/tmp-normalized/qit_scaffolded_e2e-66393ac232e17/bootstrap/bootstrap.sh: |-\n
     #!/bin/bash\n
 \n
     # Bootstrap Shell Script (Optional)\n
@@ @@
     # Note: Delete this file if it's not required for your setup.\n
     #\n
     # Documentation: Detailed instructions available at https://qit.woo.com/docs/\n
-/tmp-normalized/qit_scaffolded_e2e-NORMALIZED_ID/bootstrap/mu-plugin.php: |\n
+/tmp-normalized/qit_scaffolded_e2e-66393ac232e17/bootstrap/mu-plugin.php: |\n
     <?php\n
     /*\n
      * mu-plugin File (Optional)\n
@@ @@
      *\n
      * Documentation: For more information, please visit https://qit.woo.com/docs/\n
      */\n
-/tmp-normalized/qit_scaffolded_e2e-NORMALIZED_ID/example.spec.js: |-\n
+/tmp-normalized/qit_scaffolded_e2e-66393ac232e17/example.spec.js: |-\n
     /*\n
      * This is an example E2E test. You can write your own tests, or generate them with Codegen.\n
      * \n

/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/ObjectDriver.php:44
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199
/Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:68
/Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/ScaffoldTest.php:41

FAILURES!
Tests: 30, Assertions: 42, Failures: 11, Warnings: 2, Skipped: 1.

`./qit scaffold:e2e /tmp/foo` ✅

Example E2E test generated in: /tmp/foo/
You can now run your first test with qit run:e2e <your_slug> "/tmp/foo/" --ui
You can start writing your tests with codegen: qit run:e2e --codegen
And when you are ready, you can publish your tests with qit test-tags:upload <your_slug> <path_to_test>
Read more about it on our documentation: https://qit.woo.com/docs/

`./qit tag:upload automatewoo /tmp/foo` 🔴

Uploading zip...
 0/1 [>---------------------------]   0%
In RequestBuilder.php line 182:
                                                                        
  Error: No route was found matching the URL and request method. (404)  
                                                                        

tag:upload <test_tag> <test_path> [<test_type>]

`./qit tag:list` 🟡

Command displayed the list at the end, but before printed many instances of the warning:

PHP Warning:  Undefined array key "type" in phar:///Users/rommel/Projects/qit-cli/qit/src/Commands/Tags/ListTestTagsCommand.php on line 46

Warning: Undefined array key "type" in phar:///Users/rommel/Projects/qit-cli/qit/src/Commands/Tags/ListTestTagsCommand.php on line 46
PHP Warning:  Undefined array key "type" in phar:///Users/rommel/Projects/qit-cli/qit/src/Commands/Tags/ListTestTagsCommand.php on line 46

`./qit run:e2e automatewoo --plugin woocommerce:activate` 🔴

                                     
  Command "run:e2e" is not defined.  
                                     
  Did you mean one of these?         
      run:activation                 
      run:compatibility              
      run:malware                    
      run:phpcompatibility           
      run:phpstan                    
      run:security                   
      run:woo-api                    
      run:woo-e2e                    
      scaffold:e2e

I checked via ./qit and "run:e2e is not in the list of commands

Luc45 · 2024-05-06T20:53:00Z

Howdy @rcstr , can you please confirm that your local Manager is in the correct branch (the counterpart of this one on the Manager repo)? And when you do your manual test where run:e2e is not available, are you connected to the Local instance of that Manager?

Luc45 · 2024-05-06T21:07:49Z

For the self-tests part (not manual testing), I can see some things:

Failed to remove directory "/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/tmp_qit_config-qit_custom_tests_66393a99f183f": rmdir(/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/._YUT): Directory not empty

This is probably a racing condition where we invoke the method to delete the tmp directory (recursively), but ._YUT still exists, which seems to be resource forks from macOS. Reference 1, Reference 2

macOS stores metadata and resource forks using a technique called AppleDouble, in which the data fork is written as one file, and the resource fork and metadata are written as an entirely separate file preceded by a "._" naming convention. For example: ExampleFile.psd would contain the data fork, and ._ExampleFile.psd would contain the resource fork and metadata.

I think this only happens on the context of self-tests. I'll add a small workaround (probably a 250ms wait before deleting) to see if it addresses it in the context of self-tests.

I won't change anything in the actual production flow, so if there's a bug there about this it will continue to happen on Mac and we can catch it.

As for this failure:

+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n

From the snapshot tests it seems your instance of Docker adds some unexpected output to the terminal, which is nice to catch. I'll add a DOCKER_CLI_HINTS=false env var before running things to normalize this output.

zhongruige

Awesome work on this @Luc45!

Luc45 added 30 commits March 20, 2024 13:38

Use latest PW

f8803ba

Codegen is bootstrap only

eca6872

Throw if no tests

9e03245

Increase timeout for auth

431a711

Re-order options

fca588b

Add debug output to environment downloader, sync on env:up

ae40a1c

Run all PW tests in one run for centralized reporting

78aa38f

Run all PW tests in one run for centralized reporting, fix circular dep

21b0bd8

build

1284711

Add some helpful client-side validation when uploading test from dir

9db91ed

Tests

63007f0

Reduce verbosity on codegen command

fbf1395

Codegen output

e9481a2

Use our own Docker images with FixUID

e918d2e

FixUID

73a5e0c

Warning

80d0542

Allow compatibility to also be comma-separated slugs (eg: to skip Core)

65f156a

Tweaks

09243f4

Improvements to reporting, etc

d374523

Tests, etc

c3cbe53

typo

2d4f933

woocommerce_version RC

8281412

Delete dangling without asking

043a968

PHPCS, etc

756907a

Improve pasting on auth (was hard on Windows)

905cc59

tests

f69adf8

Move class

5d2e575

Pull zip image

adb2951

Pull docker images and inform user

11787a1

Tweak test

c424a20

Revert "Try with paratest again"

14769ae

This reverts commit 8a1c83f.

Luc45 marked this pull request as ready for review May 3, 2024 20:54

Luc45 self-assigned this May 3, 2024

Luc45 requested a review from a team May 3, 2024 21:10

Luc45 added 2 commits May 3, 2024 18:19

Test self-tests on Staging

b7e8bb8

Update syntax for features

c2846a3

Luc45 added 6 commits May 6, 2024 18:19

Disable DOCKER_CLI_HINTS

f12326d

Update how we normalize the path after scaffolding in self-tests

015ce48

build

158ca72

Update snapshots

71923d9

Trigger CI again

838bd55

Retry deleting after waiting just in case

a9cd10b

zhongruige approved these changes May 7, 2024

View reviewed changes

Luc45 added 12 commits May 8, 2024 13:33

CR Fixes

850eb0c

Improved listing environments

4e3645f

CR: Update doc links

1884108

Improve message

e58c98f

Static tests

e46d238

Update snapshots

71cf2be

Merge branch 'trunk' into 24-03/custom-tests-3

e21c9eb

Tweak error message

9d76f30

Tests

a30c3bf

Merge branch 'trunk' into 24-03/custom-tests-3

6ab1df3

PHPCompatibility snapshot

9c6df5a

Re-add all workflows

9b023a7

Luc45 merged commit e1c16a3 into trunk May 10, 2024

Luc45 deleted the 24-03/custom-tests-3 branch May 10, 2024 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom Tests #148

Custom Tests #148

Luc45 commented Mar 19, 2024 •

edited

Loading

rcstr commented May 6, 2024

Luc45 commented May 6, 2024 •

edited

Loading

Luc45 commented May 6, 2024 •

edited

Loading

zhongruige left a comment

Custom Tests #148

Custom Tests #148

Conversation

Luc45 commented Mar 19, 2024 • edited Loading

Custom Tests

Testing instructions

Manual testing instructions

Scaffold a test, upload it, list it and run it:

Publish another test tag

Documentation:

rcstr commented May 6, 2024

./vendor/bin/paratest 🔴

./qit scaffold:e2e /tmp/foo ✅

./qit tag:upload automatewoo /tmp/foo 🔴

./qit tag:list 🟡

./qit run:e2e automatewoo --plugin woocommerce:activate 🔴

Luc45 commented May 6, 2024 • edited Loading

Luc45 commented May 6, 2024 • edited Loading

zhongruige left a comment

Choose a reason for hiding this comment

Luc45 commented Mar 19, 2024 •

edited

Loading

`./vendor/bin/paratest` 🔴

`./qit scaffold:e2e /tmp/foo` ✅

`./qit tag:upload automatewoo /tmp/foo` 🔴

`./qit tag:list` 🟡

`./qit run:e2e automatewoo --plugin woocommerce:activate` 🔴

Luc45 commented May 6, 2024 •

edited

Loading

Luc45 commented May 6, 2024 •

edited

Loading