test: create colony flow #3327

olgapod · 2024-10-15T07:33:31Z

Description

Added e2e tests for the Create Colony flow
Set up playwright eslint plugin and dotenv

Testing

Prerequisites: The test script simulates user interactions with the local instance of the web app, which means that the BE is expected to be set up and running. For now, backend services should be started manually and the initial test data should be seeded . Therefore, ensure that the development environment is properly set up by following the instructions in the README file

Setup and start your local dev environment (all BE services should be up and running) npm run dev
Seed the data, if it's not already done before node scripts/create-data.js
If it's the first time you run e2e tests on your machine - run npm e2e:install
run npm run e2e or npm run e2e:ui
Ensure all tests pass

Diffs

New stuff ✨

playwright folder is added to the lint script
added loading of the .env file into playwright environment

rdig · 2024-10-15T08:26:07Z

This shouldn't be based on the master branch, but rather on the playwright changes from #3297

So the base branch should be feat/add-playwright

rdig · 2024-10-16T06:26:11Z

@olgapod please assing yourself to PRs

rdig

All the tests fail for me here, I assume it's something to do with playwright's setup as it seems to not be starting the dev env.

Either that, or your testing instructions are missing a few steps.

Speaking of the above, until everybody gets comfortable with this, please also include the steps to actually set up playwright (install, etc).

Basically what I am saying is be more comprehensive when writing your testing instructions.

olgapod · 2024-10-16T18:44:54Z

@rdig I missed the step to start the dev environment (BE services) before running the tests, added it now

rdig

Ran this with the dev environment up and running and while some initial tests work, most of them fail or time out due to your usage of a hard coded token address.

Token addresses are not static, so you cannot hard code them.

Also, please leave a comment on this PR with what I told you in our private chat, as to why you're not wiring up the dev environment to run automatically, but have to start it separately by hand.

rdig · 2024-10-17T14:55:52Z

package.json

@@ -38,7 +38,7 @@
    "postinstall": "if [ -z \"$SKIP_HOOKS\" ]; then ./scripts/lambda-functions-dependencies.sh && scripts/generate-amplify-local-config.sh; fi",
    "watch-amplify": "node scripts/watchAmplifyFiles",
    "forward-time": "node scripts/forward-time",
-    "playwright:install": "npx playwright install --with-deps",   
+    "playwright:install": "npx playwright install --with-deps",
    "playwright:run": "playwright test",


can you please change this to something like "test" or "execute" or something, just having npm run playright:run is very redundant :)

Not a major gripe but a nitpick here

rdig · 2024-10-17T14:59:55Z

playwright/e2e/create-colony.spec.ts

+test.describe('Create Colony flow', () => {
+  let colonyUrl: string;
+  const colonyName = 'testcolonyname';
+  const validToken = '0xeF841fe1611ce41bFCf0265097EFaf50486F5111';


This won't work. Addresses are not the same between dev environment runs, or even between machines at times.

So this cannot be hardcoded.

That's why this test fails

Interesting problem to solve. Even if we can get away with manually creating a token in the DB for now, will the tests have to interact with the chain like create-data script does?

The proper solution here is to use one of the "verified" tokens we deploy via the create data script and fetch it's address, either from the db, or from the chain directly

thanks, I've used your advice and updated the PR

rdig · 2024-10-17T15:04:05Z

playwright/e2e/create-colony.spec.ts

+      });
+    });
+
+    test('Should render Confirmation step as expected', async ({ page }) => {


This step fails again because of the token that doesn't exist

updated, removed hardcoding the token address

jakubcolony

Great to see a start on the create colony flow, very nice work @olgapod 🎉

Unfortunately I'm also seeing some tests fail:

jakubcolony · 2024-10-17T22:05:20Z

playwright/e2e/create-colony.spec.ts

+test.describe('Create Colony flow', () => {
+  let colonyUrl: string;
+  const colonyName = 'testcolonyname';
+  const validToken = '0xeF841fe1611ce41bFCf0265097EFaf50486F5111';


Interesting problem to solve. Even if we can get away with manually creating a token in the DB for now, will the tests have to interact with the chain like create-data script does?

jakubcolony

Getting a full pass ✅ now, great work Olha 💯

I looked through the test cases and could not see one where a correctly formatted token address is used but it cannot be found in the DB and on chain. Should that be covered as well?

olgapod · 2024-10-18T15:29:58Z

@jakubcolony I'll add that test case in the next PR. Good catch, thanks! Also, please feel free to suggest test cases going forward, it would be helpful as I'm still learning all the features of the app

iamsamgibbs

Great work on this, all working nicely and all tests are passing.

As Jakub mentioned, it would be good to test for invalid but properly formatted addresses.

It might be worth checking src/components/common/Onboarding/wizardSteps/CreateColony/validation.ts for any other relevant validation checks. For example, there is a list of RESERVED_ROUTES and it shouldn't be possible to set your colony url to any of these reserved routes. I think we should get this merged and add these in a later PR though. :)

rdig

This is getting closer. Sorry I keep finding changes and preventing you from merging this already :(

First of, on my machine, tests fail in headless mode, which might be a "me" issue. So we'll need some confirmations from other people

(I'm running this commit)

However, the UI run of the tests seem to be running fine

I've also left some comments on what other checks you need to add to make this PR complete.

Sorry again, for holding this back still.

rdig · 2024-10-21T13:23:04Z

playwright/e2e/create-colony.spec.ts

+      await page.getByLabel(/token symbol/i).fill(tokenSymbol);
+    });
+
+    test('Token Logo file uploader should work correctly', async ({ page }) => {


Please also test going back and forth and ensure the token name, symbol and logo persist

Basically testing the bug you identified here: #3300

added for the token name and symbol. for the logo I will add a test once the bug is fixed

rdig · 2024-10-21T13:24:54Z

playwright/e2e/create-colony.spec.ts

+      );
+    });
+
+    test('Should create a colony and navigate to the Complete Setup step', async ({


At this stage the colony is not yet created, so this description is a bit misleading.

I would actually have this step wait until all transactions are complete and you get the "successfully created colony" welcome modal (although that might be harder)

added this step, although it's quite slow as the transactions and transition to the new colony page take some time

rdig · 2024-10-21T13:27:38Z

playwright/e2e/create-colony.spec.ts

+      ).toBeHidden();
+    });
+
+    test('Should accept existing token and display validation message', async ({


Can we have a test where we also check invalid tokens. It should check:

a bad formatted address

a good formatted address, which is a user address

a good formatted address, which is a colony address

a good formatted address, but one that doesn't exist on the chain

address zero (it's a good formatted address, exists on the chain, but you shouldn't be allowed to add it as a native token)

olgapod · 2024-10-21T15:16:06Z

This is getting closer. Sorry I keep finding changes and preventing you from merging this already :(

First of, on my machine, tests fail in headless mode, which might be a "me" issue. So we'll need some confirmations from other people

(I'm running this commit)

However, the UI run of the tests seem to be running fine

I've also left some comments on what other checks you need to add to make this PR complete.

Sorry again, for holding this back still.

Interesting, Webkit tests are failing on your machine for some reason. To identify if it's your env specific problem or an issue with the tests please try on a clean Playwright example proj by running this command in your terminal npm init playwright mytestproject && cd mytestproject && npx playwright test. This will scaffold a test project (don't forget to remove it afterward) and run tests in headless mode. Let me know if webkit tests are also failing for you there.

rdig · 2024-10-21T15:17:41Z

Interesting, Webkit tests are failing on your machine for some reason. To identify if it's your env specific problem or an issue with the tests please try on a clean Playwright example proj by running this command in your terminal npm init playwright mytestproject && cd mytestproject && npx playwright test. This will scaffold a test project (don't forget to remove it afterward) and run tests in headless mode. Let me know if webkit tests are also failing for you there.

👍

iamsamgibbs

This could just be a me issue, but some of the tests are failing for me after the rebase unfortunately!

jakubcolony

iamsamgibbs · 2024-10-29T08:34:48Z

Just noting this here as I've tried a few times after different commits and have been unable to get all tests to pass. Seems like it could be an issue just for me if they are passing for Jakub. There also isn't any consistency to which tests pass / fail, and I can get most failed tests to pass after retrying them.

Would be helpful for other people to test and confirm if the tests pass for them or not.

mmioana · 2024-10-29T09:41:50Z

playwright/utils/create-colony.ts

I would suggest creating a helper function for getting an input field by label and also add the { force: true } option - without this option I believe the Continue button will not get enabled, but would love to hear your thoughts

Also, can we please create a constants object holding all the text strings that we are using for getting a text field/button etc?

Thanks for your suggestions! @mmioana

I don't think creating such a helper would make the code significantly shorter or easier to read.

Regarding the use of { force: true }, I'm not sure I fully understand the need for it. This option is used to bypass Playwright's actionability checks, like when an element is hidden or disabled. In our case, are we looking to click buttons regardless of their, for example, disabled state?

Creating a constants object to hold all the text strings for selectors - sure we can do this. would be interesting to see what others think about this

rdig · 2024-10-29T12:02:14Z

@olgapod please set up a call w/ me, @iamsamgibbs (and however else is interested) so that we can debug this live, as I'm experiencing the same as Sam, but @jakubcolony 's seem to work.

It's either a OS thing, or a resource thing, but either way the tests seem flaky at the moment.

So let's jump on a call and debug this live, as I want to get this merged as soon as possible

Nortsova · 2024-10-29T12:15:01Z

hey guys, these are errors that I get :)

@olgapod I am also interested in investigating this, please include me in a call as well 🙌

iamsamgibbs

I've ran this 15 or so times now and had it fail only a couple of times. I think it is working consistently enough that we should get this merged and just keep fine tuning as we go as more tests are implemented.

rumzledz

Hey @olgapod ! I had a go at this and tried my best to explain the issue I'm having with one of the tests. But if it consistently passes for everyone else including you, don't mind my comment and I'll just approve it 👌 😄

rumzledz · 2024-10-29T18:47:42Z

playwright/e2e/create-colony.spec.ts

+          for (const value of values) {
+            await inputField.fill(value);
+
+            await expect(page.getByTestId('form-error')).toBeVisible();


Hey @olgapod awesome job on this PR! I ran this locally and the test seems to consistently fail on this step, as in like 10 times in a row:

Looking at the recorded timeline, it looked like it was entering the 21-length string (A.repeat(21)) into the field which is correct and expected but for whatever reason, the input gets cleared afterwards, which causes this test to fail.

I experimented a little bit and found that the problem went away after adding a 100ms timeout to the fill function:

// I really don't know why.... 🤷🏻‍♂️ await inputField.fill(value, { timeout: 100 });

There are however also other instances where this test would just pass without this shenanigan 😅

EDIT: And now it passes 8 out of 10 times. It might just be my machine or something! The new thing that's flaking out is this test:

This might just work properly when run on CI. A good test to monitor but I have no objections on getting it merged!

EDIT:

@olgapod Both tests have passed 5 out of 5 times on my machine without any code changes whatsoever 🚀

rumzledz · 2024-10-29T18:50:10Z

playwright/e2e/create-colony.spec.ts

+
+          for (const value of values) {
+            await inputField.fill(value);
+            await inputField.blur();


I think the test will be a more accurate real-world representation if you didn't blur the field here because our validation is triggered via onChange 👌

iamsamgibbs

Tests seem to be failing more consistently for me this morning unfortunately!

rumzledz · 2024-10-30T09:48:42Z

Tests seem to be failing more consistently for me this morning unfortunately!

Hey @olgapod it seems that CI is having issues with these as well 😕 I thought it was just my machine. I had some luck "fixing" these yesterday on my local, I outlined it on my comment yesterday. No promises it works on CI though, but I guess it's worth a shot? 🤞

olgapod · 2024-10-30T11:57:46Z

Thanks all for your time to debug this. The most frustrating part is that I can't reproduce the failures on any of my three laptops (although all macOS based, but with very different specs). For now, my best idea is to close this PR and break down its changes into smaller ones - one for each step of the flow. I think this will help pinpoint the problem with flakiness

olgapod force-pushed the e2e/create-colony-flow branch from b936e41 to ff1c455 Compare October 15, 2024 09:03

olgapod changed the base branch from master to feat/add-playwright October 15, 2024 13:06

Base automatically changed from feat/add-playwright to master October 15, 2024 13:13

olgapod force-pushed the e2e/create-colony-flow branch 2 times, most recently from ffb0989 to cbdf36a Compare October 15, 2024 18:50

olgapod changed the title ~~test: create colony flow, step 1~~ test: create colony flow Oct 15, 2024

olgapod force-pushed the e2e/create-colony-flow branch from cbdf36a to dcfeb30 Compare October 16, 2024 06:48

olgapod self-assigned this Oct 16, 2024

olgapod marked this pull request as ready for review October 16, 2024 06:49

olgapod requested review from a team as code owners October 16, 2024 06:49

rdig requested changes Oct 16, 2024

View reviewed changes

rdig requested changes Oct 17, 2024

View reviewed changes

jakubcolony requested changes Oct 17, 2024

View reviewed changes

olgapod force-pushed the e2e/create-colony-flow branch 2 times, most recently from 1504d58 to 16d4a44 Compare October 18, 2024 13:42

jakubcolony previously approved these changes Oct 18, 2024

View reviewed changes

olgapod requested a review from rdig October 18, 2024 15:22

iamsamgibbs previously approved these changes Oct 21, 2024

View reviewed changes

rdig requested changes Oct 21, 2024

View reviewed changes

olgapod dismissed stale reviews from iamsamgibbs and jakubcolony via dc61fd1 October 21, 2024 15:40

olgapod force-pushed the e2e/create-colony-flow branch from 343ac17 to c08b5bb Compare October 21, 2024 21:48

olgapod and others added 5 commits October 25, 2024 16:10

test: add test case for final step of create colony flow

6d36f22

test: add nativeToken validation step test cases in create colony flow

512c5f6

test: accept cookie consent banner if it is shown

0015d46

chore: remove console log

efc5b8d

test: update create colony tests to account for dachboard redesign

f7f7f53

olgapod dismissed rdig’s stale review via f7f7f53 October 25, 2024 14:30

olgapod force-pushed the e2e/create-colony-flow branch from de81248 to f7f7f53 Compare October 25, 2024 14:30

olgapod requested a review from rdig October 25, 2024 14:33

iamsamgibbs reviewed Oct 25, 2024

View reviewed changes

test: increase test base timeout for local runs

c364a56

olgapod requested a review from iamsamgibbs October 28, 2024 09:41

olgapod added 3 commits October 28, 2024 12:04

test: generate new create colony url for each test

26d6364

test: disable workers for a single file tests and renamed script name

869ccdd

chore: updated e2e readme file

37b9977

jakubcolony previously approved these changes Oct 28, 2024

View reviewed changes

mmioana reviewed Oct 29, 2024

View reviewed changes

chore: enable trace record in playwright config

18a8ca9

olgapod dismissed jakubcolony’s stale review via 18a8ca9 October 29, 2024 13:15

iamsamgibbs previously approved these changes Oct 29, 2024

View reviewed changes

test: define customColonyUrl variable locally where it's used

2f5a7ce

olgapod dismissed iamsamgibbs’s stale review via 2f5a7ce October 29, 2024 17:24

rumzledz reviewed Oct 29, 2024

View reviewed changes

test: remove redundant actions from fillColonyNameStep helper

3ae7579

iamsamgibbs reviewed Oct 30, 2024

View reviewed changes

olgapod closed this Oct 30, 2024

test: create colony flow #3327

test: create colony flow #3327

Conversation

olgapod commented Oct 15, 2024 • edited Loading

Description

Testing

Diffs

rdig commented Oct 15, 2024

rdig commented Oct 16, 2024

rdig left a comment

Choose a reason for hiding this comment

olgapod commented Oct 16, 2024

rdig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakubcolony left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakubcolony left a comment

Choose a reason for hiding this comment

olgapod commented Oct 18, 2024 • edited Loading

iamsamgibbs left a comment

Choose a reason for hiding this comment

rdig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olgapod Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olgapod commented Oct 21, 2024

rdig commented Oct 21, 2024

iamsamgibbs left a comment

Choose a reason for hiding this comment

jakubcolony left a comment

Choose a reason for hiding this comment

iamsamgibbs commented Oct 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdig commented Oct 29, 2024

Nortsova commented Oct 29, 2024

iamsamgibbs left a comment

Choose a reason for hiding this comment

rumzledz left a comment

Choose a reason for hiding this comment

rumzledz Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iamsamgibbs left a comment

Choose a reason for hiding this comment

rumzledz commented Oct 30, 2024

olgapod commented Oct 30, 2024

olgapod commented Oct 15, 2024 •

edited

Loading

olgapod commented Oct 18, 2024 •

edited

Loading

olgapod Oct 21, 2024 •

edited

Loading

rumzledz Oct 29, 2024 •

edited

Loading