Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression tests for double suspense/double resource fetch #3103

Merged
merged 9 commits into from
Oct 25, 2024

Conversation

metatoaster
Copy link
Contributor

@metatoaster metatoaster commented Oct 12, 2024

As per advice, this has been added to the existing suspense_tests in examples, but given the complexity it's added as a new standalone module that exports the InstrumentedRoutes for inclusion in some other App. During this initial start I've noted some bugs with [aria-current] for CSR (which affects the <nav> towards the subpaths, and this is observed with the existing tests), and that the A hrefs don't seem to format properly as there seem to be subtle differences between actix and axum integration. I think the new test may benefit from gating behind feature flags to flip between them, and potentially the SsrMode as documented in a comment in the new module.

These tests target the issues originally reported in #2937, #2956 and #2961.

Tasks:

  • Set up the original components with identifiers for selection and instrument critical paths with counters.
  • Provide the base set of fixtures for the scenarios.
  • Show that the basic scenarios work.
  • Create the rest of the scenarios that previously failed.
  • Verify that the scenarios that should pass now will fail against builds against older versions.

@metatoaster metatoaster force-pushed the regression_test_2937_2956_2961 branch 3 times, most recently from 128cb1d to 87c8b65 Compare October 12, 2024 12:38
@metatoaster

This comment was marked as outdated.

@metatoaster
Copy link
Contributor Author

metatoaster commented Oct 14, 2024

I've now verified that the current test cases will fail as expected for past commits, so I am happy for this to be reviewed and merged.

That said, I think d3c3b22 should be omitted from the squash, assuming if that's going to be the case (I generally try to make good commits but I can understand having testing related commits be squashed to make a more concise history). Reason is that those tests do function as a standalone set of documentation showing the difference between the effects of CSR and Hydrated rendering on subsequent resource fetches, and if this discrepancy should be corrected later the specific commit highlighting the fact may be useful.

(Edit: I did a subsequent force push because I included a baseline example that I should have included in an earlier commit; correspondingly above commit id is updated.)

Moreover, if the [aria-current] fixes for the <A/> components are to be added, I am sure the testing for those can potentially fit in this instrumented example I've added (despite the fact that they are not suspense related, if you run the examples you will see that the CSR highlighting do not work, and that I used <a/> instead because they were buggy and in the way of the actual testing I wanted to do here).

For posterity, the following are the failures against specific key historic commits related to this issue from the past:

Test failures against 7831e4a (assumed to be beta4, with the original flaws in the reactive system that resulted in the duplicate excessive resource fetch; suspense tests were removed as not relevant yet)
Feature: Using instrumented counters for real
  Scenario: Emulate steps 1 to 5 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 0 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:18:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        5,
      >        2,
           ),
           (
               "inspect_item_root",
               0,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 6 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 41#
   ✔  And I refresh the page
   ✔  When I select the following links
       | Target 4## |
       | Target 42# |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 2 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:32:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        4,
      >        1,
           ),
           (
               "inspect_item_root",
      <        6,
      >        2,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 7 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 42#
   ✔  And I refresh the page
   ✔  When I select the following links
       | Target 4## |
       | Target 42# |
       | Target 41# |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 3 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:47:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        6,
      >        1,
           ),
           (
               "inspect_item_root",
      <        10,
      >        3,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 8, "not trigger double fetch".
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:62:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        9,
      >        2,
           ),
           (
               "inspect_item_root",
      <        2,
      >        1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Like above, for the "double fetch" which shouldn't happen
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
       | Target 3##   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 3 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:78:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        13,
      >        3,
           ),
           (
               "inspect_item_root",
      <        3,
      >        1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Like above, but using 4## instead
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
       | Target 4##   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:94:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        15,
      >        2,
           ),
           (
               "inspect_item_root",
      <        3,
      >        1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate part of step 8 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the link Item Listing
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 0 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:117:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        1,
           ),
           (
               "inspect_item_root",
               0,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate above, instead of refresh page, reset csr counters
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I click on Reset CSR Counters
   ✔  When I select the link Item Listing
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 0 |
       | inspect_item_root  | 0 |
       | inspect_item_field | 0 |
  Scenario: Start with hydration from Target 41# and go up
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 41#
   ✔  And I refresh the page
   ✔  When I select the link Target 4##
   ✔  And I select the link Item Listing
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:147:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        5,
      >        1,
           ),
           (
               "inspect_item_root",
      <        4,
      >        1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Start with hydration from Target 41# and go up
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 41#
   ✔  And I click on Reset CSR Counters
   ✔  When I select the link Target 4##
   ✔  And I select the link Item Listing
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 0 |
       | inspect_item_root  | 0 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:160:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        2,
      >        0,
           ),
           (
               "inspect_item_root",
      <        2,
      >        0,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
Test failures against 827cc0b (which demonstrates that the fix to the resource tracking drastically improved - these are mixed in synthetically as the Suspend test failures happen before and are included for comparison)
Feature: Using instrumented counters for real
  Scenario: Emulate steps 1 to 5 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 2 |
       | item_inspect  | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:18:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "item_listing",
               1,
           ),
           (
               "item_overview",
      <        3,
      >        2,
           ),
           (
               "item_inspect",
               0,
           ),
       ]
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 0 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:18:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        2,
           ),
           (
               "inspect_item_root",
               0,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
 
  Scenario: Emulate step 6 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 41#
   ✔  And I refresh the page
   ✔  When I select the following links
       | Target 4## |
       | Target 42# |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 0 |
       | item_overview | 1 |
       | item_inspect  | 2 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:37:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "item_listing",
               0,
           ),
           (
               "item_overview",
      <        2,
      >        1,
           ),
           (
               "item_inspect",
      <        3,
      >        2,
           ),
       ]
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 2 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:32:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
               1,
           ),
           (
               "inspect_item_root",
      <        3,
      >        2,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 7 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 42#
   ✔  And I refresh the page
   ✔  When I select the following links
       | Target 4## |
       | Target 42# |
       | Target 41# |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 0 |
       | item_overview | 1 |
       | item_inspect  | 3 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:57:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "item_listing",
               0,
           ),
           (
               "item_overview",
      <        2,
      >        1,
           ),
           (
               "item_inspect",
      <        4,
      >        3,
           ),
       ]
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 3 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:47:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
               1,
           ),
           (
               "inspect_item_root",
      <        4,
      >        3,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
     Scenario: Emulate step 8, "not trigger double fetch".
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 2 |
       | item_inspect  | 1 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:77:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "item_listing",
               1,
           ),
           (
               "item_overview",
      <        3,
      >        2,
           ),
           (
               "item_inspect",
      <        2,
      >        1,
           ),
       ]
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:62:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        2,
           ),
           (
               "inspect_item_root",
               1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
            
  Scenario: Like above, for the "double fetch" which shouldn't happen
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
       | Target 3##   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 3 |
       | item_inspect  | 1 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:98:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "item_listing",
               1,
           ),
           (
               "item_overview",
      <        5,
      >        3,
           ),
           (
               "item_inspect",
      <        2,
      >        1,
           ),
       ]
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 3 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:78:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        5,
      >        3,
           ),
           (
               "inspect_item_root",
               1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Like above, but using 4## instead
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
       | Target 4##   |
   ✔  And I go check the Counters
   ✘  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 3 |
       | item_inspect  | 1 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:119:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "item_listing",
               1,
           ),
           (
               "item_overview",
      <        4,
      >        3,
           ),
           (
               "item_inspect",
      <        2,
      >        1,
           ),
       ]
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:94:9
      Matched: tests/fixtures/world/check_steps.rs:83:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        2,
           ),
           (
               "inspect_item_root",
               1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
Test failures against 7c0889e (which fixed the double suspense, but naturally didn't include the Param related fix so Resource depending on Params remained broken)
Feature: Using instrumented counters for real
  Scenario: Emulate steps 1 to 5 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 2 |
       | item_inspect  | 0 |
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 0 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:23:9
      Matched: tests/fixtures/world/check_steps.rs:84:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        2,
           ),
           (
               "inspect_item_root",
               0,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 6 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 41#
   ✔  And I refresh the page
   ✔  When I select the following links
       | Target 4## |
       | Target 42# |
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 0 |
       | item_overview | 1 |
       | item_inspect  | 2 |
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 2 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:42:9
      Matched: tests/fixtures/world/check_steps.rs:84:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
               1,
           ),
           (
               "inspect_item_root",
      <        3,
      >        2,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 7 of issue #2961
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 42#
   ✔  And I refresh the page
   ✔  When I select the following links
       | Target 4## |
       | Target 42# |
       | Target 41# |
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 0 |
       | item_overview | 1 |
       | item_inspect  | 3 |
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 1 |
       | inspect_item_root  | 3 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:62:9
      Matched: tests/fixtures/world/check_steps.rs:84:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
               1,
           ),
           (
               "inspect_item_root",
      <        4,
      >        3,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Emulate step 8, "not trigger double fetch".
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 2 |
       | item_inspect  | 1 |
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:82:9
      Matched: tests/fixtures/world/check_steps.rs:84:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        2,
           ),
           (
               "inspect_item_root",
               1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Like above, for the "double fetch" which shouldn't happen
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
       | Target 3##   |
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 3 |
       | item_inspect  | 1 |
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 3 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:103:9
      Matched: tests/fixtures/world/check_steps.rs:84:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        5,
      >        3,
           ),
           (
               "inspect_item_root",
               1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      
  Scenario: Like above, but using 4## instead
   ✔> Given I see the app
   ✔> And I select the mode Instrumented
   ✔  Given I select the link Target 3##
   ✔  And I refresh the page
   ✔  When I select the following links
       | Item Listing |
       | Target 4##   |
       | Target 41#   |
       | Target 4##   |
   ✔  And I go check the Counters
   ✔  Then I see the following counters under section
       | Suspend Calls |   |
       | item_listing  | 1 |
       | item_overview | 3 |
       | item_inspect  | 1 |
   ✘  And the following counters under section
       | Server Calls (CSR) |   |
       | list_items         | 0 |
       | get_item           | 2 |
       | inspect_item_root  | 1 |
       | inspect_item_field | 0 |
      Step failed:
      Defined: features/check_instrumented_suspense_resource.feature:124:9
      Matched: tests/fixtures/world/check_steps.rs:84:1
      Step panicked. Captured output: assertion failed: `(left == right)`
      
      Diff < left / right > :
       [
           (
               "list_items",
               0,
           ),
           (
               "get_item",
      <        3,
      >        2,
           ),
           (
               "inspect_item_root",
               1,
           ),
           (
               "inspect_item_field",
               0,
           ),
       ]
      

So yeah, the tests are shown to work retroactively, while also showing the desired improvements as the issues were corrected.

@metatoaster metatoaster marked this pull request as ready for review October 14, 2024 10:47
- Based on initial concepts developed for reproducing leptos-rs#2937 and others,
  streamlined and instrumented for e2e testing and refined for inclusion
  as a standalone module to be plugged into some other App.
- Instead of using examples, just feed it the table because examples
  will rerun the whole scenario from scratch, which isn't what we are
  trying to test here.
- Provide a basic example with item listing to show how this will work.
- Keep all SSR calls on ticket 0 as a means to disambiguate them from
  CSR calls.  For the mean time the focus of tests isn't on that
  behavior but this may be modified when suitable.
- Given the new understanding, the scenerios all being the baseline
  tests they are now moved into one file.
- Have the checks against all calls at once for better diff output,
  and reword the new scenerios into more idomatic gherkin.
- Streamline the steps and provide additional ones that will help with
  feature definitions.
- Done by providing a button directly on the top level component with
  the navigation footer.  This will be useful for the next test.
- Specifically, under hydrated load, resources that shouldn't need
  refetching gets refetched, while CSR does not show this issue.
@gbj
Copy link
Collaborator

gbj commented Oct 25, 2024

Thanks very much for doing the work to add these, and to test the tests against the past examples! I really appreciate it.

@gbj gbj merged commit 0e73d18 into leptos-rs:main Oct 25, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants