Incorrect status being reported for skipped, pending, failed steps v16.0.3

[richardforjoejnr]

👓 What did you see?

I have "json": {"enabled": true,} and noticed a couple of issues regarding the status values in the json output file which seem incorrect

Wanted to check if these are expected or incorrect

Summary
1] I have a step that fails - steps[0].result.status = 'failed' but in the after block steps[1].result.status = 'skipped'
2] I have a step that is pending - steps[0].result.status = 'failed' ("error_message": "pending") but in the after block steps[1].result.status = 'skipped'
3] I have a step that returns skipped steps[0].result.status = 'passed' and the after block steps[1].result.status = 'passed'
4] I have a step that is undefined - steps[0].result.status = 'passed' but in the after block steps[1].result.status = 'passed'

✅ What did you expect to see?

1] I have a step that fails - steps[0].result.status = 'failed' after block steps[1].result.status = 'failed' - whole scenario failed
2] I have a step that is pending - steps[0].result.status = 'failed' ("error_message": "pending") but in the after block steps[1].result.status = 'failed' or "pending"
3] I have a step that returns skipped steps[0].result.status = 'skipped' and the after block steps[1].result.status = 'skipped'
4] I have a step that is undefined - steps[0].result.status = 'pending' but in the after block steps[1].result.status = 'pending'

📦 Which tool/library version are you using?

"cypress": "^12.16.0", | "@badeball/cypress-cucumber-preprocessor": "16.0.3",

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Install '.cypress": "^12.16.0' using "@badeball/cypress-cucumber-preprocessor": "16.0.3",
2. Create a feature file with the below Feature & step defs
   Feature

Feature: Example feature

  Scenario: Test various statuses - Pass
    Given I have a step that passes

  Scenario: Test various statuses - Fail
    Given I have a step that fails

  Scenario: Test various statuses - Undefined
    Given I have a step that is undefined

  Scenario: Test various statuses - Pending
    Given I have a step that is pending

  Scenario: Test various statuses - skipped
    Given I have a step that returns skipped

  Scenario: Test various statuses - undefined
    Given I have a step that returns undefined

// cypress/support/step_definitions/exampleSteps.js

import { Given } from '@badeball/cypress-cucumber-preprocessor';

Given('I have a step that passes', () => {
  cy.log('This step passes');
});

Given('I have a step that fails', () => {
  cy.log('This step fails');
  expect(true).to.be.false;
});

Given('I have a step that is undefined', () => {
  // Leave this step undefined
});

Given('I have a step that is pending', () => {
  // In Cucumber JS, you can use 'return "pending"' to mark a test as pending,
  // but Cypress doesn't support this directly. You could, for instance,
  // throw a specific error to denote a pending test and then handle this
  // in your test result reporting.
  throw new Error('pending');
});

Given('I have a step that returns pending', () => {
  // In Cucumber JS, you can use 'return "pending"' to mark a test as pending,
  // but Cypress doesn't support this directly. You could, for instance,
  // throw a specific error to denote a pending test and then handle this
  // in your test result reporting.
  return 'pending';
});

Given('I have a step that returns skipped', () => {
  // In Cucumber JS, you can use 'return "pending"' to mark a test as pending,
  // but Cypress doesn't support this directly. You could, for instance,
  // throw a specific error to denote a pending test and then handle this
  // in your test result reporting.
  return 'skipped';
});

Given('I have a step that returns undefined', () => {
  // In Cucumber JS, you can use 'return "pending"' to mark a test as pending,
  // but Cypress doesn't support this directly. You could, for instance,
  // throw a specific error to denote a pending test and then handle this
  // in your test result reporting.
  return 'undefined';
});

4. In your package.json. ensure you have

"cypress-cucumber-preprocessor": {
    "stepDefinitions": "test/e2e/**/*.{js,ts}",
    "commonPath": "test/e2e/commonSteps",
    "json": {
      "enabled": true,
      "output": "reports/cucumber-report.json"
    },...

7. Run cypress and check the json file generated

📚 Any additional context?

Json output

{
    "description": "",
    "elements": [
      {
        "description": "",
        "id": "example-feature;test-various-statuses---pass",
        "keyword": "Scenario",
        "line": 3,
        "name": "Test various statuses - Pass",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 4,
            "name": "I have a step that passes",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "passed",
              "duration": 8000000
            }
          },
          {
            "keyword": "After",
            "hidden": true,
            "result": {
              "status": "passed",
              "duration": 4000000
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      },
      {
        "description": "",
        "id": "example-feature;test-various-statuses---fail",
        "keyword": "Scenario",
        "line": 6,
        "name": "Test various statuses - Fail",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 7,
            "name": "I have a step that fails",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "failed",
              "duration": 0,
              "error_message": "expected true to be false"
            },
            "embeddings": [
              {}
            ]
          },
          {
            "keyword": "After",
            "hidden": true,
            "result": {
              "status": "skipped",
              "duration": 0
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      },
      {
        "description": "",
        "id": "example-feature;test-various-statuses---undefined",
        "keyword": "Scenario",
        "line": 9,
        "name": "Test various statuses - Undefined",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 10,
            "name": "I have a step that is undefined",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "passed",
              "duration": 4000000
            }
          },
          {
            "keyword": "After",
            "hidden": true,
            "result": {
              "status": "passed",
              "duration": 3000000
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      },
      {
        "description": "",
        "id": "example-feature;test-various-statuses---pending",
        "keyword": "Scenario",
        "line": 12,
        "name": "Test various statuses - Pending",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 13,
            "name": "I have a step that is pending",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "failed",
              "duration": 0,
              "error_message": "pending"
            },
            "embeddings": [
              {}
            ]
          },
          {
            "keyword": "After",
            "hidden": true,
            "result": {
              "status": "skipped",
              "duration": 0
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      },
      {
        "description": "",
        "id": "example-feature;test-various-statuses---skipped",
        "keyword": "Scenario",
        "line": 15,
        "name": "Test various statuses - skipped",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 16,
            "name": "I have a step that returns skipped",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "passed",
              "duration": 5000000
            }
          },
          {
            "keyword": "After",
            "hidden": true,
            "result": {
              "status": "passed",
              "duration": 3000000
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      },
      {
        "description": "",
        "id": "example-feature;test-various-statuses---undefined",
        "keyword": "Scenario",
        "line": 18,
        "name": "Test various statuses - undefined",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 19,
            "name": "I have a step that returns undefined",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "passed",
              "duration": 4000000
            }
          },
          {
            "keyword": "After",
            "hidden": true,
            "result": {
              "status": "passed",
              "duration": 4000000
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      }
    ],
    "id": "example-feature",
    "line": 1,
    "keyword": "Feature",
    "name": "Example feature",
    "tags": [],
    "uri": "test/e2e/Example/test.feature"
  }

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[jenisys]

Mmh, your expectations are at least partly wrong (IMHO):

• If a step fails in a Scenario, the remaining steps of this Scenario are normally skipped (but: the scenario.status is still failed). Therefore, the step.status = skipped reflects exactly what occurs internally
• There are some cases (or: cucumber implementations) that can be configured to continue after the first failed step. In these cases, the step.status of the remaining steps after the first failed step are not skipped but are passed or failed whatever occurs when executing these remaining steps.

[richardforjoejnr]

Thanks @jenisys , was hoping to get some of my expectations clarified, as i wasn't 100% sure myself. Regarding skipped. Since the steps are being skipped because the step / scenario failed. Feels like it should be failed, since the reason for it being skipped is because the step failed?

So
Scenario

• step - passed
• step - failed
• step skipped
  => scenario failed

Scenario

• step - skipped
• step - skipped
• step - skipped
  => scenario skipped

[mpkorstanje]

Generally speaking the way to determine the result of a scenario is to take the worst status of its steps.

https://github.com/cucumber/cucumber-jvm/blob/main/cucumber-core/src/main/java/io/cucumber/core/runner/TestCaseState.java#L54C1-L54C1

The order being

    PASSED,
    SKIPPED,
    PENDING,
    UNDEFINED,
    AMBIGUOUS,
    FAILED

https://github.com/cucumber/cucumber-jvm/blob/main/cucumber-java/src/main/java/io/cucumber/java/Status.java

If a scenario has no steps implementations currently vary a bit on what the outcome should be.

cucumber/gherkin#11

[jenisys]

HINT (just that we both have the correct understanding; as @mpkorstanje stated):
The step skipped does not cause the scenario.status = failed but any step.status = failed does (caused by the failing step that occurred first).
The step.status = skipped is just a correlated change caused by the failing step.

[richardforjoejnr]

Yea that sounds right to me. failed is caused by worst status

So even though the last step is skipped, since the worst status is failed from previous steps, final status should be failed, rather than skipped. based on @mpkorstanje ordering

Scenario
step - passed
step - failed
step skipped
=> scenario failed (worst status)

Scenario
step - passed
step - passed
step - skipped
=> scenario skipped (worst status)

Scenario
step - skipped
step - passed
step - pending
=> scenario pending (worst status)

Scenario
step - passed
step - undefined
step - passed
=> scenario undefined (worst status)

[mpkorstanje]

Since you're using @badeball/cypress-cucumber-preprocessor you'll want to take this issue there.

[richardforjoejnr]

Ooops, sorry about that. Accidentally raised it here. Thanks for your feedback