Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URI / Location values in JSON 'can not be determined with webdriver.io' ? #60

Open
raumB opened this issue Jul 9, 2021 · 29 comments
Open

Comments

@raumB
Copy link

raumB commented Jul 9, 2021

Describe your question with as much detail as possible
Hello, being not a specialist in this subject I just wanted to understand the reason for https://github.com/webdriverio-community/wdio-cucumberjs-json-reporter/blob/v6/lib/reporter.js#L238 and https://github.com/webdriverio-community/wdio-cucumberjs-json-reporter/blob/v6/lib/reporter.js#L319. Is this due to some principle technical restriction and it never would be possible to have the URI / location parameters implemented? Thanks and kind regards.

If it's about a specific piece of code, try and include some of it to support your question.

            uri: 'Can not be determined',

(https://github.com/webdriverio-community/wdio-cucumberjs-json-reporter/blob/v6/lib/reporter.js#L238)

                location: 'can not be determined with webdriver.io'

(https://github.com/webdriverio-community/wdio-cucumberjs-json-reporter/blob/v6/lib/reporter.js#L319)

Environment (please complete the following information):

  • Node.js version: 10.16.3
  • NPM version:
  • webdriver.io version: ~6.11.x
  • @wdio/cucumber-framework version: ~6.11.1
  • wdio-cucumberjs-json-reporter version: ~2.0.2

Additional context
Cucumber reporters for other technologies seem to to offer these parameters which could be used for forward navigation to the test case.

@wswebcreation
Copy link
Collaborator

Hi @raumB

WebdriverIO behaves in a different way regarding Cucumber reports in comparison to other frameworks that use CucumberJS. WebdriverIO has its own way of reporting and this module is translating that way back into JSON.
During the time I created this module there was no way to determine that, maybe there is now. That's why I added a label for people to help with this.

@christian-bromann
Copy link
Contributor

This data should be available now as we updated the payload information with the last Cucumber release. @raumB you want to give us a hand and help us fix this?

@raumB
Copy link
Author

raumB commented Jul 12, 2021

Thanks a lot for the updates. I understand that in principle an enhancement with regard to the mentioned properties would be possible. With which exact versions of WDIO and Cucumber such an enhancement would work together? I'm asking since we are still using WDIO v6 for some other reason. Maybe we would need to go to v7 and maybe also switch in parallel to higher versions of other components? Thanks and kind regards.

@christian-bromann
Copy link
Contributor

@raumB I would recommend to switch to v7 anyway. If you aren't a TypeScript user this update should be fairly simple. See more on this in our migration guide. Then we can fix this and release a new version of the reporter that you can then use in your project. Happy to help in this process.

@raumB
Copy link
Author

raumB commented Jul 13, 2021

We plan to evaluate v7 for our test setup.

@raumB
Copy link
Author

raumB commented Jul 23, 2021

@christian-bromann @wswebcreation
In an evaluation project we have now managed to migrate from WDIO v6 to v7 and would be able to test on v7 level.
To our understanding the URI parameter should contain some link or path to the corresponding feature file and the location parameter should contain some indicator for the specific corresponding test (for example, a line number from the feature file).
I assume there are some standards around about syntax and format of the cucumber.json file and what I'm suggesting for the two parameters should be checked against their common use in that file.
I myself could not contribute when it comes down to development activities, but I could try to test things, of course.
Thanks and kind regards.

@wswebcreation
Copy link
Collaborator

I've checked the output but the WDIO reporter is not giving back this information for the reporter. Will keep this one open, but this might first need to be fixed in the Cucumber framework adapter

@raumB
Copy link
Author

raumB commented Oct 4, 2021

Does also the WDIO v7 reporter does not give back this information?

@wswebcreation
Copy link
Collaborator

Nope, this is not given back, but maybe something Cucumber is giving back, but not passing it to the Cucumber framework adapter. Need to dive into this as this might be a 3rd party dependency, but if someone wants to take a look at it then be my guest

@wswebcreation
Copy link
Collaborator

wswebcreation commented Oct 15, 2021

Some info for the person who's going to pick this up

  • the uri is of type string and contains the path to the file
  • the location is of type Location and contains this ({path-to-step-file}:{column})

@wswebcreation
Copy link
Collaborator

@christian-bromann

I've looked into this again, I can determine the uri and the line for the feature file level, but the @wdio/cucumber-framework only gives back feature-file data, not the TS/JS-files date like the TS/JS-file file path and line/column from the specific step.

Do you happen to know how this could be added to the @wdio/cucumber-framework?

@christian-bromann
Copy link
Contributor

Do you happen to know how this could be added to the @wdio/cucumber-framework?

Unfortunately the @wdio/reporter doesn't propagate any spec locations (e.g. row and column of scenario or step). This would be a nice addition.

@silke-vormbrock
Copy link

@wswebcreation
Sorry for not coming back to your comment for such a long time.
We finally got the confirmation from development that the uri and the line for the feature file level would be sufficient to ensure traceability. Your support in this topic would be very much appreciated!

@silke-vormbrock
Copy link

@wswebcreation
would it be ok to invite for a meeting to discuss possible next steps?

@wswebcreation
Copy link
Collaborator

Hi @silke-vormbrock

Sorry for not getting back to this. I have it on my to-do list, which is pretty long ATM :(, but will notify you when this is released

@silke-vormbrock
Copy link

Hi @wswebcreation
thank you so much for coming back to me. Do you have a rough idea when this release can be expected?
This missing piece would allow us to visualize fully automated traceability. Thanks!

@wswebcreation
Copy link
Collaborator

Hi @silke-vormbrock

I hope before the end of the month

@silke-vormbrock
Copy link

Hi @wswebcreation,
that would be brilliant! As Product Excellence Award Winners 2021 we are invited to the devX Quality Talk on Feb 8 and one of our topics is how we ensure traceability in every pipeline run. Thank you!

@wswebcreation
Copy link
Collaborator

@silke-vormbrock

A new version has been published, see https://github.com/webdriverio-community/wdio-cucumberjs-json-reporter/releases/tag/v4.2.0

This will now add the line and uri, see the example below

[
  {
    "keyword": "Feature",
    "type": "feature",
    "description": "",
    "line": 2,
    "name": "Sample",
    "uri": "/Users/Git/wdio-cucumber-v6-v7/v7/test/features/sample.feature",
    "tags": [
      {
        "location": {
          "line": 1,
          "column": 1
        },
        "name": "@sample",
        "id": "4"
      }
    ],
    "elements": [
      {
        "keyword": "Scenario",
        "type": "scenario",
        "description": "",
        "name": "cukes",
        "tags": [
          {
            "name": "@sample",
            "astNodeId": "4"
          }
        ],
        "id": "sample;cukes",
        "steps": [
          {
            "arguments": [],
            "keyword": "Before",
            "name": "",
            "result": {
              "status": "passed",
              "duration": 1000000
            },
            "line": null,
            "match": {
              "location": "can not be determined with webdriver.io"
            }
          },
          {
            "arguments": [],
            "keyword": "Before",
            "name": "",
            "result": {
              "status": "passed",
              "duration": 0
            },
            "line": null,
            "match": {
              "location": "can not be determined with webdriver.io"
            }
          },
          {
            "arguments": [],
            "keyword": "Given",
            "name": "I have a background",
            "result": {
              "status": "passed",
              "duration": 2000000
            },
            "line": null,
            "match": {
              "location": "can not be determined with webdriver.io"
            }
          },
          {
            "arguments": [],
            "keyword": "Given",
            "name": "I have 42 cukes in my belly",
            "result": {
              "status": "passed",
              "duration": 2905000000
            },
            "line": null,
            "match": {
              "location": "can not be determined with webdriver.io"
            }
          },
          {
            "arguments": [],
            "keyword": "After",
            "name": "",
            "result": {
              "status": "passed",
              "duration": 1000000
            },
            "line": null,
            "match": {
              "location": "can not be determined with webdriver.io"
            }
          }
        ]
      }
    ],
    "id": "sample",
    "metadata": {
      "browser": {
        "name": "chrome",
        "version": "97.0.4692.71"
      },
      "device": "Test Device",
      "platform": {
        "name": "ios",
        "version": "14"
      }
    }
  }
]

Can you please confirm and then close this issue if this fixes your needs

@silke-vormbrock
Copy link

Hi @wswebcreation,

thank you so much! I forwarded this information to development and hope to get feedback very quickly.

@silke-vormbrock
Copy link

Hi @wswebcreation,
our developer is checking the solution currently, we expect his results by the end of the week.

@wswebcreation
Copy link
Collaborator

Cool, thanks for the update

@silke-vormbrock
Copy link

Hi @wswebcreation,
unfortunately, we are not yet there where we want to be, but your update helped us to move into the right direction. I scheduled a meeting with our colleagues from Cumulus and will come up with a final update after this meeting. Thank you for your effort!

@wswebcreation
Copy link
Collaborator

Hi @silke-vormbrock

can not be determined with webdriver.io is being determined per step and not per test case/scenario. So I wonder how your reporting tool works. Can you explain that a bit more?

@silke-vormbrock
Copy link

Hi @wswebcreation,
this is the answer I got from the developer who supports us in this topic:
Hi Wim, you are correct at the moment each step information is taken to display the test case. This might change in the future to the scenario name. To drill down into the scenario it could be useful to link the individual steps to its implementation. Would this be possible via webdriver.io?

@wswebcreation
Copy link
Collaborator

wswebcreation commented Feb 7, 2022

Hi @silke-vormbrock

Thanks for the answer, officially the location is connected to the step file and its data. The problem is that this can't be determined in WDIO, see the above conversation.

If they want to have the step name they need to use the combination of keyword and name, like thin this step example

{
  "arguments": [],
  "keyword": "Given",
  "name": "I have 42 cukes in my belly",
  "result": { "status": "passed", "duration": 1322000000 },
  "line": null,
  "match": { "location": "can not be determined with webdriver.io" }
},

@silke-vormbrock
Copy link

Hi Wim,
thank you so much for your support. Our dev colleagues took over for the next steps.
Please allow me to keep this GitHub issue open until we have the final solution in place.
Thanks! Best regards, Silke

@wswebcreation
Copy link
Collaborator

Sure, no problem

@christian-bromann
Copy link
Contributor

Any updates @silke-vormbrock ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants