RFC: organizing release testing efforts

[miri64]

In the past we used tick-lists in the OP like this to organize the testing efforts of a release:

• test 1
• test 2
• ...

With the 2019.04 release @danpetry tried out a spreadsheet solution.

Another alternative could be a Kanban board (aka Github Projects).

Another alternative could be to automatically generate and push results to an online dashboard, for the tests that are automated. For example, creating JUnit XML test results documents and pushing to https://www.cdash.org/.

All of these solutions have drawbacks and benefits and none of them are perfect. Here a benefits and drawbacks I found.

Solution	Benefit	Drawback
Tick-list	Immediate overview	easy to accidentally override changes of others
	GitHub integration
	easy to auto-generate (markdown template)
Google spreadsheet	Synchronizable	External to GitHub
		No immediate overview
		not as easy to auto-generate (Google access required)
Github projects	Synchronizable	No immediate overview
	GitHub integration	not as easy to auto-generate but doable (GitHub access required)
		Can easily end in chaos (?)
Automation and dashboard	Instant	Lots of upfront work
		Could require time to come to consensus on solution

So here is my question: Anyone got any other idea? Do we want to change the organization of the testing efforts at all? Are there more drawbacks/benefits to the ones I listed?

[danpetry]

I added automated dashboard, which @aabadie suggested.

[danpetry]

FYI the reason why I came up with a spreadsheet in the first place wasn't primarily to solve the overwrite issue above, it was to make it easier to be on top of the state of testing (incl passes, fails, bugfix status...). An inline conversation really means you have to keep all this in your head, re-read if necessary to find information, etc, which is kind of a drain energy- and time-wise from an organizing perspective. I initially was just going to use it for myself but then got a comment offline that I could upload it so everyone could use it. Just for some background.

[miri64]

I added automated dashboard, which @aabadie suggested.

Mh. I don't know what this should be? Some custom software or is there a link were you can point to @aabadie?

[danpetry]

Link in comments above table. https://www.cdash.org

[aabadie]

Another alternative could be to automatically generate and push results to an online dashboard, for the tests that are automated. For example, creating JUnit XML test results documents and pushing to https://www.cdash.org/.

I did some tests of this for 2019.01 (see https://my.cdash.org/index.php?project=RIOT&date=2019-02-01, not sure you can access it). But I found bugs with the integration JUnit XML in CDash. It might be fixable in CDash though, but I haven't taken the time to do it.

I added automated dashboard, which @aabadie suggested.

Do you have a link or something ? Not sure what "automated dashboard" means.

[aabadie]

Link in comments above table. https://www.cdash.org

Ok but that doesn't tell where you put the dashboard. Are you self-hosting an instance of CDash or use an application in my.cdash.org hosting offer ?

[miri64]

I did some tests of this for 2019.01 (see https://my.cdash.org/index.php?project=RIOT&date=2019-02-01, not sure you can access it). But I found bugs with the integration JUnit XML in CDash. It might be fixable in CDash though, but I haven't taken the time to do it.

Another thing I noticed also is that I have to create an account there, just to see what you did there ;-).

[miri64]

And even then I get

Error: You do not have permission to access this page.

So this seems to be a little bit too involved :(

[danpetry]

I think we can talk here about the general concept of using an online dashboard plus automated result generation and uploading - rather than specifics of where it's hosted etc, or am I missing something? I guess we could find a dashboard which is suitable to our needs if the concept makes sense?

I guess that creating an account and setting permissions would be minor in comparison to the rest of the programming and integration work required? But once it's set up it would be instant and very easy to use. Perhaps it would be up to the release manager to assign permissions.

IMO requiring completely zero-effort to learn new tools for the testers would be somewhat restrictive, but maybe I'm not considering the open-source community/voluntary context well enough :)

[aabadie]

Another thing I noticed also is that I have to create an account there, just to see what you did there ;-).

I initially (and intentionally) created the project private until I have something showable :)
What email did you use to register ?
Maybe the simplest is to make the project public ?

[aabadie]

What email did you use to register ?

I found it :)

[miri64]

I think we can talk here about the general concept of using an online dashboard plus automated result generation and uploading - rather than specifics of where it's hosted etc, or am I missing something? I guess we could find a dashboard which is suitable to our needs if the concept makes sense?

Yepp, sorry.

As far as I can see however, this requires automation of the release tests first. So that would also be (at least for now) a drawback, as our automation is, at the moment, a sketch at best.

[miri64]

As far as I can see however, this requires automation of the release tests first. So that would also be (at least for now) a drawback, as our automation is, at the moment, a sketch at best.

(Also once we are at a good automation level I expect our release tests to be included into the nightly tests so they would be included into the dashboard of what ever test framework we are using for that anyway ;-))

[danpetry]

For RC2, do we think a spreadsheet is helping or would going back to the original tick-list be better for everyone?

[danpetry]

i.e. which one would people like me to use

[miri64]

I'm indifferent to both solutions to be honest.