Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Restructure/update Release Specs #279

Open
jia200x opened this issue Apr 13, 2023 · 2 comments
Open

[RFC] Restructure/update Release Specs #279

jia200x opened this issue Apr 13, 2023 · 2 comments
Assignees

Comments

@jia200x
Copy link
Member

jia200x commented Apr 13, 2023

Thanks to the tooling around, it is nowadays way easier to manage a Release than some years ago. Therefore, we should probably get the best out of the tooling available and update the tests a bit.

I would like to propose:

  1. Try to make the tests as generic as possible, i.e no board-specific tests such as Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN. We then try to run the tests on as many boards as possible and indicate such boards in the Release notes. We can keep the current bare-minimum (samr21-xpro, iotlab-m3), although IMO there are several boards that are widely used and are not covered by the Release Specs (e.g nrf52840). This would basically obsolete Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN, as we would always try to run the tests in as many devices as possible. It would probably reduce the number of specs as well, many of the ping tests are similar and only differ in the board.
  2. In tests such as Task #02 - Subset of tests on iotlab-m3, we should aim to run all tests nowadays, as the compile_and_test_for_board.py scripts takes care of everything.
  3. Extend/update interop tests. For example, we should use other UPLINK settings for the RIOT Border Router (e.g slip, cdc-ecm). On the other side, we should really deprecate the Contiki interop test (or at least update it to Contiki-NG), as this test ends up running in outdated hardware anyway and it's not user friendly.
@MrKevinWeiss
Copy link
Contributor

I agree that we should rework the release spec tests.

One thing would be to parameterize the DUT as a generic board and specify "golden" boards, which would not really be the thing we are testing.

The most challenging example of this would be multihop, where the solution would be have 3 tests, one with the DUT as a root node, then as a middle node, then as end node.

We should also have a nice breakdown of the boards that were run through these tests, making sure that every board in the list would have undergone all applicable tests. Then we can enter that in the release notes very clearly and simply so that people know what "should" be passing for a given release.

The final thing would be to create a tracking issue for the specific release that exists in RIOT for all bugs found that would not be blocking the release (the compile_and_test_for_board.py might be hard to get everything passing but shouldn't block the release).

@MrKevinWeiss
Copy link
Contributor

We should also collect the test artifacts and add it as an asset to the release... I don't know how long the retention time is for the actions logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants