You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks to the tooling around, it is nowadays way easier to manage a Release than some years ago. Therefore, we should probably get the best out of the tooling available and update the tests a bit.
I would like to propose:
Try to make the tests as generic as possible, i.e no board-specific tests such as Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN. We then try to run the tests on as many boards as possible and indicate such boards in the Release notes. We can keep the current bare-minimum (samr21-xpro, iotlab-m3), although IMO there are several boards that are widely used and are not covered by the Release Specs (e.g nrf52840). This would basically obsolete Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN, as we would always try to run the tests in as many devices as possible. It would probably reduce the number of specs as well, many of the ping tests are similar and only differ in the board.
In tests such as Task #02 - Subset of tests on iotlab-m3, we should aim to run all tests nowadays, as the compile_and_test_for_board.py scripts takes care of everything.
Extend/update interop tests. For example, we should use other UPLINK settings for the RIOT Border Router (e.g slip, cdc-ecm). On the other side, we should really deprecate the Contiki interop test (or at least update it to Contiki-NG), as this test ends up running in outdated hardware anyway and it's not user friendly.
The text was updated successfully, but these errors were encountered:
I agree that we should rework the release spec tests.
One thing would be to parameterize the DUT as a generic board and specify "golden" boards, which would not really be the thing we are testing.
The most challenging example of this would be multihop, where the solution would be have 3 tests, one with the DUT as a root node, then as a middle node, then as end node.
We should also have a nice breakdown of the boards that were run through these tests, making sure that every board in the list would have undergone all applicable tests. Then we can enter that in the release notes very clearly and simply so that people know what "should" be passing for a given release.
The final thing would be to create a tracking issue for the specific release that exists in RIOT for all bugs found that would not be blocking the release (the compile_and_test_for_board.py might be hard to get everything passing but shouldn't block the release).
Thanks to the tooling around, it is nowadays way easier to manage a Release than some years ago. Therefore, we should probably get the best out of the tooling available and update the tests a bit.
I would like to propose:
samr21-xpro
,iotlab-m3
), although IMO there are several boards that are widely used and are not covered by the Release Specs (e.gnrf52840
). This would basically obsolete Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN, as we would always try to run the tests in as many devices as possible. It would probably reduce the number of specs as well, many of the ping tests are similar and only differ in the board.compile_and_test_for_board.py
scripts takes care of everything.slip
,cdc-ecm
). On the other side, we should really deprecate the Contiki interop test (or at least update it to Contiki-NG), as this test ends up running in outdated hardware anyway and it's not user friendly.The text was updated successfully, but these errors were encountered: