Verifying the quality of Yubico Authenticator is made through multiple levels of tests, loosely defined in the following groups.
-
Automatic tests in CI pipeline
-
Unit tests
-
Automatic tests requiring a Yubikey
-
Manual tests
The automated tests requiring a key and the manual tests are run by the developer team as part of regression tests for releasing new versions of the app. The manual tests are problematic to automate as they depend on other technology which are a bit to expensive to automate (camera, webauthn) in the flutter test framework.
To run the tests you need to specify the serial number of a Yubikey which the tests are able to run on. IMPORTANT: this key will be reset by running the tests, so don’t use one of your personal keys. The serial numbers should either be defined as an environment variable on execution, or in a CSV-file with:
$ /yubioath-flutter/integration_test/test_res/resources/approved_serial_numbers.csv
This file is part of .gitignore, and should never be committed. If you do not wish to add serial numbers to a file like above, you can also initiate the test runs by expressing a dart-define:
$ flutter --dart-define=YA_TEST_APPROVED_KEY_SN=SN1,SN2,...,SNn test integration_test
Running the tests for the CI environment:
$ flutter test integration_test/keyless_test.dart
Running all the tests for desktop (requires a Yubikey):
$ ./testrunner.sh
Running all the tests for desktop (requires a Yubikey):
$ flutter test
The testrunner interface is work in progress, but it deals with a problem of the flutter integration_test framework, i.e. that it fails to initiate multiple UI-tests.
Manual test scripts will be moved here so all quality related information is in the same place.