Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed tests still produce exit code of 0 (zero) #307

Open
Mierdin opened this issue Mar 14, 2018 · 0 comments
Open

Failed tests still produce exit code of 0 (zero) #307

Mierdin opened this issue Mar 14, 2018 · 0 comments

Comments

@Mierdin
Copy link

Mierdin commented Mar 14, 2018

Running a test that results in success, as expected, produces an exit code of 0:

~$ jsnapy --snapcheck -f jsnapyconfig.yaml -v
Tests Included : test_applications
Taking snapshot of RPC: get-config
***************************** Device: 127.0.0.1 *****************************
Tests Included: test_applications
*************************RPC is get-config*************************
----------------------Performing is-equal Test Operation----------------------
Test Succeeded!!, destination-port is <30589>
PASS | All "destination-port" is equal to "30589" [ 1 matched ]
------------------------------- Final Result!! -------------------------------
test_applications : Passed
Total No of tests passed: 1
Total No of tests failed: 0
Overall Tests passed!!!

~$ echo $?
0

However, a failing test also produces a 0:

~$ jsnapy --snapcheck -f jsnapyconfig.yaml -v
Tests Included : test_applications
Taking snapshot of RPC: get-config
***************************** Device: 127.0.0.1 *****************************
Tests Included: test_applications
*************************RPC is get-config*************************
----------------------Performing is-equal Test Operation----------------------
Test Failed!!!, destination-port is <30589>
FAIL | All "destination-port" is not equal to "12345" [ 0 matched / 1 failed ]
------------------------------- Final Result!! -------------------------------
test_applications : Failed
Total No of tests passed: 0
Total No of tests failed: 1
Overall Tests failed!!!

~$ echo $?
0

This isn't ideal for running jsnapy in a network automation pipeline, where network changes may or may not continue based on whether or not these tests passed. Should return a nonzero exit code to indicate one or more tests didn't pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants