Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

closure_js_test - Passed tests sometimes hang #406

Open
robfig opened this issue Jul 21, 2019 · 0 comments
Open

closure_js_test - Passed tests sometimes hang #406

robfig opened this issue Jul 21, 2019 · 0 comments

Comments

@robfig
Copy link
Contributor

robfig commented Jul 21, 2019

We are just starting to use rules_closure for real, and the second test suite that I've migrated shows flaky behavior on our CI system. I was hoping for some advice on how to diagnose this.

We use teamcity, and when running the suite 3 times in a row, one test finished in TIMEOUT the first run, a different test the second, and all tests passed on the third. It ran on 3 different hosts, but they are identically configured with ansible.

The tests don't take long to run, and the runner outputs "all tests passed" long before the timeout. They do not do anything that could account for a long delay.

First run:

[13:16:46]	==================== Test output for //js/corp/consumerfeedbackstorm/invites/templates:addtemplatedialog_test:
[13:16:46]	Listening http://localhost:35355/index.html
[13:16:46]	Serving /index.html
[13:16:46]	Serving /filez/corp/js/corp/consumerfeedbackstorm/invites/templates/addtemplatedialog_test_bin.js
[13:16:46]	Serving /filez/corp/js/vendor/jquery-ui-1.8.24.min.js
[13:16:46]	Serving /filez/corp/js/vendor/jquery-1.10.2.min.js
[13:16:46]	-> 13:11:33.964 : Starting tests: Untitled Test Case
[13:16:46]	-> 13:11:33.965 : Running test: testErrorRendering
[13:16:46]	-> 13:11:33.975 : testErrorRendering : PASSED
[13:16:46]	-> 13:11:33.975 : Running test: testUserInput
[13:16:46]	-> 13:11:34.003 : testUserInput : PASSED
[13:16:46]	-> 13:11:34.003 : 2 of 2 tests run in 38.78055200000017ms.
[13:16:46]	-> 2 passed, 0 failed.
[13:16:46]	19 ms/test. 4 files loaded.
[13:16:46]	-> 13:11:34.003 : Tests complete
[13:16:46]	Terminated
[13:16:46]	-- Test timed out at 2019-07-21 13:16:30 EDT --
[13:16:46]	================================================================================
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:customlegal_test         PASSED in 1.7s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:emailinvites_test        PASSED in 1.8s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:maxcontactfrequency_test PASSED in 2.3s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:quarantine_test          PASSED in 1.5s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:settingsrow_test         PASSED in 1.8s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:surveypageenabled_test   PASSED in 1.3s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsvalidator_test PASSED in 0.8s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsview_test PASSED in 1.9s
[13:16:46]	//js/corp/consumerfeedbackstorm/invites:reviewinvitationvalidator_test   PASSED in 1.2s
[13:16:46]	//js/corp/consumerfeedbackstorm/invites/templates:templatevalidator_test PASSED in 1.1s
[13:16:46]	//test/com/corp/consumerfeedback:unit                                    PASSED in 48.1s
[13:16:46]	//js/corp/consumerfeedbackstorm/invites/templates:addtemplatedialog_test TIMEOUT in 315.1s
[13:16:46]	  /home/cdbuilder/.cache/bazel/_bazel_TeamCity/df2b12f83f809abace199d393a19f0d3/execroot/corp/bazel-out/k8-fastbuild/testlogs/js/corp/consumerfeedbackstorm/invites/templates/addtemplatedialog_test/test.log
[13:16:46]	
[13:16:46]	Executed 12 out of 12 tests: 11 tests pass and 1 fails locally.
[13:16:46]	

Second run

[14:46:17]	==================== Test output for //js/corp/consumerfeedbackstorm/accountsettings:settingsrow_test:
[14:46:17]	Listening http://localhost:54287/index.html
[14:46:17]	Serving /index.html
[14:46:17]	Serving /filez/corp/js/corp/consumerfeedbackstorm/accountsettings/settingsrow_test_bin.js
[14:46:17]	Serving /filez/corp/js/vendor/jquery-ui-1.8.24.min.js
Serving /filez/corp/js/vendor/jquery-1.10.2.min.js
[14:46:17]	-> 14:41:04.476 : Starting tests: Untitled Test Case
[14:46:17]	-> 14:41:04.476 : Running test: testCancelReset
[14:46:17]	-> 14:41:04.497 : testCancelReset : PASSED
[14:46:17]	-> 14:41:04.497 : Running test: testRenderErrors
[14:46:17]	-> 14:41:04.500 : testRenderErrors : PASSED
[14:46:17]	-> 14:41:04.500 : Running test: testSubmitToggle
[14:46:17]	-> 14:41:04.502 : testSubmitToggle : PASSED
[14:46:17]	-> 14:41:04.502 : Running test: testToggleEdit
[14:46:17]	-> 14:41:04.513 : testToggleEdit : PASSED
[14:46:17]	-> 14:41:04.514 : 4 of 4 tests run in 38.01679799999988ms.
[14:46:17]	-> 4 passed, 0 failed.
[14:46:17]	10 ms/test. 4 files loaded.
[14:46:17]	Loading: 
[14:46:17]	-> 14:41:04.514 : Tests complete
[14:46:17]	Terminated
[14:46:17]	Loading: 0 packages loaded
[14:46:17]	-- Test timed out at 2019-07-21 14:46:02 EDT --
[14:46:17]	================================================================================
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:customlegal_test         PASSED in 0.9s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:emailinvites_test        PASSED in 1.0s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:maxcontactfrequency_test PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:quarantine_test          PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:surveypageenabled_test   PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsvalidator_test PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsview_test PASSED in 1.2s
[14:46:17]	//js/corp/consumerfeedbackstorm/invites:reviewinvitationvalidator_test   PASSED in 0.9s
[14:46:17]	//js/corp/consumerfeedbackstorm/invites/templates:addtemplatedialog_test PASSED in 0.9s
[14:46:17]	//js/corp/consumerfeedbackstorm/invites/templates:templatevalidator_test PASSED in 0.9s
[14:46:17]	//test/com/corp/consumerfeedback:unit                                    PASSED in 19.2s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:settingsrow_test        TIMEOUT in 315.0s

Third run didn't print anything because it was successful, taking 1m44s to build/run the java and js test suites.

This seems very much like the JS test suites succeeds, but for whatever reason Bazel doesn't get that memo and waits until the 5 minute limit expires, after which Bazel declares the test timed out.

Have you seen anything like this before?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant