Support automated test running in local browsers

[toddself]

Problem: phantomjs development has slowed significantly and it does not support any ES6 syntax. (Although istanbul also has issues, but I can do --no-coverage). What I'd like to be able to do is use zuul as my test runner locally to run my tests in a local browser (either live or via xvfb, either way is ok with me), like how testling works.

I'm happy to put together a PR for this, but would this be a welcome addition? (Or does this currently work and I just haven't figured it out?)

[parshap]

You can do this using zuul --local.

[toddself]

Wrong idea.

What I want is for zuul to listen to the browser's console, have the browser execute the tests, and then exit returning a non-zero status for failing tests along with console-based output (tap compatible would be great!)

--local requires you to open the browser (or you can use --open if you're lazy) but you have to visually parse the results. The idea here is automated testing using real browsers.

Alternatively phantomjs could kickstart dev again and actually release 2.0....

[parshap]

Ah yes, that'd be great.

I also notice that if you call process.exit() with a non-zero exit code, zuul does not recognize that as a failure.

[vvo]

--local requires you to open the browser (or you can use --open if you're lazy) but you have to visually parse the results. The idea here is automated testing using real browsers.

You can use saucelabs for this. If you really want real browsers local mode (+ exit code etc..) then you should implement selenium connector in zuul and then use selenium.

Otherwise this:

What I'd like to be able to do is use zuul as my test runner locally to run my tests in a local browser (either live or via xvfb, either way is ok with me), like how testling works.

Is exactly what http://karma-runner.github.io/0.13/index.html is

[vvo]

Problem: phantomjs development has slowed significantly and it does not support any ES6 syntax.

I do use --phantom with zuul: https://github.com/defunctzombie/zuul/wiki/Phantom-js But you have to compile to ES5 (BTW you will have to compile to ES5 no matter what you do, because not all browsers are supporting ES6, nor all of it)

[toddself]

phantom no longer runs under os x el capitan, so that's out. the project seems to be on life support (2.0 was never fully released and there is very little activity)

Also most browsers support a decent chunk of es6 at this point; we have a limited number of browsers to support so we're using ES6 syntax that is available in them. Transpiling just for testing is not a scenario I'd like to get into.

Saucelabs costs money and quite a lot of it unless you're working on open source and go through the hassle of opening an account. Even if you're working on open source entirely, I've found saucelabs to be a lot of hassle at times.

I'm like a 1/4 of the way through making a connector to run on a local browser; hopefully will be able to PR shortly.

[vvo]

Transpiling just for testing is not a scenario I'd like to get into.

Indeed, either you always transpile with babel or no transpile at all.

Even if you're working on open source entirely, I've found saucelabs to be a lot of hassle at times.

Yes indeed, mostly selenium is the hassle

[vvo]

I'm like a 1/4 of the way through making a connector to run on a local browser; hopefully will be able to PR shortly.

Sooooo cool! way to go :D

[defunctzombie]

I am happy with removing phantomjs support once this lands. I never really cared for phantomjs support anyway since it was so easy to just open a real browser.

[toddself]

Well this has come full circle. I remember talking with Thorsten to see if he could get phantom re-added as we needed it for local automated testing when we both worked at Conde Nast. :)

[Hypercubed]

I recently switched several projects to zuul (some were using karma others using jsdom). I switched specifically because it is easy to run ES6 code in phantomjs using the babelify transform. Also works well with Travis CI. I hope you will keep phantomjs integration.

Also, sauce labs is ending free accounts.

[defunctzombie]

@Hypercubed hm. Good to know! Also, are all free accounts going away or just for non-open source? If they end all free accounts that would be a bummer.

[Hypercubed]

The FAQ says they are halting Free Accounts: https://support.saucelabs.com/customer/portal/articles/2204230-free-trial-faq
But it looks like there is something called Open Sauce.