webdrivercss tests fail on travis CI

[pavloo]

Here is the content of my .travis.yml:

language: node_js
node_js:
  - "4.2.1"
env:
  - CXX=g++-4.8
addons:
  apt:
    sources:
    - ubuntu-toolchain-r-test
    packages:
    - g++-4.8

before_install:
  - sudo apt-get install graphicsmagick libcairo2-dev
  - export PKG_CONFIG_PATH=/opt/X11/lib/pkgconfig
  - npm install -g bower  

install:
  - npm install
  - bower install

before_script:
  - npm install -g gulp
  - npm install -g karma
  # see https://github.com/webdriverio/gulp-webdriver/issues/20
  - npm install webdriverio

script: npm test

My tests run ok locally, but all of them are failing with near the same error on Travis:

[ { baselinePath: 'test/css/screenshots/buttons.primary.1024px.baseline.png',
    regressionPath: 'test/css/screenshots/buttons.primary.1024px.regression.png',
    diffPath: 'test/css/diffs/buttons.primary.1024px.diff.png',
    message: 'mismatch tolerance exceeded (+8.94), image-diff created',
    misMatchPercentage: 8.99,
    isExactSameImage: false,
    isSameDimensions: true,
    isWithinMisMatchTolerance: false,
    properties: { name: 'primary', elem: '.bb-btn--primary' } } ]

Here is the code of the test:

  it('buttons should have correct styles', function* () {
    yield browser.url('/buttons.html');
    const res = yield browser.webdrivercss('buttons',[
      {
        name: 'primary',
        elem: '.bb-btn--primary'
      }
    ]);

    assert.ok(res.primary[0].isWithinMisMatchTolerance, 'primary');
  });

Webdriver config:

webdrivercss: {
      screenshotRoot: 'test/css/screenshots',
      failedComparisonsRoot: 'test/css/diffs',
      misMatchTolerance: 0.05,
      screenWidth: [1024]
    }

Tests are being run on phantomjs. Any ideas what may cause the issue? Thanks.

[christian-bromann]

@pavloo have you been able to check the actual screenshots?

[pavloo]

@christian-bromann unfortunately I haven't. any ideas on how to check them on travis? thanks

[Blackbaud-PatrickOFriel]

@pavloo you could make an after_failure step in travis that publishes the screenshot folder to a repo somewhere

[klamping]

You could also use something like https://github.com/webdriverio/webdrivercss-adminpanel.

I'm guessing the issue is small rendering differences between your local browser and what Travis uses. Using browserstack or sauce labs is almost a necessity for this sort of test.

[pavloo]

@Blackbaud-PatrickOFriel @klamping I'll check it out, thanks