Pa11y reporters do not output to console when used with Pa11y CI

[aarongoldenthal]

When pa11y loads reporters, it wraps the reporter functions so that any output from any reporter function is written to console (here). That logic is missing from pa11y-ci (it should probably be here). So, when pa11y reporters are used with pa11y-ci, any reporters that returns a value expecting it to externally be written to console have their output lost. This includes the cli, csv, html, and tsv reporters. The json reporter is not impacted since it directly writes its output.

The dedicated pa11y-ci reporters handle their own output so are not effected.

[aarongoldenthal]

After thinking about this a little, if pa11y-ci were updated to simply write the value returned from a reporter to stdout, as pa11y does (in most cases), then it would do the same for all analyzed files. The results could be piped to a file, but the result wouldn't be valid for the given format (excluding cli for the moment, and technically tsv and csv would but would have headers repeated for every file).

An alternative proposal - create a new reporter that's essentially a shim for using any pa11y reporter:

• Assume one reporter, requiring configuration to specify the pa11y reporter
• This reporter would take the output from the specified pa11y reporter results function, create a unique file name based on the URL, and save the results to the file (which would then be valid for the given format)
• The reporter configuration could also include a destination directory
• My original thought was exclude the cli reporter, although it could be used, but would probably want to strip out the ANSI escape codes
• For errors, the output from the reporter error function would be written to stderr

For the pa11y reporters:

• The csv, html, and tsv reporters would work with the above implementation
• The json reporter writes to stdout, but could be changed to simply return the results, which would then work with the implementation above. Pa11y already writes that output to stdout, so would be functionally transparent to the user.
• The cli reporter could be updated to write to stdout itself (as the json reporter does now). This would be transparent for pa11y, but would allow the reporter to be used directly with pa11y-ci with the expected output. In this case it is valid and logical output on its own.

[langalex]

It's not much but I took the JSON reporter and changet it to wrap the pa11y HTML reporter:

'use strict';

const fs = require('fs');
const { resolve, isAbsolute, dirname } = require('path');
const pa11yHtmlReporter = require('pa11y/lib/reporters/html');

function writeReport(fileName, data) {
  try {
    const dirPath = dirname(fileName);
    if (!(fs.existsSync(dirPath) && fs.lstatSync(dirPath).isDirectory())) {
      fs.mkdirSync(dirPath, { recursive: true });
    }
    fs.writeFileSync(fileName, data);
  } catch (error) {
    console.error(`Unable to write ${fileName}`);
    console.error(error);
  }
}

function resolveFile(fileName) {
  if (typeof fileName !== 'string') {
    return null;
  }
  return isAbsolute(fileName) ? fileName : resolve(process.cwd(), fileName);
}

module.exports = function htmlReporter(options = {}) {
  const fileName = resolveFile(options.fileName);
  return {
    results(report) {
      return pa11yHtmlReporter.results(report).then((html) => {
        // If reporter options specify an output file, write to file.
        // Otherwise, write to console.
        if (fileName) {
          writeReport(fileName, html);
        } else {
          console.log(html);
        }
      });
    },
  };
};

Config:

{
  "defaults": {
    "reporters": [
      "cli",
      ["./pa11y-ci-html-reporter.js", { "fileName": "./pa11y-cli-report.html" }]
    ]
  }
}

It now writes its output either to a file or console.

[aarongoldenthal]

@langalex if you're just looking for an HTML reporter for pa11y-ci, you can try https://www.npmjs.com/package/pa11y-ci-reporter-html, which produces a report for each page as well as a summary (for full disclosure, it's my npm package)

[langalex]

Thanks. I ended up creating my own Markdown report so I could attach it to a PR check summary here on GitHub.