Postpone build failure on error until end of mkdocs build to see all errors

[skwde]

With the mkdocs.yml setting in

plugins:
  - macros:
      modules: [<plugin>]
      verbose: true
      on_error_fail: true
      on_undefined: strict

we have the option to have the mkdocs build fail on error.
The build however fails exactly when the error is encountered.

When using on_error_fail: false one can see all the errors but the build does not fail

It would be better if one could see all the logs and all the error.
Otherwise one can only fix one error and build again to see the next error.

[fralau]

That's a good point, but my answer is two-fold:

1. I do not see how this could be easily changed.
2. The purpose is to stop the build process so as to warn that something went wrong, not to fully debug it. As with any other piece of code, correcting the single error displayed and restarting the whole build process is not the prescribed way to go.

1. Why this cannot be easily changed

If were look at the piece of code:

        on_error_fail = self.config['on_error_fail']
        try:
            md_template = self.env.from_string(markdown)
            # Execute the jinja2 template and return
            return md_template.render(**page_variables)

        except Exception as error:
            error_message = format_error(
                error,
                markdown=markdown,
                page=self.page,
            )

            trace('ERROR', error_message)
            if on_error_fail:
                exit(ERROR_MACRO)

            else:
                return error_message

It fails just as any compiler would, at the first error, and there is no way to resume.

The best that we could do would be to keep track that it is a failure, and still try to run other pages. Beside the fact that this would complicate the code, it might display a series of errors, but still without certainty that all of them were found (since there might be more than one error per page).

As mentioned earlier, the purpose is to stop the build process with an indication of where to start from to debug, not to fully debug the website.

2. How to debug

The prescribed way to debug an MkDocs website mirrors any other piece of code that could break during a build process. The offending piece of code must be analysed, debugged and run through its own "unit" testing, until all bugs are found and weeded out.

In that particular case, the testing tool of choice would be mkdocs serve. In that case on_error_fail should indeed be set to false so that one could observe as many errors as possible.

[skwde]

Thanks for your comments.

Wouldn't be a way around this to report the 'errors' as 'warnings' to mkdocs and have the --strict mode deal with that?

[fralau]

--strict would still fail at the first warning? Also, plugins can always raise some spurious warnings which could make the build process unreliable?

More generally, the behavior of MkDocs-Macros is built on the arguments that can be given to the Jinja2 environment.

I used a simple approach, but going beyond that would require some more serious thought.

One could perhaps define a new subclass of Undefined that stores all errors into a file or output them as warnings, and then it would call a graceful failure at the end? 🤔

I would like it if someone else studied a solution and proposed it; as long as it doesn't change the rest of the plugin's logic.

[skwde]

Regarding --strict:

I use

strict: true
validation:
  omitted_files: warn
  absolute_links: warn
  unrecognized_links: warn

and the build process runs through, at the end it prints something like

There have been X warnings

after which an error is raised and the build failed.
Then I can go through the log and fix the issues before running the build again.

Hmm, spurious warnings of plugins are bad and somehow spoil this approach.

[fralau]

Perhaps we are getting philosophical here, but the point with warnings, is that they were defined in Python as distinct from Exceptions, so as to continue the execution:

Warning messages are typically issued in situations where it is useful to alert the user of some condition in a program, where that condition (normally) doesn’t warrant raising an exception and terminating the program. For example, one might want to issue a warning when a program uses an obsolete module.

A strict policy may be warranted in an environment where the developpers have control on each component of the ecosystem and are willing/able to enforce high standards of quality. DevOps usually deals with highly heterogenous material from different internal teams and external sources; so one should strike a delicate balance between catching errors that would affect production, and not being overly demanding on formal requirements (since one would make it too difficult/time-consuming to achieve a new build). So, my impression is that strict would be too demanding in most cases, unless one is making the conscious choice of reducing the number of MkDocs components one is allowing (and therefore features, ease of use or development, etc.) for the sake of reliability.

Which is why I would prefer to (ideally) provide solutions about stopping/not-stopping that rely on exceptions; then letting the user (DevOp) decide what they want to catch or not.

[skwde]

Well in python this makes sense, but we are talking about mkdocs (even though your plugin is written in python).

• I think demanding strict already in devops makes sense, because
  • this reduces the errors / warnings to fix when it goes in production
  • errors / bad style are immediately reported and have to be fixed
• requiring to run an mkdocs serve whenever a change in the docs is done, is just not feasible, because
  • for large documentation it actually takes quite a while to reach the point where it actually serves the documentation
  • just 'quickly' doing a change is no longer possible (think of editing on the remote repo)

From my perspective the quickest / easiest way to write documentation is to 'just' write it have it built and then fix all the errors in one go.

[fralau]

I understand, though in my mind the job of the running mkdocs build (strict or not) is to ensure that it runs; it is not a debug tool. If it doesn't (as in an exam: one error means there are likely others), it should be sent back to whomever is in charge and they should fix their work in their prefered way. Waiting for the project build to fail, to detect an error in the MkDocs website would be too late on the production chain.

If there disagreement on this, it is a question that might be deferred to the MkDocs project? 🤔

[skwde]

You are right, if there is one error, there are likely more. Why making it hard / inconvenient to see all errors?

Imagine if it is sent back to you, you probably also prefer to see all the errors immediately rather than having to set up your own environment first in order to see and fix all issues.

Having all build errors displayed rather than only the first error is just more convenient. Don't you agree?

[fralau]

I understand but unfortunately that is not the way Jinja2 and MkDocs have been implemented.

There have three options with Jinja2:

1. Ignore errors
2. Display errors in the page
3. Fail with an exception at the first occurence..

As far as I know, it is adequate for the user base and there is not a sufficient case for trying to alter that behavior at the moment.