Parsing mathjax html document produced with lwarp #9615

tvercaut · 2024-03-31T11:18:30Z

tvercaut
Mar 31, 2024

I am trying to convert fairly complex latex documents to Microsoft Word docx format. Pandoc has some limitations for this and I am thus looking into first producing html using lwarp and then converting the html to docx using pandoc. Unfortunately, I can't seem to make pandoc understand the mathjax equations produced by lwarp.

For the record, here is a simple latex file I am using for testing purposes:

\documentclass{article}

\usepackage[mathjax]{lwarp}

\usepackage{amsmath}
\usepackage{blindtext}

\author{Jane Doe}
\title{Test}

\begin{document}

\maketitle

\blindtext

Testing bliography citation \cite{article-full} and testing referenced equation \eqref{eq:test}:
\begin{equation}\label{eq:test}
\frac{1}{x} = \sqrt{a^2+b^2}  
\end{equation}
    
\bibliographystyle{plain}
\bibliography{xampl}

\end{document}

The resulting mathjax html is as such:

Expand

<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="UTF-8" />
<meta name="generator" content="LaTeX Lwarp package" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Test</title>
<link rel="stylesheet" type="text/css" href="lwarp.css" />

<script>
// Lwarp MathJax emulation code
//
// Based on code by Davide P. Cervone.
// Equation numbering: https://github.com/mathjax/MathJax/issues/2427
// Starred and ifnextchar macros: https://github.com/mathjax/MathJax/issues/2428
// \left, \right delimiters: https://github.com/mathjax/MathJax/issues/2535
//
// Modified by Brian Dunn to adjust equation numbering and add subequations.
//
// LaTeX can use \seteqnumber{subequations?}{section}{number} before each equation.
// subequations? is 0 usually, 1 if inside subequations.
// section is a string printed as-is, or empty.
// number is auto-incremented by MathJax between equations.
//
MathJax = {
   subequations: "0",
   section: "",
   loader: {
      load: ['[tex]/tagformat', '[tex]/textmacros'],
   },
   startup: {
      ready() {
        // These would be replaced by import commands if you wanted to make
        // a proper extension.
        const Configuration = MathJax._.input.tex.Configuration.Configuration;
        const CommandMap = MathJax._.input.tex.SymbolMap.CommandMap;
        const Macro = MathJax._.input.tex.Symbol.Macro;
        const TexError = MathJax._.input.tex.TexError.default;
        const ParseUtil = MathJax._.input.tex.ParseUtil.default;
        const expandable = MathJax._.util.Options.expandable;

        // Insert the replacement string into the TeX string, and check
        // that there haven't been too many maxro substitutions (prevents
        // infinite loops).
        const useArgument = (parser, text) => {
          parser.string = ParseUtil.addArgs(parser, text, parser.string.slice(parser.i));
          parser.i = 0;
          if (++parser.macroCount > parser.configuration.options.maxMacros) {
            throw new TexError('MaxMacroSub1',
            'MathJax maximum macro substitution count exceeded; ' +
            'is there a recursive macro call?');
          }
        }

        // Create the command map for:
        //      \ifstar, \ifnextchar, \ifblank, \ifstrequal, \gsub, \seteqnumber
        new CommandMap('Lwarp-macros', {
           ifstar: 'IfstarFunction',
           ifnextchar: 'IfnextcharFunction',
           ifblank: 'IfblankFunction',
           ifstrequal: 'IfstrequalFunction',
           gsubstitute: 'GsubstituteFunction',
           seteqnumber: 'SeteqnumberFunction'
        }, {
           // This function implements an ifstar macro.
           IfstarFunction(parser, name) {
              const resultstar = parser.GetArgument(name);
              const resultnostar = parser.GetArgument(name);
              const star = parser.GetStar();                 // true if there is a *
              useArgument(parser, star ? resultstar : resultnostar);
           },

          // This function implements an ifnextchar macro.
          IfnextcharFunction(parser, name) {
             let whichchar = parser.GetArgument(name);
             if (whichchar.match(/^(?:0x[0-9A-F]+|[0-9]+)$/i)) {
               // $ syntax highlighting
               whichchar = String.fromCodePoint(parseInt(whichchar));
             }
             const resultnextchar = parser.GetArgument(name);
             const resultnotnextchar = parser.GetArgument(name);
             const gotchar = (parser.GetNext() === whichchar);
             useArgument(parser, gotchar ? resultnextchar : resultnotnextchar);
          },

          // This function implements an ifblank macro.
          IfblankFunction(parser, name) {
             const blankarg = parser.GetArgument(name);
             const resultblank = parser.GetArgument(name);
             const resultnotblank = parser.GetArgument(name);
             const isblank = (blankarg.trim() == "");
             useArgument(parser, isblank ? resultblank : resultnotblank);
          },

          // This function implements an ifstrequal macro.
          IfstrequalFunction(parser, name) {
             const strequalfirst = parser.GetArgument(name);
             const strequalsecond = parser.GetArgument(name);
             const resultequal = parser.GetArgument(name);
             const resultnotequal = parser.GetArgument(name);
             const isequal = (strequalfirst == strequalsecond);
             useArgument(parser, isequal ? resultequal : resultnotequal);
          },

          // This function implements a gsub macro.
          GsubstituteFunction(parser, name) {
             const gsubfirst = parser.GetArgument(name);
             const gsubsecond = parser.GetArgument(name);
             const gsubthird = parser.GetArgument(name);
             let gsubresult=gsubfirst.replace(gsubsecond, gsubthird);
             useArgument(parser, gsubresult);
          },

          // This function modifies the equation numbers.
          SeteqnumberFunction(parser, name) {
              // Get the macro parameters
              const star = parser.GetStar();                  // true if there is a *
              const optBrackets = parser.GetBrackets(name);   // contents of optional brackets
              const newsubequations = parser.GetArgument(name); // the subequations argument
              const neweqsection = parser.GetArgument(name); // the eq section argument
              const neweqnumber = parser.GetArgument(name);   // the eq number argument
              MathJax.config.subequations=newsubequations ;   // a string with boolean meaning
              MathJax.config.section=neweqsection ;           // a string with numeric meaning
              parser.tags.counter = parser.tags.allCounter = neweqnumber ;
          }

        });

        // Create the Lwarp-macros package
        Configuration.create('Lwarp-macros', {
          handler: {macro: ['Lwarp-macros']}
        });

        MathJax.startup.defaultReady();

        // For forward references:
        MathJax.startup.input[0].preFilters.add(({math}) => {
          if (math.inputData.recompile){
              MathJax.config.subequations = math.inputData.recompile.subequations;
              MathJax.config.section = math.inputData.recompile.section;
          }
        });
        MathJax.startup.input[0].postFilters.add(({math}) => {
          if (math.inputData.recompile){
              math.inputData.recompile.subequations = MathJax.config.subequations;
              math.inputData.recompile.section = MathJax.config.section;
          }
        });

          // For \left, \right with unicode-math:
          const {DelimiterMap} = MathJax._.input.tex.SymbolMap;
          const {Symbol} = MathJax._.input.tex.Symbol;
          const {MapHandler} = MathJax._.input.tex.MapHandler;
          const delimiter = MapHandler.getMap('delimiter');
          delimiter.add('\\lBrack', new Symbol('\\lBrack', '\u27E6'));
          delimiter.add('\\rBrack', new Symbol('\\rBrack', '\u27E7'));
          delimiter.add('\\lAngle', new Symbol('\\lAngle', '\u27EA'));
          delimiter.add('\\rAngle', new Symbol('\\rAngle', '\u27EB'));
          delimiter.add('\\lbrbrak', new Symbol('\\lbrbrak', '\u2772'));
          delimiter.add('\\rbrbrak', new Symbol('\\rbrbrak', '\u2773'));
          delimiter.add('\\lbag', new Symbol('\\lbag', '\u27C5'));
          delimiter.add('\\rbag', new Symbol('\\rbag', '\u27C6'));
          delimiter.add('\\llparenthesis', new Symbol('\\llparenthesis', '\u2987'));
          delimiter.add('\\rrparenthesis', new Symbol('\\rrparenthesis', '\u2988'));
          delimiter.add('\\llangle', new Symbol('\\llangle', '\u2989'));
          delimiter.add('\\rrangle', new Symbol('\\rrangle', '\u298A'));
          delimiter.add('\\Lbrbrak', new Symbol('\\Lbrbrak', '\u27EC'));
          delimiter.add('\\Rbrbrak', new Symbol('\\Rbrbrak', '\u27ED'));
          delimiter.add('\\lBrace', new Symbol('\\lBrace', '\u2983'));
          delimiter.add('\\rBrace', new Symbol('\\rBrace', '\u2984'));
          delimiter.add('\\lParen', new Symbol('\\lParen', '\u2985'));
          delimiter.add('\\rParen', new Symbol('\\rParen', '\u2986'));
          delimiter.add('\\lbrackubar', new Symbol('\\lbrackubar', '\u298B'));
          delimiter.add('\\rbrackubar', new Symbol('\\rbrackubar', '\u298C'));
          delimiter.add('\\lbrackultick', new Symbol('\\lbrackultick', '\u298D'));
          delimiter.add('\\rbracklrtick', new Symbol('\\rbracklrtick', '\u298E'));
          delimiter.add('\\lbracklltick', new Symbol('\\lbracklltick', '\u298F'));
          delimiter.add('\\rbrackurtick', new Symbol('\\rbrackurtick', '\u2990'));
          delimiter.add('\\langledot', new Symbol('\\langledot', '\u2991'));
          delimiter.add('\\rangledot', new Symbol('\\rangledot', '\u2992'));
          delimiter.add('\\lparenless', new Symbol('\\lparenless', '\u2993'));
          delimiter.add('\\rparengtr', new Symbol('\\rparengtr', '\u2994'));
          delimiter.add('\\Lparengtr', new Symbol('\\Lparengtr', '\u2995'));
          delimiter.add('\\Rparenless', new Symbol('\\Rparenless', '\u2996'));
          delimiter.add('\\lblkbrbrak', new Symbol('\\lblkbrbrak', '\u2997'));
          delimiter.add('\\rblkbrbrak', new Symbol('\\rblkbrbrak', '\u2998'));
          delimiter.add('\\lvzigzag', new Symbol('\\lvzigzag', '\u29D8'));
          delimiter.add('\\rvzigzag', new Symbol('\\rvzigzag', '\u29D9'));
          delimiter.add('\\Lvzigzag', new Symbol('\\Lvzigzag', '\u29DA'));
          delimiter.add('\\Rvzigzag', new Symbol('\\Rvzigzag', '\u29DB'));
          delimiter.add('\\lcurvyangle', new Symbol('\\lcurvyangle', '\u29FC'));
          delimiter.add('\\rcurvyangle', new Symbol('\\rcurvyangle', '\u29FD'));
          delimiter.add('\\Vvert', new Symbol('\\Vvert', '\u2980'));
    }     // ready
  },      // startup

  tex: {
    packages: {'[+]': ['tagformat', 'Lwarp-macros', 'textmacros']},
    tags: "ams",
         tagformat: {
             number: function (n) {
                 if(MathJax.config.subequations==0)
                      return(MathJax.config.section + n);
                 else
                      return(MathJax.config.section + String.fromCharCode(96+n));
             },
         },
  }
}
</script>

<script
    id="MathJax-script"
    src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js"
></script>

</head>
<body>


<div class="bodywithoutsidetoc">


<main class="bodycontainer">


<section class="textbody">

<a id="mwe-autofile-0"></a>

<!--MathJax customizations:-->
<div data-nosnippet
    style="display:none"
>

\(\newcommand{\footnotename}{footnote}\)

\(\def \LWRfootnote {1}\)

\(\newcommand {\footnote }[2][\LWRfootnote ]{{}^{\mathrm {#1}}}\)

\(\newcommand {\footnotemark }[1][\LWRfootnote ]{{}^{\mathrm {#1}}}\)

\(\let \LWRorighspace \hspace \)

\(\renewcommand {\hspace }{\ifstar \LWRorighspace \LWRorighspace }\)

\(\newcommand {\mathnormal }[1]{{#1}}\)

\(\newcommand \ensuremath [1]{#1}\)

\(\newcommand {\LWRframebox }[2][]{\fbox {#2}} \newcommand {\framebox }[1][]{\LWRframebox } \)

\(\newcommand {\setlength }[2]{}\)

\(\newcommand {\addtolength }[2]{}\)

\(\newcommand {\setcounter }[2]{}\)

\(\newcommand {\addtocounter }[2]{}\)

\(\newcommand {\arabic }[1]{}\)

\(\newcommand {\number }[1]{}\)

\(\newcommand {\noalign }[1]{\text {#1}\notag \\}\)

\(\newcommand {\cline }[1]{}\)

\(\newcommand {\directlua }[1]{\text {(directlua)}}\)

\(\newcommand {\luatexdirectlua }[1]{\text {(directlua)}}\)

\(\newcommand {\protect }{}\)

\(\def \LWRabsorbnumber #1 {}\)

\(\def \LWRabsorbquotenumber &quot;#1 {}\)

\(\newcommand {\LWRabsorboption }[1][]{}\)

\(\newcommand {\LWRabsorbtwooptions }[1][]{\LWRabsorboption }\)

\(\def \mathchar {\ifnextchar &quot;\LWRabsorbquotenumber \LWRabsorbnumber }\)

\(\def \mathcode #1={\mathchar }\)

\(\let \delcode \mathcode \)

\(\let \delimiter \mathchar \)

\(\def \oe {\unicode {x0153}}\)

\(\def \OE {\unicode {x0152}}\)

\(\def \ae {\unicode {x00E6}}\)

\(\def \AE {\unicode {x00C6}}\)

\(\def \aa {\unicode {x00E5}}\)

\(\def \AA {\unicode {x00C5}}\)

\(\def \o {\unicode {x00F8}}\)

\(\def \O {\unicode {x00D8}}\)

\(\def \l {\unicode {x0142}}\)

\(\def \L {\unicode {x0141}}\)

\(\def \ss {\unicode {x00DF}}\)

\(\def \SS {\unicode {x1E9E}}\)

\(\def \dag {\unicode {x2020}}\)

\(\def \ddag {\unicode {x2021}}\)

\(\def \P {\unicode {x00B6}}\)

\(\def \copyright {\unicode {x00A9}}\)

\(\def \pounds {\unicode {x00A3}}\)

\(\let \LWRref \ref \)

\(\renewcommand {\ref }{\ifstar \LWRref \LWRref }\)

\( \newcommand {\multicolumn }[3]{#3}\)

\(\require {textcomp}\)

\(\newcommand {\intertext }[1]{\text {#1}\notag \\}\)

\(\let \Hat \hat \)

\(\let \Check \check \)

\(\let \Tilde \tilde \)

\(\let \Acute \acute \)

\(\let \Grave \grave \)

\(\let \Dot \dot \)

\(\let \Ddot \ddot \)

\(\let \Breve \breve \)

\(\let \Bar \bar \)

\(\let \Vec \vec \)

</div>

<a id="mwe-autopage-1"></a>
<div class="titlepage">

<h1>Test</h1>


<div class="author">


<div class="oneauthor">

<p>
Jane Doe
</p>
</div>

</div>


<div class="titledate">

<p>
March 31, 2024
</p>
</div>

</div>

<p>
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Etiam lobortis facilisis sem. Nullam nec mi et neque pharetra sollicitudin. Praesent imperdiet mi nec ante. Donec ullamcorper, felis non sodales commodo, lectus velit ultrices augue, a dignissim nibh lectus placerat
pede. Vivamus nunc nunc, molestie ut, ultricies vel, semper in, velit. Ut porttitor. Praesent in sapien. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Duis fringilla tristique neque. Sed interdum libero ut metus. Pellentesque placerat. Nam rutrum augue a leo.
Morbi sed elit sit amet ante lobortis sollicitudin. Praesent blandit blandit mauris. Praesent lectus tellus, aliquet aliquam, luctus a, egestas a, turpis. Mauris lacinia lorem sit amet ipsum. Nunc quis urna dictum turpis accumsan semper.
</p>

<p>
Testing bliography citation [1] and testing referenced equation <span class="textup">(<a href="mwe.html#eq:test">1</a>)</span>:
</p>

<span class="hidden"> \(\seteqnumber{0}{}{0}\)</span>

<!--

                                                                                                                                 1  p
                                                                                                                                   = a2 + b2                                                                                                   (1)--><a id="eq:test"></a><!--
                                                                                                                                 x
-->

<p>

\begin{equation}
\label {eq:test} \frac {1}{x} = \sqrt {a^2+b^2}
\end{equation}

</p>
<!--
...... section References ......
-->
<h4 id="autosec-3">References</h4>
<a id="mwe-autopage-3"></a>


<ul class="list" style="list-style-type:none">

<li>
<p>
<span class="listmarker">[1]&#x2003;</span> L[eslie]&nbsp;A. Aamport. The gnats and gnus document preparation system. <i>G-Animal’s Journal</i>, 41(7):73+, July 1986. This is a full ARTICLE entry.
</p>
<p>

</p>
</li>
</ul>

<a id="mwe-autofile-last"></a>
</section>

</main>

</div>

</body>
</html>

This html renders well in a web browser:

However, it is not being parsed properly by pandoc. Here is a screenshot of the resulting docx conversion:

I tried converting the mathjax html document into a mathml html one before feeding this to pandoc. For this, I used the tex2chtml-page command line tool from MathJax-demos-node which seemed to do mostly what I wanted. However the equation numbers disappear in the output html from tex2chtml-page. And when I use this as input to pandoc, the generated docx file contains Undefined control sequence \seteqnumber lines.

I am not quite sure whether I am missing something in how I should use pandoc here, if this is an issue with pandoc or an issue with lwarp. As such, this discussion is somewhat mirrored in bdtc/lwarp#25

Any suggestion would be greatly appreciated.

jgm · 2024-03-31T18:10:31Z

jgm
Mar 31, 2024
Maintainer

What is the problem going straight from the tex to docx? Aside from the equation number, that works.
When reading HTML with embedded math, you should use -f html+tex_math_dollars or -f html+tex_math_single_backslash.
The lwarp output contains a big section with macro definitions using \let and \def. texmath (which does our math conversions) doesn't support these. Pandoc can handle the macro definitions when reading latex, but not when reading mathjax bits in HTML.
If you delete the macro definitions (which are just boilerplate in this case) and add $$ delimiters around the \begin{equation}...\end{equation}, and use -f html+tex_math_dollars, then it works. If mathjax allows these \begin{equation}..\end{equation} environments outside of the $$ delimiters (it didn't used to), then we may want to modify pandoc so that it can handle them.
In principle it would also be possible to modify pandoc so that the +latex_macros extension works with HTML; then the macro definitions could be interpreted.

0 replies

jgm · 2024-03-31T18:22:43Z

jgm
Mar 31, 2024
Maintainer

If +raw_tex is also enabled, then the bare equation environment will get parsed as a raw tex chunk. This will come through unscathed in tex output, but it will be dropped in docx output, so it doesn't help in this case.

I'm not sure what the best improvement would be here. We could change the HTML reader so that (perhaps depending on the extensions enabled) it parses \begin{equation}..\end{equation} as a display math environment. Then we'll get the proper equation in every format, including docx, but the number will disappear.

4 replies

tvercaut Mar 31, 2024
Author

Many thanks for the insight and prompt reply!

Going straight from tex to docx with pandoc often fails to produce appropriate outputs for me. Even in the MWE shown above, the oputput of \blindtext is for example discarded. Hence my attempts at going first to html using lwarp (or tex4ht or latexml) and then from html to docx with pandoc.

2-4. I had tried -f html+tex_math_dollars and -f html+tex_math_single_backslash but this didn't seem to help in this instance.

I managed to go a little further by using lwarp then tex2chtml-page and then pandoc with a filter to discard lwarp's bespoke handling of equation numbering. It is a bit brittle and looses the equation numbers but so far it's the best I get.

For the record, here is my pandoc lua filter:

function Span(elem)
    if elem.classes[1] == "hidden" then
        return {}
    end
    return elem
end

jgm Mar 31, 2024
Maintainer

Pandoc won't be able to handle arbitrary tex, especially using external packages like blindtext, but if you stick to the basics it should give decent results. For non-basic things you can sometimes work around its limitations by adding a few macro definitions (it can interpret standard latex macro definitions, so for example you could add your own definition of \blindtext).

jgm Mar 31, 2024
Maintainer

If you're okay with losing equation numbers, then one option could be to use -f html+raw_tex, which will give you RawInline (Format "tex") "\begin{equation}...\end{equation", and then use a Lua filter to convert this into a regular Math element, which will then be able to appear in docx.

tvercaut Apr 1, 2024
Author

Thanks. I'll look into this.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parsing mathjax html document produced with lwarp #9615

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Parsing mathjax html document produced with lwarp #9615

tvercaut Mar 31, 2024

Replies: 2 comments · 4 replies

jgm Mar 31, 2024 Maintainer

jgm Mar 31, 2024 Maintainer

tvercaut Mar 31, 2024 Author

jgm Mar 31, 2024 Maintainer

jgm Mar 31, 2024 Maintainer

tvercaut Apr 1, 2024 Author

tvercaut
Mar 31, 2024

Replies: 2 comments 4 replies

jgm
Mar 31, 2024
Maintainer

jgm
Mar 31, 2024
Maintainer

tvercaut Mar 31, 2024
Author

jgm Mar 31, 2024
Maintainer

jgm Mar 31, 2024
Maintainer

tvercaut Apr 1, 2024
Author