SVG + MathML interop for accessibility

[dginev]

I am opening an issue to discuss and develop an interop strategy for mixed SVG+MathML(+HTML) content, starting with existing examples from math practice.

Three examples:

1. SVG as a host language (<foreignObject> would be the typical host for inner MathML).

   • idea: Any diagram that is not oriented via the usual baseline fits better as an SVG. For this example, here is an up-right-diagonal arrow operator between two nodes, where the source/target math objects are wrapped in a <foreignObject>, as is the math object annotating the arrow (which explains the transformation).

     <svg height="78.64" overflow="visible" version="1.1" width="93.86">
       <g transform="translate(0,78.64) matrix(1 0 0 -1 0 0) translate(46.93,0) translate(0,39.32)" fill="#000000" stroke="#000000" stroke-width="0.4pt">
         <g transform="matrix(1.0 0.0 0.0 1.0 -42.32 -34.71)">
           <g transform="matrix(1 0 0 -1 0 57.88)">
             <g transform="matrix(1 0 0 1 0 9.34)">
               <g transform="matrix(1 0 0 -1 20.04 0) translate(55.35,0) matrix(1.0 0.0 0.0 1.0 -4.64 -4.73)" fill="#000000" stroke="#000000">
                 <foreignObject width="9.28" height="9.46" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible">
                   <math><mi>S</mi></math>
                 </foreignObject>
               </g>
             </g>
             <g transform="matrix(1 0 0 1 0 57.88)">
               <g transform="matrix(1 0 0 -1 0 0) translate(20.04,0) matrix(1.0 0.0 0.0 1.0 -15.42 -3.46)" fill="#000000" stroke="#000000">
                 <foreignObject width="30.85" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible">
                   <math><mrow><mi>R</mi><mo>/</mo><mi>I</mi></mrow></math>
                 </foreignObject>
               </g>
             </g>
           </g>
         </g>
         <path d="M -8.82 -22.9 L 23.06 5.06" style="fill:none"></path>
         <g transform="matrix(0.75182 0.65936 -0.65936 0.75182 23.06 5.06)" stroke-dasharray="none" stroke-dashoffset="0.0pt" stroke-linecap="round" stroke-linejoin="round" stroke-width="0.31999pt">
           <path d="M -1.66 2.21 C -1.52 1.38 0 0.14 0.42 0 C 0 -0.14 -1.52 -1.38 -1.66 -2.21" style="fill:none"></path>
         </g>
         <g transform="matrix(1.0 0.0 0.0 1.0 2.85 -23.21)" fill="#000000" stroke="#000000">
           <foreignObject width="9.01" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible">
             <math><mi>ψ</mi></math>
           </foreignObject>
         </g>
       </g>
     </svg>

   • sample rendering:
     

2. MathML as host language (<mtext> and <mglyph> would be usual carriers for SVG inside MathML).

   • idea: An arrow operator between two variables in <mi> nodes, where the arrow is itself annotated with a MathML object.

   <math>
     <mi>A</mi>
     <mtext>
       <svg id="S0.Ex1.m1.pic1nest" class="ltx_picture" height="17.17" overflow="visible" version="1.1" width="52.92">
         <g fill="#000000" stroke="#000000" transform="translate(0,17.17) matrix(1 0 0 -1 0 0) translate(0,2.98) translate(26.46,0) translate(0,9.34)">
           <g stroke-width="0.4pt" color="#000000">
             <g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 -22.93 -2.42)">
               <foreignObject height="9.69" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="45.85">
                 <math class="ltx_Math" alttext="(f\circ g)\circ h" display="inline">
                   <mrow>
                     <mo stretchy="false">(</mo>
                     <mrow>
                       <mi>f</mi>
                       <mo lspace="0.222em" rspace="0.222em">∘</mo>
                       <mi>g</mi>
                     </mrow>
                     <mo rspace="0.055em" stretchy="false">)</mo>
                   </mrow>
                 </math>
               </foreignObject>
             </g>
             <path d="M 25.54 -8.1 L 20.65 -8.1 L 19.47 -9.07 L 17.1 -7.13 L 14.74 -9.07 L 12.38 -7.13 L 10.02 -9.07 L 7.65 -7.13 L 5.29 -9.07 L 2.93 -7.13 L 0.57 -9.07 L -1.79 -7.13 L -4.16 -9.07 L -6.52 -7.13 L -8.88 -9.07 L -11.24 -7.13 L -13.61 -9.07 L -15.97 -7.13 L -18.33 -9.07 L -20.69 -7.13 L -23.05 -9.07 L -26.18 -8.1 L -26.18 -8.1" style="fill:none"></path>
           </g>
           <g stroke-dasharray="none" stroke-dashoffset="0.0pt" stroke-linecap="round" stroke-linejoin="round" stroke-width="0.31999pt" transform="matrix(1.0 0.0 0.0 1.0 25.54 -8.1)" color="#000000">
             <path d="M -1.66 2.21 C -1.52 1.38 0 0.14 0.42 0 C 0 -0.14 -1.52 -1.38 -1.66 -2.21" style="fill:none"></path>
           </g>
         </g>
       </svg>
     </mtext>
     <mi>B</mi>
   </math>

   • sample rendering:
     

3. Simple MathML and HTML mixing.

   • idea: As a simplest example, take a list of three variables: $x_i, y_i,\text{ and }z_i$, where and is additionally emphasized via an HTML <em> element.

   • option 1: Host in HTML, with MathML tightly capturing variable names

     <span>
       <math><msub><mi>x</mi><mi>i</mi></msub></math>,
       <math><msub><mi>y</mi><mi>i</mi></msub></math>,
       <em>and</em>
       <math><msub><mi>z</mi><mi>i</mi></msub></math>
     </span>

   • option 2: Host in MathML, using <mtext> for interleaved text (and other HTML-grade material)

     <math>
       <msub><mi>x</mi><mi>i</mi></msub>
       <mo>,</mo>
       <msub><mi>y</mi><mi>i</mi></msub>
       <mtext><span>, <em>and</em> </span></mtext>
       <msub><mi>z</mi><mi>i</mi></msub>
     </math>

minor note: the SVG examples have been generated with help from latexml's Tikz support (then trimmed down for size).

---

The question I would like to open with these examples is how to further annotate them for accessibility, and how the different AT systems should interoperate.

SVG has a range of options for annotating accessible walks through a diagram, using its <title> element on each sub-component, as well as ARIA's role, aria-labelledby and aria-describedby attributes. I likely do not know the best resource to link to, but I found a reasonable example diagram from svg-access-w3cg, namely composed-tree.svg.

1. The natural connection is that for a given MathML element <math id="mid"> one can use the id attribute in an ARIA annotation, such as from an outer SVG <g aria-labelledby="id1 ... mid ... idn"> to indicate the right traversal order. This is also the likely mechanism if we are annotating regular HTML nodes.

2. Similarly, the MathML Intent mechanism allows to indicate e.g. <mtext arg="diagram"><svg>...</svg></mtext> and point to this SVG wrapper node via the reference syntax $diagram from some outer ancestor node. A basic use is <mrow intent="$diagram(A,B)">...</mrow>.

It would be great if we could have a cross-group discussion if these kinds of constructions are already 1) possible to annotate, 2) easy to evaluate by AT and 3) good enough for MathML 4. I think most importantly we need to find out if AT vendors find it possible to interoperate.

As a possible alternative, one could consider if the intent mechanism could also be suggested for adoption and use in SVG diagrams, given the added convenience of annotating full tree structures, over the simple flat list of ids offered by ARIA (which may require additional scaffold elements).

And an explicit CC to @bkardell and @NSoiffer , who expressed interest in developing a few meaningful examples, before deciding on next steps.