sibling intent references

[davidcarlisle]

Currently we say

references prefixed with $. An argument reference such as $name references a descendent element that has an attribute arg="name". Unlike id attributes, arg do not have to be unique within a document. When searching for a matching element the search should only consider descendants, while stopping early at any elements that have a set intent or arg attribute, without descending into them.

We could say

references prefixed with $. An argument reference such as $name references a descendent or sibling element that has an attribute arg="name". Unlike id attributes, arg do not have to be unique within a document. When searching for a matching element the search should first consider descendants, while stopping early at any elements that have a set intent or arg attribute, without descending into them. If this search fails, and the current element is not <math>, the search is repeated from the parent element.

---

This would allow intents on groups of table rows:

<mtr intent="sum($a,$b,$c)"><mtd><mi arg="a">a</mi></mtd><mtd><mo>+</mo><mi arg="b">b</mi></mtd></mtr>
<mtr>                       <mtd></mtd>                  <mtd><mo>+</mo><mi arg="c">c</mi></mtd></mtr>

It would also allow (which may not be seen as an advantage) expressions missing nested mrow structure to be annotated with intent without having to add in the missing mrow.

1 + log x

<mrow>
<mn>1</mn>
<mo>+</mo>
<mi arg="l" intent="$l($x)">log</mi>
<mi arg="x">x</mi>
</mrow>

[dginev]

I have a kind of a pedantic point, but to my reading:

<mi arg="l" intent="$l($x)">log</mi>

should expand into an infinite log(log(log(...log(x)))...). The self-reference is generally a scary proposal.

---

If the descendant-only args are getting to be too constraining, why not allow an additional #id reference, for the rare cases where one has to express such "far away" arguments:

<mtr intent="plus($a,$b,#c)">
  <mtd><mi arg="a">a</mi></mtd>
  <mtd><mo>+</mo><mi arg="b">b</mi></mtd>
</mtr>
<mtr>
  <mtd></mtd>
  <mtd><mo>+</mo><mi id="c">c</mi></mtd>
</mtr>

I admit this is biased, as a common latexml workflow is to already generate id attributes.

Personally, I'm fine without either extension, and imposing a constraint that keeps some simplicity in reasoning about the expressions.

P.S. sum and plus are also good candidates for w3c/mathml-docs#40 aliases.

[davidcarlisle]

should expand into an infinite log(log(log(...log(x)))...). The self-reference is generally a scary proposal.

I think it's OK (and not dependent on the change suggested here) and covered by the don't descend in to elements with arg rule, but if not we should clarify the spec:-)

If the descendant-only args are getting to be too constraining, why not allow an additional #id

I am not sure I like id here for same reason we didn't use id for intent refereces, it introduces document scope.

But there are variant proposals, eg rather than uncontrolled searching of parent elements, something like

<mtable arg="m">
<mtr intent-scope="m" intent="...$x ">...</mtr>
<mtr> ...<mi arg="x"   ...</mtr>

meaning references in the intent are searched from the mtable rather than the mtr

[dginev]

For the double <mtr> example, there is a slightly larger problem - how would AT know to silence the pieces in the second row that are left unused?

With the proposal in this issue, you may end up producing the narration "a+b+c +c", repeating the second row. So it seems that such an approach would need either some extra annotation, or an extra bit of convention, to silence the unnecessary pieces. This problem is also there if we used #id instead.

To me falling back to annotating a common ancestor (so mtable here) has just the right balance in place.

[NSoiffer]

@dginev: the 'c' would only get referenced once because (presumably) the mtable would list the mtrs to speak.

How about this tweak to the wording:

references prefixed with $. An argument reference such as $name references a descendant or a sibling element that has an attribute arg="name". Unlike id attributes, arg do not have to be unique within a document. When searching for a matching element the search should first consider descendants, while stopping early at any elements that have a set intent or arg attribute, without descending into them. If this search fails, and the current element is not , the siblings but not their decedents are searched for the matching arg.

I'm actually not a fan of either, but at least this one minimizes the chances of accidentally picking up an arg because it was generated by some template that is used in two separate rows (I hope that makes sense -- I've run out of time).

[davidcarlisle]

the siblings but not their decendents are searched for the matching arg.

I actually started with that, but it's not enough to structurally mark up a=b+c+d split over two rows, whch was one of the original intenet use cases, and addressed in the examples in the spec with append and extend keywords, but the examples came from an intent proposal with a rather different syntax.

[brucemiller]

I'm not seeing how any of the variants of finding the referent save you from the self-reference in

<mi arg="l" intent="$l($x)">log</mi>

If non-descendants are allowed, AT will find this node for $l, without problem. The problem is how to instantiate the speech for this node. To avoid recursion, it sounds like you're wanting the rule to be "Use the intent the first time you see a node, and the content the second time". (?)

Moreover, even if you avoid the recursion, I'd expect the "x" to be spoken twice, unless you introduce an extra rule about not speaking nodes with @arg unless they're explicitly referenced (which doesn't seem like a wise rule ... or is it?).

[brucemiller]

Aside:

I share @dginev 's bias for ID's, but exactly because it allows document scope. In current MathML and HTML(5), the distinction between math and text is too strong and the restrictions upon each containing the other is very limiting. Real world documents often have groups of aligned equations, equation numbers, intertext with more than phrase content, diagrams containing math which contains graphical things, etc (this is not just a TeX problem, though painfully apparent there). Consequently, the author/generator has to make some unpleasant choices, such as using HTML tables containing math fragments, rather than a MathML mtable containing mtext fragments, or vice-versa.

One could imagine how a future version of intent which could cross these boundaries would provide a solution to this kind of problem; To assert that a particular mess of mixed markup represents a specific set of equations, for example.

Not that I'm proposing we take that up here and now. We now return you to our regularly scheduled discussion.

[davidcarlisle]

I'm not seeing how any of the variants of finding the referent save you from the self-reference

yes probably right, so back to the drawingboard...

https://w3c.github.io/mathml/#mixing_intent_examples_mtr

has "magic" intent values

<mtr intent="append">

meaning, more or less:

"read this by the default table row reading, but logically it is a continuation of the expression on the previous row."

But we don't say that in the spec. Can we make such a scheme work?

We also have the option of not addressing tables, although having an intent mechanism for "re-joining" table cells has been a reason for downplaying malignmark. I don't think saying you can use an intent on the mtable is usable, we would be better to say nothing. If you have an aligment of a few dozen rows, making a combined intent for all of them in one attribute will be a horrible fragile format that is impossible to edit.

[dginev]

If you have an aligment of a few dozen rows, making a combined intent for all of them in one attribute will be a horrible fragile format that is impossible to edit.

So I wrote this by hand and it is at least possible to edit it. Improving ergonomics is a great goal though, happy to co-brainstorm.

<mrow intent="$table">
  <mo>{</mo>
  <mtable arg="table"
     intent="system-of-equations($row1,$row2,$rows34,$row5,$row6,$row7,$row8,$row9,$row10,$row11,$row12)">`
    <mtr arg="row1"><!-- contents --></mtr>
    <mtr arg="row2"><!-- contents --></mtr>
    <mtr arg="rows34"><!-- contents --></mtr>
    <mtr><!-- contents, referenced by row above --></mtr>
    <mtr arg="row5"><!-- contents --></mtr>
    <mtr arg="row6"><!-- contents --></mtr>
    <mtr arg="row7"><!-- contents --></mtr>
    <mtr arg="row8"><!-- contents --></mtr>
    <mtr arg="row9"><!-- contents --></mtr>
    <mtr arg="row10"><!-- contents --></mtr>
    <mtr arg="row11"><!-- contents --></mtr>
    <mtr arg="row12"><!-- contents --></mtr>
  </mtable>
</mrow>

All rows are independent, except rows 3 and 4 which I decided to join together for example's sake. Since you wouldn't need to enumerate each row as an argument, unless there was a particular override - such as order being shuffled, rows getting ignored, etc.

Edit: Ha, I should not write XML before my first coffee of the day. I tricked myself and wrote the snippet as if I had "sibling references" available. But of course we don't, so this will actually be larger:

<mrow intent="$table">
  <mo>{</mo>
  <mtable arg="table"
     intent="system-of-equations($row1,$row2,equals($row3lhs,_($row3rhs,$row4rhs)),
             $row5,$row6,$row7,$row8,$row9,$row10,$row11,$row12)">`
    <mtr arg="row1"><!-- contents --></mtr>
    <mtr arg="row2"><!-- contents --></mtr>
    <mtr><!-- contents, including arg row3lhs, and arg row3rhs --></mtr>
    <mtr><!-- contents, including arg row4rhs --></mtr>
    <mtr arg="row5"><!-- contents --></mtr>
    <mtr arg="row6"><!-- contents --></mtr>
    <mtr arg="row7"><!-- contents --></mtr>
    <mtr arg="row8"><!-- contents --></mtr>
    <mtr arg="row9"><!-- contents --></mtr>
    <mtr arg="row10"><!-- contents --></mtr>
    <mtr arg="row11"><!-- contents --></mtr>
    <mtr arg="row12"><!-- contents --></mtr>
  </mtable>
</mrow>

[davidcarlisle]

equals($row3lhs,_($row3rhs,$row4rhs)

yes this, and this is a simple case with only two colums and only one row wrapped. In general you'd end up referencing each cell of the table in the top level intent.

Actually generating such a thing is no harder than generating other forms, you have to collect the data and write it out somewhere. But this is making a write-only format that you are never going to want to edit, keeping paralllel tree structures in the xml and in the attribute.

Your first form with

<mtr><!-- contents, referenced by row above --></mtr>

looked a lot nicer I still wonder if we can enable some form of that without over-expanding the search required, or introducing loops or...

[dginev]

Thinking out loud...

"read this by the default table row reading, but logically it is a continuation of the expression on the previous row."
But we don't say that in the spec. Can we make such a scheme work?

My worry is that this is a bit too ad-hoc as a construct - it only covers a very specific case of two adjacent elements that behave as appended to each other. And its "magic" only goes so far - if the appending is partial (cherry-picking some mtd nodes from one row, then some others from another couple of rows) - it requires extra markup to specify how to ignore/jump over/mute some table cells that may otherwise be picked up auto-magically.

I also generally don't like annotation directives with implied arguments on large trees, as a general design preference (sorry). It gets very hard to debug which arguments one thought about and which were actually picked up by the software. Because of all the implicit dependencies even the fixes of bugs themselves end up looking magical. The exception is with avoiding the argument list for default behavior, by virtue of avoiding any annotation at all. That makes a lot of sense, especially since most presentation mathml is raw/unannotated and needs to have some reasonable baseline.

It seems more rewarding to aim for a more general feature that allows extending the argument scope for intent references. But to otherwise remain open to the full breadth of intent expressions we can build - sometimes appending, sometimes prepending, sometimes mix-and-matching (tabular diagrams), etc. And yes, that requires explicitly naming each argument.

Bad idea 1 (search scope for each arg):

Extending the $argref scope to the parent element (or generally - some ancestor element above), remains close to our current subtree-based traversals (without reaching into the sibling axis). In that case, we have to avoid "arg" name collisions in the same search scope (e.g. unique "arg" names for the entire mtable allow conflict-free search for a mtr that has a parent scope).

I worry that doing such scoping changes automatically for any argument is prone to unpredictable outcomes - it would be safer for the annotation to request scope broadening explicitly.

As another direction of consideration, how about expanding the argument reference syntax, to allow searching from a specified height of ancestry above? I don't have great syntax ideas however. So here is an obvious one instead:

• $name searches under the current element, as we do currently
• $./name does the same
• $../name searches from the parent
• $../../name searches from the grandparent
• etc upwards,
• $//name searches from the <math> root

I like the predictability and performance of this idea, but seeing this level of syntax extension is rather scary. Heading towards XPath...

The example of two adjacent rows would then be:

<mrow intent="system-of-equations($table)">
  <mo>{</mo>
  <mtable arg="table">
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr intent="plus($a,$b,$../c)">
      <mtd><mi>a</mi></mtd>
      <mtd><mo>+</mo><mi>b</mi></mtd>
    </mtr>
    <mtr arg="c" intent="$cellc">
      <mtd></mtd>
      <mtd><mo>+</mo><mi arg="cellc">c</mi></mtd>
    </mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
    <mtr><!-- contents --></mtr>
  </mtable>
</mrow>

Still thinking on other ideas...

[davidcarlisle]

@dginev nice summary. I don't think the "magic" keywords really fits the design either.

I very nearly suggested a syntax for "search from parent" I was actually thinking of $$c where the $ adds a parent level $../../name looks more "normal" but people may expect it is some path like $../mtr/mrow/x but that's a minor point and I think I like this direction better than the "always search parent tree" version at the start of this issue

[NSoiffer]

I also like the path spec idea more. We need to decide how general it is. The more general we make it, the harder it is to implement, so I would like to keep it as simple as possible to solve the known problems. So .. is fine, but $../c/d (find arg 'c', then find 'd' inside it) seems to much.

[davidcarlisle]

@NSoiffer

, but $../c/d (find arg 'c', then find 'd' inside it) seems to much.

yes as I commented above, I worry that making it look like a path syntax makes it promise more than we want to offer.

I think I quite like $c search descendents, $$csearch descendents of parent, and possibly $$$$$c search descendents of great great grandparents. but just allowing ../ paths so $c and $../c and $../../../../c is an equivalent suggestion.

Probably any such rule would need some side constraints to avoid accessing above <math> and to avoid cyclic self references,

[davidcarlisle]

I tried writing something in my fork

so grammar description

https://davidcarlisle.github.io/mathml/#intent_reference

table examples

https://davidcarlisle.github.io/mathml/#mixing_intent_examples_mtr

This is not a PR just experimenting...

I think we still haven't got this right yet.

conceptually I just want to say "treat row 1 and 2 as an mrow and read as a=b+c-d+e-f but that is quite hard to do as it is already hard on an mrow, even without tables.

The system must be able to read an mrow but as soon as you need any kind of intent you need a prefix functional form, so perhaps intent=equal(a,sum @infix (b,c,minus(d),e,minus(f))) when really I want intent=infiix-expression($mrow) and for the merged table rows intent=infix-expression($thisrow,$$nextrow)

[dginev]

The double dollar ($$) means "scalar dereference" in perl (docs), and is an acquired taste. Especially its chained form - $$$...name, which some of us try to avoid in practice - it is hard to read accurately when you pass into 3+ dollar territory.

I think we have a good case that $../c is preferrable to $$c, just for being closer to standard path syntax.

And thinking some more, a more legible variant is to lead with the paths, as in ../$c, ../../$c, //$c.

As Neil says, we should constrain the syntax to avoid nested named lookups as in ../$d/$c, which seem too advanced for our current ambitions.

I am still skeptical we should be making intent even more complicated. My offline brainstorming is continuing in a direction without any new syntax added. But there are no obvious easy wins so far, as complicating the lookup algorithm gets messy rather quickly.

[davidcarlisle]

@dginev yes ../..$c looks better than $../../c` Although I agree the whole thing may be too complicated.

if we could address the use case of marking an infix expresson split over muliple mtr without having to refrerence each individual item I'd be happy to drop the whole idea, at least in first version, extensions can be added later if proved needed.

[davidcarlisle]

If some mathml denotes a+b-c and I'm interpreting it in a computational system I really want to interpret it as a+b-c not (a+b)-c or sum(a,b,minus(c)) it is more than just a "speech hint" imposing a particular order might be computationally wrong.

Most of the time you can avoid the issue by simply not giving an intent, but sometimes when combining with other subterms it would be convenient to have an intent that did not change the default interpretation.

if (say) math-expression($a,$b,....) meant concatenate all the arguments, and read as default,

then

a  =b + c
      + d
   = x

could be something like

<mtable intent="aligned-equations(math-expression($r1,$r2),$r3)">

which still has everything at the top but only referencing child rows, and making the wrapped line explicit. Might be enough to avoid sibling refs?

[dginev]

Computation is such an open-ended terrain for distractions that every time it gets brought up I come up with enough material for a novel-sized comment. Hiding that in a details tag, since it's largely an aside to the main issue topic. Expand at your own risk:

If some mathml denotes a+b-c and I'm interpreting it in a computational system I really want to interpret it as a+b-c not (a+b)-c or sum(a,b,minus(c)) it is more than just a "speech hint" imposing a particular order might be computationally wrong.

There is (at least) one correct mathematical "order of operations", and an underlying part of the synergy between accessibility and computation is that

1. we reuse the "accepted" technical concept names and
2. the intent expression markup follows the mathematical argument structure.

If the operators are grouped wrong, it is not just a problem with invalid computation, the speech will be misleading/broken as well. To illustrate with a fully non-commutative example, consider the left-associative arithmetic:

a-b/c-d

There is a single order of operations (and corresponding "operator tree") that is accurate mathematically - subtract(subtract(a,divide(b,c)), d). If the annotation is instead mistakenly right-associative, as in subtract(a, subtract(divide(b,c),d)) the accessible outcome is also harmed in at least two ways:

• navigation will visit the expression components in the wrong order, which would imply a parenthetical to a non-sighted user, as in a-(b/c-d).
• if AT produces an audible "pause" to delimit a longer phrase, or speaks out in full the operations with their arguments, the speech will also convey incorrect mathematics, again as if the author wrote a-(b/c-d).

So a wrong operator tree is always an annotation "bug". Both for a11y outcomes and for computational applications.

If someone insisted to pass the pieces along without any markup, as your comment got the urge to do: that will likely be the "raw presentation" default ( open topic at #253 ). If we wanted to be really explicit about it, we now have _(a,minus,b,divide,c,minus,d).

I still agree that there will be hard cases where "flat pieces" are at least initially needed: Say a (naive) grammar won't be able to parse an expression. Then the MathML generator will need to automatically mark "can not do better than pieces here". But I think we already have sufficient tools for that: 1) no intent markup at all and 2) underscore _ overrides with references.

To draw from real-world practice: I have an example where an alignment breaks a ( parenthetical opened on row 2, and offers the closing ) on row 3. The expression getting grouped is a long "Π" product. See equation 11 of ar5iv:0804.3731. A naive grammar that does not parse an aligned expression with all rows combined on input will fail to produce an operator tree.

I suspect it is exactly in expressions of such large size that having the correct operator tree will be a life-saver for accessibility. With good navigation one can get an overview of the entire baseline, then jump over irrelevant components, before encountering the barrage of indexed variable names.

So your example:

a  =b + c
      + d
   = x

to me should be either the grammatical:

<mtable intent="derivation(a,plus(b,c,d),x)">

or the semi-narrative:

<mtable intent="derivation(a,_(b,plus,c,$row2),x)">

The fully narrative variant is completely flat as an expression, and very underwhelming, I'll save myself the typing and let you imagine it.

(Aside 2: there is another formalization nitpick here, since derivation is a commonly accepted term, but current Content MathML and OpenMath work is somewhat free-spirited in using various loose notions of "equality" instead, as is the one in relation4:eqs. It is quite different if an equality asserts a premise, versus it indicating a proof step. The former one can use axiomatically, the latter needs to be verified).

[davidcarlisle]

@dginev

we reuse the "accepted" technical concept names and

the intent expression markup follows the mathematical argument structure.

Yes although starting from that shared premise we reach different conclusions. I did not intend a formal proof derivation here but simultaneous equations (or simultaneous inequalities in the example in the spec). Even had it been intended to be a proof I'd find "derivation" a slightly uncommon term to use unless (as in the wikipedia page you linked to) you are working in formal mathematical logic. So defiinitely I would use

but current Content MathML and OpenMath work is somewhat free-spirited in using various loose notions of "equality"
instead

here so equals(... or simultaneous-equations(... not derivation(....

But returning to your first point

If the operators are grouped wrong, it is not just a problem with invalid computation

That was my point: the expression may be intrinsically infix and n-ary and any binary grouping is sub-optimal 1+2+3 is neither (1+2)+3 nor 1+(2+3) infix linear expressions can be evaluated efficiently eg https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation but adding parens forces each subterm to be evaluated separately.

we now have _(a,minus,b,divide,c,minus,d)

No, that does not address the use case at all. (Which incidentally is not my use case, it predates this WG, an intention to devise intent in a way that allows the structure of terms split over alignment cells to be reconstructed.)

_( is a useful way to force a particular reading, but at the price of an anonymous function head so losing any semantic use of the intent. As @NSoiffer has commented, reading alignments is tricky but the underlying speech engines already have code to do this well. We should be able to give the semantic hint that this is a set of wrapped equations without over-riding the reading of the entire alignment. The current spec has intent="extend" to directly address this, but those examples come from a CG document based on rather a different intent grammar, so this issue is seeing how far the current version meets this requirement.

To draw from real-world practice: I have an example where an alignment breaks a ( parenthetical opened on row 2, and offers the closing ) on row 3.

Yes. That is very common and a FAQ at tex help sites eg

https://tex.stackexchange.com/q/89615/1090

So the intent="extend" in the current spec (which comes from @samdooley 's proposal originally, I think) is addressing exactly that use, just denoting that row3 is a visually wrapped extension of the expression started in row 2. It is just a simple semantic hint, not requiring the document author to generate full speech _(.... strings for the entire display.

[NSoiffer]

This is complicated! We will bring it up again for the next version. Potentially it could be solved with a tbody.

[brucemiller]

Seems closable to me; This quickly gets too complicated. Can the need be solved with a simple (simplistic) generic solution like mrow($a,$b,$c) as mentioned in #337?

[davidcarlisle]

@NSoiffer

This is complicated! We will bring it up again for the next version. Potentially it could be solved with a tbody.

I think we can close. mtbody would solve most of the issues (for mathml-next) and for now the proposed :continued-equation property (which could perhaps be replaced by a more generic :continued-row property) which allows "cosmetic" wrapped rows to be distiguished from rows reprenting a semantc unit, catches several simple but important cases without having to $-reference very cell.