ANN: jog – a Lua utility library to "take an energetic trot along the pandoc AST"

[tarleb]

I'm happy to announce the first release of jog, a pure Lua library that provides functionality similar to pandoc's :walk method, but with different properties and features.

Where pandoc's :walk leverages lazy marshaling and a functional programming style, jog embraces mutability and strict traversal of elements. Both methods have their advantages, depending on the task at hand, and jog aims to fill the niche for when walk is less suitable.

Jog's primary use-case are repeated traversals of the same document object with different filters, especially when many nodes must be inspected and "filtered". Repeated lazy marshaling and evaluation can make built-in traversals comparatively slow, and jog allows to avoid much of that overhead in such cases.

Additionally, users used to libraries such as panflute often find the lack of element contexts confusing, as finding the parent node of an AST object is not always straight-forward. With jog it's possible to enable context = true in filters, which then gives access to the list of parent nodes via a second filter function argument:

local filter = {
  context = true,
  Inline = function (inln, context)
    -- print the list of nodes that lead to this point.
    print(context)
    -- ...
  end,
}
jog(doc, filter)

---

The library should be considered experimental for now. Feedback is most welcome, as usual.

[fiapps]

The ability to look at context would definitely have simplified some of the filters I've written. I will definitely consider jog the next time I have another problem like that to solve.

[tarleb]

I should add that I'm looking for ways to integrate this (or somthing similar) into pandoc's Lua engine at some point, but that might take quite a while.

[jgm]

Do you have (even informal) benchmarks to show the performance improvement?

[jgm]

Interesting. I wonder why the difference between counting Str and counting Para?

[tarleb]

My hypothesis is that walking in Haskell is significantly faster than doing it in Lua, and that calling into Lua from Haskell is slow. But I don't know why that would be the case, and I haven't actually measured any of the steps.

[tarleb]

Or rather: marshaling is the bottleneck, and there are fewer elements to marshall when counting Para. Jog always has to marshall and evaluate all elements, but only once. Walk is more selective and marshals only the elements it needs, but must do so every time it is called.

Jog "pays for the full document upfront", which is a bad idea if only every 30th or so element is actually inspected by the filter.

[jgm]

Now we just need to create sprint, which goes much faster but gets tired out if you have to look at too many elements.

[tarleb]

Or a function stroll that walks lazily, but remembers and caches every marshaled Lua object