Is there a way to run filter concurrently?

[mrkissinger]

I'm using a filter (https://github.com/pandoc-ext/diagram) to embed diagrams genereted by plantuml.
When I wrote a long article, which contains more than 100 diagrams, pandoc took several minutes to finish convertion (md -> docx).
I have a server with 20 CPU cores, but only one core is running.

Is it possible to run filter concurrently? Just like GNU Make with "-j" option to specify the simultaneous jobs.

[wlupton]

I haven't used this filter, but I've done something similar where I run a python program to check and reformat a protobuf payload. This also runs quite slowly, but this is for the most part because the program is run separately for each code block. I have had it on my to-do list for a long time to update the lua wrapper code so the blocks are (intelligently) joined together, the python program is run once, and then the results are (again intelligently!) split up. Might the same approach be feasible here?

[alerque]

I realy don't think joining blocks would be a good approach here.

Lua has something called co-routines which can probably be leveraged for this. You might have to combine it with lua-lanes or something like that, and integrating with the Pandoc API is going to be interesting. I'm guessing the best approach will be to write a filter that handles the entire AST at once so that you don't have to worry about compiling results inside the Pandoc walk functions. This will mean iterating the tree yourself, extracting all the bits you need to transform, transforming them, then stuffing them back in an AST and feeding the whole thing back.

[alerque]

Actually if generating diagrams is generating an eternal resource and you're replacing the block, you can probably do that without your own AST parsing. You could just launch off a new background process to do the generation and as long as you know ahead of time what the output location was going to be you can used that to replace the block and not worry about waiting for the external process to finish at all.

[tarleb]

There's nothing built-in, that would require significant changes. But one could use something like luaposix to fork out a separate process and then let image generation work in the background.

Note that the diagram filter can be configured to use a cache, which should improve performance significantly after the first run.

[mrkissinger]

Seems this is the easiest way. Thanks.