Processing custom BibTeX tags with a pandoc Lua filter

[amine-aboufirass]

I have a BibTeX file that looks like this:

@inproceedings{Urban_Water_Bal,
  author = {Suping Li and Weiming Yu},
  booktitle = {Conference on Environmental Pollution and Public Health},
  file = {.\bcbd56737652faa0dda5ffe5e053ef13\test.pdf},
  title = {Urban Water Balance Analsis by Holistic Approach},
  year = {2010},
}

@article{Conceptual_Urba_Zeisl_2018,
  author = {Zeisl, Peter and Mair, Michael and Kastlunger, Ulrich and Bach, Peter and Rauch, Wolfgang and Sitzenfrei, Robert and Kleidorfer, Manfred},
  doi = {10.3390/su10030716},
  file = {.\dd56fc18ef2a05e6b6e4a33b47d34106\sustainability-10-00716.pdf},
  issue = {3},
  journal = {Sustainability},
  language = {en},
  month = {3},
  pages = {716},
  publisher = {MDPI AG},
  title = {Conceptual Urban Water Balance Model for Water Policy Testing: An Approach for Large Scale Investigation},
  url = {http://dx.doi.org/10.3390/su10030716},
  volume = {10},
  year = {2018},
}

Each entry has a custom tag called file which refers to the file path of the document on my local system.

I would like to write a simple Lua filter which will append a link to this path to a bibliography rendered in HTML. I tried inspecting the native syntax tree first with the command pandoc --citeproc --from biblatex --to native .\references.bib. Unfortunately it looks like custom tags and their corresponding values do not end up in the native tree.

Writing my own BibTeX reader is not really a road I'm prepared to go down at this point, and my hunch is that the bibliography processing is done inside the Pandoc engine (i.e. using Haskell) and not a Lua filter which I might have been able to copy and modify.

Are there any other workarounds I could try to be able to process custom tags in a simple Lua filter?

[bpj]

Been there done that!

The attached zip file contains some scripts which you may find helpful. Remember to read the enclosed README!

With them you could

1. Convert the BibTeX bibliography to CSL JSON with pandoc.

2. Reinsert the file field with pandoc-lua and get-bib-fields.lua

3. Load the modified JSON file in your filter and create an index keyed on the id field of each bibliography entry. There are functions to help with that in helper-functions.lua

4. Look up the indexed entries by the reference id. Remember that for each entry in the #refs div i.e. div with class .csl-entry

   local key = entry.identifier:gsub('^ref-', "")
   local filename = bib_index[key].file

get-bib-fields.zip

[jgm]

I noticed this relevant issue: jgm/citeproc#137

If we modified citeproc to support the custom field, then we could modify pandoc to parse the file field in bibtex as a subfield of custom. That would simplify this considerably, because you'd have the filename in the pandoc AST and you could just extract it with a filter.

If you want to put up a pandoc issue suggesting this, with a link to the associated citeproc issue, go ahead.