Language detection strategies for OCR processing

[alliomeria]

What is needed?

Logical, iterative approach for detecting the Language to be used during OCR processing.

How could this work?

(*Adapted from Archipelago Slack discussion, 2022-04-20)

1. Generate OCR based on user input (passed via an ADO's metadata).
2. If NLP is selected, detect the language from the pure text language.
3. If language detected is similar, pass; If not, re-OCR using the correct language and Log.
   • Possible option: Add a checkbox that says "Trust language detection" (in case someone really wants to avoid this at all)
4. With Old OCR (if correct) or New OCR (if was wrong) and language detected, check NLP-enabled languages. If language is there, then to NLP polyglot. If not present, simply skip.
5. For all cases, the original language passed and the one detected will be part of the resulting Solr doc.

To Dos:

• Determine how to know a detected language is similar
• Determine which NLP language detection library/tool to use. Possible options to consider:
  • https://github.com/saffsd/langid.py
  • https://github.com/shuyo/language-detection (or not, as JAVA 🤮 )
  • https://py-googletrans.readthedocs.io/en/latest/ (free tier of GTranslate API: https://cloud.google.com/translate/pricing)
  • https://spacy.io/usage/models
  • https://fasttext.cc/docs/en/language-identification.html
  • OR forking 🍴 https://github.com/web64/nlpserver and re-using ♻️ https://github.com/web64/nlpserver/blob/master/nlpserver.py (use the nlp64 client already in place and fetch via a unified interface)

@DiegoPino & @mbennett-uoe, does this cover the highlight reel?