Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Processor.resolve_resource: support on-demand download of URL values #799

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

kba
Copy link
Member

@kba kba commented Feb 14, 2022

With this in place, users can use URL directly for parameter values:

ocrd-tesserocr-recognize -P model https://github.com/tesseract-ocr/tessdata_best/raw/main/bos.traineddata

and it should download on demand the first time it encounters and registers the URL in the user resource_list.yml. Subsequent calls will use the cached download.

In practice though I cannot seem to find an example where this works:

  • ocrd_{tesserocr,cis-ocropy} have a different mechanism of model storage. It's still compatible with ocrd resmgr download in tesserocr's case but does not use the self.resolve_resource method this PR extends
  • ocrd_calamari requires a directory of files, or an archive which is too complex to do on demand in a generalized way IMHO
  • ocrd-page-transform is a bashlib processor and won't support this.

So if anybody has a good idea on how to test and/or generalize this to make it available to all the processors, pls let me know.

Copy link
Collaborator

@bertsky bertsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Perhaps the resource_type can also be guessed from the URL suffix (/ for directory, .zip etc for archive).

(As a test case, because we would not want to be dependent on an external module, you could define a subclass of DummyProcessor with a file parameter and some resolve_resource logic in the constructor, e.g. printing the file to stdout.)

@bertsky
Copy link
Collaborator

bertsky commented Feb 14, 2022

In practice though I cannot seem to find an example where this works:

  • ocrd_{tesserocr,cis-ocropy} have a different mechanism of model storage. It's still compatible with ocrd resmgr download in tesserocr's case but does not use the self.resolve_resource method this PR extends

Yes. For Ocropy recognition, your cisocrgroup/ocrd_cis#83 is long overdue.

And for Tesseract, I believe your OCR-D/ocrd_tesserocr#176 could be rewritten such that instead of overriding the constructor (for the list_resources and show_resource cases), one would directly override module_dir, so (assuming core will have a mechanism of ensuring that list_all_resources and resolve_resource abide by its Processor.ocrd_tool['resource_locations']) everything will automagically make only the files from the module directory survive.

  • ocrd_calamari requires a directory of files, or an archive which is too complex to do on demand in a generalized way IMHO

Yes, Github directory downloads would be hard to implement. But IMO we can assume that model deployment for Calamari and and eynollah and sbb_binarize will involve release archives in the future.

  • ocrd-page-transform is a bashlib processor and won't support this.

With the recent changes to bashlib, this should work out of the box, though. (What happens is that it delegates to ocrd.cli.ocrd_tool's list-resources and show-resource, which in turn add the ocrd-tool.json directory, but also do the usual lookup under the other resource locations. Since we do not have a ocrd__resolve_resource builtin in bashlib yet, I delegate to ocrd__list_resources. But to ensure maximum interoperability, I just commited bertsky/workflow-configuration@9f68fe8.)

A Pythonic test scenario would be running-downloading blla.mlmodel under OCR-D/ocrd_kraken#33, or one of the ocrd_detectron2 config/model combinations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants