Move image retrieval code here

[thequicksort]

This repository hosts the feature vector representations of the image data set used for similarity search. The resulting HDF5 files are orders of magnitude more compact than storing the raw images. As such, we should move the scripts/notebooks for downloading raw images to this repository.

Here are the repositories that use this:

• https://github.com/uwmisl/cas9-similarity-search
• https://github.com/uwmisl/primo-similarity-search

High Level

Low Level

Open Images

• migrate download/feature extraction code from cas9 similarity search

Hybridization Similarity Search

• delete notebooks/01_datasets/01_download.ipynb
• delete notebooks/01_datasets/02_extract_features.ipynb
• create interface for accessing feature vectors from open images

Cas9 Similarity Search

• migrate notebooks/01_datasets/01_download.ipynb to Open Images
• migrate notebooks/01_datasets/02_extract_features.ipynb to Open Images
• copy docker.sh and Dockerfile to Open images
• create interface for accessing feature vectors from open images

[thequicksort]

• Make the Image feature vector download a separate process from checking out the repository / starting the docker image.
• Allow the user to point to the location of the feature vectors (e.g. a different location on disk, a location in the Docker container).

Q: Why?
A: Because users might want to utilize different parts of the pipeline, like sequencing analysis, that shouldn't require downloading the gigabytes of feature vector data.

[thequicksort]

Open question: How do we want the similarity search repositories to access the feature vectors? What should we recommend to users checking out the repository who want to reproduce our results (perhaps even without downloading all the images from scratch)?

1 - Git submodules
2 - Manually specify locations (requires extra steps of user checking out repository, running git lfs, etc)
3 - As part of this pipeline, publish to external bucket
4 - Other approaches?

[thequicksort]

High-level overview of the migration proposal: