Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SS-1: Robust Parallelization Downloads of Pubchem Annotations **Incomplete** #2

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

SS-1: Robust Parallelization Downloads of Pubchem Annotations **Incomplete** #2

wants to merge 6 commits into from

Conversation

Sulstice
Copy link

In this PR, I modified the pubchem annotations to parallelize on the downloads and be more robust.

  • The first step is creating the links.txt file to show how much data we need to download.
  • Second is use of a parallelized software like aria2c (this is incomplete).

@Sulstice Sulstice added the enhancement New feature or request label Jul 18, 2024
@Sulstice Sulstice self-assigned this Jul 18, 2024
@tomlue
Copy link
Contributor

tomlue commented Jul 18, 2024

probably links.txt should go in a cache folder and added to the dvc output or ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zmughal zmughal Awaiting requested review from zmughal

@nibafanfan nibafanfan Awaiting requested review from nibafanfan

Assignees

@Sulstice Sulstice

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Sulstice @tomlue