Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent file paths returned from multi_download when resumed. #346

Open
h-a-graham opened this issue Sep 12, 2024 · 0 comments
Open

Comments

@h-a-graham
Copy link

Hello,

Firstly, thanks for this package it is hugely appreciated!

I recently noticed that when using curl::multi_download the destfile column of the returned data.frame object differs depending on whether the file is resumed on completed on first go. When a file is resumed, the destfile is absoloute but when it completes in one it is relative. See here for a reprex:

suppressPackageStartupMessages({
  library(curl)
  library(R.utils)
})

# Example: some large files
urls1 <- sprintf(
  "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-%02d.parquet", 1:2
)

# cancel download after 5 seconds
res1 <- try(withTimeout(multi_download(urls1, resume = TRUE), timeout = 5)) # You can interrupt (ESC) and resume
# now escape the download and resume
res1 <- multi_download(urls1, resume = TRUE)

urls2 <- sprintf(
  "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-%02d.parquet", 3:4
)
# Do not escape this.
res2 <- multi_download(urls2, resume = FALSE) # You can interrupt (ESC) and resume

# full filepaths returned when resumed
print(res1$destfile)
#> [1] "/tmp/RtmpbPZ3Zs/reprex-146af8714e091d-awake-topi/yellow_tripdata_2021-01.parquet"
#> [2] "/tmp/RtmpbPZ3Zs/reprex-146af8714e091d-awake-topi/yellow_tripdata_2021-02.parquet"
# relative filepaths returned when completed in one go.
print(res2$destfile)
#> [1] "yellow_tripdata_2021-03.parquet" "yellow_tripdata_2021-04.parquet"

Created on 2024-09-12 with reprex v2.1.1

Is the solution perhaps as simple as setting destfiles to normalizePath(destfiles, mustWork = FALSE ) in:
https://github.com/jeroen/curl/blob/master/R/multi_download.R#L186

Cheers,
Hugh

@h-a-graham h-a-graham changed the title Inconsistent filepaths returned from multi_download when resumed. Inconsistent file paths returned from multi_download when resumed. Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant