retry with the multi suite of functions?

[sckott]

hey @jeroen 👋🏽

Someone asked about retries with the async methods I have in crul, which use your multi functions

We have retry for synchronous requests, but that's straightforward to handle.

It's not clear to me how we'd handle this. We'd need some way to put retry logic into the handle (via curl::handle_setopt?)?

[jeroen]

I don't fully understand the how you are doing the async-queue in crul, but the trick is you re-add the request to the multi pool from within the done() callback. You can even use the same handle, or construct a new one.

I don't use R6 myself but here is a proof-of-concept using recursion in the same way as you do in your retry:

retry_async <- local({
  pool <- curl::new_pool()
  out = NULL
  retry <- function(url, times = 3){
    curl::curl_fetch_multi(url, pool = pool, done = function(res){
      if(res$status_code >= 400 && times > 0){
        message("Retrying ", times)
        retry(url, times = times -1)
      } else {
        out <<- res
      }
    })
    curl::multi_run(pool = pool)
    out
  }
})

retry_async('https://httpbin.org/status/404')

[sckott]

Thanks! I think that makes sense.

[sckott]

Do you think there's any way to combine other requests with a retry request (following approach you gave above) using async? For example, if a user wants to do a couple of GET requests, and a POST request, I can do something like:

pool <- curl::new_pool()
curl::multi_add(handle1, pool = pool) # GET request 1
curl::multi_add(handle2, pool = pool) # GET request 2
curl::multi_add(handle3, pool = pool) # POST request
curl::multi_run(pool = pool)

But if I want to add a retry reqeust to pool before running curl::multi_run, I'm not sure I could do that because the multi_run is within the retry function; which does seem necessary, but then I don't see a way to register a retry request like this with a pool of other requests. Maybe I just have to run retry requests separately from all others (non-retry)?

[jeroen]

Actually multi_run does not need to be in the retry function, that was just the example. The trick is simply that you can add new requests to the currently running pool from within the done or failure callbacks.

Here is a (slighlty more complicated) example that calls multi_run only once:

retry_async <- local({
  pool <- curl::new_pool()
  out = NULL
  retry <- function(url, times){
    curl::curl_fetch_multi(url, pool = pool, done = function(res){
      if(res$status_code >= 400 && times > 0){
        message("Retrying ", times)
        retry(url, times = times -1)
      } else {
        out <<- res
      }
    })
  }
  function(url, times = 3){
    retry(url, times)
    curl::multi_run(pool = pool)
    out
  }
})

retry_async('https://httpbin.org/status/404')

[sckott]

thanks! I think that does it.