[FR] Download Rate Limiting

[fritolays]

How can I set download sleep intervals to avoid rate limits?

yt-dlp has:

--sleep-requests SECONDS        Number of seconds to sleep between requests
                                during data extraction
--sleep-interval SECONDS        Number of seconds to sleep before each
                                download. This is the minimum time to sleep
                                when used along with --max-sleep-interval
                                (Alias: --min-sleep-interval)
--max-sleep-interval SECONDS    Maximum number of seconds to sleep. Can only
                                be used along with --min-sleep-interval
--sleep-subtitles SECONDS       Number of seconds to sleep before each
                                subtitle download

Could I set these in yt-dlp-configs/base-config.txt"?
Or would this not work as expected across multiple sources in pinchflat?

Otherwise global sleep requests and interval min-max in Config -> Settings?
Seems better than per Media Profile but your call.

[kieraneglin]

Hey there! Thanks for the report (:

Those flags won't do anything since Pinchflat (intentionally) handles the download queue itself. A global rate limit isn't something that's easy to implement, so before I look into that can you provide me with some more info? Are you actively being rate limited or is this just a precaution? Have you tried setting the YT_DLP_WORKER_CONCURRENCY to 1?

[fritolays]

Back from work.

A download got stuck with the prove your not a bot message from YT. I did not notice for a while and my vpn ip got banned. No biggy, just cycle to a new one.

However, pinchflat seems to just keep hammering regardless of the error ytdlp gives. A max download retries would be handy here. Also maybe display the actual error encounter in the gui someplace? Like a history of commands for each source.

As for rate limiting, I have compiled a few playlists that have 500+ items and suspect that downloading in one go will likely burn another ip. I would like to add some element of randomness to the download pattern. So even if its not global but source specific would be nice.

As a sidenote, will you add a poToken option?
https://github.com/yt-dlp/yt-dlp/wiki/Extractors#po-token-guide

[kieraneglin]

However, pinchflat seems to just keep hammering regardless of the error ytdlp gives. A max download retries would be handy here. Also maybe display the actual error encounter in the gui someplace? Like a history of commands for each source.

Fair point! There is already a list of errors from yt-dlp which the app considers non-recoverable and won't try again - I'll look into updating that list to handle the "prove you're not a robot" message

As for rate limiting, I have compiled a few playlists that have 500+ items and suspect that downloading in one go will likely burn another ip. I would like to add some element of randomness to the download pattern. So even if its not global but source specific would be nice.

Another fair point! This is a really hard thing to architect since each download is (intentionally) independent and I don't know ahead-of-time how long they'll take so I can't preemptively schedule them in a staggered fashion. There are other ways to achieve this that I'll look into

(note to self: look into an Elixir Registry or Agent initialized in the main app's supervision tree)

As a sidenote, will you add a poToken option?

I'll check with the yt-dlp team to see if this is the official solution yet. I know there's been a LOT of discussion around it but I'm not aware if it's finally crossed over to being their recommendation. I'll dig in!

[fritolays]

Another fair point! This is a really hard thing to architect since each download is (intentionally) independent and I don't know ahead-of-time how long they'll take so I can't preemptively schedule them in a staggered fashion. There are other ways to achieve this that I'll look into

So you dont know when they will end, but could you add a random sleep to before each download starts?

I'll check with the yt-dlp team to see if this is the official solution yet. I know there's been a LOT of discussion around it but I'm not aware if it's finally crossed over to being their recommendation. I'll dig in!

Ah sorry, no rush or need to pressure on this one. I understand it is an evolving situation.

[kieraneglin]

Thank you for the update! I just wanted to let you know that I'll be (mostly) out of cell service for just over a week starting tomorrow. But I'll keep thinking about this and see what I can come up with once I'm back