-
Notifications
You must be signed in to change notification settings - Fork 215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generator for chunks of content stream #96
Comments
Adding a bit of context here. he observation was that all download methods would load the entire file into memory, unless you pass a file handle to download_to. Perhaps that is fine. But if you wanted to download the file, but not download it to a file on-disk, but not load the entire contents into memory all at once, there was no way to do that. If you wanted to:
then something like what I proposed would possibly be necessary. It looks like I opened that issue originally in response to this user comment: #56 (comment) This request is logged in our Backlog as SDK-307 ticket. |
The stream processing case is already possible with the
Since The approach also has a few other benefits, like allowing for canceling the download (by raising an exception), but it doesn't give full control to the caller (like exposing a generator might). |
Today, there are two options for downloading a file:
File.content()
, which requests the entire file at once, and loads it all into memory.File.download_to()
, which requests the file in chunks, and dumps it into a stream (file
handle or another instance ofio.IOBase
).The first option always loads the full contents into memory. The second option always loads the full contents into memory except in the case where the stream is a file handle. Additionally, both options always download the full file before handing control back to the caller.
There should be a third option for download, which only downloads and loads individual chunks at a time, and is a generator and therefore hands control back to the caller for each chunk.
I've implemented this both locally and in a comment on #93, and it looks something like this:
This could be added as-is, but I haven't created a PR yet because I wanted to think more about #87. I was wondering if there might be a better way to expose streams in the abstract
Network
interface. Right now we have a genericresponse_as_stream()
method onNetworkResponse
, but it isn't really generic because:stream=True
to be passed torequest()
onNetwork
, which isn't documented because it is specific to the requests library.response_as_stream()
, you must call itsstream()
method, which isn't documented because it is specific to the requests library.We could just add this as-is, because it doesn't technically add any additional dependencies on requests that weren't already added via
download_to()
.The text was updated successfully, but these errors were encountered: