Enable feather and parquet in S3 #361
Conversation
mamo3gr
force-pushed
the
feat/feather-and-parquet
branch
from
d43deef
to
93ae346
Compare
gokart/file_processor.py
Outdated
|
|
||
| def load(self, file): | ||
| loaded_df = pd.read_feather(file.name) | ||
| loaded_df = pd.read_feather(BytesIO(file.read())) |
There was a problem hiding this comment.
Can we avoid file.read() here? I think it will read all content onto memory.
I hope BytesIO accept file (reader-like) itself
There was a problem hiding this comment.
I hope BytesIO accept file (reader-like) itself
Unfortunately it seems not to do so.
https://docs.python.org/3/library/io.html#binary-i-o
Binary I/O (also called buffered I/O) expects bytes-like objects and produces bytes objects.
There was a problem hiding this comment.
It is a little bit heavy to enable streaming read because we need to modify codes in luigi. I leave FIXME comment there in 7abbc33. Can we merge this PR with these comments?
There was a problem hiding this comment.
Additionally I added a conditional branching with respect to whether the passed file is reader-like or not in 65bb5b0.
There was a problem hiding this comment.
reverted the way of loading except S3 in 0a6880f.
I tried it and found to need the time. I wrote down the details to #363 and would proceed to merge. May I do so? |
Problem
When I use
.featheror.parquetwith S3, got errorToDO