-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support efficient conversion of Tensor and Complex Data in Python #2
Comments
FYI, if it's a fixed size list array under the hood, you can do this conversion to numpy a bit easier than accessing the buffers:
|
Ah, that's much better. Thanks @jorisvandenbossche. |
Currently, conversion of Tensors and Complex data in Python is inefficient:
produces the following output
This is because the extension types are defined in C++, and the
to_numpy()
method on the default Python Extension Type wrapper isn't overridden. See daskms.experimental.arrow.extension_types.to_numpy for a possible implementation.Two possible solutions exist
Provide wrappers with richer features within Apache Arrow
The Arrow maintainers are aware of this issue:
And the following exploratory PR's suggest initial solutions:
Provide wrappers at the casa-arrow level
Provide a table wrapper that creates numpy arrays directly from the arrow column buffers: e.g.
The text was updated successfully, but these errors were encountered: