Read from clud storage using geoarrow.pyarrow.dataset

[nagyrobir]

Hi!

I was wondering if geoarrow-python has support for reading from geoparquet that is stored on a Azure blob storage? I know Pyarrow has it but uncertain if geoarrow-python has it. If it does have could you show an example of some sort?

Thank you !

[paleolimbot]

geoarrow-pyarrow is just a thin wrapper around pyarrow, so any call to pyarrow.dataset.dataset() should be supported by geoarrow.pyarrow.dataset.dataset() (you can see how the args are passed down here: https://github.com/geoarrow/geoarrow-python/blob/main/geoarrow-pyarrow/src/geoarrow/pyarrow/dataset.py#L499-L513 ). I don't have an example of how to open an Azure cloud file dataset off the top of my head and I'm not 100% certain that support has merged into pyarrow yet.

Note that geoarrow.pyarrow.dataset is pretty experimental and might change at any time 🙂 . I'm a little torn because the concept of geospatial-augmented dataset is very useful; however, part of the philosophy of geoarrow.pyarrow is that geometry should "just work" as a first-class data type (and special casing the dataset seems like a rather flagrant violation of this philosophy).

[nagyrobir]

Hi again!

Are there any plans to implement spatial predicates in the future? Like being able to join two geopyarrow tables based on inteserction of geometries or similar?

[kylebarron]

I can't speak for @paleolimbot of whether there will be a c-based spatial join in geoarrow-pyarrow, but I'm planning to have a spatial join on geoarrow tables in geoarrow-rust within the next couple releases. It'll interop seamlessly with geoarrow-pyarrow via #38

[nagyrobir]

I am a bit slow so type so be patient with me :), The code written in geoarrow-rust will be exposed/usable in geoarrow-pyarrow without any kind of extra coding effort?

[paleolimbot]

Anything Kyle exposes in geoarrow-rust will be accessible here eventually! But for the time being, there is no spatial join implementation in geoarrow-pyarrow (and it will be a while until there is a battle-tested version accessible here).

[nagyrobir]

Roger that! Thank you again for the work that you are doing!