Community for the Daft project and all things distributed data

Distributed Data Community

Is anyone familiar with our Daft arrow “super_extension”? Trying to figure out why it’s necessary — it’s causing some issues with a refactor I’m trying to do on `<http://DataFrame.to|DataFrame.to>_arrow()`

Super extension? Could you point me to where specifically this is?

image.png

Defined at: `daft-core/src/datatypes/dtype.rs:118`

It seems we wrap arrow extension types in our own custom Daft super_extension… But we’re also a little inconsistent with applying it

Looks like all this is is a custom Arrow extension type with the name `super_extension`, a bunch of (binary) metadata, and an actual underlying arrow type.

I’m guessing this helps inform us when we read data that a given columns should be semantically something like an image or embedding

But this seems to get a little odd with tensors and fixed size lists, where arrow and ray have their own support for these types… I think we might be a little inconsistent with where we apply the super_extensions