This message was deleted.

Haha the rest of the team might have some more insight here as well… For some backstory when we first started building Daft we considered using Polars, Datafusion or Velox as an engine Unfortunately we had various use-cases that were hard to fit into these frameworks. For example we allow for Python object columns (mostly as a escape hatch for arbitrary user-provided Python objects) which are hard to fit into other purpose-built frameworks And we also wanted to innovate heavily on types, I/O and functionality (building in tensor types, image types, eventually types for documents and video as well) which felt like it would be really difficult to extend in a 3rd party engine 😞

The velox team at the time was also not taking many external contributions outside of meta. So it was hard to extend to what it needed it to do

It looks like a great architectural decision to go with a fairly generic solution...This way the space for innovation is much larger I guess 🤔. Otherwise you will depend on the external ones which might go in other directions.