Hey everyone, extremely new to Daft. Only stumbled across it today. I’m processing a dataset of a few hundred million rows and applying some python string manipulation functions to some of the columns. I have been using snowflake for this, but my python-based UDFs are hard to manage in snowflake, so I’ve moved to using modal.com to iterate over my dataset and call their .map on my list of batches, and it automatically fans out my python calls to hundreds of machines. It works fine, but it’s pretty manual, so I was hoping for something a bit more “standard” via an api purpose-built for this sort of pythonic fan-out processing. Daft seems to be that, but as I understand it, it’s based on Ray. I presume that is a pretty strong dependency? If I wanted to have dask call my asynchronous udf and leave the remote stuff to me/modal, how hard would that be?

Welcome! Do you have some example code of what you are trying to achieve? The way you can think of Daft is as if it is one giant dataframe. Today that giant dataframe runs on Ray if you want it to be distributed. We are also working on a proprietary service (kind of like Modal) next year, so you won’t need Ray for what you’re trying to do.

will daft continue to support ray (for open source ver sion) or will the support drop in future ?