Community for the Daft project and all things distributed data

Distributed Data Community

<@U041QSEF2H2> <@U07B9PD9W7J> could I get a re-review on this PR? I also commented in the PR about some of the new decisions I made based on our discussions
<https://github.com/Eventual-Inc/Daft/pull/3437>

Made all the fixes from the latest reviews, ready to merge? <@U041QSEF2H2> <@U07B9PD9W7J>

Can you produce a chrome trace and verify the cross join is parallel 

which runner(s) would we want to look at?

`DAFT_DEV_ENABLE_CHROME_TRACE=1` for native btw

image.png

looks like it's beautifully parallelized (the gaps I believe are from round robin dispatching)

I added a sort at the end and (it was taking forever so I killed it, but) the trace shows that it's perfectly parallelized with the unordered dispatcher

Yeah the sort rn is a naive collect and sort :smiling_face_with_tear:

it was trying to sort 150 million rows so I don't blame it haha