If possible, could we disable using the Window function for now if it is not easier to make it more scalable to large table. As it is now preventing us from upgrading daft.

For the sake of discussion, for compute partition boundaries for a large tables, we could consider implementing a probabilistic algorithm that based on sampling of the table to estimate partition boundary to avoid using Windowing and Sorting over the entire table. This will allows trade off btw accuracy and computational complexity (both time and space complexities). The parameter to control this trade off is the sampling size. The large the sampling size, the more accurate, but it will be more computational complex. If smaller sampling size will be more likely to have uneven sized partition (skew).

For now, I think support only the fallback partition scheme is good for most use cases where data is already roughly evenly distribute across partition column.