@Sammy Sidhu 2 PRs for your review:
• [BUG] Fix ScanTask memory estimations when limits are provided:
https://github.com/Eventual-Inc/Daft/pull/2735
• [PERF] Make merging of ScanTasks be more conservative when provided with a LIMIT:
https://github.com/Eventual-Inc/Daft/pull/2758
The first fixes our ScanTask’s
estimate_in_memory_size_bytes
to take into account limit pushdowns
The second uses some of that prior work to make our scan task merger more conservative when merging ScanTasks with LIMIT pushdowns