Another question on Daft... When running large jobs on ray, I am seeing that the denominator keeps moving:

ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3246/3437 [30:25<01:20,  2.38it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3250/3441 [30:26<01:11,  2.68it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3255/3446 [30:27<01:03,  3.03it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3259/3450 [30:30<01:23,  2.28it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3265/3456 [30:31<01:07,  2.83it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3269/3460 [30:33<01:17,  2.46it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3272/3463 [30:35<01:16,  2.50it/s]
ScanWithTask-Aggregate [Stage:3]:  94%|█████████▍| 3277/3468 [30:36<01:05,  2.93it/s]

Is this expected? Any insight as to what is happening under the hood? Is Daft only scheduling a few partitions at a time?

Correct! Daft is scheduling your partitions in waves, and sometimes will be able to short-circuit certain executions (e.g. during a limit)