I have a cluster that's running a single job and seems to be underutilized. According to the physical plan there's >10000 tasks but these are not getting populated and the number of tasks being run seems to consistently be below the number of cores available. I think maybe each worker is requesting a large amount of memory but biting off more than it can chew with just a single core. Any recommendations on configurations which could be adjusted to improve the utilization rate?

Interesting, what kind of plan are you running here? There are a few possibilities here: • The job is nearing the end, and there are only 224 straggler tasks that need to be completed • Overly conservative memory estimations (I think this is unlikely, Daft is probably overly aggressive today) • Not enough partitions available to saturate the machines (unlikely, I see a lot of tasks in your report) Could you share the plan as well?