If ScanWithTask is requesting for { "memory": 37407051360, "CPU": 1 } by default which causes my worker to OOM, how can I reduce this?

Is it OOMing or failing to schedule? I’m guessing it’s failing to schedule since you’re able to see the error message? The reason it’s requesting for that memory is because the file it’s reading is pretty large. You likely need bigger machines to process this dataset!

I'm working on a 30TB dataset and each file is indeed >10G but I'm not sure why some tasks succeed and some don't..

How large are your machines, and how many machines are in your cluster?

I have 70 machines which have 200gb memory each

Gotcha — yeah this is a really large workload. Perhaps we should set up some time to chat about it? There are some things that we need to be doing here that we aren’t doing yet to work well with these workloads: • Splitting of these 10G files into smaller partitions (we currently have a limit of 10 files for scan splitting because of metadata fetching overheads) • When performing shuffles (joins, sorts, groupbys, repartitions), there will be a really large Ray graph being constructed. We are in the process of designing a new shuffle system to handle these larger shuffles. LMK what timezone you’re operating out of? I’d love to get some of the Daft team involved and learn more about your workload to see if we can help stabilize it.

I'm based in GMT +8 😅

Hey, I’m from Singapore 😛 very familiar with that tz

Great okay! I'm also okay with night calls

LMK if tomorrow @9AM is good for you? That would be our Monday@6PM PST / Your Tuesday@9AM

Okay sure!

(Sent you a calendar invite — chat soon!)

Cool! See you soon!