Can I be reminded what the issues were with using

mmap

? I thought it was in relation to CSV reading, but I've kind of forgotten the details. If I recall correctly, it was related to Ray interpreting memory-mapped files as occupying physical memory, even though it's virtual memory. This misconception led to Ray performing actions that we don't want.

Yeah I can speak about the local side, but the ray side I think Jay would have a more definitive answer. When you mmap a file, heap size goes up which is not an issue, but as you read the file from front to back the data gets paged in and resident set size increases. Even if you pass madvise::free to the os, the os won't remove the pages from memory until the system is under pressure and it needs to reclaim memory. This is fine for the os, it'll just reclaim memory. But meanwhile RSS is up and Ray doesn't understand that the data can be paged out anytime

Yeah I think the worry was that Ray would see the large amount of memory being used and kill the worker When running on Ray, there are actually 2 OOMKillers at work. One is the Ray OOMKiller which monitors and kills Ray task processes, and one is the OS’ OOMKiller on each machine which monitors and kills normal processes.

Can you mununmap or whatever it is called previous parts of the file you have already read

And tbf this is something we do need to actually try on Ray and see what the behavior is

Yeah munmap would remove the mappings. I didn't see a good ergonomic way to do it though. You'd have to do some pointer arithmetic, and it's most definitely unsafe rust. Or another approach might be mmapping small chunks that are unmapped. I'm not sure if it would be fast though