@jay I checked the memory usage for

to_pylist

function using both

iter_rows

and materializing everything at once using

<http://result.to|result.to>_pylist()

for the issue: (#2578) and here are my findings. In both approaches, the memory usage is fairly similar. For streaming approach, the code uses ~790 MB and for the other approach, it uses ~795MB. This is tested on a parquet data with 100 partitions. I also tried this with 1000 partitions, Even with increase in the number of partitions, the memory usage did not increase much in both the approaches.

I would expect peak memory usage to be the same, but I’m guessing during the materialization process it should be more stable!

could you please let me know which approach we can use?

You should be able to get memray to produce a graph of memory usage over time! I also commented on your PR again — we’re unnecessarily calling

.collect()

now that you’re correctly using

.iter_rows()

, which would result in an unnecessary intermediate materialization

After removing

.collect

,

iter_rows()

is using ~650MB.

I think it should be good to go

Sure