Hey @Colin Ho For the issue: (https://github.com/Eventual-Inc/Daft/issues/2578), I would like to do some memory profiling on the solutions. Can you please give me some information on that?

we use memray: https://github.com/bloomberg/memray to profile. Here's an example script you can try:

with memray.Tracker("memray.bin", native_traces=True):
    df = df.to_pylist()

This will produce a

memray.bin

file, which you can analyze via a flamegraph. For example:

memray flamegraph memray.bin