I ran a flamegraph on the parquet readers, It looks like we are spending a significant amount of time in

_platform_memmove

calls. for the code snippet

let file = "Daft/lineitem.parquet";

        let io_config = IOConfig::default();

        let io_client = Arc::new(IOClient::new(io_config.into())?);
        let runtime_handle = daft_io::get_runtime(true)?;

        let table = read_parquet(
            file,
            None,
            None,
            None,
            None,
            None,
            io_client,
            None,
            runtime_handle,
            Default::default(),
        )?;

Fun…is this running on the “malformed” polars data with thousands of rowgroups?

yeah

BTW How does this runtime compare to the “rest of Daft”? I’m curious if the problem might be further up the stack than the Rust-level

read_parquet

call. E.g. how much % does the Rust

read_parquet()

call take up in a Daft

daft.read_parquet("…").collect()

end-to-end call through the dataframe API?

hmm interesting point. I'll see if i can get a flamegraph of the entire stack. I don't have much experience profilling python scripts, but i'm sure it can't be too difficult.

Re: `_platform_memmove`: I guess Polars probably doesn’t have the overhead of needing to concat all the arrays on read since they do array chunking and Daft doesn’t, so that’s definitely a potential culprit…

Re: `_platform_memmove`: yeah I was thinking the same thing. It looks like we create a table/batch for every row group, then concat them all at the end into a single table. I was thinking we could potentially be a bit more intelligent while reading, and instead of creating a table for every row group, we do some batching at the reader, so like

N

row groups would be processed as a single table.

That’s going to hurt…

I think that should be a pretty easy fix though. I'll see if I can get a PR together for it this weekend.

Did you manage to get e2e profiling of a

daft.read_parquet

working? 😮

yeah I did with

memray

but it doesn't capture the native symbols as well as

cargo flamegraph

import daft

print(daft.read_parquet('../Daft/lineitem.parquet').collect())

That is indeed surprising, given that we do a bunch of concats

Here's polars for reference.