I have a glob to s3 with thousands of parquets and it takes Distributed Data Community #general

I have a glob to s3 with thousands of parquets and...

Kyle

09/09/2024, 12:58 PM

I have a glob to s3 with thousands of parquets and it takes very long to run show. Is that expected?

jay

09/09/2024, 5:23 PM

How many thousand are you running on, and which version of Daft are you using?

jay

09/09/2024, 5:23 PM

We added some optimizations in 0.3.2 to help with

.show()

of (many) small Parquet files