.explain()

already estimates the dataset size. Any way to get programatic access to this besides parsing text output of explain?

@Desmond Cheong for cardinality estimation

We can add a

.describe()

api that would work similar to how

DESCRIBE TABLE

works in sql, and show summary stats on dataframes

That sounds great. My use-case is to be able to compute some common-sense parameters on behalf of the user based on their dataset size.

Cool, I'll add it to the backlog. I expect this will naturally be implemented as we do the cardinality work planned for the next few weeks. Will keep you in the loop!

Thanks!

Did this get implemented? As

df.schema()

seems to just return the types of each column and not any statistics?

ah we didn't do a specific

.describe()

api yet

Is there an issue to track or planned timeline. I ask as would like to get summary stats for a df for each column with values such as approx_unique, number_nulls, min, max. If you have suggestions on how to do this efficiently be good to hear.

Decided to just bang out a cheap way to do it: https://github.com/Eventual-Inc/Daft/pull/3711 @Colin Ho if you could help take a look? I'm also going to push out a PR shortly for a

.stats()

API that will simply return the statistics for the logical plan (currently only estimated num rows and estimated num bytes)