Crazy idea… What if we exposed the Daft schema itself as a logical type in Daft? Some shenanigans you could do:

df = daft.from_glob_paths("...")
df = df.with_column("is_parquet", df["path"].url.is_parquet())
df = df.with_column("schemas", df["path"].url.parquet_schema())

# Merge schemas
df = df.agg(df["schemas"].merge_schemas())

# Check schema compatibility
df = df.with_column("compatible", df["schemas"].schema_compat(other_df.schema()))

Could be cool if we wanted to extend our schema inference behaviors to be even more sophisticated… Definitely also strays dangerously close to maybe also wanting to doing this for statistics over files? 😬

I believe iceberg does something like this where their schema is just a massive struct dtype

Iceberg has some convenient metadata which helps with table maintenance and analytics for sure. I think I’d find the most use out of extra encoding metadata for binary types or some sort of artifact type for files with helpful info like extension, mime type, size, and gzip compression. Then the binary can feed into the ImageType with its own predefined/optimized serialization scheme. My current approach to multi modality in my documents data frames is a content node schema of:

[

“node_id”: pa.string(),

“node_type”: pa.string(),

“node_binary”:pa.binary(),

“node_meta”:pa.map_(pa.string(),pa.string())

]

If the

pa.binary()

class contained a bunch of encoding/serialization metadata, that would be huge.

IOConfigs shouldn’t be persisted — they contain potentially credentials! We do have separate internal configs for things like parquet files though