Community for the Daft project and all things distributed data

Distributed Data Community

BTW <@U041QSEF2H2> looks like Iceberg might require us to apply a different partitioning scheme on different files/ScanTasks depending on when they were written: <https://github.com/Eventual-Inc/Daft/pull/2084>

The situation looks like:
1. Table has `months(dt)` partitioning
2. Old file is written with `months(dt)` partitioning
3. The table is changed to `days(dt)` partitioning, but the old files are left untouched
4. New files are now written with `days(dt)` partitioning
When Daft reads the data, we currently error out because we try to find the `days(dt)` key on old files.

Any thoughts on what the correct way of dealing with this should be?

Would it always be safe if we just rely on the file’s partitioning info and ignored the one in the table-level?