Community for the Daft project and all things distributed data

Distributed Data Community

What is the behavior for `overwrite` when writing partitioned tables? Does the whole table get deleted, or do only the partitions you're writing to get deleted?

Currently it overwrites the entire thing. However we want to also add a `dynamic overwrite` mode which will overwrite only given partitions because this seems to be a common ask…

What kind of data/table are you writing to?

Parquet with partition columns although we'll probably switch to iceberg at some point.

The use case for us would be incremental updates in case it is helpful for thinking about it. Maybe there is a better way to achieve that outcome.

Let me check with the team on the best way of doing this!

Right now I'm hacking it by querying for the unique partitions in a batch and then deleting any files in those partitions with boto. Not the worst think in the world but definitely would be nice if there was an `overwrite_partitions` like spark has.