Hey Folks, I was there at Daft's office for the Iceberg meetup. I was just digging into how Iceberg writes are done in Daft https://github.com/Eventual-Inc/Daft/blob/main/src/daft-writers/src/catalog.rs#L68 . Are the writes done using the Python Iceberg Client?

Hi Karthic! We use PyIceberg to read and write table metadata for our iceberg writes, but the table data itself is written in a distributed manner by Daft directly

but the table data itself is written in a distributed manner by Daft directly

What's used underneath for the protocol writes? Iceberg-rs or you have your own implementation of the protocol?

We use PyIceberg for most of the iceberg protocol logic including writing metadata

Thanks for the answer. Is there a short list of functionalities that are missing in Iceberg-rs because of which you picked PyIceberg instead of the Rust library?

Last time we checked I think it was missing support for some catalogs as well as proper write support. Based on some conversations we've had, I believe write support is still not complete

Got it, thank you.