hi all, how daft deals with snapshots in Iceberg table ? Would appends to Iceberg table require daft to scan and load the entire table again, or it can deal with deltas between snapshots ?

Hi Ravi, are you asking about how Daft does appends to an Iceberg table? If you have a Daft Dataframe and you call DataFrame.write_iceberg, Daft will not do a scan of the Iceberg table, but will only read the table metadata before writing out a new snapshot. The method also takes in a

mode

argument, which by default is set to append.

Hey Kevin, Thanks for the response, but my question is around reading the iceberg table data between snapshots by daft during a query. Does daft handle iceberg table refresh automatically and read the delta between snapshots ?

Ah thanks for clarifying. Daft receives a PyIceberg in

daft.read_iceberg

, so it will only be aware of the snapshots that the table object is loaded with. PyIceberg tables do not automatically refresh, but you may call Table.refresh to do so before passing it into Daft. Daft does not cache data between queries, so if you call daft.read_iceberg multiple times, Daft will read all of the data it needs to perform the query each time.