I just got read_deltalake and write_deltalake with databricks unity catalog on azure working. Curious if there's any way yet to write into a managed table and or create a new managed table?
j
jay
10/09/2024, 1:03 AM
Creation is tricky… we are looking very seriously at building a catalog/table abstraction in Daft to be able to do this over Glue, Unity etc. any thoughts here on API?
j
jordan fox
10/09/2024, 1:27 AM
The api in DataCatalogTable looks pretty good to me. I almost thought, hey, maybe this'll work and tried it anyway. Would likely need a table_provider param somehow.
j
jay
10/09/2024, 2:09 AM
Yeah the trickiest part is getting this to work across “all” catalogs and table formats.
We’ll cook something up and get your thoughts!
j
jordan fox
10/09/2024, 2:29 AM
Yeah for sure. The way I was thinking about it is if table=string, it's a local path, if it's of type DataCatalogTable, it's a path to a catalog provider location. Then DataCatalogType would need to be used to handle implementation.
Complicated for sure though.