Does Daft support reading/writing from Hive Metastore?

Hi! Not at the moment, are you thinking purely for hive tables?

we have hive and iceberg tables, and I guess iceberg tables are already supported, right?

I would second the suggestion to add hive support to daft sooner. Despite open table formats, particularly iceberg, becoming more and more popular, Hive tables are still widely used. It will be great if Daft could support HMS, AWS glue catalog for reading/writing hive tables. We also have mixed hive tables and iceberg tables and could completely replace Spark/Athena for many of our use cases if Daft supports AWS glue hive table.

Those are good points, thanks for bringing this up @Tianji Li @Phil Chen! It should not be hard for us to add full support for HMS, we'll take a look soon

do you support unity Catalog? https://github.com/unitycatalog/unitycatalog/

We do! Take a look at our integration docs

@Tianji Li are you using Databricks?

Nope we run all compute engines by ourselves. I saw Unity Catalog is recently open sourced and was looking at it…

Haven't had the time to look into the deltalake issues yet but it's in my backlog

FYI all on this thread — we’re going to prioritize doing hive-style folder reads first, and then target HMS shortly after. So something like:

daft.read_parquet(..., hive_folder_partitioning=True)

This will then attempt to auto-detect the partitioning info on the folder paths.

That is great to hear @jay Jay and @Kevin Wang. Thank!!! Quick question, will Glue catalog be supported the next? I know if we use glue server-less job with ray, daft will be able to work with glue catalog today. But many people don’t want to use glue job. Hence it will be great to decouple glue catalog support from glue serverless computing.

Yes I believe it shouldn’t be too hard to do that since the Glue Catalog supports the HMS interface. The API might look something like:

daft.register_catalog(“my_glue_catalog”, type=“aws_glue”)

df = daft.read_table(“my_glue_catalog.x.y.z”)

We should be able to use the glue API to find all the tables I think, and perhaps we can start with supporting just Hive tables and Iceberg tables. We’re still iterating on the possible API spec on our table/catalog support, would love to get your inputs once we have some proposals here @Phil Chen !

Super! Looking forward to seeing the proposal. Thanks @jay !