hey folks, loving the Unity Catalog Table support! Is there already an issue tracker for Volume support as well?

No issue tracker yet, but definitely create one! I’m interested in figuring out the API. Was thinking we could leverage

.url.download()

like how we do for S3 URLs, but the tricky thing to try and make work is the credentials story as well as the URL scheme. Maybe we can recognize URLs like

unity://<volume_name>/...

and perform a 2-step process: 1. Convert these URLs to S3 URLs. Grab S3 credentials from Unity Catalog. 2. Run a normal

.url.download()

on the S3 URLs using those S3 credentials

created an issue here https://github.com/Eventual-Inc/Daft/issues/2482