Hi - I'm looking to use a UDF to run embedding computations. Given this could take a long time on large data or have possible GPU provisioning delays etc: is there any way to get progress/updates about a daft computation (probably with Ray executor)?

Hey @Clive Cox , currently it's possible to get updates by inspecting the Daft progress bar (which logs to stdout), or the Ray Dashboard. Additionally, any stdout/stderror from your UDFs should be streamed out to log files: https://docs.ray.io/en/latest/ray-observability/user-guides/configure-logging.html Unfortunately we don't have any kind of event based notification system if that's what you're looking for, though we are working on a better observability system other than progress bars.

Thanks. Look forward to updates here as seems quite important when integrating these longer running tasks.