Finally cleaned up my auto-RayRunner PR: https://github.com/Eventual-Inc/Daft/pull/2282 @Kevin Wang it’s ready for another look, here’s the logic: • If a user explicitly calls

daft.context.set_runner_ray/py

, this overrides all behavior ◦ If a user calls

daft.context.set_runner_ray

with a specified address, but Ray is already initialized, we warn them that their address is being ignored • Otherwise, on first execution Daft will attempt to retrieve the runner config from the current environment: ◦ Check for the

DAFT_RUNNER

environment variable for `RAY`/`PY` ◦ Check to see if Ray is initialized, and if we aren’t running in a Ray worker:

RAY

◦ Fallback onto:

PY

◦ If a user calls

daft.context.set_runner_ray

with a specified address, but Ray is already initialized, we warn them that their address is being ignored

If Ray is already initialized, and the user wants to connect to a different Ray cluster, would that prevent this?

If Ray is already initialized, and the user wants to connect to a different Ray cluster, would that prevent this?

That isn’t allowed right now. I don’t think we can make that work either as the Ray SDK can only be initialized once!

Ok fair enough