Hey folks, have there been any recent changes to how the daft runner is set? I used to be able to do it safely in a notebook. Now, I can only do it immediately after I import

daft

- If i leave it until later (ie. after we build some dataframes and imported them from a different file), daft complains with

230         warnings.warn(
    231             "Calling daft.context.set_runner_ray(noop_if_initialized=True) multiple times has no effect beyond the first call."
    232         )
    233         return ctx
--> 234     raise RuntimeError("Cannot set runner more than once")
    236 from daft.runners.ray_runner import RayRunner
    238 ctx._runner = RayRunner(
    239     address=address,
    240     max_task_backlog=max_task_backlog,
    241     force_client_mode=force_client_mode,
    242 )

RuntimeError: Cannot set runner more than once

Hey, yes we did make some changes! Setting the runner does affect global state, which is very iffy. We highly recommend doing this in your application’s

__main__

entrypoint, or similar.

hmm, okay - I'll simply add this to the top of our notebooks in this case. Sadly, from a user experience perspective, this requires adding specific linting exclusions or work-arounds. It may be nice for users to get this information from daft (including some potential config exception examples) so it's easier to deal with

Have you considered using the

DAFT_RUNNER=ray/py/native

environment variable to set the runner? That could be a much better user experience overall

Oh this is cool - yes, we can do that. We like users being able to decide on a notebook-specific or run-specific level. Perhaps we'll bake it into our

direnv

setup.

If you need to connect to Ray, Daft respects the canonical

RAY_ADDRESS=…

that Ray uses for detecting Ray connections too.

DAFT_RUNNER=ray
RAY_ADDRESS=...