https://www.getdaft.io logo
Join Slack
Powered by
Im having trouble connecting to a local ray head n...
# general
k
Im having trouble connecting to a local ray head node (
ray start --head --port=6379
). I can confirm that the node is active and can also reach the dashboard (http://127.0.0.1:8265/#/cluster) However, I cannot connect to it on daft using its 
<host>:<port>
format
Copy code
import daft
daft.context.set_runner_ray(address="<ray://127.0.0.1:6379>")
I get a connection timeout error
Copy code
daft.context.set_runner_ray(address="auto")
works, and automatically connect to 
127.0.0.1:6379
is this a bug? wanted to check in here first before opening an issue
oh,
Copy code
daft.context.set_runner_ray(address="127.0.0.1:6379")
works
👀 1
j
Yeah there’s actually 2 different ports here: 
--port 6379
is the port for Ray internally to talk to each other Looks like you still have to do 
<ray://127.0.0.1:10001>
(Just tried it out locally with using Ray directly 
ray.init(address=…)
k
oh interesting,
Copy code
daft.context.set_runner_ray(address="127.0.0.1:6379")
worked somehow
j
Hmm let me play around with it
🙏 1
k
ya np! i was just wondering if im doing something wrong
j
Oh you’re right! It does work. I think what’s happening is that there might be two ways to connect to the Ray cluster. Port 10001 exposes the 
ray://
protocol, and then 
6379
exposes a raw one… Hmm
image.png
Interesting I didn’t know 😛 cc @Raunak Bhagat too who is working on simplifying the Daft/Ray experience
k
Copy code
daft.context.set_runner_ray(address="<ray://127.0.0.1:10001>")
works!
j
😭
k
i can open an issue to track/formalize what works and what doesnt
j
Ray why you do this….
k
ya i didnt know about the 10001 port
j
Thanks!
Yeah that’s what I normally use, didn’t realize the 6379 port was a common pattern
Let us know how your first impressions of Daft goes 🙂
🙌 1
k
@Raunak Bhagat would love to get your opinion on spinning up a ray cluster; thats my next goal 🙂
❤️ 1
j
Also we just shipped Iceberg partitioned writes. Feel free to give that a spin
We actually shipped partitioned writes for bucket/truncation transforms too… Hahaha
k
ooo nice, i’ll give it a try
j
So technically a superset of what PyIceberg itself can do
👏 1
r
Spinning up ray clusters using an AWS backend should be pretty simple using our daft-launcher tool. I'm releasing a cleaner version today (v0.2.0), and some additional docs (including an mdBook!), so be on the lookout for that. We also have a little sync tomorrow where I'll be going over the entire launching demo, so feel free to hop on in if you'd like! Fun times for sure.
k
is the “daft-launcher tool” available already? would love to try it out
btw, opened an issue to track this thread https://github.com/Eventual-Inc/Daft/issues/2927
r
@Kevin Liu The current version (v0.1.0) has a bug in it that I found. The version that I'm releasing later today (v0.2.0) will address that bug. I'll try to release it asap for you!
🙌 1
k
ty! where can i find either versions?
r
Should be available by pip installing 
daft-launcher
!
Copy code
# if you're using pip
pip install daft-launcher

# if you're using uv (recommended)
uv pip install daft-launcher
I'm going to publish v0.2.0 in around 30min!
Published! The docs are still in works, and we're planning on OSSing this tool soon as well, but as of right now, you can view the project here: https://pypi.org/project/daft-launcher/0.2.0/
👀 1
🙌 1
Just a small nit: the 
daft init-config
file isn't implemented just yet. I accidentally included that prematurely in the readme 😞 Will try to release another patched version asap. However, the rest of the commands should work!
You need a 
.daft.toml
file in your CWD. The structure of the file is as such:
There are more configuration parameters that you can specify; however, those are the ones that should get you up and running. (This is what the 
daft init-config
command would have outputted in the first place. I'll try to get that working asap 🙂)
k
thank you! i’ll take a look. and congrats on the launch 🙂
🚀 1
r
Aw thanks!
Hey @Kevin Liu! We have a contributor sync today at 12pm PST! If you are available, you should try to hop on in. I've released a couple of patches and bug fixes to daft-launcher (v0.2.4) and you can see me run through it real-time.
2 Views
Open in Slack
PreviousNext
Powered by