I'm getting a horrendous amount of panics with the...
# general
k
I'm getting a horrendous amount of panics with the new version release 👀 Update: Nothing wrong with the new version, but having a cluster that installs getdaft in the ray runtime env without specifying the version could cause unexpected panics when there's version mismatches within the cluster
❕ 3
j
What’s going on? Any chance you could share the error message?
k
There's a whole bunch of things being printed I have no idea about, but rolling back to 0.3.5 seems to work
j
Yeah panics are usually internal errors so they wouldn’t make much sense to a user. This is helpful let me take a look
k
image.png
image.png
image.png
j
Ok we’re investigating now
👍 1
Any chance you might have a different version of Daft in the cluster vs local?
k
This is done via the Ray Job so there shouldn't be a dependency
👍 1
k
I usually just leave it as getdaft[all] in my runtime env yaml so today it got upgraded when i sent my job in
👍 1
j
Could you do us a favor and run again with the Ray job, on a fresh Ray cluster specifying specifically
getdaft==0.3.6
across the entire job? Trying to rule out a version mismatch issue which would almost definitely cause problems that look like these
And if that is still panicking, you can. try specifying this wheel to install for your Ray job (I’m assuming you machines are x86 intel machines):
Copy code
<https://anaconda.org/daft-nightly/getdaft/0.3.5+dev0007.396c004e/download/getdaft-0.3.5+dev0007.396c004e-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
If this version of Daft works, this will help us isolate the only commit(s) in this release that we think is causing the issue.
k
Might need to let you know again later! It's currently running a few jobs 🫠
j
😛 alright
Thanks!!
Really appreciate the patience
k
Np!!! Thanks for the quick action!
Indeed it seems like it's the versioning! I've tried 0.3.6 with a fresh cluster and it works without panicking!
j
Ahhhh good. Thanks @Kyle!
Yeah highly recommend that when you specify Daft versions on Ray, you do so using pinned versions.
k
Yep! Haha