https://www.getdaft.io logo
Join Slack
Powered by
I just started a Ray Client and found that I am ab...
# general
k
I just started a Ray Client and found that I am able to find my local parquet files on the ray workers using from_glob_path on /tmp/ray/xxx but was not able to do the same with read_parquet on the same folder. Any way to still read the files in using the paths from the glob? I get this error
Copy code
FileNotFoundError: File: /tmp/ray/xxx/0.parquet not found
No such file or directory (os error 2)
From the glob:
Copy code
╭────────────────────────────────┬────────┬──────────╮
│ path                           ┆ size   ┆ num_rows │
│ ---                            ┆ ---    ┆ ---      │
│ Utf8                           ┆ Int64  ┆ Int64    │
╞════════════════════════════════╪════════╪══════════╡
│ file:///tmp/ray/xxx/0.parquet  ┆ 358370 ┆ None     │
╰────────────────────────────────┴────────┴──────────╯
j
Hmm so you’re saying: This works
Copy code
daft.from_glob_paths("/tmp/ray/xxx/*.parquet")
But this doesn’t
Copy code
daft.read_parquet("/tmp/ray/xxx/*.parquet")
k
Yep! but I did ** for both
Only for the Ray Client though. Both work when I submit the Ray job
j
So when you do 
daft.read_parquet/from_glob_paths
on your laptop, it’s going to do the globbing from your laptop/dev machine…
k
But the from_glob_path didn't
😱 1
j
i.e. you need the HDFS mounts both on your cluster, as well as the dev machine that is launching the job
Are you certain? That is very odd…. Hahaha 😛 (They use the same code-paths, I think. Let me double check.)
OH
You’re right! 
from_glob_path
is a special code-path to run the globbing in the cluster. I think we did this specifically because people wanted to use cluster credentials for globbing
Interesting.
k
Well in this case the from_glob_path one works better for me haha
j
Yeah the tricky bit also is that 
read_parquet
does a bunch of other things besides globbing (we read the first file to get the schema as well). All of this today happens on the dev machine instead of the cluster.
k
When I do decide set my runner to the ray runner it would be great to be able to access my runner's paths 😅
👍 1
Oh I see 😮
j
This does mean that we need to do this schema inference on the cluster as well… because it needs access to HDFS
k
True..
Which would be faster actually
j
Ok, yeah it’s not impossible to do, but we need to think very hard about the implementation semantics
Let me chat with the team
Could you file a ticket here with some brief details?
k
Cool! Okay!
❤️ 1
j
Thank you!!
k
Thank you!
4 Views
Open in Slack
PreviousNext
Powered by