Is there a way to run from_glob_path with more than one worker?

Currently

from_glob_path

can only run on one worker. Are you looking to run globs for multiple paths across workers?

It's only one big folder with many files so it's taking very long but yeah I guess it'll be hard to parallelize

Yeah we do a bunch of tricks there already, but it works only if you have your files in folders. Are you listing S3 or local storage?

Ah I see, it's currently in one big folder in local

S3 is probably slower actually, because it’s paginated and lists things one at a time in pages of about 1,000

I see.. I tried running it locally for a folder with 700K files but I couldn't tell how much it had completed after half an hour so I ended it to run something else

@Kyle are you running a simple glob or any other additional operations chained to it? Running glob by itself proved to be really fast (0.1 to 190 seconds max), so I started globbing the different paths/buckets I'm working with (~500k-2m objects each) and then saving/working with those tables.

Yeah i thought it was really weird as well.. it's just a simple glob.. @Ammar Bitar are you running this locally or on Ray?

locally. I also used it on both S3 and local folders

I see! I'm thinking maybe because I'm running it on Ray it's actually much slower because it only uses one core

Couldn't help you with that, still in the process of correctly setting up ray 🙂 But if that's the issue, might make sense to glob locally save the table somwhere and re-read it with ray ? Otherwise, sorry can't help further 😞 just for reference, I'm just using glob like this

df = daft.from_glob_path(
    s3_url,
    io_config=io_config
)

And on various buckets which vary from being very shallow and wide to very convoluted structures - and it seems to work (weirdly, even better than boto3's own listobjectsv2 + paginator...)

That's very cool! I'll try it out on the local machine then! Thanks for sharing your experience!!

what happens for ‘from_glob_path’ , if the pattern matches millions of S3 objects ? Does daft paginate automatically ?

Yes! We have a paginated + parallel globber