jay
06/25/2024, 3:00 AM.url.download()
don’t schedule with a SPREAD strategy unlike our ScanTasks/ReduceTasks
@Sammy Sidhu do you think we should corner-case projections with URL downloads to also run with SPREAD?jay
06/25/2024, 3:06 AMdf = daft.from_glob_paths(...)
df = df.into_partitions(32)
df = df.with_column("data", df["path"].url.download())
Sammy Sidhu
06/25/2024, 3:07 AMSammy Sidhu
06/25/2024, 3:07 AMjay
06/25/2024, 3:08 AM