Hello again folks, I see that Daft makes it easy to pull data from a URL column into memory using the

url.download()

expression. What would be the most efficient/daft-friendly way to perform an equivalent

url.upload()

? The current API only seems to support uploading to a directory/S3 prefix and doesn't allow setting up a column for desired output URLs.

Hey @Neil Wadhvana! You should be able to upload the data and get a new column of “uploaded URLs”:

import daft

df = daft.from_pydict({"foo": [b"data1", b"data2", b"data3"]})
df = df.with_column("bar", df["foo"].url.upload("my_dir/"))
df.collect()

I'm looking to upload the column to row-specific urls instead of writing n-thousand randomly named objects and then moving them again. Is there a way to do this? ie. What's the easiest way to upload to a column of row-specific S3 paths instead of a dataframe-wide S3 prefix?

Oh I see what you mean

That's the exact syntax I tried to use 👍

What expression would you expect to be returned?

The output column of URLs seems like a good option. How does

download()

handle/report failures?

we expose an

on_error="raise,null"

flag which dictates behavior

Makes sense. I think the same logic can go for the upload functionality. This is technically an "in-place" operation on the dataframe itself. Do you have any paradigms you follow for this class of operations?

No 😛

Haha - in that case, successfully uploaded S3 URI or

null

seems like a meaningful output. That way, the user can still control the behavior of

raise

without it breaking API consistency with

download()

?

Ok this all sounds great

Issue incoming

Do you mind making an issue for us to track please?

https://github.com/Eventual-Inc/Daft/issues/3320 I've elected for

No

simply because the rest of the year is busy for me, but I expect I may have some small contributions to offer next year to

daft

!

cc @Desmond Cheong for context as well