https://www.getdaft.io logo
Join Slack
Powered by
Another q! I'm trying to use a transformation cla...
# general
n
Another q! I'm trying to use a transformation class from 
torchvision
or 
kornia
on a daft dataframe column. To do this, I have 3 options: • wrap it in a function and apply it using 
transform()
• wrap it in a stateful udf and take care of all the typing (this needs to be done for every class I want to use) to make it behave nicely • try applying it with 
.apply()
directly. I think the latter should be the best bet, especially if I know that the transform state does not change with each application. Is there any better API you could suggest for applying such transforms without wrapping them in extra boilerplate code? The blocker is specifically the 
__qualname__
check in 
.apply()
.
Example syntax:
Copy code
from kornia.augmentation import RandomHorizontalFlip

df = df.with_column(
    "image",
    df["image"].apply(
        RandomHorizontalFlip(p=0.7, keepdim=True), return_dtype=daft.DataType.python()
    ),
)
can the qualname chack silently/safely fall back to name?
j
We usually recommend not using apply because it forces application of your function per-row If you wrap it in a UDF you get access to a batch of data instead, which could be more efficient!
You can wrap it in a stateless UDF here since there isn’t really any state or expensive initializations
n
It definitely could, however, a lot of the "quick" wins can be had by simply plugging in transforms from these libraries directly instead of trying to maintain a "helper" function which introduces more friction to the API
👍 1
Also, even if you do get access to the batch of data, the series still needs to be packed into the "batched" format inside the udf - no? That operation could potentially add overhead.
j
The batched format would give pytorch better performance I think And yes, making a helper for something like pytorch transforms would be super cool.
n
It would then need to be maintained for "any such library"
Perhaps there's a cleaner way to do this in 
apply()
directly
Here's what I've introduced for now, for anyone who ventures into this thread:
Copy code
def with_qualname(obj: object) -> object:
    if not hasattr(obj, "__qualname__"):
        obj.__qualname__ = obj.__class__.__name__
    return obj
2 Views
Open in Slack
PreviousNext
Powered by