https://www.getdaft.io logo
Join Slack
Powered by
Hey Daft folks! I was looking at your <architectur...
# general
r
Hey Daft folks! I was looking at your architecture, and I noticed that the vast amount of functionality is implemented in rust, with python simply as a convenient API interface. I was wondering if you've considered having a way to serialize operations defined in python (thus materialized into plans, I believe, in rust?) and load the operational plan and execution in a non-python environment for high performance data transformation? (ie real time serving or streaming)
1. This is not a feature request! 2. It's clear to me that this is not in scope for the current library's goals (it seems the rust API is not publicly advertised, and thus implicitly not intended to be exposed) 3. Despite the above, I'm curious to get your insight for the possibility of doing this, if you think there are any obvious pitfalls etc!
concrete use case: define operations to process raw data for training a ML model (python environment/notebook) and then packaging the transformations as some serialized artifact that can be loaded at serving time outside of python
j
Are you thinking of doing something like numba/JIT-compiling a Python UDF? We’ve definitely thought of that, and it’s possible to do but hasn’t been a high priority for us
Today our Python UDFs are very naively packaged into the plan (as a cloudpickle), which works for most workloads that don’t require serializability of the plan that lives beyond the lifetime of a single query (since pickles are not meant to be used for persistent storage)… I guess this feature could be useful if we want to re-use the plan at some (much later) time
j
What use cases might there be for wanting to re use the plan at a future time?
r
> What use cases might there be for wanting to re use the plan at a future time? As I said before: > concrete use case: define operations to process raw data for training a ML model (python environment/notebook) and then packaging the transformations as some serialized artifact that can be loaded at serving time outside of python The simplest example would be something like having a bunch of raw numeric data for an ML model which is sensitive to the scale of the numeric features (feature = column). The most common approach for this is to normalize each feature so that the mean is 0 and the standard deviation is 1. This requires an aggregation over the dataset to calculate mean and standard deviation for each feature, and then to apply 
(x-mean)/std_dev
to the column. When we serve the model in real time, we want to ensure the same transformations we did during training can be applied at serving time. In this specific case, it wouldn't be so hard to take the aggregate metadata (mean+std dev for each column) and "manually" apply the function, but many transformations are more complex than this, so being able to serialize it and apply to future data is ideal
Are you thinking of doing something like numba/JIT-compiling a Python UDF?
I can't even say I've thought that far about it, I'm very new to the daft project and exploring its possibilities. It sounds to me that specifically support for python UDFs would block this from being possible
👍 1
j
Yes, Python UDFs would make this very tricky because we need to exactly reproduce the Python environment that is needed to run it…
r
yeah, coming into this, I assumed all/the majority of operations were actually some kind of rust primative. Is this the case, but with the added extension of python UDFs?
j
Yes correct. All of our expressions are executed using Rust code with the addition of python UDFs for more complex operations that only our users can provide
9 Views
Open in Slack
PreviousNext
Powered by