hello :wave: . Does daft support user defined aggr...
# generalm
Martin Bomio
10/04/2024, 7:25 PMhello 👋 . Does daft support user defined aggregation functions? Concretely, i want to compute standard deviation of a column in daft but couldn't find any implemented expression of std
🎉 1
Martin Bomio
10/04/2024, 7:26 PMto formulate better my questions:
1. Does daft supports std aggregation?
2. If not, is there a way to define a custom aggregation function?
j
k
Kevin Wang
10/04/2024, 7:30 PMLooks like something that shouldn't be too difficult to add!
We do not yet have a user defined global aggregation, although if your data fits into one machine, you can do a groupby on a common key and then a map_groups
Kevin Wang
10/04/2024, 7:40 PM@Martin Bomio made an issue to track progress on this: https://github.com/Eventual-Inc/Daft/issues/2997
j
jay
10/04/2024, 7:41 PMI think stddev should totally be on our roadmap though. Who’s ready to do some math 😛
🙌 1
jay
10/04/2024, 7:42 PMOk @Raunak Bhagat got nerdsniped by the math
😂 2
jay
10/04/2024, 7:42 PMHe’s taking care of it haha
🙌 2
r
Raunak Bhagat
10/08/2024, 3:46 PMHey @Martin Bomio, I've taken on this request and almost have everything working. Right now, just waiting on the CI pipeline and some incoming reviews.
The PR is opened here
https://github.com/Eventual-Inc/Daft/pull/3005
Hopefully this gets merged in ASAP and we unblock you.
🙌 1
Raunak Bhagat
10/08/2024, 11:11 PMHey @Martin Bomio, stddev has been released in the latest version of daft!
https://github.com/Eventual-Inc/Daft/releases/tag/v0.3.6
Please let me know if you get the chance to try this feature and if you have any comments or concerns.
🎉 1
j
jay
10/08/2024, 11:12 PMOh btw @Raunak Bhagat let’s also get an example up on the API docs!
r
Raunak Bhagat
10/08/2024, 11:18 PMOh hmm, looks like I missed getting an example for expressions. I added examples for dataframes, however.
I'll open up a follow-up PR to get that done.
🙌 1
m
Martin Bomio
10/09/2024, 2:30 AM@Raunak Bhagat thank you for the quick addition! I'll try it out tomorrow!
❤️ 1
r
Raunak Bhagat
10/09/2024, 2:49 AMNo worries! If any other features are needed, let me know!
Raunak Bhagat
10/09/2024, 10:23 PMHey Martin, were you able to try out the stddev feature? How's it holding up? Nothing unexpected, I hope.
m
Martin Bomio
10/10/2024, 5:33 PMhey @Raunak Bhagat tried the implementation today and seems to be working for me (at least on some small data, still need to try full scale)
Martin Bomio
10/10/2024, 5:36 PMI do have some other functions that been writing as udfs for now to get unblock but that maybe there's already a way to do it natively.
For example, we need a way to gather multiple cols into a single col, this will be used for one-hot encoding. We want to go from daft.Dataframe({"colA": [1, 2, 3], "colB": [4, 5, 6]}) to daft.Dataframe({"gathered": [[1, 4], [2, 5], [3, 6]]})
also let me know if you want me to start another thread
r
Raunak Bhagat
10/10/2024, 5:41 PMSure, if you don’t mind, a new thread would help organize our convo a bit better!
3 Views