https://www.getdaft.io logo
Join Slack
Powered by
Hello! I was wondering if there's any transform to...
# general
m
Hello! I was wondering if there's any transform to gather multiple columns int a new one. I want to go from daft.Dataframe({"colA": [1, 2, 3], "colB": [4, 5, 6]}) to daft.Dataframe({"gathered": [[1, 4], [2, 5], [3, 6]]})
cc @Raunak Bhagat
r
Synced with Jay about this. We have a struct version for this, i.e.:
Copy code
{ "col_a": [1,2,3], "col_b": [4,5,6] }

# after gathering

{ "gathered": [{"col_a":1,"col_b":4}, {"col_a":2,"col_b":5}, {"col_a":3,"col_b":6}] }
But no list version currently. I will take on implementing this asap. We will have this done by next release cycle!
m
thank you!
related to this topic (one_hot encoding), finding the unique value counts for a column is also an aggregation that would be useful to have (right now I am counting uniquye values outside of daft and using those stats to one hot encode)
👍 2
j
We currently have 
.approx_count_distinct()
as an aggregation, but I’m guessing you want an exact variant of that?
m
@jay looking at the docs, that would give us a count of distinct values, I was thinking more of a for each unique value, get me the distinct counts
❤️ 1
j
I see! We do have a very similar function interestingly, it does it per-row for list columns right now.
But we should be able to generalize it to work over columns
c
just checking regarding 'unique value counts for a column', could you use a group by and count on the same column? eg.
Copy code
df = daft.from_pydict({"unique_values": ["a", "b", "c", "b", "c", "c"]})
df = df.groupby("unique_values").agg(df["unique_values"].count().alias("count"))
df.show()

╭───────────────┬────────╮                                                                                                                                             
│ unique_values ┆ count  │                                                                                                                                             
│ ---           ┆ ---    │
│ Utf8          ┆ UInt64 │
╞═══════════════╪════════╡
│ c             ┆ 3      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ a             ┆ 1      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ b             ┆ 2      │
╰───────────────┴────────╯
Also on the note of one hot encoding (i'm not an expert on this but I just thought maybe it might help), you could see if pivot can work for you.
Copy code
import daft

# Sample data dictionary
data = {
    "id": [1, 2, 3, 4],
    "platform": ["macos", "macos", "windows", "linux"]
}

# Create Daft DataFrame from the dictionary
df = daft.from_pydict(data)

# Add a column with constant values of 1 to indicate the presence of each category
df = df.with_column("value", daft.lit(1))

# Perform the pivot operation to create one-hot encoded columns
# We pivot on the "id" to keep rows unique and create columns for each "platform" with values of 1 where present, 0 otherwise
df_onehot = df.pivot(group_by="id", pivot_col="platform", value_col="value", agg_fn="sum")

# Display the result
df_onehot.show()

╭───────┬─────────┬───────┬───────╮                                                                                                                                    
│ id    ┆ windows ┆ macos ┆ linux │                                                                                                                                    
│ ---   ┆ ---     ┆ ---   ┆ ---   │
│ Int64 ┆ Int64   ┆ Int64 ┆ Int64 │
╞═══════╪═════════╪═══════╪═══════╡
│ 2     ┆ None    ┆ 1     ┆ None  │
├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
│ 4     ┆ None    ┆ None  ┆ 1     │
├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
│ 1     ┆ None    ┆ 1     ┆ None  │
├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
│ 3     ┆ 1       ┆ None  ┆ None  │
╰───────┴─────────┴───────┴───────╯
https://www.getdaft.io/projects/docs/en/stable/api_docs/doc_gen/dataframe_methods/daft.DataFrame.pivot.html#daft.DataFrame.pivot
j
Also if this works, we should totally make it into a helper. This is such a common use-case
m
@Colin Ho thank you! I think that together with a gather would totally do it.
j
@Martin Bomio let us know if that works for you, perhaps we should turn this into a helper method on the `DataFrame`… Also would love to take a contribution here if you’re up for it 😉
2 Views
Open in Slack
PreviousNext
Powered by