https://www.getdaft.io logo
Join Slack
Powered by
Noticed that the list.value_counts is in and it's ...
# daft-dev
k
Noticed that the list.value_counts is in and it's working for me! Thanks @Andrew Gazelka! Could we also have it added in the docs on the website?
daft party 2
d
Andrew could you add the one line entry to 
docs/source/api_docs/expressions.rst
?
k
I do have a follow-up problem now: How can I get the key and value of the item with the highest value from the map series?
I'm currently going to do it this way but if you have any optimization do let me know!
Copy code
@daft.udf(return_dtype=daft.DataType.struct({"top_key":daft.DataType.string(), "max_value": daft.DataType.uint64()}))
def get_most_frequent_key(count_series):
    counts_list = count_series.to_pylist()
    results = []
    for counts in counts_list:
        top_key = None
        max_value = 0
        for count in counts:
            key, value = count
            if value > max_value:
                top_key = key
                max_value = value
        results.append({"top_key": top_key, "max_value": max_value})
    return results
d
Hmmm if we're talking about native kernel support, I think we're missing a few pieces that other engines might support: ā€¢ some map -> array of map entries function ā€¢ some array sort function that takes in a predicate for sorting (we already have 
list.sort
, but it only takes primitive types and does not support custom predicates) with the two pieces above, we could naturally sort the value counts in descending order of counts, then just call 
list.get
on the first element to achieve what you want
k
yes šŸ˜… i was thinking along the lines of a custom comparator for list.sort and also the ability to call for .keys/.values/.items
plus one 1
d
Yeah let's see what the team thinks. Funny enough, for this specific pattern, it seems that lots of engines don't have a good story end-to-end despite being super reasonable. For example, duckdb has 
map_entries(map)
that gives a list of 
(key, values)
, but then you can't really sort this after (that I'm aware of). Spark has 
map_entries
and 
array_sort
, but for 
value_counts
you'd have to 1. get distinct elements for each list, 2. roll your own lambda function in a 
transform
to aggregate the counts for each distinct value.
šŸ™ 1
a
7 Views
Open in Slack
PreviousNext
Powered by