https://www.getdaft.io logo
Join Slack
Powered by
Found an interesting behavior of hashing and modul...
# general
y
Found an interesting behavior of hashing and modulo (Maybe I am using it wrong?)
Copy code
daft.col(join_key_left).hash() % n_partitions)

VS

daft.col(join_key_left).apply(lambda val: hash(val) % n_partitions, return_dtype=daft.DataType.int64()))
For the first, I get a very un-even distribution, for the second I get an even distribution. My guess is % expression is not doign what I think it's doing.
First:
second:
cc @Henry T
j
What’s the type of the 
join_key_left
column?
y
UTF8
Copy code
import daft
from daft import col
import random
import string

def rand_str():
    return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(20))


n_partitions = 10
df = daft.from_pydict({"col": [rand_str() for _ in range(10000)]})

print("Uniformly distributed")
df.with_column(
    "hash",
    daft.col("col").apply(lambda val: hash(val) % n_partitions, return_dtype=daft.DataType.uint16())) \
.groupby("hash").agg(
    col("hash").count().alias("count")
) \
.show()


print("Non-uniformly distributed")
df.with_column(
    "hash",
    daft.col("col").hash() % n_partitions) \
.groupby("hash").agg(
    col("hash").count().alias("count")
) \
.show()
output:
Copy code
Uniformly distributed
╭────────┬────────╮                                                                                                                                                                                                                                                                                                                  
│ hash   ┆ count  │                                                                                                                                                                                                                                                                                                                  
│ ---    ┆ ---    │
│ UInt16 ┆ UInt64 │
╞════════╪════════╡
│ 9      ┆ 1001   │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 5      ┆ 990    │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 2      ┆ 994    │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 7      ┆ 984    │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 1      ┆ 968    │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 3      ┆ 1033   │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 8      ┆ 992    │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 4      ┆ 993    │
╰────────┴────────╯

(Showing first 8 rows)
Non-uniformly distributed
╭─────────┬────────╮                                                                                                                                                                                                                                                                                                                 
│ hash    ┆ count  │                                                                                                                                                                                                                                                                                                                 
│ ---     ┆ ---    │
│ Float64 ┆ UInt64 │
╞═════════╪════════╡
│ 4       ┆ 1994   │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 2       ┆ 2061   │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 6       ┆ 1965   │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 5       ┆ 1      │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 9       ┆ 3      │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 1       ┆ 1      │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 0       ┆ 1976   │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 8       ┆ 1999   │
╰─────────┴────────╯
s
that's odd that it is showing up as a float. I think there might be a bug here. Also can you try the same query but with like a 100 rows? It may be the role of the hash function we are using. In this case, xxhash
@Yuri Gorokhov So I found an issue with how we did type widening for 
UInt64
and the signed literal that was passed in. The resultant type ended up being the float that you saw above which when was rehashed for the group by led to odd distribution you saw. This however only affected the 
%
operator when used with a literal like what you had. When performing joins, group by or other partitioning operations, the distribution should be what we expect. Heres the PR to fix this It looks correct now! 🙂 Thanks for raising this issue!
🔥 1
y
Thanks a lot @Sammy Sidhu!!
Now half my workers will not mysteriously run out of memory 🙂
🫠 1
h
great thank you for the quick fix
5 Views
Open in Slack
PreviousNext
Powered by