Can the minhash function accept a different hashin...
# generalk
Kyle
09/25/2024, 3:58 AMCan the minhash function accept a different hashing algorithm for the minhashing?
j
jay
09/25/2024, 4:09 AMWhat hashing algorithm are you thinking of?
k
Kyle
09/25/2024, 4:10 AMThe current one we're using is the first four bytes of sha1 like in https://github.com/bigcode-project/bigcode-dataset/blob/main/near_deduplication/minhash_deduplication_spark.py
Kyle
09/25/2024, 4:10 AMThe deduplication results are currently vastly different so it makes it a bit difficult to switch over
j
jay
09/25/2024, 4:10 AMWe currently use mumurhash
cc @Sammy Sidhu
jay
09/25/2024, 4:11 AMYeah shouldn’t be too difficult to make configurable on our end I think
s
Sammy Sidhu
09/25/2024, 4:11 AMWhat hash functions would you like to see implemented?
k
Kyle
09/25/2024, 4:14 AMJust this sha1 first 4 bytes
j
jay
09/25/2024, 4:14 AMsha1 is kind of wasteful 😛 so much compute just to take the 32 bits
k
Kyle
09/25/2024, 4:14 AMYeah..
j
jay
09/25/2024, 4:14 AMyou’ll likely see some performance degradation I’m guessing
jay
09/25/2024, 4:14 AMAlso are you doing normalization the same way as you were before? That might affect results
k
Kyle
09/25/2024, 4:14 AMI'm wondering if murmurhash's first four bytes will give equivalent results
Kyle
09/25/2024, 4:15 AMNormalization is mostly similar except tokenization which is slightly different because it splits on non-alpha regex instead of on whitespace
s
Sammy Sidhu
09/25/2024, 4:17 AMWe also do have
xxhash3_64murmurSammy Sidhu
09/25/2024, 4:19 AMfor context, this is the hash function we use for all internal Daft partitioning and it seems to work well
k
Kyle
09/25/2024, 4:20 AMCool! I think because of the truncation to 4 bytes the collision rate is much higher and hence the pyspark implementation is identifying a lot more duplicates
Kyle
09/25/2024, 4:21 AMSo the truncation to 4 bytes i think is important
2 Views