Joins with different typed join keys turn out to b...
# daft-dev
k
Joins with different typed join keys turn out to be pretty broken right now and I'm not sure what the best approach is to fix it. The main issue is that if your left_on and right_on values are different types, that will cause problems with various things in the join, such as hash bucketing. My initial solution was to put a cast on all of the expressions if they needed one, but that causes issues with our logic around determining common join keys, since a col(a) may be turned into a col(a).cast(int64), which is no longer considered a possible common join key. An alternative I've considered is to insert projects before the join to do the casting, but that seems a bit messy. Wondering if anyone had any other ideas about how to approach this
j
Feels like translating it into a Project step with casting could be the way to go, and it could just get optimized out if it's a no-op...
k
Maybe raise a warning or an error with an option to allow the automated cast? All of the times I had joins with different types it was because they were wrongly typed and because it could continue to run I had a hard time figuring out why my results were bad or why the joins were taking forever
k
The results are incorrect now but they should not be and it should not take significantly more time @Kyle
After an offline chat with Sammy what I think I will do is my initial solution but with the join op augmented with an explicit field for the common join keys instead of using the left_on and right_on expressions to determine it
k
I think the one I had was between a string and an int32. Will it get casted from int32 to string in this case?
k
Yes, the int32 will be casted to a string
k
Okay cool looking forward to it!