Is string equality not null safe in daft?

col = "foo" # both c.foo and p.foo are all nulls
df.where(df[f"c.{col}"] == df[f"p.{col}"]).show()

returns 0 rows

Equality actually falls back on

null

if either LHS/RHS is a

null

I believe. We should check what postgres behavior is and follow that cc @Kevin Wang

hmm, so I can't have the expression short circuited with a true values earlier in the expression?

df.where(
    (col(f"p.{col_name}").is_null() & col(f"c.{col_name}").is_null())
    | (col(f"p.{col_name}").not_null() & col(f"c.{col_name}").not_null())
    | (col(f"p.{col_name}") == col(f"c.{col_name}"))
).show()

That is correct, it propagates null values up. Could you try using fill_null instead?

OK will try fill_null, but isn't there a core issue here with boolean logic? df.where(True | False) should return everything

Hm, will investigate what standard behavior is among other libraries. The general rule we try to follow is to propagate nulls but I think this may be a case where it should not do that

there's no type checking with

fill_null

? Not giving an error filling an int64 col with strings or vice versa

It does implicit supertype-casting (notice how the type changes from int32 to string in this example)

what's the trick for filling null on a decimal type?

DaftCoreException: DaftError::External Unable to create logical plan node.
Due to: DaftError::TypeError Expected expr and fill_value arguments for fill_null to be castable to the same supertype, but received amount#Decimal(precision=10, scale=0) and literal#Decimal(precision=1, scale=0)

df[f"c.{col}.amount"].fill_null(Decimal(-1.0)) != df[f"p.{col}.amount"].fill_null(Decimal(-1.0))

EDIT: nvm, realized what precision vs scale are. Need a 10 digit number and it works.