https://www.getdaft.io logo
Join Slack
Powered by
does sql.expr() not support comparing lists? seein...
# general
t
does sql.expr() not support comparing lists? seeing this error:
Copy code
thread 'Compute-Thread-10' panicked at src/daft-core/src/series/ops/comparison.rs:63:5:
List(Utf8) not implemented
ERROR:daft_local_execution:Error when running pipeline node ProjectOperator
for this expr:
Copy code
when_neq_condition = " OR ".join(
    [f"c.{col} <> p.{col}" for col in value_columns]
)

cdc = daft.sql_expr(
f"""
CASE
    WHEN c.uuid IS NULL THEN 'D'
    WHEN p.uuid IS NULL THEN 'I'
    WHEN ({when_neq_condition}) THEN 'U'
    ELSE NULL
END AS cdc
"""
)
seeing this issue also for columns that contain structs:
Copy code
thread 'Compute-Thread-10' panicked at src/daft-core/src/series/ops/comparison.rs:63:5:
Struct([Field { name: "amount", dtype: Decimal128(10, 0), metadata: {} }, Field { name: "currency", dtype: Utf8, metadata: {} }]) not implemented
ERROR:daft_local_execution:Error when running pipeline node ProjectOperator
Hmm, seems to be a core issue I guess. Tried also using daft method directly and same issue:
Copy code
df = df.with_column("cdc", (
    df["c.uuid"].is_null().if_else("D",
        df["p.uuid"].is_null().if_else("I", 
            (reduce(lambda x, y: x | y, (df[f"c.{col}"] != df[f"p.{col}"] for col in value_columns))).if_else("U", None)
        )
    )
))
Since my schema is predictable, I'm able to work around this by explicitly comparing the struct fields and converting the list to a string using 
.list.join()
j
What’s the expected behavior for list/struct comparisons? Indeed I don’t think we have them defined today.
t
Same as most SQL engines, and Spark I suppose. Compares both lists by elements. I think structs are handled the same way; compares both by fields.
j
Ah ok, I think we can support eq/neq but the general case for comparisons (
>/>=
etc) might be hard cc oncall @Kevin Wang
I’m surprised equality doesn’t already work though, that seems like a pretty easy fix
k
Hi @Tabrez Mohammed, I created an issue for this to track its progress and put it into our backlog! Will take a look at implementation soon https://github.com/Eventual-Inc/Daft/issues/3510
🙏🏽 1
2 Views
Open in Slack
PreviousNext
Powered by