Is Iceberg table predicate pushdown applicable to is_in expression? More specifically, my use case needs equivalent of pandas_df[pandas_df.column_a.isin(python_list) ] on an iceberg table bucketed by column_a

Hmm interesting. I’m actually not sure 😛 @Sammy Sidhu any idea if this is the case?

We currently don't have that as a rule but it should be straightforward to add here! Would you be interested in making a PR to add this?

yes, thanks for the work around. Is there a way to support pyiceberg's in clause in daft as well? My work-around is to leverage pyiceberg for "in" clause use cases

inter_table = catalog.load_table("mydb.mytable_iceberg")
    inter_df = daft.from_pandas(inter_table.scan(
       row_filter= f" lookup_id in ( {lookup_list_string} ) " 
    )

Is the above example different from

df.where(df["customer_id"].is_in(customer_list_string))

?

it is another example