This message was deleted.

Not right now in a lazy fashion on the dataframe. If you don't need a distributed setup I would recommend using duckdb on an arrow table (produced by daft) for the SQL component

df = daft.read_iceberg(my_table)

# Partition Filter to prune out much needed data
df = df.where(df["partition_col"] < SOME_VAL)

# selected columns (all others will be pruned out)
df = df.select("i", "j", "k")

con = duckdb.connect()
# Zero copy version to arrow
my_arrow_table = df.to_arrow()

results = con.execute("SELECT * FROM my_arrow_table WHERE i = 2")

We plan on specing out SQL support soon! We want to ensure that we do a good job!

Thanks @Sammy Sidhu, didnt knew about to_arrow function , will try this . Looking forward to the distributed sql support