@jay @Sammy Sidhu a follow up to our discussion yesterday on UDFs - if we want to be explicit about the API for stateful/pool-based UDFs, should we actually maybe expose it as a dataframe-level operation? we need to eventually pull it out into its own plan node anyways. we could have something like

df.with_actor_pool_columns(my_udf(df[“a”]), n_actors=4, parallelism_strategy=“thread”)

Let’s not overcomplicate it — this is going to be really difficult extending to SQL Keep it simple! We can just add the behavior we discussed wrt .with_concurrency making it a separate “pool” of resources (

parallelism_strategy="thread"

can also come as a follow-on PR if need be)