I'm looking to implement monotonically increasing id's, right now it's

df = df._add_monotonically_increasing_id("id")

but I want to adapt it to be something along the lines of

df = df.with_column("id").mid()

Would

.mid()

belong in

daft.expressions

or something new like

daft.functions

or something else entirely?

@Colin Ho did you work on monotonically increasing sequences recently? I thought I saw a related commit.

I did the original monotonically increasing id implementation, we implemented it as a dataframe method

Nice, a “generator” expression is a cool idea!

from pyspark.sql import functions as sf
spark.range(0, 10, 1, 2).select(sf.monotonically_increasing_id()).show()

this is how it works in spark

I think “sequence” is the term of art here, and could eventually take parameters.

I think the question I had was where the monotonically_increasing_id expression would live

Nice. I think daft.functions makes sense for a namespace in general. Others are builtins, routines, info_schema but those are a little arcane. For the port, do you intend to support both chaining and composition?

How would chaining and composition look like for an example use case? But yes, I am intending to support both,

I think we would want to generally want users to use composition other than for functions that are tied to the origin expression such as .alias("x") or .is_null(). Implementation-wise we can probably do this in a gradual manner, where we incrementally move our expressions and add deprecation warnings to the corresponding methods in

Expression

IMO I would prefer the functions in

daft.functions

rather than

daft.expression

so they are grouped, but I don't think there are issues with re-exporting? I believe we could easily support both composition and chaining automagically with some monkey patching. Here's an example, Nishant.

col("my_str").lower()   # chaining

lower(col("my_str"))    # composed

Functional languages often support both, and I think deprecating all current function APIs might be jarring to customers.

Additionally, there are ways to do even regular function composition that are preferred over the

h(g(f(x)))

formulation. Scala has the

f andThen g andThen h

. Haskell has

h . g . f

. F# has a forward composition operator

f >> g >> h

that returns a composed function and something similar called the forward pipe operator (

|>

). Functional programmers prefer these formulations because of the lack of nesting. Nesting is a lot harder cognitively to track than juxtaposition.

thanks, taking a look right now! i agree about continuing support for chaining

I'm a little hesitant on providing multiple ways to do the same thing. There are expressions at the moment that definitely do not make sense in chaining-form (e.g. if_else). That along with the expression type namespacing (e.g. col("x").str.length()) which we want to remove, means users will probably see some breaking changes in our API anyway

You might be conflating expressions with functions. Functions make sense in either composition/chaining, but not all expressions do. In your example,

if_else

is a control-flow expression and not a function.

What do you mean by normal-form?

No not expressions, but scalar functions. A normal strict function is one that evaluates all of its arguments before invoking the routine itself. Control-flow does not do this e.g. if-else, coalesce

Ah I see. What would you consider boolean and/or?

binary expressions

I'm admittedly not familiar with these concepts. We might want to move to a different thread though, this is perhaps off topic

Feel free to send me a DM if you have any additional questions and we can work through examples.