@Cory Grinstead I'm thinking of doing some work to always parse expressions using our sql expr parser, which would enable string sql expressions everywhere that takes in an expression. this would include moving our expression resolving logic to the sql parser (things like parsing

a.b

to

col("a").struct.get("b")

and parsing

a.*

to a list of expressions). any thoughts about that?

i think that could be interesting, but could also make for some weird/unwanted flexibility such as:

col("a as b")

when we'd likely want people to just use

sql_expr("a as b")

why not just keep

col

the way it is since we also have

sql_expr

Hm, imo that might be fine? If someone wants to do

col("a as b")

I would be okay with it. What are your thoughts? The main motivations for doing this is to 1. unify column name parsing for syntactic sugar such as struct getters and wildcards (right now we have something that is honestly kind of hard to understand on the non-sql side) 2. bring the sql expr integration to places other than .where

hmm, I'm leaning towards explicit usage as i think there's a lot of edge cases with using the sql parser for column expansion. Also, the sql parser relies on the column expr, (the rust implementation) so you may run into a circular dependency as well.

What kind of edge cases are you thinking about?

most sql dialects are case insensitive with columns, unless wrapped in quotes.

select NAME

is the same is

select name

, but different from

select "Name"

, but i'm wondering how that'd all work in dataframes as

col

is case sensitive. so

col("NAME")

,

col("Name")

and

col("name")

refer to 3 distinct columns. I suppose you could require the user to do

col('"Name"')

but that feels weird to me, and would likely introduce a ton of backwards incompatible changes.

I see. That makes sense, thanks for the explanation

I did have a lengthy conversation about case sensitivity with Robert Howell (Amazon) Even within the SQL world this is apparently not standardized. I think we just have to take an opinionated stance and go with it there.

I think most engines have some global parameter that configures whether or not it's case sensitive. we can likely do the same