We got a cool feature request to support <gzip dec...
# daft-dev
r
We got a cool feature request to support gzip decode which has me excited for additional binary string features like, • base64 encode/decode • gzip/lz4/zstd/snappy Are there other features in this domain that people are excited about or have wanted? 🧵
SQL also defines OVERLAY and TRIM for binary strings which are kind of interesting to parallel from character strings. OVERLAY semantics are a bit odd, but I could see a "MASK" function being useful too or clever things like applying bitwise operators to binary strings with a mask, repeat, and step size.
https://www.postgresql.org/docs/9.6/functions-binarystring.html • encode/decode with encoding as string arg • md5 • get_bit / set_bit • get_byte / set_byte
c
i saw your comments on the issue specifically
# decode as "gzip" or return NULL on failure
col("my_bytes").try_decode("gzip")
in another issue/PR you mentioned the need for
try_cast
An expression that I really want to add is a
try
expression, instead of individual
try_
methods.
I'm not exactly sure what that'd look like. maybe something like this:
Copy code
try_(col("my_bytes").decode("gzip")).except_(lit(b"an error decoding"))
r
This could be pretty cool to flesh out alongside the existing SQL control-flow operations: coalesce, case-when, nullif
For context, some popular SQL impls have
try_cast
and it would have came in handy for SUMMARIZE to permissively support more types.
j
I also wonder if our casts should be more appropriately defined under the hood... e.g. • str to int cast should call out to a
stoi
expression or similar • int to string cast should call a
to_string
expression • string to date should call a
strptime
expression • date to string can call a
strftime
expression, with a default setting (e.g.
YYYY-MM-DDTHH-MM-SS
) etc etc This would make our casting docs easier (we can call out exactly which expression we call out to under the hood for various cast operations) and also allow people to be more explicit when moving between different datatypes...
If we can make creating new expressions easier maybe this would be a lot lower of a lift šŸ˜›
r
Named cast methods are quite nice and are common in some SQL dialects too. The named cast methods can also be nice when you want a cast that has different semantics and optional parameters which differ than an SQL CAST which is highly prescriptive in the standard. The only thing to solve would be argument handling, but this plays well with the receiver comment I just made šŸ™‚
Copy code
class Expr:
   def to_str(self):
       # a cast with no type args

   def to_decimal(self, precision, scale):
        # cast with args

# daft.functions

def to_str(expr): ...

def to_decimal(expr, precision, scale): ...
j
All great ideas!
r
I love language design šŸ˜„
@Cory Grinstead Here's a PR for the encode/decode with deflate, gzip, zlib which should hopefully cover all deflate related cases. I set us up to trivially add more codecs, and hopefully compression options as well! I did not add a "try_encode/try_decode" since you had a different, more general proposal but could easily include in this PR. https://github.com/Eventual-Inc/Daft/pull/3907
šŸ‘€ 1
c
Just left a review, Functionally it looks fine, but I did leave some comments about performance concerns.
r
I'll update because I'd like to improve my contributions here. I followed precedent from the previous functions which, applying your feedback, could also be improved.
šŸ™Œ 1
I went ahead and did the buffers directly and made it all binary->binary transforms. For convenience, I kept utf8 as an allowable input to encode so customers would not have to cast.
šŸ™Œ 1