https://www.getdaft.io logo
Join Slack
Powered by
Hey guys, I have a series of json files which do ...
# daft-dev
n
Hey guys, I have a series of json files which do not follow a jsonl format. We can't expect them to be re-formatted. I've been working through getting daft to load them from remote storage (instead of pre-processing them in 
pandas
and then moving the data into 
daft
. This is a requirement due to the remote storage loading which 
daft
supports with 
download()
Once the files are loaded, I need to perform an 
explode()
on the object column, which requires that I first 
cast(daft.DataType.list(daft.DataType.python())
. Daft is not happy. Is there any other first-party way to work with this data?
The current approach I'm trying to use is 
download()
apply(..., json.loads)
, 
cast(...)
, 
explode()
j
Seems about right… explode works on list columns so if you can get your apply to return a list type you should be good to go!
n
When I tried to set the return_dtype on the 
apply
, I get: 
Daft casting from Struct ... to Python not implemented
. Is there a way to get 
daft
to auto-cast the struct?
j
What return_dtype are you using? If you intend to explode it later on, you should perhaps use a list type?
n
Copy code
def parse_json_binary(df: daft.DataFrame) -> daft.DataFrame:
    return df.with_column(
        "json_objects",
        df["json"].apply(
            json.loads,
            return_dtype=daft.DataType.list(daft.DataType.python()),
            # return_dtype=daft.DataType.python(),
        ),
    )
j
Ah I see, I’m guessing the inner data is some kind of struct and it can’t be cast to Python right now We can probably make that happen so it just becomes Python dicts. For now if you’d like to workaround it, you could probably fully specify the data type with Daft types instead of falling back on Python
👍 1
n
Yeah I'm getting Claude to do my job for me - let's see how that goes
Is there a way to retrieve the inferred struct type intentionally without running into the error? I'd like a future user to be able to print out the daft struct representation but I can't seem to get to it unless I "break" it
j
Could you elaborate a little what you mean/maybe provide an example of what you’re thinking of?
n
If there's a way to run 
df.infer_python()
, it may be possible to understand what 
daft
may internally convert a column's datatype to. Perhaps, 
df["col_name"].infer_python()
would be more efficient and precise.
6 Views
Open in Slack
PreviousNext
Powered by