Is there a way to load multiline json files in daft ? I can load easily psv and one-line json files. I can load them in some convoluted way I think, say apply glob, then download, then maybe some udf. Is there a way to collapse them to one line and read them more easily ? Is that something that we can fix in daft, at some place just add a flag that removes newlines? Thank you for your help

Hi might you be thinking of our

daft.read_json

?

Hi Jay, I am using daft.read_json and the error I've get is daft.exceptions.DaftCoreException: DaftError::External JSON deserialization error: Syntax at character 0 ('{')

Yes, Daft reads new line-delimited JSON (one object per row). The only way to read the JSON you presented would be to load it all into memory, extract the “records” field and then explode the inner data. For the JSON you just presented, what is your expected dataframe output?

@jay, In that example, just one column idx with one row. Ideally I would like patch the daft to remove new lines if you can point me where in the code. I guess it would break JSONL but it would be fast for me :) I did get a workaround df1 = daft.from_glob_path("<s3://p/d-**.json.gz>") df1 = df1.with_column("data", df1["path"].url.download(max_connections=8)) df1 = df1.with_column("output", df1["data"].apply(lambda data: process_file(data), return_dtype=daft.DataType.python())) and in process file I read file, ungzip and parse json and convert to pandas and then collect to daft. I still have to measure speed of this. We did found that pyarrow read_json has a parsing option newlines_in_values but it is not working as desired yet. Thank you for your help!

Yup that’s a great solution!