Hello, is there a plan to support to hierarchical json reads ? I understand that currently only newline delimited json reads are supported. Thanks

Out of curiosity, what are you referring to when you say hierarchical json reads?

I am referring to nested json objects or arrays.

I believe we can do what you want to do today So either we have a record in a json file with nested objects:

{
  "a": 1,
  "b": {
    "c": 3,
    "d": [
      1.1,
      2.2,
      3.3
    ]
  }
}

that we can read

>>> daft.read_json("my.json").collect()
╭───────┬────────────────────────────────────╮                                                                                                                                                                                                       
│ a     ┆ b                                  │
│ ---   ┆ ---                                │
│ Int64 ┆ Struct[c: Int64, d: List[Float64]] │
╞═══════╪════════════════════════════════════╡
│ 1     ┆ {c: 3,                             │
│       ┆ d: [1.1, 2.2, 3.3],                │
│       ┆ }                                  │
╰───────┴────────────────────────────────────╯

(Showing first 1 of 1 rows)

or alternatively, let's say you have a string column that you want to treat as json with nested json objects, you can also use our

.json.query

expression to query the json column

>>> import daft
>>> df = daft.from_pydict({"col": ['{"a": 1, "b": {"c": 2}}', '{"a": 2, "b": {"c": 20}}', '{"a": 3, "b": {"c": 300}}']})
>>> df.with_column("res", df["col"].json.query(".b.c")).collect()
╭───────────────────────────┬──────╮                                                                                                                                                                                                                 
│ col                       ┆ res  │
│ ---                       ┆ ---  │
│ Utf8                      ┆ Utf8 │
╞═══════════════════════════╪══════╡
│ {"a": 1, "b": {"c": 2}}   ┆ 2    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤
│ {"a": 2, "b": {"c": 20}}  ┆ 20   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤
│ {"a": 3, "b": {"c": 300}} ┆ 300  │
╰───────────────────────────┴──────╯

(Showing first 3 of 3 rows)

(see https://www.getdaft.io/projects/docs/en/latest/api_docs/doc_gen/expression_methods/daft.Expression.json.query.html#daft-expression-json-query for more details)

I gather that json line format (JSONL) is supported in Daft. This is practical because reading a big json array object would mean loading the whole (possibly very big) file into memory. For performance, line-delimited would enable streaming reads - so far so good. However, I am dealing with some modeling data (JSON-LD) which is a graph. The files themselves are not too big, but can be a huge number of them. Pandas can read these files in a way that they are flattened. Daft could read them too after a bit of formatting, but it does not flatten the complex object as you also demonstrate in your example above. So far, I see one option to handle this on my part - preprocessing with another big data tool like Spark/ Dask and save the intermediate result (JSONL format) - possibly also batching to deal with small file inefficiency.

Any chance you could point me to an example of the data you're working with and what you've tried with daft? As far as I'm aware, spark doesn't expose any special json api that daft doesn't also support, but I could be wrong on this