Also, am I correctly assuming that: `.write_deltal...
# general
a
Also, am I correctly assuming that:
.write_deltalake()
locally, it seems it does save / write each part over time separaetly, but on S3, it will upload only at the end?
j
The behavior should be the same whether doing a write locally or on S3
a
Could you maybe elaborate, when the writing happens? How does it decide how to split the parquet files?
j
Daft writes each partition to disk/cloud, but also respects an estimated target file size (so it will try to split up partitions into multiple files if they get too large)
✅ 1
Each partition is written as a separate file though, so we can do distributed writes