Hi, wanted to check whether checkpointing is supported to recover from crashes/failures, while processing a ton of json objects in S3

Hi @Ravi Anne! We don't do checkpointing just yet, but did you have an example of how you'd like this to work?

Hi @jay, the primary use-case is primarily ETL. Read the jsons, transform them and append to an Iceberg table. Daft can be either run on single node or multiple nodes using Ray, to process the files. In both the cases, if there is any crash of node or cluster, how to resume processing from the point of failure ?

Iceberg appends are atomic, so it's an all-or-nothing operation If there are node failures/worker failures in between, Ray has fault tolerance which allows it to recompute specific parts of the pipeline that was lost!