I've been getting this error very very occasionally, not too sure how to recreate it. Sometimes at the end of the batch of write tasks the write_parquet fails.. (I've changed some of the folder paths in this error message)

df.write_parquet(output_path, write_mode="overwrite")
 File "/tmp/ray/xxx/virtualenv/lib/python3.10/site-packages/daft/api_annotations.py", line 26, in _wrap
 return timed_method(*args, **kwargs)
 File "/tmp/ray/xxx/virtualenv/lib/python3.10/site-packages/daft/analytics.py", line 198, in tracked_method
 return method(*args, **kwargs)
 File "/tmp/ray/xxx/virtualenv/lib/python3.10/site-packages/daft/dataframe/dataframe.py", line 563, in write_parquet
 overwrite_files(write_df, root_dir, io_config)
 File "/tmp/ray/xxx/virtualenv/lib/python3.10/site-packages/daft/filesystem.py", line 370, in overwrite_files
 paths = [info.path for info in fs.get_file_info(file_selector) if info.type == pafs.FileType.File]
 File "pyarrow/_fs.pyx", line 582, in pyarrow._fs.FileSystem.get_file_info
 File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
 File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Cannot list directory '/tmp/ray/zzz'

This is the code for the file overwriter. I think there's 2 possibilities: 1. There was no data to write, so the output directory was not created. Therefore when the overwriter tries to look into the directory, it fails with a

FileNotFoundError

. This is a bug that I can fix. 2. Some other process deleted the output directory in the middle of

write_parquet

. Not very likely probably. Do you know if it's possible that your query sometimes produces empty outputs?

Ah yes you're right it's due to it having zero rows. I also tried with append mode and it gives a similar error but with no found parquet despite having a path and name for the parquet.

Yeap ok I see that bug as well, have a fix here: https://github.com/Eventual-Inc/Daft/pull/3278