https://www.getdaft.io logo
Join Slack
Powered by
Any tips for dealing with S3 throttling when using...
# general
y
Any tips for dealing with S3 throttling when using 
write_parquet
?
Copy code
/arrow/cpp/src/arrow/filesystem/s3fs.cc:753: CompletedMultipartUpload got error embedded in a 200 OK response: SlowDown ("Please reduce your request rate."), retry = 1
j
Is this a log or is this erroring out the job? Arrow already does its own retries, so perhaps we need a way to configure the retrying logic
y
it's a log
I think the bigger issue is that it's creating many small files
I am wondering if there is a way to make use of Pyarrow's:
Copy code
min_rows_per_group int, default 0
Minimum number of rows per group. When the value is greater than 0, the dataset writer will batch incoming data and only write the row groups to the disk when sufficient rows have accumulated.
Actually, Another question
I am using:
Copy code
.write_parquet(
    tmp_doc_partition_path,
    partition_cols=[partition_col, ])
but am getting way more small files than expected
does partition_cols do any sorting/batching?
j
Partition_cols does this per-Daft-partition! If you want larger file chunks, you have to do a shuffle with repartition first so that the data is colocated first before the write.
(We already do respect a target file size)
k
If there's already a version written successfully that is partitioned by the partition column and has many small files in each partition folder, what would be the best way to write another copy which reduces each partition folder into a single file or a small number of big files per partition without calling for repartition on the entire dataset across partitions?
j
Daft does attempt to coalesce reads of small files into larger files already, which means that if you perform a simple 
.read -> .write
of the fragmented files, it should do pretty well in terms of reducing fragmentation! Funny enough, in effect you’re using the filesystem/S3 here as effectively a shuffle service 😛
k
Nice haha 😆 is there a config for the definition of small? Is it the target file size?
j
The configs we default to (execution config) are 
_*parquet_target_filesize=*_512MB
Also when we read, we do a coalescing with these configs: 
scan_tasks_min_size_bytes/scan_tasks_max_size_bytes
 targeting to bring the data sizes to within 
96/384MB
k
I see! Cool!
3 Views
Open in Slack
PreviousNext
Powered by