Any suggestions, clues or guide how to best optimi...
# generala
Ammar Bitar
11/29/2024, 10:02 PMAny suggestions, clues or guide how to best optimize the partitionning and/or other parameters (max connections, num_cpu, concurrency etc.) to not kill memory when;
1. downloading + decoding images from MinIO
a. Applying some chained computations on those images
2. wiriting deltalake into that same minio store?
3. (not using ray - yet)
Specs:32 cores, 64Gb RAM
Objects: ~2.5mil images (mix of RGB, Grayscale, RGBA,...) of varying resolutions (800x600 - 2000x1800)
j
jay
11/29/2024, 10:05 PMI think it’s easiest if you try the new native executor @Colin Ho can guide you here
The problem with memory is usually you need to know your partitioning upfront (total number of partitions). The new executor does streaming execution which lets you set morsel sizes instead.
a
Ammar Bitar
11/29/2024, 10:10 PMIs that using `daft.context.set_runner_native()`right?
This works with all versions >=0.3.11 right?
I've tried using it but didn't seem to fix the OOM issues - or do I need to go through
daft.set_execution_configj
jay
11/29/2024, 10:11 PMI think you might still need to reduce the sizes of your morsels using set_execution_config, especially on a 32 core machine (we’re probably running too aggressively).
👍 1
a
Ammar Bitar
11/29/2024, 10:13 PMWould appreciate if there are any guidelines or rule of thumbs to help set the best size ? 😄 Otherwise I'll go through trial and error 🙂
j
jay
11/29/2024, 10:13 PMYou can think of the total number of rows running in parallel as: num_cpus * morsel size
jay
11/29/2024, 10:14 PMSo given you have 32 CPUs, maybe try something like 256? Really depends how large your images are 😝
We’re working on more automated mechanisms to do dynamic sizing/parallelism based on memory pressure but right now it’s a little bit manual!
a
Ammar Bitar
11/29/2024, 10:16 PMAs I said, the worst case scenario (AFAIK so far) is images of size ~2000x1800x4
Ammar Bitar
11/29/2024, 10:17 PMAnd yeah looking forward to that more automated stuff ;D - quite liking daft in general so far thank you!
Ammar Bitar
11/29/2024, 10:17 PM(and still learning a lot on the side)
Ammar Bitar
11/29/2024, 10:19 PMJust to get this straight, you suggest to set the following to 256? (and the default is 131072)
• default_morsel_size – Default size of morsels used for the new local executor. Defaults to 131072 rows.
Ammar Bitar
11/29/2024, 10:22 PMAlso note, again, I'm not keeping the images in the rows! Just downloading them processing, computing and hoping that they get flushed after getting the results
c
Colin Ho
11/29/2024, 10:25 PMYes, try 256, likely the lower the better if you're doing url-downloads and decodes.
👍 1
a
Ammar Bitar
12/02/2024, 2:06 PMCan we use the morsel settings while using ray clusters? Seems to not work correctly?
Ammar Bitar
12/02/2024, 2:18 PMAlso, for single node, thanks that worked great! 😄
c
Colin Ho
12/02/2024, 4:38 PMno the morsel size only works when using the native runner, i..e
set_runner_native()a
Ammar Bitar
12/10/2024, 2:23 PMWhat would be the best approach when using ray then?
Currently, just partitionning, but it doesn't seem efficient / optimal at all?
c
Colin Ho
12/10/2024, 7:41 PMFor ray, using
into_partitionsa
Ammar Bitar
12/11/2024, 7:52 AMYes, I compute chunk size in worst case scenario based, but of course that's not optimal most of the time
5 Views