I am running Daft in AWS Lambda and it is working well but limitation is 10GB of memory and that due to small number of file handlers and lack of shared memory in AWS lambda I cannot switch to ray. processing. There is an option in future that we connect to some bigger Ray compute but in the meanwhile is there a way to limit memory in Daft or to keep memory low with some slowdown? . I guess we could reduce number of CPUs (I forgot how though)?

hey @A Ivanovic! just for some context, what kind of workload are you running? in the meantime, some general solutions you could try: • Using

iter

or

iter_partitions

and specify the

results_buffer_size

to be a value less than the number of CPUs. https://www.getdaft.io/projects/docs/en/stable/api_docs/doc_gen/dataframe_methods/daft.DataFrame.iter_partitions.html#daft.DataFrame.iter_partitions • specify resource requests on ur UDF (if you are using one) https://www.getdaft.io/projects/docs/en/stable/user_guide/udf.html#resource-requests • Split up your dataframe into smaller partitions via

into_partitions

https://www.getdaft.io/projects/docs/en/stable/api_docs/doc_gen/dataframe_methods/daft.DataFrame.into_partitions.html#daft-dataframe-into-partitions