https://www.getdaft.io logo
Join Slack
Powered by
Would anyone know why `tests/integration/sql/test_...
# daft-dev
k
Would anyone know why 
tests/integration/sql/test_sql.py::test_sql_read_with_binary_filter_pushdowns
would be OOMing with Ray on Python 3.9? The only changes I made that I believe are relevant to this test is using 3.9 instead of 3.8. Logs here: https://github.com/Eventual-Inc/Daft/actions/runs/12383372435/job/34566179049?pr=3592
c
Not sure, rerunning the test to double check
j
Hmm it’s using all 15G? That’s a lot
c
Screenshot 2024-12-17 at 4.13.45 PM.png
its only the trino ones that are failing
j
Is it failing with a memory issue too?
c
they are all OOM errors
the same issue is hapening on the latest merge tho: https://github.com/Eventual-Inc/Daft/actions/runs/12382439325/job/34563515869 which is still 3.8
k
Yeah not a 3.9 issue
I see non-trino failures too nvm those were warnings and not failures
trino's docker image updated recently, that is probably why the behavior changed: https://hub.docker.com/r/trinodb/trino/tags unsure what the new version does differently that causes OOM though
j
Ah… got it. Yeah we can maybe pin our trino docker container for now
The Lance errors on the other hand look concerning too @Kevin Wang — are we able to upgrade lance?
k
Lance seems to fail on 3.9, not sure why
@Colin Ho could you take on the task of pinning our docker image versions/investigating the new trino version issues?
c
yeop
🙏 1
j
If lance continues to fail we can also skip the test for 3.9
k
That's what Im doing for now but I'll take a look at the issue for a bit before we go with that
c
pr to pin the docker images: https://github.com/Eventual-Inc/Daft/pull/3594 @Kevin Wang Also the fix for the sql intersect and distinct test to assert ordering: https://github.com/Eventual-Inc/Daft/pull/3596 (its breaking a lot of prs so might as well get it in first)
the docker pr is failing cuz of ordering sql tests ... and the ordering sql tests PR is failing cuz of the trino OOMs ... 🤡
j
Sooo we couldn’t do the deprecation? Haha
s
I believe lance is breaking because lance wasn't pinned to an exact version here https://github.com/Eventual-Inc/Daft/blob/main/requirements-dev.txt#L49 This means that when we upgraded python, the version of lance was also upgraded. For some reason, we cant pickle the newer version of lance
k
Yeah the newer versions of lance error specifically on the ray tests:
Copy code
FAILED tests/io/lancedb/test_lancedb_writes.py::test_lancedb_minio - RuntimeError: Failed to serialize: ValueError: LanceError(IO): Generic N/A error: Encountered internal error. Please file a bug report at <https://github.com/lancedb/lance/issues>. Failed to get AWS credentials: an error occurred while loading credentials, /home/runner/work/lance/lance/rust/lance-io/src/object_store.rs:180:31, /home/runner/work/lance/lance/rust/lance/src/dataset.rs:343:35
Any ideas why?
It's the only thing blocking the upgrade 😔
I think actually, our lance writes have always been broken on Ray and we just never tested it since the integration tests have always been on python 3.8 which skips this test. I will just disable the test for now since I don't think the version bump will actually break any existing behaviors.
4 Views
Open in Slack
PreviousNext
Powered by