Error Type TASK EXECUTION EXCEPTION User exception 163 in ev Distributed Data Community #general

Error Type: TASK_EXECUTION_EXCEPTION User excepti...

Kyle

09/29/2024, 11:58 AM

Error Type: TASK_EXECUTION_EXCEPTION User exception: 163, in eval_expression_list return MicroPartition._from_pymicropartition(self._micropartition.eval_expression_list(pyexprs)) File "/tmp/ray/session_2024-09-29_14-48-32_270438_4696/runtime_resources/pip/a3c6247ce98cce590cd464321f9935b4df9dddd2/virtualenv/lib/python3.9/site-packages/daft/udf.py", line 120, in run_udf raise RuntimeError( RuntimeError: User-defined function

<function model_predict_udf at 0x7f80199e9670>

failed when executing on inputs with lengths: (1, 300000) Is this an OOM?

Kyle

09/29/2024, 12:30 PM

Copy code

File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/worker.py", line 2639, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/worker.py", line 864, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DaftCoreException): [36mray::ScanWithTask-Project-WriteFile [Stage:1]()
  File "/tmp/ray/session_2024-09-29_14-48-32_270438_4696/runtime_resources/pip/6ed82d08d6f74619c74a84ad767ecd74043abeeb/virtualenv/lib/python3.9/site-packages/daft/runners/ray_runner.py", line 494, in single_partition_pipeline
    return build_partitions(instruction_stack, partial_metadatas, *inputs)
  File "/tmp/ray/session_2024-09-29_14-48-32_270438_4696/runtime_resources/pip/6ed82d08d6f74619c74a84ad767ecd74043abeeb/virtualenv/lib/python3.9/site-packages/daft/runners/ray_runner.py", line 472, in build_partitions
    partitions = instruction.run(partitions)
  File "/tmp/ray/session_2024-09-29_14-48-32_270438_4696/runtime_resources/pip/6ed82d08d6f74619c74a84ad767ecd74043abeeb/virtualenv/lib/python3.9/site-packages/daft/execution/execution_step.py", line 522, in run
    return self._project(inputs)
  File "/tmp/ray/session_2024-09-29_14-48-32_270438_4696/runtime_resources/pip/6ed82d08d6f74619c74a84ad767ecd74043abeeb/virtualenv/lib/python3.9/site-packages/daft/execution/execution_step.py", line 526, in _project
    return [input.eval_expression_list(self.projection)]
  File "/tmp/ray/session_2024-09-29_14-48-32_270438_4696/runtime_resources/pip/6ed82d08d6f74619c74a84ad767ecd74043abeeb/virtualenv/lib/python3.9/site-packages/daft/table/micropartition.py", line 172, in eval_expression_list
    return MicroPartition._from_pymicropartition(self._micropartition.eval_expression_list(pyexprs))
daft.exceptions.DaftCoreException: DaftError::External Sender of OneShot Channel Dropped before sending data over: channel closed

jay

09/29/2024, 4:48 PM

Hmm do you have full trace? That error is usually indicative of a user code issue in the UDF

Kyle

09/29/2024, 11:32 PM

Oh I see! Hmm it works well for most of my data but for a particular dataset it seems to fail

Kyle

09/30/2024, 1:13 AM

Oh I think I have found the problem - there are some Nones in the column which led to this error

Copy code

AttributeError: 'NoneType' object has no attribute 'find'

but the error doesn't seem to show up in the main ray dashboard logs

Kyle

09/30/2024, 1:13 AM

It should be from running .str.replace

jay

09/30/2024, 1:25 AM

Hmm! Interesting. I tried to make it such that any user-facing error should be part of the trace you get when you see the

User defined function failed…

Do you not see that if you scroll up on that message? It could be that Ray breaks the error lineage for some reason hmm.

jay

09/30/2024, 1:25 AM

I’ll try it later with Ray as well to see if that is the case.

3 Views

Open in Slack

Previous Next