https://www.getdaft.io logo
Join Slack
Powered by
Hi, I am a new daft user! I'm hoping to get it wor...
# general
m
Hi, I am a new daft user! I'm hoping to get it working with my data. I am trying to set it up to use an AWS profile and read a table from iceberg but am running into an error that looks like this: 
OSError: When reading information for key 'my-key.avro' in bucket 'my-bucket': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
Is there a better way to debug this? All logs are turned on and I don't see any requests going out to S3.
Code:
Copy code
import daft
from daft.io.config import IOConfig
from daft.io.aws_config import S3Config
from pyiceberg.catalog import load_catalog


catalog = load_catalog(
    "name",
    **{
        "uri": "endpoint",
        "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
    }
)
io_config = IOConfig(s3=S3Config(region_name="us-west-2", profile_name="my-profile-name"))

# this works:
# df = daft.read_parquet("<s3://bucket/><..>.parquet", io_config=io_config)
# df.show()

table = catalog.load_table("namespace.table_name")
df = daft.read_iceberg(table, io_config=io_config)
# this fails with error
df.show()
The last logs that are printed:
Copy code
2024-11-01 21:55:07,470.470 - 1032 - 7988980736 -     INFO - __init__.py:331 - load_file_io - Loaded FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
2024-11-01 21:55:07,490.490 - 1032 - 7988980736 -     INFO - pyrunner.py:401 - run_iter - Using python executor
Traceback:
Copy code
Traceback (most recent call last):
  File "main.py", line 92, in <module>
    main()
  File "main.py", line 87, in main
    df.show()
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/api_annotations.py", line 26, in _wrap
    return timed_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/analytics.py", line 202, in tracked_method
    result = method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/dataframe/dataframe.py", line 2553, in show
    dataframe_display = self._construct_show_display(n)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/dataframe/dataframe.py", line 2510, in _construct_show_display
    for table in get_context().runner().run_iter_tables(builder, results_buffer_size=1):
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/runners/pyrunner.py", line 415, in run_iter_tables
    for result in self.run_iter(builder, results_buffer_size=results_buffer_size):
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/runners/pyrunner.py", line 403, in run_iter
    plan_scheduler = builder.to_physical_plan_scheduler(daft_execution_config)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/logical/builder.py", line 71, in to_physical_plan_scheduler
    return PhysicalPlanScheduler.from_logical_plan_builder(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/plan_scheduler/physical_plan_scheduler.py", line 34, in from_logical_plan_builder
    scheduler = _PhysicalPlanScheduler.from_logical_plan_builder(builder._builder, daft_execution_config)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/daft/iceberg/iceberg_scan.py", line 152, in to_scan_tasks
    iceberg_tasks = self._table.scan(limit=limit, snapshot_id=self._snapshot_id).plan_files()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 1960, in plan_files
    for manifest_file in snapshot.manifests(<http://self.io|self.io>)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/pyiceberg/table/snapshots.py", line 255, in manifests
    return list(read_manifest_list(file))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/pyiceberg/manifest.py", line 633, in read_manifest_list
    with AvroFile[ManifestFile](
  File ".pyenv/versions/daft/lib/python3.11/site-packages/pyiceberg/avro/file.py", line 170, in __enter__
    with self.input_file.open() as f:
         ^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/daft/lib/python3.11/site-packages/pyiceberg/io/pyarrow.py", line 270, in open
    input_file = self._filesystem.open_input_file(self._path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_fs.pyx", line 789, in pyarrow._fs.FileSystem.open_input_file
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
I am able to perform a HEAD request on the obj requested using AWS CLI with the same AWS profile name
j
Hi! Welcome to the club :) This error I believe is actually coming from PyIceberg. You should make sure to provision it with the correct credentials as well when you do catalog.load_table
m
Thanks for the hint. If I'm understanding these issues correctly, it seems like it currently does not support AWS profiles šŸ«  https://github.com/apache/iceberg-python/issues/1207
j
šŸ˜­ this is why relying on 3rd party libraries sucks sometimesā€¦
6 Views
Open in Slack
PreviousNext
Powered by