👋 given the current state of the Iceberg integration, how much would work would it be use Nessie https://projectnessie.org/guides/ given its support for Iceberg REST?

I haven't played around with it yet but if it supports Iceberg REST, I believe it should work out of the box with PyIceberg, which is what we use for Iceberg integration!

yeah, I see this tutorial using pyiceberg https://www.dremio.com/blog/intro-to-pyiceberg/

Yep that would be my guess. If you try it out, do let us know how it goes!

cool will do!

Hm. Does writing with PyIceberg work?

yeah, writing with pyiceberg works, something like below, and I can read back the updated results in daft

table.append(
    pa.Table.from_pydict(
        {
            "id": np.array([7, 8, 9], dtype="int32"),
            "name": ["George", "Henry", "Isaac"],
        },
        schema=schema.as_arrow(),
    )
)

Gotcha. Is there anything more in the error message? I believe the next line should be

Details:

ah, it says

No such file or directory (os error 2)

Interesting, Daft seems to think that this is a local file, instead of on S3. Might be that we're not getting the path properly from S3. It's also a read error, wonder why it would happen on a write. Everything else other than the write works with Daft?

looking in

data

, the formatting is different for some reason? the pyiceberg file has the

00000-0-

prefix

[2025-02-07 13:19:43 PST]   914B STANDARD 00000-0-2025b60b-6a19-42b2-b326-9d10e8d17d63.parquet
[2025-02-07 13:21:19 PST]   926B STANDARD 8a69900c-5ace-4f64-843f-91d2a4d73889-0.parquet

Was it working before and then it stopped?

it stopped working after I restarted the interpreter and tried to read the table again

Ah. Was the Daft error from before or after the restart?

before the restart, I am guessing with

table = catalog.load_table("demo.test_daft_table")

,

table

is maybe pointing at an older snapshot after the daft write, which is why the pyiceberg read looked like it worked.

Oh wait so the initial Daft read and write work, but the read after the write does not?

yeah,

pyiceberg

is looking for the file locally?

>>> catalog.load_table("demo.test_daft_table").scan().to_pandas()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 1471, in to_pandas
    return self.to_arrow().to_pandas(**kwargs)
           ^^^^^^^^^^^^^^^
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 1438, in to_arrow
    ).to_table(self.plan_files())
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 1456, in to_table
    if table_result := future.result():
                       ^^^^^^^^^^^^^^^
  File "/Users/garrett.weaver/monorepo/.css/bin/python/3.12.4+20240713/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/garrett.weaver/monorepo/.css/bin/python/3.12.4+20240713/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/garrett.weaver/monorepo/.css/bin/python/3.12.4+20240713/python/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 1437, in _table_from_scan_task
    batches = list(self._record_batches_from_scan_tasks_and_deletes([task], deletes_per_file))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 1517, in _record_batches_from_scan_tasks_and_deletes
    for batch in batches:
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 1228, in _task_to_record_batches
    with fs.open_input_file(path) as fin:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_fs.pyx", line 789, in pyarrow._fs.FileSystem.open_input_file
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
  File "pyarrow/_fs.pyx", line 1559, in pyarrow._fs._cb_open_input_file
  File "/Users/garrett.weaver/Library/Caches/pypoetry/virtualenvs/marketintel-enrichment-pipeline-0EJ9G2K5-py3.12/lib/python3.12/site-packages/pyarrow/fs.py", line 419, in open_input_file
    raise FileNotFoundError(path)
FileNotFoundError: /Users/garrett.weaver/monorepo/py/ds/marketintel/marketintel_enrichment_pipeline/data-services-experimentation/nessie-test/demo/test_daft_table_d0af3d92-cd4d-4b82-8277-02cbcfcd5a53/data/a1fa17c4-21bd-4aaf-8beb-de0f6b4303ba-0.parquet

That's interesting. So the PyIceberg table changes to point to the wrong thing after the Daft write

latest for both 0.4.3 / 0.8.1

@Robert Howell would you be interested in investigating this?

oh I see it, the file written by daft is missing the

s3://

prefix in the manifest files:

{'content': 0, 'file_path': 'data-services-experimentation/nessie-test/demo/test_daft_table_4ef4d783-bb1a-434b-8b6f-a464e7ce9f6b/data/8a69900c-5ace-4f64-843f-91d2a4d73889-0.parquet', 'file_format': 'PARQUET', 'partition': {}, 'record_count': 3, 'file_size_in_bytes': 926, 'column_sizes': [{'key': 1, 'value': 93}, {'key': 2, 'value': 101}], 'value_counts': [{'key': 1, 'value': 3}, {'key': 2, 'value': 3}], 'null_value_counts': [{'key': 1, 'value': 0}, {'key': 2, 'value': 0}], 'nan_value_counts': [], 'lower_bounds': [{'key': 1, 'value': b'\x04\x00\x00\x00'}, {'key': 2, 'value': b'Dave'}], 'upper_bounds': [{'key': 1, 'value': b'\x06\x00\x00\x00'}, {'key': 2, 'value': b'Frank'}], 'key_metadata': None, 'split_offsets': [4], 'equality_ids': None, 'sort_order_id': None}

Yeah that's true. It might be an issue with Daft Iceberg writes in general? We should definitely look into this

I have been doing a lot of S3 Iceberg writes with daft and only encountered this while testing rest api, so hopefully just something around that 🤞

Ok, could either of you two please file an issue with these details so we don't lose the thread? Thank you.

@Robert Howell added an issue https://github.com/Eventual-Inc/Daft/issues/3783

Awesome, thank you!

stepping through, the resulting data files returned after collect have no prefix:

data_files[0].file_path
'data-services-experimentation/nessie-test/demo/test_daft_table_d0af3d92-cd4d-4b82-8277-02cbcfcd5a53/data/e94293a7-6c1c-4c9b-b08d-a3b370963d88-0.parquet'