https://www.getdaft.io logo
Join Slack
Powered by
# daft-dev
  • g

    Garrett Weaver

    04/27/2026, 8:37 PM
    đź‘‹ looking to add Jaccard similarity and I was curious if it would be preferred to "bake in" generating work/n-grams with some configurable params (similar to https://www.jpt.sh/projects/jellyfish/functions/#jaccard-similarity) or expect users to handle themselves and pass in two columns with type
    list(str)
    ? I somewhat prefer the explicit handling by users over hiding some details, but curious on thoughts.
    c
    • 2
    • 1
  • m

    Mehul Batra

    05/06/2026, 8:33 AM
    Hi Team, Mehul from DigitalOcean's . We're in the middle of evaluating Daft & Raydata for a multimodal ingestion & transformation on Kubernetes (Heterogeneous GPU/CPU, Batch Inference, sinking to Lance + Iceberg with Gravitino catalog registration). Daft is technically the better fit on most dimensions that matter to us, RayData seems to be good too but need alot of wrapper on low level API, but surely winning in corporate marketing with anyscale summit. Before we commit, I'm trying to get a clearer picture on a few things, and the docs only go so far: 1. Real-world production deployments on self-hosted K8s The public reference set (Amazon BDT, CloudKitchens, Together AI, Essential AI) is well-documented but mostly 12–18 months old. Are there newer teams running Daft on self-hosted Kubernetes not Daft Cloud, not Anyscale at meaningful scale? On that note: we're aware that ByteDance's LAS (Lake for AI Service) team has been one of the most active contributors on the Daft↔️Lance integration surface distributed FTS, compaction, schema evolution, limit pushdown, and most recently
    GravitinoGvfs
    local filesystem writes in v0.7.10 (
    @qingfeng-occ
    ). Their pattern Lance as storage, Ray as runtime, Daft hardening the client layer is close to our target architecture. If anyone from that team or others running a similar stack is in this Slack, I'd love to connect. 2. Flotilla vs set_runner_ray() — current recommendation for DOKS We're open-source-only. Is the Flotilla native K8s path production-ready today for a team that can't fall back to Daft Cloud? Or is the practical answer still
    set_runner_ray()
    + KubeRay for anything serious? Any known stability cliffs under sustained load? 3. Checkpointing roadmap for Lance/Iceberg sinks
    daft-checkpoint
    scaffolding landed in v0.7.6 — great signal. Our write path is Lance + Iceberg + OpenSearch, and the current Parquet-only limitation means we'd still need external idempotency for v1 regardless. Is Lance/Iceberg write support on the near-term checkpoint roadmap, or is that explicitly out of scope for now? We'd plan to contribute back when things fall in place. Happy to share our UC matrix and PoC findings once we're further along.
    đź‘‹ 1
    e
    r
    c
    • 4
    • 7
  • d

    Desmond Cheong

    05/14/2026, 4:32 PM
    @Robert Howell could you help take a look at this PR? https://github.com/Eventual-Inc/Daft/pull/6760 It adds support for our native extensions on the distributed runner
    đź‘‹ 6
    đź‘€ 2
    ❤️ 1
    r
    • 2
    • 1
  • u

    北南

    06/04/2026, 5:01 PM
    https://github.com/daft-engine/daft-lance/pull/18 @Robert Howell could you help take a look at this this PR? thanks!
    âś… 1
  • u

    北南

    06/05/2026, 4:59 AM
    https://github.com/daft-engine/daft-lance/pull/21 @Robert Howell Support Distributed Segmented BTREE Index Building
    âś… 1
  • u

    北南

    06/05/2026, 9:54 PM
    https://github.com/daft-engine/daft-lance/pull/23 @Robert Howell Integrates Lance's log-structured ingestion framework (MemWAL) with
    daft-lance
    to enable high-throughput parallel writes. thanks a lot!!
    âś… 1
    🙌 1
    r
    j
    • 3
    • 8
  • a

    Abhijeet Patil

    06/09/2026, 5:23 PM
    Hi Team, is there a drop-in replacement of spark streaming in daft?
    âž• 1
    u
    j
    • 3
    • 4
  • u

    北南

    06/13/2026, 5:10 AM
    Need help for code review: https://github.com/daft-engine/daft-lance/pull/24 https://github.com/daft-engine/daft-lance/pull/25 https://github.com/daft-engine/daft-lance/pull/26
    j
    • 2
    • 5
  • a

    Abhijeet Patil

    07/02/2026, 6:12 PM
    Hi Team, can someone please help with code review: github.com/Eventual-Inc/Daft/pull/7187
    s
    • 2
    • 5
  • a

    Abhijeet Patil

    07/10/2026, 5:47 PM
    Hi Team, do we have functions in daft to detect PII?
    s
    • 2
    • 3
  • a

    Abhijeet Patil

    07/13/2026, 10:13 AM
    Hello Team, we are writing an engineering blog about the gains we saw after migrating from spark to daft at MakeMyTrip. We would like to submit it to daft's website's blog section. (eventual.ai/blog) Can someone please tell us steps to do that and guidelines for the same?
    • 1
    • 1
  • s

    Satyendra Kumar

    07/13/2026, 10:21 AM
    Hi Team, please take a look at these: github.com/Eventual-Inc/Daft/discussions/7259 github.com/Eventual-Inc/Daft/issues/7260
  • a

    Abhijeet Patil

    07/17/2026, 3:33 PM
    Folks with experience running Daft on GCP, which family of compute engine offers best price to performance ratio (intel, amd, axion, etc)? Asking for ETL pipelines.
  • d

    dsohong

    09/02/2026, 3:34 AM
    Hi Team, I need help with a code review for Daft's Iceberg read/write features. github.com/Eventual-Inc/Daft/pull/7419 github.com/Eventual-Inc/Daft/pull/7421 github.com/Eventual-Inc/Daft/pull/7429 thx
    👍 2
  • p

    peter-tang

    09/08/2026, 2:18 PM
    Hi Daft team — my PR #7439 (push  Limit  down to the row-preserving side of Left/Right joins, addressing #2275) is green and mergeable; could someone take a look when you have a chance? https://github.com/Eventual-Inc/Daft/pull/7439:arrow_upper_right:
    âś… 1
    c
    • 2
    • 1
  • d

    dsohong

    09/09/2026, 3:43 AM
    Hi all — no urgency at all, but if anyone has a spare cycle I'd really appreciate a look at two small Iceberg fixes: - #7421 — count_rows() on an Iceberg table silently ignored a filter on an identity-partitioned column and returned the whole table's row count. - #7429 — after a partition spec evolved to add a field, a filter on that field returned rows that don't actually match it. github.com/Eventual-Inc/Daft/pull/7421 github.com/Eventual-Inc/Daft/pull/7429 Happy to walk through the reasoning on either one, or to split them further if that makes review easier.
  • p

    peter-tang

    09/09/2026, 5:41 AM
    Hi team — could I get a re-review on PR #7398 (fix: keep pushed filter columns in projection pushdown)? https://github.com/Eventual-Inc/Daft/pull/7398↗ It fixes #6757:  read_parquet(...).where(col("b") < 1).select("a")  used to fail because projection pushdown stopped reading the filtered column. @rchowell both your earlier asks are addressed — the greptile JSON concern is fully reverted, and the count-pushdown CI failure is fixed. All checks green, mergeable with no conflicts, well-tested (unit + fixed-point race + JSON/Parquet e2e). Been quiet since Aug 24 — would appreciate a look when someone has a moment. Thanks!
  • p

    peter-tang

    09/09/2026, 7:27 AM
    Hi team — I've opened #7489, a small two-line fix for the  integration-test-catalogs  failures that have been affecting PRs since ~09-06 (tracked in #7476). The cause is infrastructure rather than anyone's code: the Iceberg test image builds  FROM python:3.10-bullseye , whose apt suite has gone stale (openjdk-11 packages 404), so  docker compose up  aborts and the catalog tests fail. The fix moves it to  python:3.10-bookworm  and CI is fully green, including that job — would anyone have a moment to review and merge it? Thank you!
  • f

    fanng

    09/10/2026, 8:25 AM
    Hi team, We’re building a multimodal processing pipeline with Daft + Lance. The following PRs add support for Lance table update and insert_overwrite semantics in Daft. Appreciate your review. github.com/daft-engine/daft-lance/pull/58 github.com/daft-engine/daft-lance/pull/62
    âś… 1
    🙌 1
  • f

    fanng

    09/10/2026, 8:32 AM
    Daft-lance was maintaining a second implementation of Lance’s merge-column write path. This exposed too many Lance internals, making it difficult to maintain and extend, and introduced known bugs such as #59 and #20. github.com/daft-engine/daft-lance/pull/60 proposes using Lance’s existing merge_columns API to implement a fast path for adding columns, Please help to review, thx!
    âś… 1
  • p

    peter-tang

    09/12/2026, 1:48 PM
    Hi Team — I added the e2e test you asked for on #7441 (prefix  starts_with  pruning via min/max stats). Would you mind taking another look when you have a moment? Thanks! https://github.com/Eventual-Inc/Daft/pull/7441:arrow_upper_right:
  • f

    fanng

    09/14/2026, 2:23 AM
    Hi! Is there a plan for a new daft-lance release? 0.4.0 went out on June 5, and main has 11 unreleased commits since — Lance Namespace support (#35), atomic insert_overwrite (#62), segmented bitmap indexes (#31), plus several merge fast-path correctness fixes (#44, #45, #46, #56, #60). Since daft's lance extra tracks the latest daft-lance, none of this reaches daft users until a release is cut. If maintainers are short on time, I'm happy to help — I can open the release PR (version bump + changelog) so all that's left is publishing the GitHub Release. Just ping me for whatever else is needed.
    r
    • 2
    • 10
  • f

    fanng

    09/18/2026, 1:09 AM
    Hi team, could you help review update lance column PR? Typically, it's used to re-running a model and writing fresh embeddings or scores back to the same column, backfilling or correcting a column at scale, it's import in our Daft&Lance pipeline, thanks! github.com/daft-engine/daft-lance/pull/58
    • 1
    • 1
  • j

    Jianyao Ma

    09/22/2026, 1:43 PM
    Hi team, I reproduced #7458 locally and have a prototype fix. It keeps one writer per input while allowing bounded concurrency across inputs. Details are in my comment. Could someone familiar with BlockingSink take a look and let me know if this approach makes sense? Thanks!
  • f

    fanng

    09/23/2026, 7:29 AM
    Hi team,
    daft-lts
    release is not updated for the limit of pypi storage limit, could any one help to handle this? thanks. github.com/Eventual-Inc/Daft/issues/7546
    s
    • 2
    • 5
  • j

    Jianyao Ma

    09/24/2026, 6:14 AM
    Quick follow-up on #7458 — the key invariant in the prototype is that each InputId still has exactly one writer, while concurrency is bounded across different inputs rather than within one input. If that matches the intended semantics for BlockingSink, I can clean this up and open a PR.
    b
    j
    • 3
    • 3
  • p

    portofino4526

    10/06/2026, 9:03 AM
    Hi team, could someone please review PR #7576? It adds
    daft.read_orc()
    for local and remote ORC files, with lazy batched reads, schema alignment, and support for paths, directories, and globs. This lets users process Hive/Spark ORC data directly in Daft without a separate read-and-materialize or format-conversion step. Feedback on the API and edge cases would be very welcome. Thanks!
    🙌 1
    r
    • 2
    • 1
  • r

    Robert Howell

    10/08/2026, 6:55 PM
    @fanng thank you for the PR's, I have cut a 0.6 release of daft-lance.
    f
    • 2
    • 1
  • j

    junling du

    10/15/2026, 9:13 AM
    hi all, github.com/Eventual-Inc/Daft/pull/7369 please help to review pr to support dynamic udf actor pool autoscaling
  • p

    portofino4526

    11/04/2026, 4:18 AM
    Hi team, could someone please review PR #7597? It adds explicit schema and schema hints to
    daft.read_orc()
    through the
    schema
    and
    infer_schema
    options, following the behavior of
    read_parquet()
    . This lets users define a known output schema, override inferred types, and include expected fields absent from the first file. Feedback on the API and edge cases would be very welcome. Thanks!
    đź‘€ 1