As a followup, tried running a 200m 75column table...
# generalj
jordan fox
10/09/2024, 1:41 AMAs a followup, tried running a 200m 75column table write_deltalake() to local with an nvme gen4 ssd, lots of space, went really fast for the first 150 or so iterations and has since stalled at 98%. Have 24 cores and 128gb memory, isn't a resource issue. More than happy to find a log of what happened / why it didn't complete if anyone knows how.
jordan fox
10/09/2024, 1:43 AMScanWithTask-WriteDeltaLake [Stage3] 98%|▉| 155/158 [2044<0223, 47.75s/it]
Hasn't moved from here since it stopped. Don't have the deltalog folder in there yet.
j
jay
10/09/2024, 3:09 AMHmm
jay
10/09/2024, 3:10 AMYes, we perform a write of all the Parquet files first before performing the delta commit, which is why the delta log folder hasn’t appeared
jay
10/09/2024, 3:10 AMWhat data source are you reading from?
j
jordan fox
10/09/2024, 3:53 AMThis is from unity catalog, Azure databricks.
jordan fox
10/09/2024, 3:53 AMConfirmed it works just fine on smaller tables, they write nearly instantaneously.
j
jay
10/09/2024, 3:53 AMI see, my guess is that the read is taking a really long time (e.g. because of really long retries/timeouts from Azure storage)
jay
10/09/2024, 3:54 AMWe’re very aggressive with our reads 😛 sometimes cloud storage gets very unhappy with us
And we’ve mostly tested/built against AWS S3 which seems to be the most resilient cloud backend out of all the clouds so far
j
jordan fox
10/09/2024, 3:57 AMYeah, I had to use service principal settings in an AzureConfig passed to IOConfig(azure=azure_config) when I went to run df.write_deltalake(table=table,mode='overwrite',io_config=custom_io_config)
jordan fox
10/09/2024, 3:57 AMSo potentially even token timeout?
jordan fox
10/09/2024, 3:59 AMGotta start testing against the least resilient storage :P
😭 1
jordan fox
10/09/2024, 4:08 AMTried again, gets to 80+% in 2 mins then slows right on down. Am at 92% after 6. Will see if I can ray submit.
j
jay
10/09/2024, 4:10 AMYeah pretty sure this is an I/O issue and we’re getting throttled to oblivion 😛
j
jordan fox
10/09/2024, 4:11 AMWell, if there's any log/data I can provide just let me know. Can keep testing this table on future versions.
🙌 1
j
jay
10/09/2024, 4:11 AMWill let you know! We’re currently in the process of stabilizing our GCS reads… will move to Azure soon 🙂
jay
10/09/2024, 4:12 AMappreciate you giving us a whirl!
jay
10/09/2024, 4:12 AMLet us know what you think of the tool
j
jordan fox
10/09/2024, 4:14 AMPlan is dagster daft for data engineering w dbx unity catalog to serve our enterprise stakeholders. Having one tool that can read from mssql, oracle, postgres, easily pick up api responses, and write em all into one catalog all at low hardware cost is great.
j
jay
10/09/2024, 4:16 AMThat sounds… really cool actually 😮
jay
10/09/2024, 4:16 AMAny chance we could grab you on a call maybe sometime next week?
jay
10/09/2024, 4:17 AMWould love to talk through your current use-cases and maybe build out some good APIs for you.
Also to battle-test these Azure integrations
j
jordan fox
10/09/2024, 2:59 PMSure thing. FYI I thought the table stalled again at 98% so I just left it running overnight. It did complete successfully but I don't know when. Gonna check the modified dates of the files.
jordan fox
10/09/2024, 3:01 PMStarted 10:02, looks like it's last modified file is 10:30. Previous run just never completed so that's a win. Can't guarantee it actually finished in 28mins though.
jordan fox
10/09/2024, 3:04 PMRunning a test using ray runner so I have concrete info.
jordan fox
10/09/2024, 4:24 PMI have an example of a stall:
🔥 1
j
jay
10/09/2024, 4:25 PMCool, yeah that’s almost definitely a bad retry policy then. It’s probably doing some really long exponential retry somewhere in the azure SDK’s defaults.
Thanks for digging in!! We’ll take a closer look today
4 Views