So.,.. apparently the recent S3 change to support ...
# daft-devj
jay
08/22/2024, 5:21 PMSo.,.. apparently the recent S3 change to support conditional writes means that DeltaLake might not need dynamodb locks anymore if I’m understanding this correctly:
cc @Kevin Wang
(From Denny on Delta Slack) Heads up all - finally Amazon S3 to supports conditional writes (i.e, S3 now supports putIfAbsent guarantees). The implication is that we will change (or no longer need) the S3LogStore. Still digging into this and will get back to y’all about the implications but very likely positive
🚀 1
k
Kevin Wang
08/22/2024, 5:22 PMyep saw this! once this ships in deltalake we can just remove the dynamodb parameter in write_deltalake
d
Desmond Cheong
08/22/2024, 5:23 PMFinally....
j
jay
08/22/2024, 5:23 PMNarrator: @Desmond Cheong’s pain was palpable even through the virtual medium of Slack
🤣 1
jay
08/22/2024, 5:25 PMHonestly I feel like if Delta could get its act together and become a more independent project from Spark (e.g. be opinionated about the type-system, better docs, REST APIs etc) it could beat iceberg….
It is definitely much easier to use on standalone mode without a catalog
plus one 2