So.,.. apparently the recent S3 change to support ...
# daft-dev
j
So.,.. apparently the recent S3 change to support conditional writes means that DeltaLake might not need dynamodb locks anymore if I’m understanding this correctly: cc @Kevin Wang
(From Denny on Delta Slack) Heads up all - finally Amazon S3 to supports conditional writes (i.e, S3 now supports putIfAbsent guarantees). The implication is that we will change (or no longer need) the S3LogStore. Still digging into this and will get back to y’all about the implications but very likely positive
🚀 1
k
yep saw this! once this ships in deltalake we can just remove the dynamodb parameter in write_deltalake
d
Finally....
j
Narrator: @Desmond Cheong’s pain was palpable even through the virtual medium of Slack
🤣 1
Honestly I feel like if Delta could get its act together and become a more independent project from Spark (e.g. be opinionated about the type-system, better docs, REST APIs etc) it could beat iceberg…. It is definitely much easier to use on standalone mode without a catalog
plus one 2