arrow2 schema JSON str is different than spark sch...
# daft-deva
Andrew Gazelka
12/02/2024, 8:23 PMarrow2 schema JSON str is different than spark schema JSON str... think I can write a conversion from spark schema str to arrow2 schema JSON? or will i likely run into major issues where it is not 1:1
c
Cory Grinstead
12/02/2024, 9:08 PMinstead of manually converting the string, could we first deserialize it to the spark schema, then convert from spark schema to daft schema using our existing functionality.
a
Andrew Gazelka
12/02/2024, 9:09 PMspark schema to daft schema using our existing functionality.where is this?
Andrew Gazelka
12/02/2024, 9:09 PMlike the one I made?
c
Cory Grinstead
12/02/2024, 9:10 PMwouldn't it just be the inverse of this
a
Andrew Gazelka
12/02/2024, 9:11 PMok sure
Andrew Gazelka
12/02/2024, 9:11 PMI think I get what you are saying
Andrew Gazelka
12/02/2024, 9:11 PMyea that sounds good to me
c
Cory Grinstead
12/02/2024, 9:11 PMwe'll very likely need that elsewhere for other spark operations too.
6 Views