How would it be possible to load in data from a hd...
# generalk
Kyle
08/26/2024, 10:26 AMHow would it be possible to load in data from a hdfs address? It works in pyarrow's parquetdataset as just the hdfs filepath but i wasn't able to use the same path in daft
Kyle
08/26/2024, 10:29 AMIt's kind of like a mounted path but daft rejects it because it starts with hdfs://.. pandas also allows the path to be read
c
Cory Grinstead
08/26/2024, 2:25 PMWe don't yet support hdfs, if you want to open up an issue on github, we can work on getting it added in an upcoming release!
j
jay
08/26/2024, 5:17 PMhdfs is very interesting… I’m not sure what the best way of supporting it would be tbh 😬 and I don’t think we can squeeze out a ton of performance from it since I believe we have to go through the JVM.
Something like: https://github.com/Xuanwo/hdrs
Feel free to open up an issue @Kyle!
k
Kyle
08/27/2024, 1:15 AMOkay thanks!!!
6 Views