for local execution impl a range means having a st...
# daft-deva
Andrew Gazelka
11/15/2024, 12:58 AMfor local execution impl a range means having a stream
https://github.com/Eventual-Inc/Daft/blob/6e28b3f39b38ddaa1d2696a22d733eb5c5567f44/src/daft-local-execution/src/sources/source.rs#L16
range def in spark is
https://github.com/Eventual-Inc/Daft/blob/f566125f0fabc48c342888e58f0676d563adaf84/src/generated/spark-connect/src/lib.rs#L1975-L1991
so think I would just be returning a
num_partitionsnum_partitionc
Colin Ho
11/15/2024, 1:06 AMFor local execution you don’t need to care about the partitions, that’ll only be for distributed.
a
Andrew Gazelka
11/15/2024, 1:13 AMah ok
Andrew Gazelka
11/15/2024, 1:13 AMim thinking MVP will just be with local executor
Andrew Gazelka
11/15/2024, 1:14 AMthink it will work if I make
Andrew Gazelka
11/15/2024, 1:14 AMand then add the distributed part later?
Andrew Gazelka
11/15/2024, 1:14 AMor do you see me running into roadblocks
Andrew Gazelka
11/15/2024, 1:14 AMand it making more sense to do it all at once
c
Colin Ho
11/15/2024, 1:20 AMI think the local vs distributed implementations of range will be different, so makes sense to separate
2 Views