https://www.getdaft.io logo
Join Slack
Powered by
<https://github.com/Eventual-Inc/Daft/blob/8e16c00...
# daft-dev
a
c
Copy code
let py_micropartition = py
                .import_bound(pyo3::intern!(py, "daft.table"))?
                .getattr(pyo3::intern!(py, "MicroPartition"))?
                .getattr(pyo3::intern!(py, "_from_pymicropartition"))?
                .call1((PyMicroPartition::from(micro_partition),))?;
Copy code
let py_plan_builder = py
            .import_bound(pyo3::intern!(py, "daft.dataframe.dataframe"))?
            .getattr(pyo3::intern!(py, "to_logical_plan_builder"))?
            .call1((py_micropartition,))?;
py_micropartition should be a pyobject when you call 
to_logical_plan_builder
a
(forgot to push code)
getting > Cache key not found
c
Copy code
let mut result_stream = native_executor
                    .run(HashMap::new(), cfg.into(), None)?
                    .into_stream();
it's cuz the hashmap is empty
a
ohh yea
c
populate it with the cachekey -> micropartition
a
it appears to be using the cachekey 
e9a0c5d157d343e191b32b8e473d32f7
how would I get which cache key it is using? grab the logical plan https://github.com/Eventual-Inc/Daft/blob/7922d2d810ff92b00008d877aa9a6553bc0dedab/src/daft-logical-plan/src/logical_plan.rs#L14-L34 and find the first source with https://github.com/Eventual-Inc/Daft/blob/7e89850b8b276a32119f2d59ccbeeb39dc009557/src/daft-logical-plan/src/source_info/mod.rs#L19-L40 ? or is there a more idiomatic way
c
maybe return a hashmap from 
pub fn local_relation(plan: spark_connect::LocalRelation)
?
a
@Colin Ho what I am asking is how do I get the key though... the key is being set outside of 
daft-connect
c
actually yeah, you can just match on the logicalplan and extract out the cache key
4 Views
Open in Slack
PreviousNext
Powered by