and i notice that the format of

MicroPartition

is different from arrow. I want to know the consideration why we not use arrow as local format directly. Modin&ray-data are using arrow as the local format. @Colin Ho @jay thx

Yes! We do this because we have our own types and internal representations of data For example, we allow for columns of Python objects. Also, having our own implementations of nested types makes it much easier to work with.

understand. but we cannot use datafusion or other existing library for operators.

Umbra strings: https://cedardb.com/blog/german_strings/

while we store data in plasma store, can

Table

support in-place computation?

We don’t do in-place computation, it’s really difficult to reason about when doing distributed data processing. Instead, we are moving to a streaming-based model for memory stability