def to_partition_tasks(self, psets: dict[str, list[PartitionT]]) -> physical_plan.MaterializedPhysicalPlan:
    return physical_plan.materialize(self._scheduler.to_partition_tasks(psets))

could you please walk through this code snippet? i stuck on this logic.... help me @Colin Ho @jay

Yup, so there are two types of generators that will be produced from the physical plan: 1.

InProgressPhysicalPlan

2.

MaterializedPhysicalPlan

InProgressPhysicalPlan

is a generator that can emit: • `None`: indicating that the plan is waiting on more work to be done before it can proceed • `PartitionTaskBuilder`: indicating that this is a pipeline of instructions that can be further appended to (allowing us to do fusing of operations such as Project -> Filter -> …) • `PartitionTask`: indicating that this is a task that has been “built”, and that the runner should run this task.

MaterializedPhysicalPlan

is very similar, but only emits: •

None

•

PartitionTask

•

MaterializedResult

: indicates that this is a final result that is completed, and the runner should store this in the final set of partitions Essentially the

materialize(plan: InProgressPhysicalPlan) -> MaterializedPhysicalPlan

function will just finalize any

PartitionTaskBuilders

that were emitted, and emit those as

PartitionTasks

. This way the runner can just run any

PartitionTasks

that are emitted from this plan.

it seems that

materialize

can also generate

MaterializedResult[PartitionT]

?

From the runner’s point of view, it has this black-box

MaterializedPhysicalPlan

that it can now call

next()

on. It will receive either a •

None

(this is the plan’s way of saying — “please do more work so I can proceed”) •

PartitionTask

(this is the plan’s way of saying — “please run this task”) • Or a

StopIteration

(this is the plan’s way o saying — no more work left) •

MaterializedResult

: indicates that this is a final result that is completed, and the runner should store this in the final set of partitions

I will compare the code to understand what you said, thx

The plan is waiting on previous work that is submitted, it doesn’t have to do anything At that point, the runner should have pending work in the queue that it is still working on (previous

PartitionTasks

that it received)

thx @jay i have grasp the processing of generating partition tasks using a whole day, hh

Yeah that’s the worst part of our code 😅

the code is clean, what we need is a doc