I'm thinking of making a bot for my processing task updates - any way to periodically get some update on the status of the current daft task being run and execute a POST?

Do you run on Ray?

Yes

If you run daft using ray jobs, you can get the job ID and then poll the ray cluster for the status of that job. If you could let me know how you run daft via ray right now (e.g. command line via

ray job submit

, python sdk via

ray.job_submission.JobSubmissionClient

), I could point you towards some specific examples!

I'm currently doing that! But it's sequential and due to the lazy execution all of the tasks actually run when I call write_parquet. I would like to call for the status in the middle of write_parquet haha

Do you mean that the job submission only terminates once the job is done? For that, you could look into using the

--no-wait

flag

I mean that I want to know for example the progress for all stages of the process every 5 minutes

ReduceMergeAndSort-Aggregate-FanoutHash [Stage:2]:  79%|███████▊  | 3507/4456 [09:13<02:20,  6.75it/s]

Ah. I don't think we have any built-in way to do this, to my knowledge you'd have to parse the progress bars somehow

But I also can't parse it in realtime because daft would be taking up all the available cores I guess?

Each of these also corresponds to some set of Ray tasks in the job. Perhaps there is a way to infer the status for the stage that way too

Yes I'm thinking the current task count would be the numerator and the divisor can be from the physical plan

Better observability of each stage is something we are definitely thinking about, however I would caution against programmatically observing the state of specific stages in the pipeline, since optimizations we make in future versions may change the physical plan quite a bit

Yes that would be ideal. I was actually thinking more of a periodic heartbeat kind of thing

I guess another question I have is what are you trying to ultimately get the status of? Are you looking to see if Daft has written a specific file or something similar?

I'm trying to send notifications on the job status to a bot so that I can view it on the go

I see. That is indeed not very easy to do at this moment

Yeah I'll leave it to next time then! 😅