If I'm trying to time a certain operation like wri...
# general
k
If I'm trying to time a certain operation like write_parquet is there a way to let it generate a report of the time taken in each stage after the job is completed instead of relying on the progress bar? (In the ray runner)
j
Hey sorry we missed this, but we're working on better visualizations (basically our own dashboard)
Stay tuned!
k
@jay I saw that as well! But i have a logstore which automatically captures stdout that i refer to for comparisons across time. Any way I can pick/produce logs via the new dashboard mechanism to stdout?
j
For storing historical runs?
k
Just selected stats of historical runs (like all runs ideally)
j
🤔 it would require some sort of persistent storage (e.g. a database)
Honestly best suited most likely for a platform offering we can deploy
What kinds of stats would you like?
k
Oh i already automatically save it to a database so all i would need is for the stats to go into stdout (just like the current status printouts except that the current version is very verbose and doesn't give time)
I would want to know the time taken for each big step in the plan, especially the UDF and the shuffles
j
gotcha gotcha
😛
image.png
sneak peek at some work coming up
k
Oh nice! Haha. Does it also show the stats in the Ray runners?