If I'm trying to time a certain operation like write_parquet is there a way to let it generate a report of the time taken in each stage after the job is completed instead of relying on the progress bar? (In the ray runner)

Hey sorry we missed this, but we're working on better visualizations (basically our own dashboard)

@jay I saw that as well! But i have a logstore which automatically captures stdout that i refer to for comparisons across time. Any way I can pick/produce logs via the new dashboard mechanism to stdout?

For storing historical runs?

Just selected stats of historical runs (like all runs ideally)

🤔 it would require some sort of persistent storage (e.g. a database)

Oh i already automatically save it to a database so all i would need is for the stats to go into stdout (just like the current status printouts except that the current version is very verbose and doesn't give time)

gotcha gotcha

Oh nice! Haha. Does it also show the stats in the Ray runners?