Just curious, why is it that the total number of t...
# generalk
Kyle
09/28/2024, 12:33 AMJust curious, why is it that the total number of tasks in the logs are always increasing and not identical to the physical plan right from the start?
j
jay
09/29/2024, 1:36 AMHmm @Sammy Sidhu do you know?
jay
09/29/2024, 6:51 PMI thought about this a little more. It has to do with the way that we run our tasks in “batches” at the moment
We’re looking at building much better tooling around workload observability. More news on that soon.
Hoping to have something similar to the BigQuery UI or Spark UI. Lmk if you’d like to maybe collaborate on that!
k
Kyle
09/30/2024, 2:31 AMI actually think having an alternative to the Databricks UI would be cool
j
jay
09/30/2024, 2:56 AMCan you send me a screenshot of what you’re currently thinking about? Is it the Spark one like this:
k
Kyle
09/30/2024, 3:08 AMthe logs are within the cell output and split by stages, which you can then click into to see the spark ui
Kyle
09/30/2024, 3:10 AMproblem though is that the ray jobs now arent in notebooks so probably this would not work unless there's a wrapper that just sends the jobs in and returns the results to the notebook
Kyle
09/30/2024, 3:12 AMwould be super sleek if all the ray stuff could just happen under the hood though
j
jay
09/30/2024, 6:45 AMYeah the end goal for us is to try and abstract Ray from the user as much as possible
jay
09/30/2024, 6:46 AMWe’re thinking something more like the BQ UI:
jay
09/30/2024, 6:46 AMEasier to understand than the Spark one I think (it mirrors the query plan quite a bit more)
k
Kyle
09/30/2024, 7:04 AMOh yeah I've never seen the bigquery one! Looks nice!
2 Views