PSA didn’t get to do a demo today, but @Raunak Bhagat has made it super easy to run scripts on distributed mode Ray job in CI! I’ve been using this quite extensively for running dogfooding/testing of some code I’m writing. Here’s an example of how to do it:

uv run tools/gha_run_cluster_job.py \
    benchmarking/ooms/big_file_reads.py \
    --daft-wheel-url=<https://github-actions-artifacts-bucket.s3.us-west-2.amazonaws.com/builds/491054b0455307e161012e4d48c9a5198b1334e3/getdaft-0.3.0.dev0-cp38-abi3-manylinux_2_31_x86_64.whl> \
    -- \
    --enable-optimized-splits

(Step 0): grab a github access token and set it to

$GITHUB_OAUTH_TOKEN

1. Pass it a script (I simply use the parent folder of your script as the “working directory” which we will upload to the Ray job) 2. Pass it either a

--daft-wheel-url

or a

--daft-version

to indicate what Daft to install across the cluster 3. After

--

you can pass it any arguments that should be forwarded to your script Here’s an example: