If I want to run tokenizing with a sentencepiece model instead of tiktoken what would be the recommended way to do so?

this would likely be a good use case for UDFs

UDF is probably the way to go!

Cool, thanks!