tokenx

Estimates how many tokens a language model reads in a string, with a few characters-per-token rules in place of a vocabulary or BPE table. That makes it far smaller and cheaper than a real tokenizer, and the count is an estimate that can diverge from any specific tokenizer on some text.