# tokenx

Estimates how many tokens a language model reads in a string, with a few characters-per-token rules in place of a vocabulary or BPE table. That makes it far smaller and cheaper than a real tokenizer, and the count is an estimate that can diverge from any specific tokenizer on some text.

## Install

```sh
cargo add tokenx
```

## Links

- [Repository](https://github.com/hey-jj/tokenx)
- [crates.io](https://crates.io/crates/tokenx)
- [docs.rs](https://docs.rs/tokenx)