# english-tokenizer

Splits a sentence into an ordered list of tokens and tags each one by type, such as word, number, ordinal, URL, mention, hashtag, emoji, time or currency. It handles Latin-1 and Devanagari scripts and splits common English contractions and possessives.

## Install

```sh
cargo add english-tokenizer
```

## Links

- [Repository](https://github.com/hey-jj/english-tokenizer)
- [crates.io](https://crates.io/crates/english-tokenizer)
- [docs.rs](https://docs.rs/english-tokenizer)