english-tokenizer

Splits a sentence into an ordered list of tokens and tags each one by type, such as word, number, ordinal, URL, mention, hashtag, emoji, time or currency. It handles Latin-1 and Devanagari scripts and splits common English contractions and possessives.