nltk tweettokenizer
(nltk)Natural Language Toolkit
Difficulty Level : Easy

TweetTokenizer and word_tokenize are tokenizers almost work the same way, to split a given sentence into words. But you can think of TweetTokenizer as a subset of word_tokenize. TweetTokenizer keeps hashtags intact while word_tokenize doesn’t.

Example

Alternatives

  • RegexpTokenizer
  • SExprTokenizer
  • stanford_segmenter

Download and Install NLTK

Alternative On Linux
Run the command

To ensure central installation, run the command