Module tokenize

Classes

BlanklineTokenizer
LineTokenizer
PunktSentenceTokenizer
PunktWordTokenizer
RegexpTokenizer
SExprTokenizer
SpaceTokenizer
TabTokenizer
TextTilingTokenizer
TreebankWordTokenizer
WhitespaceTokenizer
WordPunctTokenizer

Functions

blankline_tokenize
line_tokenize
regexp_tokenize
sent_tokenize
sexpr_tokenize
word_tokenize
wordpunct_tokenize

Variables

BLOCK_COMPARISON
DEFAULT_SMOOTHING
Deprecated
HC
LC
VOCABULARY_INTRODUCTION
WordTokenizer
[hide private]