Package nltk :: Package tokenize :: Module simple :: Class TabTokenizer
[hide private]
[frames] | no frames]

type TabTokenizer

source code

     object --+        
              |        
 api.TokenizerI --+    
                  |    
api.StringTokenizer --+
                      |
                     TabTokenizer

A tokenizer that divides a string into substrings by treating any single tab character as a separator. If you are performing the tokenization yourself (rather than building a tokenizer to pass to some other piece of code), consider using the string split() method instead:

>>> words = s.split('\t')
Instance Methods [hide private]

Inherited from api.StringTokenizer: span_tokenize, tokenize

Inherited from api.TokenizerI: batch_span_tokenize, batch_tokenize

Class Variables [hide private]
  _string = '\t'