Package nltk :: Package tokenize :: Module simple :: Class SpaceTokenizer
[hide private]
[frames] | no frames]

type SpaceTokenizer

source code

     object --+        
              |        
 api.TokenizerI --+    
                  |    
api.StringTokenizer --+
                      |
                     SpaceTokenizer

A tokenizer that divides a string into substrings by treating any single space character as a separator. If you are performing the tokenization yourself (rather than building a tokenizer to pass to some other piece of code), consider using the string split() method instead:

>>> words = s.split(' ')
Instance Methods [hide private]

Inherited from api.StringTokenizer: span_tokenize, tokenize

Inherited from api.TokenizerI: batch_span_tokenize, batch_tokenize

Class Variables [hide private]
  _string = ' '