Package nltk :: Package tokenize :: Module api :: Class StringTokenizer

type StringTokenizer

object --+    
         |    
TokenizerI --+
             |
            StringTokenizer

Known Subclasses:

A tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses).

Instance Methods

tokenize(self, s)
Divide the given string into a list of substrings.

source code

span_tokenize(self, s)
Identify the tokens using integer offsets (start_i, end_i), where s[start_i:end_i] is the corresponding token.

source code

Method Details

Divide the given string into a list of substrings.

Identify the tokens using integer offsets (start_i, end_i), where s[start_i:end_i] is the corresponding token.