nltk.tokenize.punkt._PunktBaseClass

type _PunktBaseClass

object --+ | _PunktBaseClass

Includes common components of PunktTrainer and PunktSentenceTokenizer.

Instance Methods

__init__(self, lang_vars=PunktLanguageVars(), token_cls=<class 'nltk.tokenize.punkt.PunktToken'>, params=PunktParameters())

source code

Word tokenization

_tokenize_words(self, plaintext)
Divide the given text into tokens, using the punkt word segmentation regular expression, and generate the resulting list of tokens augmented as three-tuples with two boolean values for whether the given token occurs at the start of a paragraph or a new line, respectively.

source code

Annotation Procedures

_annotate_first_pass(self, tokens)
Perform the first pass of annotation, which makes decisions based purely based on the word type of each word:

source code

_first_pass_annotation(self, aug_tok)
Performs type-based annotation on a single token.

source code

Instance Variables

[hide private]

_Token
The collection of parameters that determines the behavior of the punkt tokenizer.

init(self, lang_vars=PunktLanguageVars(), token_cls=<class 'nltk.tokenize.punkt.PunktToken'>, params=PunktParameters())
(Constructor)

source code

Overrides: object.__init__: (inherited documentation)

_annotate_first_pass(self, tokens)

source code

Perform the first pass of annotation, which makes decisions based purely based on the word type of each word:

'?', '!', and '.' are marked as sentence breaks.
sequences of two or more periods are marked as ellipsis.
any word ending in '.' that's a known abbreviation is marked as an abbreviation.
any other word ending in '.' is marked as a sentence break.

Return these annotations as a tuple of three sets:

sentbreak_toks: The indices of all sentence breaks.
abbrev_toks: The indices of all abbreviations.
ellipsis_toks: The indices of all ellipsis marks.

type _PunktBaseClass

__init__(self, lang_vars=PunktLanguageVars(), token_cls=<class 'nltk.tokenize.punkt.PunktToken'>, params=PunktParameters()) (Constructor)

_annotate_first_pass(self, tokens)

init(self, lang_vars=PunktLanguageVars(), token_cls=<class 'nltk.tokenize.punkt.PunktToken'>, params=PunktParameters())
(Constructor)