Package nltk :: Package metrics :: Module agreement :: Class AnnotationTask

classobj_type AnnotationTask

Represents an annotation task, i.e. people assign labels to items.

Notation tries to match notation in Artstein and Poesio (2007).

In general, coders and items can be represented as any hashable object. Integers, for example, are fine, though strings are more readable. Labels must support the distance functions applied to them, so e.g. a string-edit-distance makes no sense if your labels are integers, whereas interval distance needs numeric values. A notable case of this is the MASI metric, which requires Python sets.

Instance Methods

[hide private]

__init__(self, data=None, distance=<function binary_distance at 0x10db7b0>)
Initialize an empty annotation task.

source code

__str__(self)

source code

load_array(self, array)
Load the results of annotation.

source code

agr(self, cA, cB, i)
Agreement between two coders on a given item

source code

N(self, k=None, i=None, c=None)
Implements the "n-notation" used in Artstein and Poesio (2007)

source code

Ao(self, cA, cB)
Observed agreement between two coders on all items.

source code

avg_Ao(self)
Average observed agreement across all coders and items.

source code

Do_alpha(self)
The observed disagreement for the alpha coefficient.

source code

Do_Kw_pairwise(self, cA, cB, max_distance=1.0)
The observed disagreement for the weighted kappa coefficient.

source code

Do_Kw(self, max_distance=1.0)
Averaged over all labelers

source code

S(self)
Bennett, Albert and Goldstein 1954

source code

pi(self)
Scott 1955

source code

pi_avg(self)

source code

kappa_pairwise(self, cA, cB)

source code

kappa(self)
Cohen 1960

source code

alpha(self)
Krippendorff 1980

source code

weighted_kappa_pairwise(self, cA, cB, max_distance=1.0)
Cohen 1968

source code

weighted_kappa(self)
Cohen 1968

source code

Method Details

[hide private]

load_array(self, array)

source code

Load the results of annotation.

The argument is a list of 3-tuples, each representing a coder's labeling of an item:
    (coder,item,label)

Do_alpha(self)

source code

The observed disagreement for the alpha coefficient.

The alpha coefficient, unlike the other metrics, uses this rather than observed agreement.