109 年 國立臺灣大學語言學研究所《語言與計算方法》

1. Էրֆէրէ պկտ մե $ htիենդ mg/In եսդ
   dear/expensive gift my friend of/from Tokyo lost
   (An expensive gift to my friend from Tokyo was lost.)

2. h სევ ინ ըրպէն կարտէն thnnლე გირლ0
   I See in urban garden little boy
   (I saw a little boy in the city garden.)

3. ტრონგ ჰუმან ეასილყ რაისէ դհիս լոատ
   Strong human easily raise this load
   (A strong man will easily lift this load.)

4. Uht վիլլ 378 ტჰის არnhait ინ ոըր ლოხ
   She will_put this article in our box
   (She will put this article in our box.)

B. (50%) inter-annotator agreement

An agreement coefficient calculates the amount that annotators agreed on label assignments beyond what is expected by chance. In a prototypical linguistic annotation task, annotators (a.k.s. coders) assign predefined labels (or values) to specific units (words, sentences, chunks, etc.) in the source. To measure the reliability of the annotation, a simple way of reporting agreement between annotators is called raw agreement measure which counts the number of items for which they provide identical labels, and report that number as a percentage of the total to be annotated.

Problem.1 (15%)
Why raw agreement does not imply that the annotation process is reliable? Provide examples to support your arguments.

In corpus and computational linguistics, a more accepted way is to measure the reliability of agreement by using a coefficient from the kappa/alpha family. For instance, Krippendorff's alpha (a)'s general form is:
$\alpha = 1 - \frac{D_o}{D_e}$
where $D_o$ is the observed disagreement among values assigned to units of analysis:
$D_o = \sum_{k} \sum_{l} o_{kl} \frac{n_k n_l}{n(n-1)}$
and $D_e$ is the disagreement one would expect when the coding of units is attributable to chance rather than to the properties of these units:
$D_e = \frac{1}{n(n-1)} \sum_{k} n_k (n_k - 1)$
The arguments in the two disagreement measures, $o_{kl}$, $n_k$, $n_l$ and $n$, refer to the frequencies of values in coincidence matrices, to be defined below.
(Krippendorff, 2011)

Take the following nominal data coded by two annotators for example, the computational steps involve:

1. Construct a 2-by-12 rating table
   Items judged: 1 2 3 4 5 6 7 8 9 10 11 12
   Ben: a a b b d c c e d d a
   Gerry: a b a b b c c e d d a

2. Transform it into a coincidence matrix.

Values:
1..k
$o_{11}..o_{1k}$
$n_1..n_k$
$n = \sum \sum o_{kl}$

Where $o_{kl} = \sum_u$ Number of c-k pairs in unit u specifically:
$o_{ab} = 1$ a-b pair in unit 1
$o_{ba} = 1$ b-a pair in unit 1
$o_{aa} = 2$ a-a pairs in unit 2
$o_{bb} = 4 = 2$ b-b pairs in unit 3 + 2 b-b pairs in unit 4
and so forth.
$n_a = 4$ is the number of a's
$n_b = 6$ is the number of b's
and so forth.
$n = 24$ is the total number of values for two observers: $n = 2N$

3. Compute alpha (a):
   $\alpha_{nominal} = 1 - \frac{D_o}{D_e}$

Problem.2 (35%)
We can use NLTK metrics package to calculate the inter-annotator agreement for this toy example. What'd be expected in the result? Calculate it manually based on the formulas.

from nltk.metrics.agreement import AnnotationTask
from nltk.metrics import binary_distance
annotation_triples = [{'Ben', '1', '1'),

Problem.2 (35%)
We can use NLTK metrics package to calculate the inter-annotator agreement for this toy
example. What'd be expected in the result? Calculate it manually based on the formulas.

from nltk.metrics.agreement import AnnotationTask
from nltk.metrics import binary_distance
annotation_triples = [{'Ben', '1', '1'),
('Gerry', '1', '2'),
('Ben', '2', '1'),
('Gerry', '2', '1'),
('Ben', '3', '2'),
('Gerry', '3', '2'),
('Ben', '4', '2'),
('Gerry', '4', '2'),
('Ben', '5', '4'),
('Gerry', '5', '2'),
('Ben', '6', '3'),
('Gerry', '6', '3'),
('Ben', '7', '3'),
('Gerry', '7', '3'),
('Ben', '8', '3'),
('Gerry', '8', '3'),
('Ben', '9', '5'),
('Gerry', '9', '5'),
('Ben', '10', '4'),
('Gerry', '10', '4'),
('Ben', '11', '4'),
('Gerry', '11', '4'),
('Ben', '12', '1'),
('Gerry', '12', '4')]
t = AnnotationTask (annotation_triples, distance = binary_distance)
result = t.alpha()