Skip to main content

Table 2 Inner annotator agreement between annotator group 1, 2 and gold set in precision (\({\mathcal {P}}\)), recall (\({\mathcal {R}}\)) and Macro \(F_1\) score (\({\mathcal {F}}_1\))

From: ChemTables: a dataset for semantic classification on tables in chemical patents

Label Annotator 1 Annotator 2 Random
\({\mathcal {P}}\) \({\mathcal {R}}\) \({\mathcal {F}}_1\) \({\mathcal {P}}\) \({\mathcal {R}}\) \({\mathcal {F}}_1\) \({\mathcal {P}}\) \({\mathcal {R}}\) \({\mathcal {F}}_1\)
SPEC 97.20 97.74 97.47 96.77 96.67 97.47 24.27 26.14 25.17
PHYS 79.02 89.13 83.77 88.43 89.15 88.79 9.43 3.79 5.41
IDE 88.47 94.51 91.39 94.04 85.54 89.59 16.64 18.87 17.68
RX 62.32 82.45 70.98 80.68 83.72 82.17 7.14 4.84 5.77
PHARM 80.94 93.67 86.84 87.55 91.69 89.57 14.76 17.42 15.98
COMPOSITION 85.71 85.41 85.56 79.20 79.94 79.57 6.97 8.04 7.47
PROPERTY 45.56 61.89 52.49 36.68 46.35 40.95 3.13 3.45 3.28
OTHER 69.92 25.05 36.88 62.21 58.37 60.23 13.43 15.32 14.31
Overall 76.14 78.73 75.67 78.20 78.93 78.54 11.97 12.23 11.88
  1. “Random” refers to randomly sampled label from the label distribution in the final gold standard dataset