Skip to main content

Table 2 Inner annotator agreement between annotator group 1, 2 and gold set in precision (\({\mathcal {P}}\)), recall (\({\mathcal {R}}\)) and Macro \(F_1\) score (\({\mathcal {F}}_1\))

From: ChemTables: a dataset for semantic classification on tables in chemical patents

Label

Annotator 1

Annotator 2

Random

\({\mathcal {P}}\)

\({\mathcal {R}}\)

\({\mathcal {F}}_1\)

\({\mathcal {P}}\)

\({\mathcal {R}}\)

\({\mathcal {F}}_1\)

\({\mathcal {P}}\)

\({\mathcal {R}}\)

\({\mathcal {F}}_1\)

SPEC

97.20

97.74

97.47

96.77

96.67

97.47

24.27

26.14

25.17

PHYS

79.02

89.13

83.77

88.43

89.15

88.79

9.43

3.79

5.41

IDE

88.47

94.51

91.39

94.04

85.54

89.59

16.64

18.87

17.68

RX

62.32

82.45

70.98

80.68

83.72

82.17

7.14

4.84

5.77

PHARM

80.94

93.67

86.84

87.55

91.69

89.57

14.76

17.42

15.98

COMPOSITION

85.71

85.41

85.56

79.20

79.94

79.57

6.97

8.04

7.47

PROPERTY

45.56

61.89

52.49

36.68

46.35

40.95

3.13

3.45

3.28

OTHER

69.92

25.05

36.88

62.21

58.37

60.23

13.43

15.32

14.31

Overall

76.14

78.73

75.67

78.20

78.93

78.54

11.97

12.23

11.88

  1. “Random” refers to randomly sampled label from the label distribution in the final gold standard dataset