By hand calculation of Krippendorff's alpha for a ratio level variable with 7 values
This variable is "What medium is analyzed? Radio" from Lombard, Snyder-Duch and Bracken (2002).
The contingency matrix is copied from Simstat output but could be obtained from other software as well.
Contingency matrices:
V24_HMJ_R->
Count | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
Row Pct | | | | | | | |
| 2.00 | 3.00 | 4.00 | 5.00 | 6.00 | 10.00 | 16.00 | Total
V24_HMC_R —————————+—————————+—————————+—————————+—————————+—————————+—————————+—————————+————————
1 2.00 | 24 | | | | | | | 24
| 100.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 46.2
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
2 3.00 | 1 | 13 | | | | | | 14
| 7.1 | 92.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 26.9
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
3 4.00 | | | 5 | | 1 | | | 6
| 0.0 | 0.0 | 83.3 | 0.0 | 16.7 | 0.0 | 0.0 | 11.5
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
4 5.00 | | | | 2 | | | | 2
| 0.0 | 0.0 | 0.0 | 100.0 | 0.0 | 0.0 | 0.0 | 3.8
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
5 6.00 | | | | | 3 | | | 3
| 0.0 | 0.0 | 0.0 | 0.0 | 100.0 | 0.0 | 0.0 | 5.8
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
6 10.00 | | | | | | 2 | | 2
| 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0 | 0.0 | 3.8
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
7 16.00 | | | | | | | 1 | 1
| 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0 | 1.9
—————————+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
Column 25 13 5 2 4 2 1 52
Total 48.1 25.0 9.6 3.8 7.7 3.8 1.9 100.0
V24_HM_R->
Count | | | | | | | |
Row Pct | | | | | | | |
| 2.00 | 3.00 | 4.00 | 5.00 | 6.00 | 10.00 | 16.00 | Total
V24_HMJ_R —————————+—————————+—————————+—————————+—————————+—————————+—————————+—————————+————————
2.00 | 24 | 1 | | | | | | 25
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
1 3.00 | | 13 | | | | | | 13
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
2 4.00 | | | 5 | | | | | 5
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
3 5.00 | | | | 2 | | | | 2
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
4 6.00 | | | 1 | | 3 | | | 4
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
5 10.00 | | | | | | 2 | | 2
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
6 16.00 | | | | | | | 1 | 1
—————————+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
Column 24 14 6 2 3 2 1 52
Total
OBSERVED Coincidence matrix:
Count | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| | | | | | | |
| 2.00 | 3.00 | 4.00 | 5.00 | 6.00 | 10.00 | 16.00 | Total
—————————+—————————+—————————+—————————+—————————+—————————+—————————+—————————+————————
1 2.00 | 48 | 1 | | | | | | 49
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
2 3.00 | 1 | 26 | | | | | | 27
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
3 4.00 | | | 10 | | 1 | | | 11
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
4 5.00 | | | | 4 | | | | 4
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
5 6.00 | | | 1 | | 6 | | | 7
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
6 10.00 | | | | | | 4 | | 4
+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
7 16.00 | | | | | | | 2 | 2
—————————+—————————+—————————+—————————+—————————+—————————+—————————+—————————+
Column 49 27 11 4 7 4 2 104
Total
So
|
o11 |
= |
48 |
|
o12 |
= |
1 |
|
o13 |
= |
0 |
|
o14 |
= |
0 |
|
o15 |
= |
0 |
|
o16 |
= |
0 |
|
o17 |
= |
0 |
|
o22 |
= |
26 |
|
o23 |
= |
0 |
|
o24 |
= |
0 |
|
o25 |
= |
0 |
|
o26 |
= |
0 |
|
o27 |
= |
0 |
|
o33 |
= |
10 |
|
o34 |
= |
0 |
|
o35 |
= |
1 |
|
o36 |
= |
0 |
|
o37 |
= |
0 |
|
o44 |
= |
4 |
|
o45 |
= |
0 |
|
o46 |
= |
0 |
|
O47 |
= |
0 |
|
o55 |
= |
6 |
|
o56 |
= |
0 |
|
o57 |
= |
0 |
|
o66 |
= |
4 |
|
o67 |
= |
0 |
|
o77 |
= |
2 |
|
|
|
|
EXPECTED Coincidence matrix:
erc = nr(nc-1)/(n-1) iff r=c (i.e., cells on diagonal)
nr*nc/(n-1) iff r not equal to c
r = row; c = column; nr = row marginal; nc = column marginal
|
e11 |
= 49(49-1)/(104-1) |
= 49(48)/103 |
= 2352/103 |
= |
22.834952 |
|
e12 |
= 49*27/(104-1) |
= 49*27/103 |
= 1323/103 |
= |
12.84466 |
|
e13 |
= 49*11/(104-1) |
= 49*11/103 |
= 539/103 |
= |
5.2330097 |
|
e14 |
= 49*4/(104-1) |
= 49*4/103 |
= 198/103 |
= |
1.9223301 |
|
e15 |
= 49*7/(104-1) |
= 49*7/103 |
= 343/103 |
= |
3.3300971 |
|
e16 |
= 49*4/(104-1) |
= 49*4/103 |
= 198/103 |
= |
1.9223301 |
|
e17 |
= 49*2/(104-1) |
= 49*2/103 |
= 98/103 |
= |
.95145 |
|
e22 |
= 27(27-1)/(104-1) |
= 27(26)/103 |
= 702/103 |
= |
6.815534 |
|
e23 |
= 27*11/(104-1) |
= 27*11/103 |
= 297/103 |
= |
2.8834952 |
|
e24 |
= 27*4/(104-1) |
= 27*4/103 |
= 108/103 |
= |
1.0485437 |
|
e25 |
= 27*7/(104-1) |
= 27*7/103 |
= 189/103 |
= |
1.8349515 |
|
e26 |
= 27*4/(104-1) |
= 27*4/103 |
= 108/103 |
= |
1.0485437 |
|
e27 |
= 27*2/(104-1) |
= 27*2/103 |
= 54/103 |
= |
.52427 |
|
e33 |
= 11(11-1)/(104-1) |
= 11(10)/103 |
= 110/103 |
= |
1.0679612 |
|
e34 |
= 11*4/(104-1) |
= 11*4/103 |
= 44/103 |
= |
.42718 |
|
e35 |
= 11*7/(104-1) |
= 11*7/103 |
= 77/103 |
= |
.74757 |
|
e36 |
= 11*4/(104-1) |
= 11*4/103 |
= 44/103 |
= |
.42718 |
|
e37 |
= 11*2/(104-1) |
= 11*2/103 |
= 22/103 |
= |
.21359 |
|
e44 |
= 4(4-1)/(104-1) |
= 4(3)/103 |
= 12/103 |
= |
.11650 |
|
e45 |
= 4*7/(104-1) |
= 4*7/103 |
= 28/103 |
= |
.27184 |
|
e46 |
= 4*4/(104-1) |
= 4*4/103 |
= 16/103 |
= |
.15533 |
|
e47 |
= 4*2/(104-1) |
= 4*2/103 |
= 8/103 |
= |
.077669 |
|
e55 |
= 7(7-1)/(104-1) |
= 7(6)/103 |
= 42/103 |
= |
.40776 |
|
e56 |
= 7*4/(104-1) |
= 7*4/103 |
= 28/103 |
= |
.27184 |
|
e57 |
= 7*2/(104-1) |
= 7*2/103 |
= 14/103 |
= |
.13592 |
|
e66 |
= 4(4-1)/(104-1) |
= 4(3)/103 |
= 12/103 |
= |
.11650 |
|
e67 |
= 4*2/(104-1) |
= 4*2/103 |
= 8/103 |
= |
.077669 |
|
e77 |
= 2(2-1)/(104-1) |
= 2(1)/103 |
= 2/103 |
= |
.019417 |
|
|
|
|
|
|
|
NOTE: TO CALCULATE A COLUMN TOTAL
RATIO DIFFERENCE FUNCTION matrix
δrc = ( r-c / r+c )2
r = row value; c = column value
(so all diagonal cells are 0)
|
δ11 |
= ( 2-2 / 2+2 )2 |
= (0/4)2 |
= |
0.0 |
|
δ12 |
= ( 2-3 / 2+3 )2 |
= (-1/5)2 |
= |
0.04 |
|
δ13 |
= ( 2-4 / 2+4 )2 |
= (-2/6)2 |
= |
0.11111 |
|
δ14 |
= ( 2-5 / 2+5 )2 |
= (-3/7)2 |
= |
0.18367 |
|
δ15 |
= ( 2-6 / 2+6 )2 |
= (-4/8)2 |
= |
0.25 |
|
δ16 |
= ( 2-10 / 2+10 )2 |
= (-8/12)2 |
= |
0.44444 |
|
δ17 |
= ( 2-16 / 2+16 )2 |
= (-14/18)2 |
= |
0.604938 |
|
δ22 |
= ( 3-3 / 3+3 )2 |
= (0/6)2 |
= |
0.0 |
|
δ23 |
= ( 3-4 / 3+4 )2 |
= (-1/7)2 |
= |
0.020408 |
|
δ24 |
= ( 3-5 / 3+5 )2 |
= (-2/8)2 |
= |
0.0625 |
|
δ25 |
= ( 3-6 / 3+6 )2 |
= (-3/9)2 |
= |
0.11111 |
|
δ26 |
= ( 3-10 / 3+10 )2 |
= (-7/13)2 |
= |
0.53846 |
|
δ27 |
= ( 3-16 / 3+16 )2 |
= (-13/19)2 |
= |
0.46814 |
|
δ33 |
= ( 4-4 / 4+4 )2 |
= (0/8)2 |
= |
0.0 |
|
δ34 |
= ( 4-5 / 4+5 )2 |
= (-1/9)2 |
= |
0.012345 |
|
δ35 |
= ( 4-6 / 4+6 )2 |
= (-2/10)2 |
= |
0.04 |
|
δ36 |
= ( 4-10 / 4+10 )2 |
= (-6/14)2 |
= |
0.18367 |
|
δ37 |
= ( 4-16 / 4+16 )2 |
= (-12/20)2 |
= |
0.36 |
|
δ44 |
= ( 5-5 / 5+5 )2 |
= (0/10)2 |
= |
0.0 |
|
δ45 |
= ( 5-6 / 5+6 )2 |
= (-1/11)2 |
= |
0.0082644 |
|
δ46 |
= ( 5-10 / 5+10 )2 |
= (-5/15)2 |
= |
0.11111 |
|
δ47 |
= ( 5-16 / 5+16 )2 |
= (-11/21)2 |
= |
0.27437 |
|
δ55 |
= ( 6-6 / 6+6 )2 |
= (0/12)2 |
= |
0.0 |
|
δ56 |
= ( 6-10 / 6+10 )2 |
= (-4/16)2 |
= |
0.0625 |
|
δ57 |
= ( 6-16 / 6+16 )2 |
= (-10/22)2 |
= |
0.20661 |
|
δ66 |
= ( 10-10 / 10+10 )2 |
= (0/20)2 |
= |
0.0 |
|
δ67 |
= ( 10-16 / 10+16 )2 |
(-6/26)2 |
= |
0.053254 |
|
δ77 |
= ( 16-16 / 16+16 )2 |
(0/32)2 |
= |
0.0 |
|
|
|
|
|
|
α = 1 – (Do / De) = 1 – ( (orc * δrc) / (erc * δrc) )
so to calculate:
(orc * δrc) = Do
|
11 |
48 |
0.0 |
0 |
|
12 |
1 |
0.04 |
0.04 |
|
13 |
0 |
0.11111 |
0 |
|
14 |
0 |
0.18367 |
0 |
|
15 |
0 |
0.25 |
0 |
|
16 |
0 |
0.44444 |
0 |
|
17 |
0 |
0.604938 |
0 |
|
22 |
26 |
0.0 |
0 |
|
23 |
0 |
0.020408 |
0 |
|
24 |
0 |
0.0625 |
0 |
|
25 |
0 |
0.11111 |
0 |
|
26 |
0 |
0.53846 |
0 |
|
27 |
0 |
0.46814 |
0 |
|
33 |
10 |
0.0 |
0 |
|
34 |
0 |
0.012345 |
0 |
|
35 |
1 |
0.04 |
0.04 |
|
36 |
0 |
0.18367 |
0 |
|
37 |
0 |
0.36 |
0 |
|
44 |
4 |
0.0 |
0 |
|
45 |
0 |
0.0082644 |
0 |
|
46 |
0 |
0.11111 |
0 |
|
47 |
0 |
0.27437 |
0 |
|
55 |
6 |
0.0 |
0 |
|
56 |
0 |
0.0625 |
0 |
|
57 |
0 |
0.20661 |
0 |
|
66 |
4 |
0.0 |
0 |
|
67 |
0 |
0.053254 |
0 |
|
77 |
2 |
0.0 |
0 |
|
|
|
|
0.08 |
(erc * δrc) = De
|
11 |
22.834952 |
0.0 |
.0 |
|
12 |
12.84466 |
0.04 |
.5137864 |
|
13 |
5.2330097 |
0.11111 |
.58143 |
|
14 |
1.9223301 |
0.18367 |
.35307 |
|
15 |
3.3300971 |
0.25 |
.83252 |
|
16 |
1.9223301 |
0.44444 |
.85436 |
|
17 |
.95145 |
0.604938 |
.575568 |
|
22 |
6.815534 |
0.0 |
.0 |
|
23 |
2.8834952 |
0.020408 |
.058846 |
|
24 |
1.0485437 |
0.0625 |
.065533 |
|
25 |
1.8349515 |
0.11111 |
.20388 |
|
26 |
1.0485437 |
0.53846 |
.56459 |
|
27 |
.52427 |
0.46814 |
.24543 |
|
33 |
1.0679612 |
0.0 |
.0 |
|
34 |
.42718 |
0.012345 |
.0052735 |
|
35 |
.74757 |
0.04 |
.0299028 |
|
36 |
.42718 |
0.18367 |
.078460 |
|
37 |
.21359 |
0.36 |
.0768924 |
|
44 |
.11650 |
0.0 |
.0 |
|
45 |
.27184 |
0.0082644 |
.0022465 |
|
46 |
.15533 |
0.11111 |
.017258 |
|
47 |
.077669 |
0.27437 |
.021310 |
|
55 |
.40776 |
0.0 |
.0 |
|
56 |
.27184 |
0.0625 |
.01699 |
|
57 |
.13592 |
0.20661 |
.028082 |
|
66 |
.11650 |
0.0 |
.0 |
|
67 |
.077669 |
0.053254 |
.0041361 |
|
77 |
.019417 |
0.0 |
.0 |
|
|
|
|
5.1295647 |
α = 1 – (Do / De) = 1 – ( (orc * δrc) / (erc * δrc) )
= 1 – (Do / De) = 1 – ( .08 / 5.1295647 )
= 1 -.0155958
= .9844041
= .984
For comparison, here is output from Krippendorff's Alpha 3.12a.
RELIABILITY V3.12 OUTPUT
***********************
Variable Name: V24_HM_R
ACCOUNT OF THE RELIABILITY DATA
Number of Observed Values: 121
Number of kinds of values: 8
Number of Coders: 2
Number of Units : 129
Nominal Ordinal Interval Ratio
Observed Disagreement 02.000 00.024 05.000 00.080
Expected Disagreement 36.667 01.258 255.127 03.956
Krippendorff's Alpha 00.945 00.981 00.980 00.980
CODER NAMES:
01 1
02 2
UNIT NAMES:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
RAW DATA LISTING:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
1 3 3 3 3 2 3 16 3
2 3 3 2 2 3 16 3
COINCIDENCE MATRICES:
---------------------
Observed Coincidence Matrix:
1 2 3 4 5 6 10 16
================================================================
1 |
2 | 24.000 00.500
3 | 00.500 13.000
4 | 05.000 00.500
5 | 02.000
6 | 00.500 03.000
10 | 02.000
16 | 01.000
Expected Coincidence Matrix:
1 2 3 4 5 6 10 16
================================================================
1 |
2 | 11.289 06.485 02.642 00.961 01.681 00.961 00.480
3 | 06.485 03.309 01.456 00.529 00.926 00.529 00.265
4 | 02.642 01.456 00.485 00.216 00.377 00.216 00.108
5 | 00.961 00.529 00.216 00.039 00.137 00.078 00.039
6 | 01.681 00.926 00.377 00.137 00.172 00.137 00.069
10 | 00.961 00.529 00.216 00.078 00.137 00.039 00.039
16 | 00.480 00.265 00.108 00.039 00.069 00.039
MATRIX WEIGHTING:
-----------------
Nominal:
1 2 3 4 5 6 10 16
================================================================
1 | 00.000 01.000 01.000 01.000 01.000 01.000 01.000 01.000
2 | 01.000 00.000 01.000 01.000 01.000 01.000 01.000 01.000
3 | 01.000 01.000 00.000 01.000 01.000 01.000 01.000 01.000
4 | 01.000 01.000 01.000 00.000 01.000 01.000 01.000 01.000
5 | 01.000 01.000 01.000 01.000 00.000 01.000 01.000 01.000
6 | 01.000 01.000 01.000 01.000 01.000 00.000 01.000 01.000
10 | 01.000 01.000 01.000 01.000 01.000 01.000 00.000 01.000
16 | 01.000 01.000 01.000 01.000 01.000 01.000 01.000 00.000
Ordinal
1 2 3 4 5 6 10 16
================================================================
1 | 00.000 00.009 00.059 00.100 00.119 00.134 00.150 00.159
2 | 00.009 00.000 00.022 00.049 00.063 00.074 00.086 00.093
3 | 00.059 00.022 00.000 00.005 00.011 00.015 00.021 00.025
4 | 00.100 00.049 00.005 00.000 00.001 00.003 00.005 00.007
5 | 00.119 00.063 00.011 00.001 00.000 00.000 00.002 00.003
6 | 00.134 00.074 00.015 00.003 00.000 00.000 00.000 00.001
10 | 00.150 00.086 00.021 00.005 00.002 00.000 00.000 00.000
16 | 00.159 00.093 00.025 00.007 00.003 00.001 00.000 00.000
Interval:
1 2 3 4 5 6 10 16
================================================================
1 | 00.000 01.000 04.000 09.000 16.000 25.000 36.000 49.000
2 | 01.000 00.000 01.000 04.000 09.000 16.000 25.000 36.000
3 | 04.000 01.000 00.000 01.000 04.000 09.000 16.000 25.000
4 | 09.000 04.000 01.000 00.000 01.000 04.000 09.000 16.000
5 | 16.000 09.000 04.000 01.000 00.000 01.000 04.000 09.000
6 | 25.000 16.000 09.000 04.000 01.000 00.000 01.000 04.000
10 | 36.000 25.000 16.000 09.000 04.000 01.000 00.000 01.000
16 | 49.000 36.000 25.000 16.000 09.000 04.000 01.000 00.000
Ratio:
1 2 3 4 5 6 10 16
================================================================
1 | 00.000 00.111 00.250 00.360 00.444 00.510 00.563 00.605
2 | 00.111 00.000 00.040 00.111 00.184 00.250 00.309 00.360
3 | 00.250 00.040 00.000 00.020 00.063 00.111 00.160 00.207
4 | 00.360 00.111 00.020 00.000 00.012 00.040 00.074 00.111
5 | 00.444 00.184 00.063 00.012 00.000 00.008 00.028 00.053
6 | 00.510 00.250 00.111 00.040 00.008 00.000 00.006 00.020
10 | 00.563 00.309 00.160 00.074 00.028 00.006 00.000 00.004
16 | 00.605 00.360 00.207 00.111 00.053 00.020 00.004 00.000