By hand calculation of percent agreement, Scott's pi, Cohen's kappa and Krippendorff's alpha for a nominal level variable with 2 values

This variable is "What medium is analyzed? Radio" from Lombard, Snyder-Duch and Bracken (2002).

The contingency matrix is copied from Simstat output but could be obtained from other software as well.

INTER-RATERS: V5_4 by V5_4J

  V5_4J->

             Count |         |         |

           Row Pct |         |         |

                   |       0 |       1 |   Total

V5_4      —————————+—————————+—————————+————————

                 0 |     127 |       1 |     128

                   |    99.2 |     0.8 |    99.2

                   +—————————+—————————+

                 1 |         |       1 |       1

                   |     0.0 |   100.0 |     0.8

          —————————+—————————+—————————+

            Column       127         2       129

             Total      98.4       1.6     100.0

_______________________________________________________________________________

Percent agreement:

PA₀ = Total A’s    =    127+1   =    128  =  .99224 = .99

               n                      129           129

_______________________________________________________________________________

Scott's pi:

Scott’s pi = PA₀ - PA_E         where PA_E = åp_i²    and    p_i = each joint marginal proportion

                    _________

                         1 - PA_E

WITH ROUNDING:

                  ----------------Marginals------------

Category   n for coder A      n for coder B         Product of marginals:       Sum of marginals          Joint marginal proportions

       1                 128                  127                            16256                              255                          255/258   =   .99

       2                     1                      2                                      2                                  3                              3/258    =    .01

                          129                  129                                                                     258                                                1.00 

So PA_E = åp_i²

= (.99)² + (.01)²

=  .98 + .0001

= .98

And, Scott’s pi =

= .99 - .98

     1 - .98

= .01

   .02

= .5

WITHOUT ROUNDING:

                   ----------------Marginals------------

Category   n for coder A      n for coder B         Product of marginals:       Sum of marginals          Joint marginal proportions

       1                 128                  127                            16256                              255                          255/258   =   .98837

       2                     1                      2                                      2                                   3                              3/258    =   .011627

                          129                  129                                                                      258                                                 .99999 

So PA_E = åp_i²

= (.98837)² + (.011627)²

=  .97687 + .00013520

= .977

 And, Scott’s pi =

= .99224 - .977

        1 - .977

= .01524

   .023

= .6626087  = .663

_______________________________________________________________________

Cohen's kapp:

Cohen’s kappa =   PA₀ - PA_E       where PA_E = (1/n²)(åpm_i)   and    n = number of units coded in common by coders

                                 ________                                                      and    pm_i = each product of marginals

                                    1 - PA_E  

WITH ROUNDING:

So:   PA_E = (1/n²)(åpm_i)

                = (1/129²)(16256 + 2)

                = (1/16641)(16258)

                = (.00006)(16258)

                = .98

And, Cohen’s kappa = PA₀ - PA_E 

                                           1 - PA_E

= .99 - .98

     1-.98

= .01

   .02

= .5

WITHOUT ROUNDING:

So:   PA_E = (1/n²)(åpm_i)

                = (1/129²)(16256 + 2)

                = (1/16641)(16258)

                = (.000060092)(16258)

                = .97698

And, Cohen’s kappa = PA₀ - PA_E 

                                          1 - PA_E

= .99224 - .97698

     1 -.97698

= .01526

   .02302

= .6629  = .663

_______________________________________________________________________

Krippendorff’s Alpha:

Contingency matrices (the first is copied from Simstat output):

INTER-RATERS: V5_4 by V5_4J

  V5_4J->

             Count |         |         |

           Row Pct |         |         |

                   |       0 |       1 |   Total

V5_4      —————————+—————————+—————————+————————

                 0 |     127 |       1 |     128

                   |    99.2 |     0.8 |    99.2

                   +—————————+—————————+

                 1 |         |       1 |       1

                   |     0.0 |   100.0 |     0.8

          —————————+—————————+—————————+

            Column       127         2       129

             Total      98.4       1.6     100.0

  V5_4->

             Count |         |         |

                   |         |         |

                   |       0 |       1 |   Total

V5_4J     —————————+—————————+—————————+————————

                 0 |     127 |       0 |     127

                   |         |         |   

                   +—————————+—————————+

                 1 |       1 |       1 |       2

                   |         |         |   

          —————————+—————————+—————————+

            Column       128         1       129

             Total

Coincidence matrix:

             Count |         |         |

                   |         |         |

                   |       0 |       1 |   Total

          —————————+—————————+—————————+————————

                 0 |     254 |       1 |    255

                   |         |         |   

                   +—————————+—————————+

                 1 |       1 |       2 |     3

                   |         |         |   

          —————————+—————————+—————————+

            Column        255       3       258

             Total

D_o = 1 – 1/n (∑ O_cc)

      = 1 – 1/258 (254 + 2)

      = 1 - .0038759 (256)

      = 1 - .992248

      = .007752

D_e = 1 – 1/ (n(n-1)) (∑ n_c(n_c-1))

     = 1 – 1/ 258(258-1) ( 255(255-1) + 3(3-1) )

     = 1 – 1/ 258(257) ( 255(254) + 3(2) )

     = 1 – 1/ 66306 (64770 + 6)

     = 1 – .000015081 (64776)

     = 1 - .9769251

     = .0230749

α   = 1 – (D_o / D_e)

     = 1 – (.007752/ .0230749)

     = 1 - .3359494

     = .6640506

     = .664