Intercoder reliability is a measure of agreement or consistency among two or more coders who independently code a transcript. It measures if two or more coders apply the same codes to the same text in the same way.

To calculate an intercoder reliability score in Delve you will need two or more researchers to code the same transcript. Follow the steps below to calculate intercoder reliability.

Intercoder reliability measures how your team will apply the same codebook. So before you begin coding, you will need a codebook with well-defined code descriptions.

Learn how to create a project

After creating the project with a codebook, invite your team to the project. You can invite them using the share button in the upper right-hand corner.

Learn how to share a project

Your research team should code the same transcript without looking at each other's work. They can do that using the "Coded By Me" feature, which hides the work of all other team members.

Learn how to use the "Coded By Me" feature

Note: As a team, you should agree if the coders are allowed to add more codes to the codebook. Intercoder reliability is most appropriate for deductive coding exercises, where the team does not add new codes to the codebook.

To then calculate intercoder reliability use the Transcript Navigation Dropdown and click on Coding Comparison.

There you will see a button for Inter Coder Reliability.

Clicking it will provide you with your intercoder reliability score for that transcript.

The intercoder reliability score is calculated using Krippendorff's Alpha, a standard statistical measure for calculating intercoder reliability. Krippendorff's Alpha will range between -1 and 1. With 1 being perfect agreement, 0 is considered no better than chance, and below 0 is considered worse than chance.

Krippendorff's Alpha has a number of benefits over other intercoder reliability scores such as percent agreement. They include:

A good intercoder reliability score depends on what type of research you are conducting and how it is being used. That being said a score of 0.8 is commonly said to demonstrate consistency.

No your team's snippets do not need to perfectly overlap to be included in the intercoder reliability calculation. Delve takes into account overlapping snippets when calculating the score.

If only one person codes a piece of text it is excluded from Krippendorff's Alpha score. Only text that is coded by at least two coders is included in the score. Any text with one or zero coders is essentially ignored by the calculation.

There are a couple of scenarios where the intercoder reliability score cannot be calculated. They are all scenarios where not enough of the transcript has been coded. Here are the following scenarios where the intercoder reliability score cannot be calculated:

In the generic Krippendorff's Alpha formula, there is a concept of "weights". Weights are essentially "partial credit", where coders may get an increase in score if they are "close enough".

Delve uses "identity" weighting, which means there is no "partial credit". The alpha score will only increase if the two coders coded the same segment in exactly the same way.

We recommend that if you are conducting a strict intercoder reliability test, that your team should create a codebook where codes are conceptually separate. That way your team can take an approach where you only assign one code to a particular piece of text.

That being said our implementation of Krippendorff's Alpha can handle simultaneous coding, where you apply multiple codes to the same piece of text. As mentioned above, Delve uses "identity" weighting. So if one person codes a piece of text with two codes, another person will only be in agreement if they code the same piece of text with the same two codes.

No, your team should not necessarily use intercoder reliability. Not all qualitative researchers agree that intercoder reliability should be universally used. In their book "Thematic Analysis: A Practical Guide" Virginia Braun and Victoria Clarke state that they believe that coding reliability procedures such as inter-coder reliability can "lead to themes that are relatively superficial and underdeveloped" (page 240). Underpinning the concept of Intercoder Reliability is the idea that researcher subjectivity is an undesirable bias. On the contrary researcher subjectivity is viewed by many qualitative researchers as an inevitable and even valuable aspect of qualitative research, providing unique insights and depth to the analysis.

Instead of using intercoder reliability consider using Delve's code comparison feature where you can compare and discuss each other's work, without reducing your unique perspective to a quantitative score.

There are no hard and fast rules for when to use intercoder reliability. However, we will list a couple of scenarios where intercoder reliability could be helpful to your research analysis: