Interrater reliability
Budget: $250 – $750 CAD
I was wondering whether someone might be able to help me out with the following issue.
For my research project, we are using picture ratings (ratings of the extent to which an individual in the picture is dominant, trustworthy, etc) to predict another variable.
To do so, we had 60 participants rate the picture on several traits (e.g., dominant, trustworthy, attractive, etc). Overall we had 500+ pictures to rate and, being afraid of the participant fatigue, we had each individual rate only a few of the pictures. As such, we ended up with a dataset where each picture has been rated by 3-8 raters. Now we need to calculate interrater reliability.
The standard cronbach's alpha is not an option because it requires each picture being rated by two of the same raters (I think). The only metrics I found that might potentially work are Krippendorff's alpha and icc1, both of which have relaxed assumption about the number of raters and whether the ratings are dependent (ie. in other words, they allow for raters to be independent and different from picture to picture). For both of this, a script can be downloaded from Andrew Haey's website for SPSS.
The difficulty for me is a) reconfiguring the data from its current format and b) not knowing whether variable number of raters is allowed for each picture or whether each picture has to be rated by the same number of raters - if the latter, I'd be fine with deleting raters beyond 3 for each picture such that each picture is rated by the the 3 raters which provided most consistent ratings. If you could do that as well as obtain either Krippendorff's alpha or icc, it would be great.
I am attaching a sample of the data. The first column indicates the picture number. As you can see each picture is rated by 3 or more raters.
For my research project, we are using picture ratings (ratings of the extent to which an individual in the picture is dominant, trustworthy, etc) to predict another variable.
To do so, we had 60 participants rate the picture on several traits (e.g., dominant, trustworthy, attractive, etc). Overall we had 500+ pictures to rate and, being afraid of the participant fatigue, we had each individual rate only a few of the pictures. As such, we ended up with a dataset where each picture has been rated by 3-8 raters. Now we need to calculate interrater reliability.
The standard cronbach's alpha is not an option because it requires each picture being rated by two of the same raters (I think). The only metrics I found that might potentially work are Krippendorff's alpha and icc1, both of which have relaxed assumption about the number of raters and whether the ratings are dependent (ie. in other words, they allow for raters to be independent and different from picture to picture). For both of this, a script can be downloaded from Andrew Haey's website for SPSS.
The difficulty for me is a) reconfiguring the data from its current format and b) not knowing whether variable number of raters is allowed for each picture or whether each picture has to be rated by the same number of raters - if the latter, I'd be fine with deleting raters beyond 3 for each picture such that each picture is rated by the the 3 raters which provided most consistent ratings. If you could do that as well as obtain either Krippendorff's alpha or icc, it would be great.
I am attaching a sample of the data. The first column indicates the picture number. As you can see each picture is rated by 3 or more raters.