Team Performance in a Simulated Obstetrical Emergency: Interrater Reliability of the Communication and Team Skills (CATS) Assessment Tool
Abstract Number: 164
Abstract Type: Original Research
BACKGROUND: The Communication and Team Skills (CATS) assessment tool is a behavior-based metric for assessing team skills in real or simulated clinical events involving patients in health care settings. The CATS tool was designed to provide quantifiable objective information about team performance and identify specific behaviors contributing to ineffective teamwork. Such information can help inform design of educational endeavors that promote effective communication and teamwork skills. An assessment tool that is reliable and has sound psychometric properties is essential to this process.
OBJECTIVES: To assess the interrater reliability and utility of the CATS tool when used to assess perinatal team performance during a simulated obstetrical emergency.
METHODS: Four raters, 2 obstetric anesthesiologists and 2 obstetrician-gynecologists, with expertise in team training and obstetrical simulation independently observed 40 randomly selected videotaped cases of multi-disciplinary perinatal teams who managed the same simulated obstetrical emergency. Team performance was scored using the CATS tool and interrater reliability was assessed with intraclass correlation. Correlation between the total CATS scores and a global Team Performance Assessment (global TPA) score was also explored.
RESULTS: Interrater reliability was acceptable for the total CATS score (intraclass correlation = 0.73 for 4 judges). Total CATS scores for raters of the same specialty correlated highest with each other (Spearman’s coefficient rho = 0.95; p < 0.01). Interrater reliability was acceptable for global TPA score (intraclass correlation=0.84 for 4 judges); and correlations were high (Spearman’s rho = 0.95, p < 0.01) between global TPA and total CATS Scores for all four raters.
CONCLUSIONS: The CATS tool was found to be reliable, both for the total CATS scores and the global team performance ratings, when assessed by experienced raters. Correlation between total CATS scores and global TPA ratings indicated concurrent validity. This study is an essential step towards fully characterizing psychometric properties of the CATS tool and its utility in assessing perinatal team performance in simulated and real healthcare environments.
1. Frankel A, Gardner R, Maynard L, Kelly A. Using the Communication and Teamwork Skills
(CATS) Assessment to Measure Health Care Team Performance. Jt Comm J Qual Patient Saf. 2007; 33(9): 549-558.