Index to Chiropractic Literature
Index to Chiropractic Literature
My ICL     Sign In
Tuesday, April 7, 2020
Index to Chiropractic LiteratureIndex to Chiropractic LiteratureIndex to Chiropractic Literature

ICL Home

For best results switch to Advanced Search.
Article Detail
Return to Search Results
ID 18764
  Title Applying Generalizability Theory to High-stakes Objectively Structured Competency Examinations in a Naturalistic Environment [platform presentation; the Association of Chiropractic Colleges' Thirteenth Annual Conference, 2006]
Journal J Chiropr Educ. 2006 Spring;20(1):30
Peer Review Yes
Publication Type Meeting Abstract
Abstract/Notes BACKGROUND: Objective structured competency examinations (OSCEs) are commonly used for licensure decisions and competency assessments. OSCEs have many sources of variance (candidates, raters, patients, and stations). Generalizability Theory (G-Theory) has been applied to OSCEs in artificial environments where raters are aware that their scores will be compared to others. Due to the high cost of using multiple raters, G-Theory is generally applied as a “pilot project” with a small number of candidates, raters and cases. The artificial environment ( in comparison to high-stakes OSCEs) makes it difficult to generalize the results of the pilot project to the actual testing setting (a naturalistic environment).

OBJECTIVE: The purpose of this project was to determine if G-Theory could be successfully applied to a high-stakes, licensure OSCE as part of its normal administrative procedures and whether the analysis could yield useful information with regard to sources of variance.

METHODS: The anonymous data received from the Canadian Chiropractic Examining Board for its June 2005 Clinical Skills Examination were analyzed with G-Theory. Variance components were estimated with SPSS 11.5 as partially nested data. The data included 182 candidates, 43 raters, 40 standardized patient actors, and 18 individual cases. The examinations were administered twice per day for two days, with 4 parallel tracks of 10 stations each. Raters filled in checklists in each station using three ratings scales: fourteen to sixteen 3-point rating scales (0=not performed, 1=performed but not correctly, 2=performed correctly), a 5-point rating scale on professionalism, and a 10-point rating scale on overall technique. Both the professionalism scale and overall technique scale were anchored at the borderline pass and borderline fail levels.

RESULTS: Both days of testing resulted in high measures of internal consistency (Alpha day 1=0.86, day 2=0.91). Internal consistency measures for each individual station averaged Alpha=0.68 for day 1 and 0.74 for day 2. Generalizability coefficients for the day 1 stations averaged 0.63 and the generalizability coefficient for the day 1 examination was 0.65. Generalizability coefficients for the day 2 stations averaged 0.74 and the generalizability coefficient for the day 2 examination was 0.42. On day 1, the raters contributed 7% of the variance of candidate measures, and the standardized patients contributed 1%. On day 2, the raters contributed 8% of the variance of candidate measure, and the standardized patient measures could not be estimated.

DISCUSSION: The application of G-Theory in the naturalistic environment can contribute to the understanding of sources of variance, and provide direction for the improvement of individual stations. D-Studies can be used to determine the effect on reliability of using multiple raters in a room. The size of the rater variance in a station may also indicate the need for increased training in that station or the need to make the scoring checklist more clear and definitive. G-theory, however, must be cautiously applied, and requires careful selection of the floating raters and vigorous training of the raters in each station.

This abstract is reproduced with the permission of the publisher.

   Text (Citation) Tagged (Export) Excel
Email To
HTML Text     Excel

To use this feature you must register a personal account in My ICL. Registration is free! In My ICL you can save your ICL searches in My Searches, and you can save search results in My Collections. Be sure to use the Held Citations feature to collect citations from an entire search session. Read more search tips.

Sign Into Existing My ICL Account    |    Register A New My ICL Account
Search Tips
  • Enclose phrases in "quotation marks".  Examples: "low back pain", "evidence-based"
  • Retrieve all forms of a word with an asterisk*, also called a wildcard or truncation.  Example: chiropract* retrieves chiropractic, chiropractor, chiropractors
  • Register an account in My ICL to save search histories (My Searches) and collections of records (My Collections)
Advanced Search Tips