Abstract
Background
The conduct of the OSCE exams in the Division of Dentistry, The University of Manchester underwent major changes in 2014 in the sense that global marking was introduced to allow setting the pass score using borderline regression analysis (BRA). Due to the nature of the changes, training and calibration of the OSCE examiners became paramount; however, due to the time-strain and occupational commitments of the OSCE examiners, an online training and calibration package was developed and launched and the examiners were advised to use it prior to the exam.
Since the latter was a “desirable” exercise, a number of the examiners chose to do the training while some chose not to. This gave us a unique opportunity to have a large dataset of two groups: calibrated (Cal) and non-calibrated (nCal) examiners. What we did not know was that if the exposure to online training had an effect on the pass mark of the stations set by BRA.
Methodology
A panel of six determined the pass score for each station using the modified Angoff (mAngoff) method. This score was used as a point of reference for the future analyses. The objective and global score data for the Cal (study) and nCal (control) groups were collected. Using BRA, the pass mark for each station was calculated. To take the station difficulty into consideration, the difference between the BRA and mAngoff pass marks were calculated for each examiner.
Result
The difference between the pass marks set by Cal and nCal groups was not statistically different when the BRA was used.
Conclusion
Although training and calibration of OSCE examiners is considered good practice; however, when the BRA method of standard setting is used, we can potentially rely on the professional judgement of the examiners, regardless of their training history.
The conduct of the OSCE exams in the Division of Dentistry, The University of Manchester underwent major changes in 2014 in the sense that global marking was introduced to allow setting the pass score using borderline regression analysis (BRA). Due to the nature of the changes, training and calibration of the OSCE examiners became paramount; however, due to the time-strain and occupational commitments of the OSCE examiners, an online training and calibration package was developed and launched and the examiners were advised to use it prior to the exam.
Since the latter was a “desirable” exercise, a number of the examiners chose to do the training while some chose not to. This gave us a unique opportunity to have a large dataset of two groups: calibrated (Cal) and non-calibrated (nCal) examiners. What we did not know was that if the exposure to online training had an effect on the pass mark of the stations set by BRA.
Methodology
A panel of six determined the pass score for each station using the modified Angoff (mAngoff) method. This score was used as a point of reference for the future analyses. The objective and global score data for the Cal (study) and nCal (control) groups were collected. Using BRA, the pass mark for each station was calculated. To take the station difficulty into consideration, the difference between the BRA and mAngoff pass marks were calculated for each examiner.
Result
The difference between the pass marks set by Cal and nCal groups was not statistically different when the BRA was used.
Conclusion
Although training and calibration of OSCE examiners is considered good practice; however, when the BRA method of standard setting is used, we can potentially rely on the professional judgement of the examiners, regardless of their training history.
| Original language | English |
|---|---|
| Publication status | Published - 23 Aug 2017 |
| Event | Association of Dental Education in Europe - Vilnius University, Vilnius, Lithuania Duration: 23 Aug 2017 → 25 Aug 2017 http://www.adee.org/meetings/vilnius2017/ |
Conference
| Conference | Association of Dental Education in Europe |
|---|---|
| Abbreviated title | ADEE |
| Country/Territory | Lithuania |
| City | Vilnius |
| Period | 23/08/17 → 25/08/17 |
| Internet address |