Abstract
Background
Portfolios are commonly used in healthcare education; they allow students to showcase their experience and learning journey whilst allowing them to demonstrate reflective practice, ethical principles and self-directed learning. The large volume of information contained within portfolios can, however, make them onerous and difficult to mark and present challenges in marking consistency.
This study analysed several marking strategies to ascertain the optimum that achieved a similar outcome to marking in its entirety, and if so what is the optimal sampling interval to ensure validity, reliability and fairness whilst ensuring time efficient marking.
Summary of Work
2 cohorts of student portfolios (n=34) were analysed by a researcher who had no prior knowledge of the student’s work using a marking rubric of 10 domains, each marked using a 0-4 scale. Each portfolio consisted of an average 5 entries per week across 2 semesters. The portfolios were marked randomly one week every 3 months and then using progressively shorter intervals until every week had been marked. The progression of marks over time for each domain was also investigated.
Summary of Results
Progression analysis identified when student marks plateaued, which identified differences across domains and across cohorts.
Pairwise analysis of the different assessment intervals found that lowest marking interval that did not impact results was to mark 6 random weeks in every 3-month period. This achieved results that were not significantly different to marking every entry (p>0.2 – p=1.0), agreement was higher for clinical and practical skill domains than higher order skills.
Discussion and Conclusion
Based on the student portfolio sample, the most time efficient marking interval that maintained acceptable levels of validity, reliability, fairness was 6 weeks every 3 months. This amounted to 35% of the total portfolio. Care however must be taken when assessing higher order skills with more weight given to the later part of the portfolio.
Take home message
Portfolio marking should be standardised across markers using a rubric and specified marking intervals identified as appropriate for the programme.
Portfolios are commonly used in healthcare education; they allow students to showcase their experience and learning journey whilst allowing them to demonstrate reflective practice, ethical principles and self-directed learning. The large volume of information contained within portfolios can, however, make them onerous and difficult to mark and present challenges in marking consistency.
This study analysed several marking strategies to ascertain the optimum that achieved a similar outcome to marking in its entirety, and if so what is the optimal sampling interval to ensure validity, reliability and fairness whilst ensuring time efficient marking.
Summary of Work
2 cohorts of student portfolios (n=34) were analysed by a researcher who had no prior knowledge of the student’s work using a marking rubric of 10 domains, each marked using a 0-4 scale. Each portfolio consisted of an average 5 entries per week across 2 semesters. The portfolios were marked randomly one week every 3 months and then using progressively shorter intervals until every week had been marked. The progression of marks over time for each domain was also investigated.
Summary of Results
Progression analysis identified when student marks plateaued, which identified differences across domains and across cohorts.
Pairwise analysis of the different assessment intervals found that lowest marking interval that did not impact results was to mark 6 random weeks in every 3-month period. This achieved results that were not significantly different to marking every entry (p>0.2 – p=1.0), agreement was higher for clinical and practical skill domains than higher order skills.
Discussion and Conclusion
Based on the student portfolio sample, the most time efficient marking interval that maintained acceptable levels of validity, reliability, fairness was 6 weeks every 3 months. This amounted to 35% of the total portfolio. Care however must be taken when assessing higher order skills with more weight given to the later part of the portfolio.
Take home message
Portfolio marking should be standardised across markers using a rubric and specified marking intervals identified as appropriate for the programme.
| Original language | English |
|---|---|
| DOIs | |
| Publication status | Published - 2023 |
| Event | 2023 Association for Medical Education in Europe conference: Transforming health care education through inclusivity and innovation - Glasgow, United Kingdom Duration: 26 Aug 2023 → 30 Aug 2023 |
Conference
| Conference | 2023 Association for Medical Education in Europe conference |
|---|---|
| Abbreviated title | AMEE 2023 |
| Country/Territory | United Kingdom |
| City | Glasgow |
| Period | 26/08/23 → 30/08/23 |