Abstract
BACKGROUND: Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.
OBJECTIVES: We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.
METHODS: As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.
RESULTS: Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95).
CONCLUSIONS: We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
| Original language | English |
|---|---|
| Pages (from-to) | e10601 |
| Journal | AEM Education and Training |
| Volume | 5 |
| Issue number | 3 |
| DOIs | |
| Publication status | Published - Jul 2021 |
Access to Document
Fingerprint
Dive into the research topics of 'The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver
}
In: AEM Education and Training, Vol. 5, No. 3, 07.2021, p. e10601.
Research output: Contribution to journal › Article › peer-review
TY - JOUR
T1 - The revised Approved Instructional Resources score
T2 - An improved quality evaluation tool for online educational resources
AU - Grock, Andrew
AU - Jordan, Jaime
AU - Zaver, Fareen
AU - Colmers-Gray, Isabelle N
AU - Krishnan, Keeth
AU - Chan, Teresa
AU - Thoma, Brent
AU - Alexander, Charlotte
AU - Alkhalifah, Mohammed
AU - Almehlisi, Abdulaziz S
AU - Alqahtani, Saeed
AU - Anderson, Scott
AU - Anderson, Shelaina
AU - Andrews, Colin
AU - Andruko, Jocelyn
AU - Antony, Nikytha
AU - Aryal, Diptesh
AU - Backus, Barbra
AU - Baird, Jennifer
AU - Baker, Andrew
AU - Batty, Sarah
AU - Baylis, Jared
AU - Beaumont, Braeden
AU - Belcher, Chris
AU - Benavides, Brent
AU - Benham, Michael
AU - Botta, Julian
AU - Berger Pelletier, Elyse
AU - Bouchard, Nicholas
AU - Brazil, Victoria
AU - Brumfield, Emily
AU - Bryson, Anthony
AU - Bunchit, Wisarut
AU - Butler, Kat
AU - Buzikievich, Lindy
AU - Calcara, David
AU - Carey, Rob
AU - Carrillo, Maria Rosa
AU - Carroll, Stephen
AU - Lyons, Casey
AU - Cassidy, Louise
AU - Challen, Kirsty
AU - Chan, Kathryn
AU - Chaplin, Tim
AU - Chatham-Zvelebil, Natasha
AU - Chen, Eric
AU - Chen, Lucy
AU - Chhabra, Sushant
AU - Chin, Alvin
AU - Chochi, Eric
AU - Choudhri, Tina
AU - Christensen, Jeremy
AU - Connors, Kimberly
AU - Coppersmith, Veronica
AU - Cosgrove, Abby
AU - Costello, Gregory
AU - Cullison, Kevin
AU - D'Alessandro, Andrew
AU - Wit, Kerstin
AU - Decock, Marie
AU - Delbani, Rayan
AU - Denq, William
AU - Deutscher, Julianna
AU - Devine, Brendan
AU - Dorsett, Maia
AU - Duda, Taylor
AU - Dueweke, Justin
AU - Dunphy, Teresa
AU - Dyer, Sean
AU - Eastley, Karthryn T
AU - Edmonds, Marcia
AU - Edwards, Ken
AU - Ehrman, Robert
AU - Elkhalidy, Youness
AU - Fedor, Preston
AU - Ficiur, Brian
AU - Flynn, Caley
AU - Fraser, Bill
AU - Fu, Meagan
AU - Fukakusa, James
AU - Funk, Eric
AU - Gaco, Damjan
AU - Gawlik, Viktor
AU - Ghaffarian, Kenn
AU - Gharahbaghian, Laleh
AU - Griffith, Andrew
AU - Griffith, Phil
AU - Gronowski, Tanner
AU - Grossman, Cathy
AU - Gucwa, Jaroslaw
AU - Gupta, Pawan
AU - Gustafson, Alexandra
AU - Guy, Andrew
AU - Haas, Mary
AU - Haciski, Stanislaw
AU - Hajdinjak, Emina
AU - Hall, Andrew K
AU - Hammock, Regina
AU - Hansel, Jan
AU - Hart, Alexander
AU - Hattin, Larissa
AU - Herb, Brandon
AU - Hilbert, SueLin
AU - Hill, Jeff
AU - Hill, Jesse
AU - Ho, Amy
AU - House, Emily
AU - House, Nina
AU - Huang, Simon York Ming
AU - Huffman, James
AU - Inboriboon, Charlie
AU - Ireland, Alex
AU - Jamal, Ali
AU - Jamil, Mohammad Ali
AU - Jansen, Victor
AU - Jarou, Zach
AU - Jia, Vivian
AU - Johnston, Levi
AU - Kalnow, Drew
AU - Kapur, Puneet
AU - Kelly, Seth
AU - Kelson, Kyle
AU - Kent, William
AU - Khakhkhar, Rishi
AU - Khurana, Jaasmit
AU - Kilp, Ashley
AU - Knapp, Scott
AU - Kohler, Sebastian
AU - Kruhlak, Ivanna
AU - Lalani, Nadim
AU - Lam, Samantha
AU - Lank, Patrick
AU - Laurie, Zander
AU - Lea, Kristina
AU - Leber, Ernest
AU - Lee, Ching-Hsing
AU - Lenes, Haakon
AU - Lenora, Nilantha
AU - Leontowicz, Jesse
AU - Lien, Kelly
AU - Lin, Michelle
AU - Lin, Yingchun
AU - Little, Andrew
AU - Liu, Harry
AU - Liu, Ivy
AU - Liu, Steve
AU - Louka, Stephanie
AU - Lovell, Elise
AU - Lowe, David
AU - Lubberdink, Ashley
AU - Luc, Jessica
AU - Ma, Sheng-Hsiang
AU - MacLeod, Hugh
AU - Mancuso, Nick
AU - Maneshi, Anali
AU - May, Jesse
AU - Mayo, John
AU - McDonnell, Mike
AU - McLellan, Susan
AU - McQuarrie, Carolyn
AU - Mead, Therese
AU - Meeuwisse, Cory
AU - Meloy, Patrick
AU - Menzies, Perry
AU - Messman, Anne
AU - Miazga, Stephen
AU - Mills, Logan
AU - Mix, Allan
AU - Montag, Steve
AU - Moore, Brendan
AU - Morgenstern, Justin
AU - Mott, Sarah
AU - Mukherj, P
AU - Mulla, Ali
AU - Nandalal, Sheena
AU - Nikel, Taylor
AU - Nood, Julia
AU - Nugent, Sean
AU - Oakland, Morgan
AU - Oberholzer, Werner
AU - Otugo, Onyeka
AU - Oyedokun, Taofiq Segun
AU - Pardhan, Alim
AU - Patel, Kinjal
AU - Paterson, Quinten
AU - Patocka, Catherine
AU - Patterson, Christine
AU - Pearlman, James
AU - Pelletier-Bui, Alexis
AU - Phan, Marc
AU - Poonja, Zafrina
AU - Powell, Aubrey
AU - Premkumar, Kamini
AU - Prosen, Gregor
AU - Puri, Vishal
AU - Quaife, Tanis
AU - Raffel, Ryan
AU - Raja, Ali
AU - Ramunno, Randi
AU - Rang, Louise
AU - Rannazzisi, Suzanne
AU - Regan, Shauna
AU - Rezaie, Salim R
AU - Ridderikhof, Milan
AU - Rogers, Vanessa
AU - Roh, Christine
AU - Rosenberg, Keith
AU - Roure, Marina
AU - Rudinsky, Sherri
AU - Rudner, Joshua
AU - Saleh, Adeeb
AU - Sanderson, Will
AU - Scheirer, Owen
AU - Schofield, Paul
AU - Schunk, Paul
AU - Schwarz, Evan
AU - Shahrabadi, Parisa
AU - Shappell, Eric
AU - Sheffield, Julia
AU - Singh, Manpreet
AU - Singson, Hector C
AU - Slessor, Dave
AU - Smith, Sam
AU - Sneath, Paula
AU - Sobehart, Robert
AU - Spearing, Kerry
AU - Stempien, James
AU - Sternard, Britni
AU - Stratton, Tara
AU - Stuart, Katherine
AU - Stuntz, Bob
AU - Susalla, Michael
AU - Sweeney, Colleen
AU - Swisher, Loice
AU - Swoboda, Henry
AU - Syed, Shahbaz
AU - Taira, Taku
AU - Tambe, Nikhil
AU - Tang, Richard
AU - Targonsky, Elisha
AU - Taylor, Alan
AU - Taylor, Rachel
AU - Taylor, Todd
AU - Ting, Paxton
AU - Tiwald, Gerhard
AU - Tran, Evelyn
AU - Tran, Kelvin
AU - Trickovic, Jason
AU - Trinquero, Paul
AU - Tyagi, Aaron
AU - Umana, Manrique
AU - Vallance, Patrick
AU - Van den Berg, Patricia
AU - van Diepen, Kelly
AU - Vargas, Luis
AU - Verbeek, Rene
AU - Viggers, Sandra
AU - Vlodaver, Zlata
AU - Wagner, Matthew
AU - Walji, Noorin
AU - Walter, Joe
AU - Wan, Miranda
AU - Wang, Rachel
AU - Wanner, Gregory
AU - Warawa, Wyatt
AU - Ward, Mike
AU - Weekes, Jennifer
AU - Weersink, Kristen
AU - Weessies, Cara
AU - Whalen-Browne, Anna
AU - Whiteside, Brian
AU - Willis, Matthew
AU - Wilmer, Jonas
AU - Wong, Nelson
AU - Woodcroft, Mark
AU - Woods, Rob
AU - Yau, Lawrence
AU - Yee, Jessica
AU - Yeh, Calvin
AU - Yurkiw, Katherine
AU - Zaver, Fareen
AU - Zozula, Alexander
N1 - © 2021 by the Society for Academic Emergency Medicine.
PY - 2021/7
Y1 - 2021/7
N2 - BACKGROUND: Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.OBJECTIVES: We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.METHODS: As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.RESULTS: Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95).CONCLUSIONS: We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
AB - BACKGROUND: Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.OBJECTIVES: We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.METHODS: As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.RESULTS: Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95).CONCLUSIONS: We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
U2 - 10.1002/aet2.10601
DO - 10.1002/aet2.10601
M3 - Article
C2 - 34141997
SN - 2472-5390
VL - 5
SP - e10601
JO - AEM Education and Training
JF - AEM Education and Training
IS - 3
ER -