Agreement of exposure assessors, heuristics and factors affecting exposure in a desktop exercise to evaluate exposure to asthmagens in the cotton and textile, and other industries

In the absence of personal exposure measurements, expert assessment, generally on a case-by-case basis, is often used to estimate exposures. However, the decision processes of individual experts when making assessments is unknown, making it difficult to assess the quality of these assessments or to compare different assessments to each other. We conducted a study combining quantitative estimation with qualitative methods to investigate the mental decision making processes and heuristics used in exposure assessment of experts and non-experts in primarily the textile and cotton industries, but also in baking, metal work and agriculture industries. This study focusses on the quality of the desktop exposure assessment exercise and the agreement between the independent experts.
Intra-Class Correlations of absolute agreement (ICC1) and consistency (ICC3) between raters were calculated; stratified for several important factors that could influence assessments, while further sensitivity and specificity were estimated using a probabilistic simulation methodology.
14 occupational hygienists and exposure assessors with complete data for all 48 job descriptions 8 non-experts participated. Absolute agreement between expert raters was fair-good, but was somewhat better for intensity (ICC1=0.61) than for probability (ICC1=0.44) of exposure and was better for experts than non-experts. Sensitivity was 0.95 and specificity 0.82 for intensity, while these were 0.91 and 0.78 for probability of exposure, respectively. Agreement between raters (both expert on non-experts) dropped for medium levels of information compared to little or extensive information. Inclusion of a photo or video generally improved agreement between experts but not between non-experts, while also for the experts the year of the job description had no influence on the assessments whereas non-experts agreed stronger on jobs in the more distant past.
These data indicate that that the desktop exposure assessment exercise was of similar quality to previously reported levels of agreement. As such, agreements between experts’ assessments were independent of the time period, providing confidence in retrospective exposure assessment exercises of long latency diseases, and can be improved by inclusion of visual material. Importantly, agreement between experts and non-experts does not linearly increase with the detail of provided job information.