Sensor fusion for analysis of gait under cognitive load: deep learning approach

Human mobility requires substantial cognitive resources, thus elevated complexity in the navigated environment instigates gait deterioration due to naturally limited cognitive load capacity. This work uses deep learning methods for 116 sensors fusion to study the effects of cognitive load on human gait of healthy subjects. We demonstrate classifications, achieving 86% precision with Convolutional Neural Networks (CNN), of normal gait as well as 15 subjects’ gait under two types of cognitive demanding tasks. Floor sensors capturing multiples of up to 4 uninterrupted steps were utilized to harvest the raw gait spatiotemporal signals, based on the ground reaction force (GRF). A Layer-Wise Relevance Propagation (LRP) technique is proposed to interpret the CNN prediction in terms of relevance to standard events in the gait cycle. LRP projects the model predictions back to the input gait spatiotemporal signal, to generate a “heat map” over the original training set, or an unknown sample classified by the model. This allows valuable insight into which parts of the gait spatiotemporal signal have the heaviest influence on the gait classification and consequently, which gate events are mostly affected by cognitive load.