Towards Biologically-Plausible Neuron Models and Firing Rates in High-Performance Deep Spiking Neural Networks

Spiking neural networks (SNNs) have the potential to reach the same accuracy level as their counterpart artificial neural networks (ANNs) through ANN-to-SNN conversion. However, this relies sig- nificantly on artificially modifying the voltage reset mechanism in the Integrate-and-Fire (IF) neuron model to support bias and batch normalization. This paper shows that our proposed MCR-Norm (minimum chain rule normalization) can enable the modeling of these two key elements by using the standard IF model. The SNNs built by the IF model can thus reach the same accuracy level as the SNNs built by the artificially modified IF model. While previous approaches tended to push deep SNNs towards very high firing rates, we found that the IF neuron is suitable to run in a low firing rate range. This is in line with biological observations and is also crucial to go beyond SNN simulation and apply it to real neuro- morphic hardware. In addition, tuning the neural network inputs is dismissed in earlier work, but we claim that it is closely related to the success of parameter normalization inside the SNNs built by the standard IF model, and can be integrated naturally into MCR-Norm.