On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays

Yanan Liu, Laurie Bose, Rui Fan, Piotr Dudek, Walterio Mayol-Cuevas

Research output: Contribution to journalArticlepeer-review

Abstract

Many types of Convolutional Neural Network (CNN) models and training methods have been proposed in recent years aiming to provide efficiency for embedded and edge devices with limited computation and memory resources. The wide variety of architectures makes this a complex task that has to balance generality with efficiency. Among the most interesting camera-sensor architectures are Pixel Processor Arrays (PPAs). This study presents two methods that are useful for embedded CNNs in general but particularly suitable for PPAs. The first is for training purely binarized CNNs, the second is for deploying larger models with a model swapping paradigm that loads model components dynamically. Specifically, this study trains and implements networks with batch normalization and adaptive threshold for binary activations. Then, we convert batch normalization and binary activations into a bias matrix which can be parallelly implemented by an add/sub operation. For dynamic model swapping, we propose to decompose applications that are beyond the capacity of a PPA into sub-tasks that can be solved by tree networks that can be loaded dynamically as needed. We demonstrate our approaches to various tasks including classification, localization, and coarse segmentation on a highly resource constrained PPA sensor-processor.

Original languageEnglish
Article number909448
JournalFrontiers in Neuroscience
Volume16
DOIs
Publication statusPublished - 15 Aug 2022

Keywords

  • convolutional neural network
  • embedded computer vision
  • on-sensor computing
  • pixel processor array
  • SCAMP vision system

Fingerprint

Dive into the research topics of 'On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays'. Together they form a unique fingerprint.

Cite this