An Energy-Efficient CMOS Image Sensor with Embedded Machine Learning Algorithm

Abstract

In this thesis, an energy-efficient CMOS (Complementary Metal–Oxide Semiconductor) image sensor for embedded machine-learning algorithms has been studied to provide low-power consumption, minimized hardware resources, and reduced data bandwidth in both digital and analog domains for power-limited applications. In power-limited applications, image sensors for embedded machine learning algorithms typically have these challenges to address: low-power operation from limited energy sources such as batteries or energy harvesting units, large hardware area due to complicated machine-learning algorithms, and high-data bandwidth due to video streaming images and large data movement for evaluating the machine-learning algorithms. This research focuses on developing the architectures, algorithm optimization, and associated electronic circuits for an energy-efficient CMOS image sensor with the embedded machine-learning algorithms. Three interdependent prototypes have been developed to solve major challenges: minimization of energy consumption and hardware resources while preserving a high degree of precision in machine-learning algorithm evaluation. Three prototypes have been fabricated and fully characterized to address these challenges. In the first chip, we implemented 2 bit spatial difference imaging, a customized look-up table (LUT) based gradient orientation assignment, and a cell-based supporting-vector-machine (SVM) to achieve both low-data bandwidth and higher area efficiency for a histogram-of-oriented-gradient (HOG)-based object detection. The proposed HOG-based object detection core operates with the 2D optic flow core to provide the vision-based navigation functionality for the nano-air-vehicle (NAV) application. The system operates at 244 pJ/pixel in 2D optic flow extraction mode and at 272.49 pJ/pixel in hybrid operation mode, respectively. The system achieved 75% reduction in memory size with proposed HOG feature extraction method and cell-based supporting-vector-machine (SVM). In the second chip, a mixed-mode approximation arithmetic multiplier-accumulator (MAC) is built to reduce power consumption for the most power-hungry component in a convolution neural network image sensor. The proposed energy-efficient convolution neural networks (CNN) imager operation is as follows. The pixel array gathers photons and converts them to electrons. The individual pixel values are transferred to a column-parallel mixed-mode MACs in a rolling shutter fashion. The column-parallel mixed-mode MACs conduct the convolution operation in the analog-digital mixed-mode signal domain. Each convolution layer in the neural network is processed in a pipeline fashion. In the last stage, an analogue-to-digital converter (ADC) converts the result of the MACs operation to digital signals. Consequently, the column-parallel mixed-mode MACs and the pipeline operation allow the imaging system to achieve real-time imaging with low-power operation during runtime. The system operates at 5.2 nJ/pixel in normal image extraction mode and at 4.46 GOPS/W in a convolution neural network (CNN) operation mode, respectively. In the third chip, a self-sustainable CMOS image sensor with concurrent energy harvesting and imaging has been developed to extend the operation time of the machine-learning imager in the energy-limited environment. The proposed CMOS image sensor employs a 3T pixel which deploys vertically both hole-accumulation photodiode and energy harvesting diode in the same pixel to achieve a high fill-factor (FF) and high-energy harvesting efficiency. The sensor achieved -13.9 pJ/pixel at 30 Klux (normal daylight), 94% FF for the energy harvesting diode, and 47% FF for the imaging sensing diode.