The University of Science has just announced an exciting development: a binarised neural network (BNN) that uses ternary gradients to tackle the computing challenges faced by IoT edge devices.
This breakthrough has real potential for powerful IoT devices that can harness artificial intelligence (AI) in new ways. Imagine wearable health-monitoring devices that are smaller, more reliable, and don’t need constant cloud connectivity. Picture smart homes that can handle more complex tasks and respond faster. Plus, this new design could lower energy consumption, helping meet sustainability goals.
At the core of this innovation is a magnetic random access memory (RAM)-based computing-in-memory architecture. The research team claims it significantly cuts down circuit size and power use. They have achieved almost the same accuracy and quicker training times compared to traditional BNNs, making it a strong contender for effective AI on resource-limited devices like those in IoT.
The researchers shared that the last decade has seen rapid advancements in two main areas: AI and IoT. Many believe we’re moving toward a world where devices are everywhere, forming the backbone of a highly interconnected society. However, integrating AI capabilities into IoT edge devices poses a major challenge because artificial neural networks (ANNs), from which BNNs are derived, typically need a lot of computing power.
IoT edge devices, on the other hand, are small, with limited power and processing capacity. Finding a way to make ANNs operate efficiently on these devices has been a significant roadblock.
In their latest paper published in IEEE Access, Takayuki Kawahara and Yuya Fujiwara from the Tokyo University of Science explained their approach to this problem. They introduced a training algorithm for BNNs along with a fresh implementation of this algorithm within a computing-in-memory (CiM) architecture suitable for IoT devices.
Kawahara described BNNs as a type of ANN that uses weights and activation values of just -1 and +1, significantly cutting down on the computational resources required. But when it comes to learning, weights and gradients are real numbers, which complicates matters, especially for IoT edge devices.
To tackle this, the research team developed the ternary gradient BNN (TGBNN) algorithm with three main innovations. First, it uses ternary gradients during training while keeping weights and activations binary. Second, they improved the straight-through estimator (STE) to manage gradient backpropagation better. Finally, they incorporated a probabilistic method for updating parameters using the behavior of magnetic RAM (MRAM) cells.
Next, they implemented the TGBNN algorithm in a CiM architecture, where calculations happen directly in memory, saving both circuit space and power. To achieve this, they created a new XNOR logic gate as a building block for a MRAM array, leveraging magnetic tunnel junctions to store information.
According to Kawahara, their ternarised gradient BNN exceeded 88% accuracy with Error-Correcting Output Codes-based learning. It matched the accuracy of standard BNNs of the same structure and showed quicker convergence during training. They are optimistic that this design will make efficient BNNs viable on edge devices, allowing them to learn and adapt effectively.