Combining Neural Network Processors and DSPs to Achieve the Best AI Performance

As shorthand for computational performance capabilities to process AI models, Giga Operations Per Second (GOPs) or Tera Operations Per Second (TOPS), where 1 TOPS equals 1000 GOPS, are often used. TOPS is, at best, a first order metric which provides a relative ideal performance capability of a processor. For AI processing, TOPS is usually compared for INT8 calculations (20 INT16 TOPS would require much more memory and data movement than 20 INT8 TOPS). You can see in Figure 1 that there is a wide range of performance requirements for different AI applications. 

TOPS is calculated by taking the number operations that can be done in one-cycle (with a multiply-accumulate counting as two operations) and multiplies it by the maximum frequency. This is a good first estimate on performance since the bulk of the computations are driven by the need for matrix multiplies, which require multiply-accumulate operations. A CPU with DSP extensions that can perform one MAC per clock cycle and runs at 2 GHz, would have a TOPS of 2 GHz x 2 operations (multiply, accumulate) x 1 MAC/cycle = 4 GOPS or 0.004 TOPS. Table 1 shows the relative performance you can expect. Neural processing units (NPUs) are clearly the best choice for the highest computational results.

Table 1 – Approximate performance ranges by processor type

Although not on the list, GPUs can also provide high levels of performance, but at a much higher power and area cost that are hard for real-time applications to absorb. In fact, each processor type in the table requires different levels of power and area to achieve the desired TOPS. For real-time applications, power (sometimes better expressed as thermal impact) and area (directly related to cost and manufacturability) are almost equally as important as performance. NPUs have been designed and optimized to be the most performance, power and area efficient processors to execute neural network algorithms.

Not every AI application needs the highest levels of neural network performance that an NPU provides. Referring to Figure 1, if you have a microcontroller or CPU with DSP extensions or a vector DSP in your system already, it might be perfectly fine to perform lower TOPS neural network tasks in between other typical processing tasks. If you need a mix of some AI performance – say less than 1 TOPS – as well as the non-AI processing capabilities of a CPU or DSP, they would be good choices for your application. Above 1 TOPS, the AI performance efficiency, power efficiency and area efficiency of an NPU is hard to argue for real-time applications. 

An NPU’s best-in-industry efficiency comes from the significant number of multiplies it can do each cycle plus some dedicated hardware for other neural network operations like activation functions. The challenge for an NPU is to have as much hardware acceleration as possible to maximize neural network efficiency while maintaining some amount of programmability. Certainly, a fully hardwire neural network ASIC would be more efficient than a programmable NPU. However, with the unrelenting pace of academic research into AI producing new techniques and models and activation functions, maintaining some programmability is critical given the multi-year cycle to produce an AI SoC.

An NPU is a dedicated neural processor engine, while a CPU or DSP can perform AI as an additional task. Since an NPU only performs AI, it is recommended for AI requirements over 1 TOPS to combine a vector DSP with an NPU. This provides the maximum performance with additional programmability. For example, in an autonomous vehicle, while the NPU is looking for pedestrians or identifying street signs or using neural networks for part of the radar or LiDAR processing, a vector DSP in the system is needed for additional filtering, radar or LiDAR processing as well as pre- and post-processing for the NPU.

In Figure 2, you can see the range of possibilities for combining NPUs and vector DSPs in AI applications. In all three cases, a high-resolution image frame is sitting in DDR memory waiting to be processed before the next frame arrives. In the first configuration, on the left, a vector DSP by itself can be used for DSP processing and some amount of AI processing – consider this the <1 TOPS use case: big DSP, small AI. A specific example of this use case is a vector DSP performing sensorless field-oriented control (FOC) for a permanent magnet synchronous motor (PMSM). DSP-based motor control is extended with AI processing, which performs position monitoring that is fed back into the control loop. The sampling rate and computational complexity of the AI model allow it to fit with the AI capabilities of a vector DSP.

In the second configuration, in the middle, the AI SoC requires both a large amount of vector DSP performance and AI performance:big AI, big DSP. While the vector DSP is processing DSP-heavy tasks, it needs to be supplemented with the neural network acceleration provided by an NPU for the AI-intensive tasks. One example use case is a digital still camera. The vector DSP can perform vision processing and pre- and post-processing support of the NPU, while the NPU is dedicated to CNN or transformer processing (object detection, semantic segmentation, super resolution, etc.) on high-resolution images. These use cases require vector DSP and NPU solutions that are closely integrated and can scale to fit the performance targets. 

In the third configuration, on the right, all the processing is focused on neural networks and the DSP processing is only needed for NPU support: small DSP, big AI.  The NPU is needed to process neural networks which often can be completely executed within the NPU. Standard neural network models or graphs like ResNet-50, MobileNet v2, etc., can run entirely on Synopsys’ ARC® NPU. There are some more complicated neural network models with floating point operations requiring support from the vector DSP. For example, the ROI pooling and ROI alignment for Mask-RCNN or the non-integer scale factors use by Deeplab v3. Even if the AI SoC doesn’t need any additional DSP processing, there is still a benefit to include some amount of vector DSP performance to support the NPU for future proofing. 

While there are multiple options for vector DSPs and NPUs on the market, for the second and third configuration types, it is best to choose your AI solution to include closely integrated processors. Some neural network accelerators embed a vector DSP into their neural network solution limiting its use for external programming. Synopsys’ ARC EV7x vision processors are heterogeneous processors closely coupling vector DSPs with option neural network engines. To increase customer flexibility and programmability, the ARC EV7x family is evolving into the ARC VPX family of vector DSPs and the ARC NPX family of NPUs. The VPX and NPX are a closely coupled solution for AI. Figure 3 shows the high-level block diagrams of these two processors and how they are interconnected.