CN110717575B - Frame buffer free convolutional neural network system and method - Google Patents

Abstract

The invention relates to a convolutional neural network system and a convolutional neural network method without a frame buffer. The frame buffer-less convolutional neural network system includes: a region-of-interest unit for extracting features to generate a region of interest of the input image frame; a convolutional neural network unit processing a region of interest of an input image frame to detect an object; and a tracking unit for comparing the features extracted at different times so that the convolutional neural network unit selectively processes the input image frame accordingly.

Description

Framebufferless Convolutional Neural Network System and Method

technical field

The present invention relates to a convolutional neural network (CNN), in particular to a framebufferless convolutional neural network system.

Background technique

Convolutional neural network (CNN) is a kind of artificial neural network, which can be used for machine learning. Convolutional neural networks can be applied to signal processing, such as image processing and computer vision.

Figure 1 shows a block diagram of a conventional convolutional neural network 900, which is disclosed in the "AReconfigurable Streaming DeepConvolutional Neural Network Accelerator for Internet of Things" proposed by Li Du et al. Things”, August 2017, IEEE Transactions on Circuits and Systems I: Periodic paper, the content of which is considered a part of this specification. The convolutional neural network 900 includes a buffer bank 91, which includes a single-port static random access memory (SRAM) for storing intermediate data and exchanging data with a frame buffer 92, The frame buffer 92 includes dynamic random access memory (DRAM), such as double data rate synchronous dynamic random access memory (DDR DRAM), for storing entire image frames for convolutional neural network operations. The buffer group 91 is divided into two parts: an input layer and an output layer. The convolutional neural network 900 includes a column buffer 93 for remapping the output of the buffer bank 91 to a convolution unit (CU) engine array 94 . The convolution unit engine array 94 contains a plurality of convolution units to perform highly parallel convolution operations. The convolutional unit engine array 94 includes a pre-fetch controller 941 for periodically fetching parameters from a direct memory access (DMA) controller (not shown) and updating the convolutional unit engine array 94 Weight and bias values. The convolutional neural network 900 also includes an accumulation buffer 95 with scratchpad memory for storing partial convolution results of the convolutional unit engine array 94 . The accumulation buffer 95 contains a max pool 951 to pool the output layer data. Convolutional neural network 900 includes command decoder 96 for storing commands pre-stored in frame buffer 92 .

In a conventional convolutional neural network system as shown in Figure 1, the frame buffer contains dynamic random access memory (DRAM), such as double data rate synchronous dynamic random access memory (DDR DRAM), to store the entire image frame for Convolutional neural network operations. For example, an image frame with a resolution of 320x240 takes up a frame buffer of 320x240x8 bits. However, double data rate synchronous dynamic random access memory (DDR DRAM) is not suitable for low power applications such as wearable or Internet of Things (IoT) devices. Therefore, it is urgent to propose a novel convolutional neural network system suitable for low-power applications.

SUMMARY OF THE INVENTION

In view of the above, one of the objectives of the embodiments of the present invention is to provide a frame buffer-free convolutional neural network system. This embodiment can use a simple system architecture to perform convolutional neural network operations on high-resolution image frames.

According to an embodiment of the present invention, a frame bufferless convolutional neural network system includes a region of interest unit, a convolutional neural network unit and a tracking unit. The region of interest unit extracts features to generate a region of interest of the input image frame. A convolutional neural network unit processes the region of interest of the input image frame to detect objects. The tracking unit compares the features extracted at different times, so that the convolutional neural network unit selectively processes the input image frame, wherein the region-of-interest unit generates block-based features to determine whether to perform convolutional neural network for each image block. network operations.

According to another embodiment of the present invention, a method for a framebufferless convolutional neural network includes: extracting features from which to generate a region of interest of an input image frame; performing a convolutional neural network operation on the input a region of interest of an image frame to detect objects; and comparing features extracted at different times to selectively process the input image frame, wherein the step of generating the region of interest comprises: generating block-based features to determine each Whether an image block performs convolutional neural network operations.

Description of drawings

Figure 1 shows a block diagram of a traditional convolutional neural network.

FIG. 2A shows a block diagram of a frame buffer-less convolutional neural network system according to an embodiment of the present invention.

FIG. 2B shows a flowchart of a method for a frame bufferless convolutional neural network according to an embodiment of the present invention.

FIG. 3 is a block diagram showing the specific structure of the region of interest unit of FIG. 2A .

Figure 4A illustrates a decision map, which includes 4x6 blocks.

FIG. 4B illustrates another decision diagram, which is updated after FIG. 4A.

FIG. 5 is a block diagram showing a specific structure of the register of FIG. 2A .

FIG. 6 is a block diagram showing the specific structure of the convolutional neural network unit of FIG. 2A .

Detailed ways

FIG. 2A shows a block diagram of a framebuffer-less convolutional neural network (CNN) system 100 according to an embodiment of the present invention, and FIG. 2B shows a framebuffer-less convolutional neural network (CNN) according to an embodiment of the present invention. ) flow chart of method 200.

In this embodiment, the frame buffer-less convolutional neural network system (hereinafter referred to as the system) 100 may include a region of interest (ROI) unit 11 for generating a region of interest in the input image frame (step twenty one). Since the system 100 of the present embodiment does not contain a frame buffer, the region of interest unit 11 can use the scanline-based technology and the block-based mechanism to find the region of interest in the input image frame. The input image frame is divided into a plurality of image blocks, which are arranged in a matrix form, such as 4×6 image blocks.

In this embodiment, the region-of-interest unit 11 generates block-based features to determine whether each image block performs a convolutional neural network (CNN) operation. FIG. 3 is a block diagram showing the specific structure of the region of interest unit 11 of FIG. 2A . In this embodiment, the region of interest unit 11 may include a feature extractor 111, for example, for extracting shallow features from the input image frame. In one example, the feature extractor 111 generates (shallow) features of a block according to a block-based histogram. In another example, the feature extractor 111 generates (shallow) features of the block based on frequency analysis.

The region of interest unit 11 may also include a classifier 112, such as a support vector machine (SVM), for determining whether each block of the input image frame performs a convolutional neural network operation. Thereby, a decision map 12 can be generated, which includes a plurality of blocks (which can be arranged in a matrix form) representing the input image frame. Figure 4A illustrates a decision diagram 12, which includes 4x6 blocks, where X indicates that the relevant block does not need to perform a convolutional neural network operation, C indicates that the relevant block needs to perform a convolutional neural network operation, and D indicates that the relevant block detects an object (eg a dog). From this, regions of interest can be determined and convolutional neural network operations performed.

2B, the system 100 may include a register 13, such as a static random access memory (SRAM), for storing (shallow) features generated by the feature extractor 111 (of the region of interest unit 11) (step 22). FIG. 5 is a block diagram showing the specific structure of the register 13 of FIG. 2A . In this embodiment, the register 13 may include two feature maps, that is, the first feature map 131A, for storing the features of the previous image frame (at the previous time t-1); and the first feature map 131A; Two feature maps 131B are used to store the features of the current image frame (at the current time t). The register 13 may also include a sliding window 132, which may be 40x40x8 bits in size, for storing the block of the input image frame.

Referring to FIG. 2A , the system 100 of the present embodiment may include a convolutional neural network (CNN) unit 14 that receives and processes (region of interest unit 11 ) the region of interest of the input image frame to detect objects (step 23 ). ). Wherein, the convolutional neural network unit 14 of the present embodiment is only executed on the region of interest, instead of executing on the entire input image frame as in the conventional system with a frame buffer.

FIG. 6 is a block diagram showing the specific structure of the convolutional neural network unit 14 of FIG. 2A . The convolutional neural network unit 14 may include a convolution unit 141, which includes a plurality of convolution engines for performing convolution operations. The convolutional neural network unit 14 may include an activation unit 142, which may perform an activation function when a default feature is detected. The convolutional neural network unit 14 may also include a pooling unit 143 for performing down-sampling or pooling on the input image frames.

The system 100 of this embodiment may include a tracking unit 15 for comparing the first feature map 131A (of the previous image frame) with the second feature map 131B (of the current image frame), and then updating the decision map 12 (step 24 ). The tracking unit 15 analyzes the content change between the first feature map 131A and the second feature map 131B. FIG. 4B illustrates another decision map 12, which is updated after FIG. 4A. In this example, at the previous time, the blocks located in the 5th to 6th columns and the 3rd row had detected objects (as indicated by D in FIG. 4A ), but at the current time, the objects disappeared (as in the fourth B marked by X). Accordingly, the convolutional neural network unit 14 does not need to perform the convolutional neural network operation on the block without feature change. In other words, the convolutional neural network unit 14 selectively performs convolutional neural network operations on the blocks with characteristic changes. Thus, the system 100 can greatly speed up operation.

Compared to conventional convolutional neural network systems, the convolutional neural network operations of the above-described embodiments can be greatly reduced (and accelerated). Furthermore, since the embodiment of the present invention does not require a frame buffer, this embodiment may be better suited for low-power applications, such as wearable or Internet of Things (IoT) devices. For image frames with a resolution of 320x240 and a (non-overlapping) sliding window size of 40x40, conventional systems with frame buffers require an 8x6 sliding window to perform convolutional neural network operations. In contrast, the system 100 of the present embodiment requires only few (less than 10) sliding windows to perform convolutional neural network operations.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the scope of patent protection of the present invention; all other equivalent changes or modifications made without departing from the spirit disclosed in the invention shall be included in the following within the scope of the claims.

[Description of reference numerals]

100 Framebufferless Convolutional Neural Network Systems

11 Region of Interest Unit

111 Feature Extractor

112 Classifiers

12 Decision map

13 Register

131A first characteristic map

131B second characteristic map

132 Sliding window

14 Convolutional Neural Network Units

141 Convolutional units

142 Excitation Unit

143 Pooling unit

15 Tracking Unit

200 Methods for Framebufferless Convolutional Neural Networks

21 Generate regions of interest in input image frames

22 Store features in feature map

23 Process regions of interest to detect objects

24 Compare features and perform convolutional neural network operations on blocks with feature changes

900 Convolutional Neural Networks

91 Buffer group

92 frame buffer

93 column buffer

94 Convolutional Cell Engine Array

941 Prefetch Controller

95 Accumulation buffer

951 Max Pooling

96 instruction decoder

Claims (18)

1. A framebufferless convolutional neural network system comprising: a region of interest unit for extracting features to generate a region of interest of the input image frame; a convolutional neural network unit that processes the region of interest of the input image frame to detect objects; and a tracking unit that compares features extracted at different times so that the convolutional neural network unit selectively processes the input image frame, The region-of-interest unit generates block-based features to determine whether each image block performs a convolutional neural network operation. 2 . The frame bufferless convolutional neural network system of claim 1 , wherein the region of interest unit adopts a scanline-based technique and a block-based mechanism to find the input image frame. 3 . A region of interest, wherein the input image frame is divided into a plurality of image blocks. 3. The framebufferless convolutional neural network system of claim 2, wherein the region of interest unit comprises: a feature extractor to extract the feature from the input image frame; and The classifier determines whether each image block performs a convolutional neural network operation, thereby generating a decision map for determining the region of interest. 4. The framebufferless convolutional neural network system of claim 3, wherein the feature extractor generates shallow features of the image block according to block-based histogram or frequency analysis. 5. The frame bufferless convolutional neural network system of claim 3, further comprising a register for storing the feature. 6. The frame bufferless convolutional neural network system of claim 5, wherein the register comprises: a first feature map for storing features of a previous image frame; and a second feature map for storing Characteristics of the current image frame. 7. The frame bufferless convolutional neural network system of claim 5, wherein the buffer comprises a sliding window for storing blocks of the input image frame. 8. The frame bufferless convolutional neural network system of claim 6, wherein the tracking unit compares the first feature map and the second feature map to update the decision map accordingly. 9. The framebufferless convolutional neural network system of claim 1, wherein the convolutional neural network unit comprises: A convolution unit, including multiple convolution engines, for performing convolution operations on the region of interest; an excitation unit that performs an excitation function when the default feature is detected; and The pooling unit is used to perform downsampling on the input image frame. 10. A method for a framebufferless convolutional neural network, comprising: extracting features to generate a region of interest of the input image frame; performing a convolutional neural network operation on the region of interest of the input image frame to detect objects; and comparing features extracted at different times to selectively process the input image frame, The step of generating the region of interest includes: generating block-based features to determine whether each image block performs a convolutional neural network operation. 11. The method for a frame bufferless convolutional neural network of claim 10, wherein the region of interest is generated using a scanline-based technique and a block-based mechanism, wherein the input image frame Divide into multiple image blocks. 12. The method for a framebufferless convolutional neural network of claim 11, wherein the step of generating the region of interest comprises: extract the feature from the input image frame; and Whether each image block is subjected to a convolutional neural network operation is determined by a classification method, thereby generating a decision map for determining the region of interest. 13. The method for a framebufferless convolutional neural network of claim 12, wherein the step of extracting the feature comprises: Shallow features of the image block are generated based on block-based histogram or frequency analysis. 14. The method for a framebufferless convolutional neural network of claim 12, further comprising the step of temporarily storing the feature. 15. The method for a framebufferless convolutional neural network of claim 14, wherein the step of temporarily storing the feature comprises: generating a first feature map for storing the features of the previous image frame; and generating a second feature map for storing the features of the current image frame. 16. The method for a framebufferless convolutional neural network of claim 14, wherein the step of temporarily storing the feature comprises: A sliding window is generated for storing the block of the input image frame. 17. The method for a framebufferless convolutional neural network of claim 15, wherein the step of comparing the features comprises: The first feature map and the second feature map are compared to update the decision map accordingly. 18. The method for a framebufferless convolutional neural network of claim 10, wherein the step of performing the convolutional neural network operation comprises: use multiple convolution engines to perform convolution operations on the region of interest; When the default feature is detected, perform an activation function; and Downsampling is performed on the input image frame.

Images