KR102833795B1 - Method and apparatus for processing convolution neural network - Google Patents

Abstract

A method and device for processing a convolution operation in a neural network generate differential windows from a plurality of row windows, and obtain elements of an output feature map by using the results of a convolution operation between a reference row window and a kernel and the results of a convolution operation between the differential windows and the kernel.

Description

Method and apparatus for processing convolution neural network {Method and apparatus for processing convolution neural network}

A method and device for processing a convolution operation between an input feature map and a kernel in a convolutional neural network.

Neural network refers to a computational architecture that models the biological brain. Recently, with the development of neural network technology, research is actively being conducted to analyze input data and extract valid information by utilizing neural networks in various types of electronic systems. Devices that process neural networks require a large amount of computation on complex input data. Therefore, in order to analyze a large amount of input data in real time using neural networks and extract desired information, a technology that can efficiently process computations related to neural networks is required.

The present invention provides a method and device for processing a convolutional neural network. The technical problems to be solved by the present embodiment are not limited to the technical problems described above, and other technical problems can be inferred from the following embodiments.

According to one aspect, a method for processing a convolutional neural network in a neural network processing device includes the steps of grouping a plurality of raw windows of an input feature map into a plurality of differential groups for a differential operation, generating differential windows by performing a differential operation between raw windows belonging to each of the plurality of grouped differential groups, obtaining a reference element of an output feature map corresponding to the reference raw window by performing a convolution operation between a reference raw window and a kernel among the raw windows, and obtaining remaining elements of the output feature map by performing a summation operation with respect to results of the convolution operations between each of the differential windows and the kernel and the reference element.

According to another aspect, a neural network processing device includes a memory in which an input feature map is stored, and a neural network processor for processing a convolutional neural network using the input feature map stored in the memory, wherein the neural network processor groups a plurality of raw windows of the input feature map into a plurality of differential groups for a differential operation, generates differential windows through a differential operation between raw windows belonging to each of the plurality of grouped differential groups, performs a convolution operation between a reference raw window and a kernel among the raw windows, thereby obtaining a reference element of an output feature map corresponding to the reference raw window, and performs a sum operation with respect to results of the convolution operations between each of the differential windows and the kernel, thereby obtaining the remaining elements of the output feature map.

FIG. 1 is a diagram illustrating the architecture of a neural network according to one embodiment.
Figures 2a and 2b are diagrams for explaining the convolution operation of a neural network.
FIG. 3 is a block diagram illustrating a hardware configuration of a neural network processing device according to one embodiment.
FIG. 4 is a flowchart of a method for processing a convolutional neural network in a neural network processing device according to one embodiment.
FIG. 5 is a diagram illustrating an input feature map according to one embodiment.
FIG. 6 is a diagram for explaining a differential group and a differential window according to one embodiment.
FIG. 7 is a diagram for explaining a cascading method of summation operation according to one embodiment.
FIG. 8 is a diagram for explaining a convolution operation using a differential window according to one embodiment.
FIG. 9 is a diagram illustrating an implementation example of a neural network processing device according to one embodiment.
FIG. 10 is a drawing for explaining a differential window output unit according to one embodiment.
Figure 11 is a graph for explaining the improvement in the operation processing speed of the operation processing method according to the present disclosure.
Figure 12 is a graph for explaining the improvement in frame rate of the operation processing method according to the present disclosure.

The terms used in these embodiments are selected from the most widely used and general terms possible while considering the functions in these embodiments, but this may vary depending on the intention of engineers in the relevant technical field, precedents, the emergence of new technologies, etc. In addition, in certain cases, there are terms that are arbitrarily selected, and in this case, the meanings thereof will be described in detail in the description of the relevant embodiments. Therefore, the terms used in these embodiments should be defined based on the meanings of the terms and the overall contents of these embodiments, rather than simply the names of the terms.

In the description of embodiments, when it is said that a part is connected to another part, this includes not only cases where they are directly connected, but also cases where they are electrically connected with another component in between. Also, when it is said that a part includes a certain component, this does not mean that other components are excluded, unless there is a specific description to the contrary, but that other components can be further included.

The terms “comprises” or “comprising” used in these embodiments should not be construed to necessarily include all of the various components or various steps described in the specification, and should be construed to mean that some of the components or some of the steps may not be included, or that additional components or steps may be included.

The description of the following examples should not be construed as limiting the scope of the rights, and what can be easily inferred by a person skilled in the art should be construed as falling within the scope of the rights of the examples. Hereinafter, examples for illustrative purposes only will be described in detail with reference to the attached drawings.

FIG. 1 is a diagram illustrating the architecture of a neural network according to one embodiment.

Referring to FIG. 1, the neural network (1) may be an architecture of a deep neural network (DNN) or an n-layer neural network. The DNN or n-layer neural network may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzman machine, etc. For example, the neural network (1) may be implemented as a convolutional neural network (CNN), but is not limited thereto. In FIG. 1, some convolutional layers are illustrated in a convolutional neural network corresponding to an example of the neural network (1), but the convolutional neural network may further include a pooling layer, a fully connected layer, etc. in addition to the illustrated convolutional layers.

A neural network (1) can be implemented as an architecture having multiple layers including an input image, feature maps, and output. In the neural network (1), a convolution operation is performed on an input image with a filter called a kernel, and feature maps are output as a result. The output feature maps generated at this time are again convolutionally operated with a kernel as input feature maps, and new feature maps are output. As a result of repeatedly performing such convolution operations, a recognition result for the features of the input image through the neural network (1) can be output as a final result.

For example, when an image of 24x24 pixels is input to the neural network (1) of Fig. 1, the input image can be output as 4-channel feature maps of 20x20 pixels through a convolution operation with the kernel. Thereafter, the 20x20 feature maps can be reduced in size through repeated convolution operations with the kernel, and ultimately features of 1x1 pixels can be output. The neural network (1) filters and outputs robust features that can represent the entire image from the input image by repeatedly performing convolution operations and sub-sampling (or pooling) operations in multiple layers, and can derive a recognition result of the input image through the output final features.

Figures 2a and 2b are diagrams for explaining the convolution operation of a neural network.

In the example of FIG. 2a, it is assumed that the input feature map (210) for the input image has a size of 6x6 pixels, the original kernel (220) has a size of 3x3 pixels, and the output feature map (230) has a size of 4x4 pixels, but it is not limited thereto, and the neural network can be implemented with feature maps and kernels of various sizes. In addition, the values defined in the input feature map (210), the original kernel (220), and the output feature map (230) are all exemplary values, and the present embodiments are not limited thereto.

The original kernel (220) performs a convolution operation while sliding in a window unit of 3x3 pixel size on the input feature map (210) (sliding window fashion). The convolution operation means an operation of obtaining each pixel value of the output feature map (230) by adding up all values obtained by multiplying each pixel value of a window of the input feature map (210) and the weight of each element of the corresponding position in the original kernel (220). Specifically, the original kernel (220) first performs a convolution operation with the first window (211) of the input feature map (210). That is, each pixel value 1, 2, 3, 4, 5, 6, 7, 8, 9 of the first window (211) is multiplied by each element's weight -1, -3, +4, +7, -2, -1, -5, +3, +1 of the original kernel (220), respectively, and as a result, -1, -6, 12, 28, -10, -6, -35, 24, 9 are obtained. Next, 15, which is the result of adding all of the obtained values -1, -6, 12, 28, -10, -6, -35, 24, 9, is calculated, and the pixel value (231) of the 1st row and 1st column of the output feature map (230) is determined as 15. Here, the pixel value (231) of the 1st row and 1st column of the output feature map (230) corresponds to the first window (211). In the same manner, a convolution operation is performed between the 2nd window (212) of the input feature map (210) and the original kernel (220), thereby determining the pixel value (232) of the 1st row and 2nd column of the output feature map (230), which is 4. Finally, a convolution operation is performed between the 16th window (213), which is the last window of the input feature map (210), and the original kernel (220), thereby determining the pixel value (233) of the 4th row and 4th column of the output feature map (230), which is 11.

That is, a convolution operation between one input feature map (210) and one original kernel (220) can be processed by repeatedly performing multiplication of values of corresponding elements in the input feature map (210) and the original kernel (220) and summing the results of the multiplication, and an output feature map (230) is generated as a result of the convolution operation.

Meanwhile, although Fig. 2a describes a two-dimensional convolution operation, the convolution operation may correspond to a three-dimensional convolution operation in which input feature maps, kernels, and output feature maps of multiple channels exist. This will be described with reference to Fig. 2b.

Referring to FIG. 2b, input feature maps (201) have X channels, and the input feature map of each channel can have a size of H rows and W columns (X, W, H are natural numbers). Each of the kernels (202) has a size of R rows and S columns, and the kernels (202) can have a number of channels corresponding to the number of channels (X) of the input feature maps (201) and the number of channels (Y) of the output feature maps (203) (R, S, Y are natural numbers). The output feature maps (203) are generated through a 3D convolution operation between the input feature maps (201) and the kernels (202), and there can be Y channels depending on the convolution operation.

The process of generating an output feature map through a convolution operation between one input feature map and one kernel is as described above in FIG. 2a, and the two-dimensional convolution operation described in FIG. 2a is repeatedly performed between the input feature maps (201) of all channels and the kernels (202) of all channels, so that output feature maps (203) of all channels can be generated.

FIG. 3 is a block diagram illustrating a hardware configuration of a neural network processing device (300) according to one embodiment.

Referring to FIG. 3, a neural network processing device (300) may include a neural network processor (310) and a memory (320).

The neural network processing device (300) illustrated in FIG. 3 only illustrates components related to the present embodiments. Therefore, it is obvious to those skilled in the art that the neural network processing device (300) may further include other general components in addition to the components illustrated in FIG. 3.

The neural network processor (310) may be implemented as a CPU (central processing unit), a GPU (graphics processing unit), an AP (application processor), etc., but is not limited thereto.

Additionally, the memory (320) may include random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

The neural network processing device (300) is included in the neural network device and serves to control overall functions for driving the neural network. For example, the neural network processing device (300) can control an operation processing process for extracting an output feature map from an input feature map of the neural network device.

Here, the neural network device can be implemented with various types of devices such as a personal computer (PC), a server device, a mobile device, and an embedded device, and as specific examples, it can correspond to a smartphone, a tablet device, an Augmented Reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, robotics, a medical device, etc. that perform voice recognition, image recognition, and image classification using a neural network, but is not limited thereto. Furthermore, the neural network processing device (300) can correspond to a dedicated hardware accelerator (HW accelerator) mounted on the above-mentioned device, and the neural network device can be a hardware accelerator such as an NPU (neural processing unit), a TPU (Tensor Processing Unit), a Neural Engine, etc., which are dedicated modules for driving a neural network, but is not limited thereto.

The neural network device may include a neural network processing device (300) and an external memory. The neural network processing device (300) including a neural network processor (310) and a memory (320) may be implemented as a single chip, in which case the memory (320) included in the neural network processing device (300) is an on-chip memory, and the external memory is an off-chip memory.

In order to reduce the size of the chip, etc., the memory (320) included in the neural network processing device (300) may have a relatively small capacity compared to an external memory.

Off-chip memory with a relatively large capacity can store all input feature maps, weight values of kernels, output feature maps, etc. The neural network processing device (300) can access an external memory to obtain data required for calculation and store the obtained data in the memory (320), which is an on-chip memory. In addition, the neural network processing device (300) can store intermediate calculation results for generating an output feature map and a part of the output feature map in the memory (320), which is an on-chip memory.

As the size of the on-chip memory (320) decreases, the size of the chip can be reduced, but as the frequency of access to the off-chip memory increases, traffic can increase. Therefore, it is necessary to reduce the data capacity of the intermediate operation result by considering the capacity of the on-chip memory (320) and to reduce the frequency of access to the off-chip memory in order to reduce the traffic generated during the operation processing.

The neural network processor (310) processes a convolution operation between the input feature map and the kernel using elements of the input feature maps stored (or buffered) in the memory (320), weights of the kernels, etc. At this time, the input feature map may be related to image data, and elements of the input feature map may represent pixels, but are not limited thereto.

In a neural network processing device (300), each of a neural network processor (310) and a memory (320) may be provided one or more times, and each of the neural network processors (310) and the memory (320) may be used to process a convolution operation in parallel and independently, thereby enabling the convolution operation to be processed efficiently.

A logic circuit implementing a convolution operator for a convolution operation may be provided within the neural network processor (310). The convolution operator is an operator implemented by a combination of a shifter or a multiplier, an adder, and an accumulator. Each of the shifter, the multiplier, and the adder within the convolution operator may be implemented by a combination of a plurality of sub-shifters, sub-multipliers, and sub-dubbers, respectively.

The neural network processor (310) can group multiple raw windows of an input feature map into multiple differential groups for differential operations.

A window represents a sub-feature map of a smaller unit than a feature map. For example, each of a plurality of windows of an input feature map may be composed of some elements among a plurality of elements that constitute the input feature map.

Each of the plurality of windows may overlap some of the elements of the input feature map. For example, the input feature map may be block data in which elements are arranged in a multidimensional space, and two adjacent windows in the multidimensional space may overlap some of the elements of the input feature map. As another example, the plurality of windows may not overlap each other.

The multidimensional space of the input feature map can be determined so that spatially adjacent elements have high correlations. For example, adjacent elements in the multidimensional space of the input feature map can have similar values. Therefore, the multidimensional space of the input feature map can be determined differently depending on the type of the input feature map. For example, if the input feature map is about image data, the multidimensional space can represent the pixel space of the image data.

Raw windows represent windows in which no difference operation is performed between windows, unlike raw windows.

The neural network processor (310) can determine a plurality of row windows according to a sliding window fashion. The sliding window fashion is a method in which a sliding window having a predetermined size and shape, i.e., a predetermined shape, slides an input feature map at a constant sliding interval to determine a plurality of windows. For example, each of the plurality of windows can be determined by scanning elements of the input feature map partitioned according to the shape of the sliding window at each of the plurality of sliding positions. At this time, the sliding direction can indicate the spatial directionality between the plurality of windows in the multidimensional space of the input feature map. However, the sliding direction does not indicate the temporal order in which the plurality of windows are determined or acquired. For example, a window at a preceding position and a window at a succeeding position in the sliding direction can be determined or acquired simultaneously.

The shape, which is the size and shape of the plurality of windows of the input feature map, can be determined according to the shape of the sliding window. For example, the input feature map can be a three-dimensional block arranged in a three-dimensional space defined by mutually orthogonal row (or length) axes, column (or width) axes, and depth axes. When the sliding window has a rectangular shape having a predetermined size in the column and row directions of the input feature map, the plurality of windows can be sub-blocks having the same size as the sliding window in the column and row directions and having a rectangular shape. In this case, the shape of the sliding window does not determine the size of the plurality of windows in the depth direction. The size of the plurality of windows in the depth direction can be the same as the size of the input feature map in the depth direction.

The neural network processor (310) can group multiple windows of an input feature map into multiple differential groups in which differential operations are performed. For example, the neural network processor (310) can group multiple windows into differential groups consisting of two adjacent windows. At this time, the differential group can be composed of differential subject windows and differential reference windows that are differential from the differential subject windows. The differential subject windows can be windows that follow in the sliding direction, and the differential reference windows can be windows that precede the differential subject windows in the sliding direction.

Among two different difference groups, a difference target window belonging to one difference group can be the same window as a difference reference window belonging to another difference group. For example, for three windows, a leading window, a middle window, and a trailing window, which are arranged adjacently in a sliding direction, two difference groups can be defined. At this time, the middle window, which is a difference target window in the difference group consisting of the leading window and the middle window, can be a difference reference window of the next difference group consisting of the middle window and the trailing window.

The neural network processor (310) can generate differential windows through differential operations between row windows belonging to each of a plurality of grouped differential groups.

The neural network processor (310) can generate each of the differential windows through an element-wise differential operation between the differential reference window and the differential target window belonging to each of the plurality of grouped differential groups. For example, the neural network processor (310) can obtain the differential window by performing a differential operation between two elements at corresponding positions of the differential target window and the differential reference window having the same shape. Accordingly, the shape of the differential window is the same as the shapes of the differential target window and the differential reference window. The neural network processor (310) can generate each of the differential windows corresponding to each of the differential target windows of the differential groups.

The neural network processor (310) can obtain a reference element of an output feature map corresponding to the reference low window by performing a convolution operation between a reference low window and a kernel among the low windows.

The neural network processor (310) can determine at least one reference raw window among the plurality of windows. For example, the neural network processor (310) can group the plurality of windows into a plurality of cascading groups. Each of the windows within the plurality of cascading groups can be adjacent to at least one other window within the cascading group.

The neural network processor (310) can determine a reference window in each of a plurality of cascading groups. The neural network processor (310) can perform a convolution operation between the reference row window and the kernel to obtain a reference element of an output feature map corresponding to the reference row window. The reference element of the output feature map is obtained directly from the result of the convolution operation between the reference row window and the kernel, independently of the differential window.

The neural network processor (310) can obtain the remaining elements of the output feature map by performing a sum operation on the results of convolution operations between each of the differential windows and the kernel with the reference element of the output feature map.

The neural network processor (310) can perform a convolution operation between each of a plurality of differential windows corresponding to a plurality of differential groups and a kernel. In addition, the neural network processor (310) can obtain the remaining elements of the output feature map by performing a sum operation with a reference element on the result of the cascading method of the convolution operation results between each of the differential windows and the kernel. Therefore, the neural network processor (310) does not need to perform a convolution operation between each of the row windows, i.e., the differential target windows, and the kernel in order to obtain the remaining elements of the output feature map corresponding to each of the differential target windows of each of the plurality of differential groups.

Since the elements of the difference window have relatively smaller values than the elements of the low window based on the similarity between the elements at corresponding positions in the difference target window and the difference reference window, the difference window can be stored using a relatively small amount of memory.

In addition, the neural network processor (310) may convert the data format of the differential windows in order to increase the convolution operation speed between each of the differential windows and the kernel. For example, if each of the differential windows is in a bit data format, the neural network processor (310) may convert each of the differential windows into a data format including information on the number of digits of a significant bit representing a bit value of 1. In addition, the neural network processor (310) may preprocess the differential windows according to a booth algorithm for reducing the number of significant bits as a preprocessing for converting the data format. The neural network processor (310) may perform a bit-shift operation based on the information on the number of digits of a significant bit of each of the converted differential windows, and may calculate the convolution operation result between the differential window and the kernel from this. In calculating the convolution operation result, the memory capacity may be reduced and the operation processing speed may be increased by performing the conversion into a data format having a relatively small amount of information and using a shift operation having a small load on the operation processing.

FIG. 4 is a flowchart of a method for processing a convolutional neural network in a neural network processing device according to one embodiment.

The processing method of the convolutional neural network illustrated in FIG. 4 can be performed by the neural network processing device (300 of FIG. 3) described in the drawings described above, so even if the contents are omitted below, the contents described in the drawings described above can also be applied to the method of FIG. 4.

In step 410, the neural network processing device (300) can group multiple raw windows of the input feature map into multiple differential groups for differential operation.

The neural network processing device (300) can determine multiple row windows according to a sliding window method.

The neural network processing device (300) can group multiple windows of an input feature map into multiple differential groups in which differential operations are performed. For example, the neural network processing device (300) can group multiple windows into differential groups consisting of two adjacent windows.

In step 420, the neural network processing device (300) can generate differential windows through differential operations between row windows belonging to each of the plurality of grouped differential groups.

The neural network processing device (300) can generate each of the differential windows through an element-wise differential operation between the differential reference window and the differential target window belonging to each of the plurality of grouped differential groups. For example, the neural network processing device (300) can obtain the differential window by performing a differential operation between two elements at positions corresponding to each of the differential target window and the differential reference window having the same shape. Accordingly, the neural network processing device (300) can generate each of the differential windows corresponding to the differential target window for each of the differential groups.

In step 430, the neural network processing device (300) can obtain a reference element of an output feature map corresponding to the reference low window by performing a convolution operation between a reference low window and a kernel among the low windows.

The neural network processing device (300) can determine at least one reference raw window among the plurality of windows. For example, the neural network processing device (300) can group the plurality of windows into a plurality of cascading groups. Each of the windows within the plurality of cascading groups can be adjacent to at least one other window within the cascading group. In addition, the neural network processing device (300) can determine a reference window in each of the plurality of cascading groups.

A neural network processing device (300) can obtain a reference element of an output feature map corresponding to a reference low window by performing a convolution operation between elements of a kernel at a corresponding position and elements of a reference low window.

In step 440, the neural network processing device (300) can perform a sum operation with the reference element on the convolution operation results between each of the differential windows and the kernel to obtain the remaining elements of the output feature map.

The neural network processing device (300) can perform a convolution operation between each of a plurality of differential windows corresponding to a plurality of differential groups and a kernel. In addition, the neural network processing device (300) can obtain the remaining elements of the output feature map by performing a sum operation with a reference element on the result of the cascading method of the convolution operation results between each of the differential windows and the kernel.

In addition, the neural network processing device (300) can convert the data format of the differential windows. For example, if each of the differential windows is in a bit data format, the neural network processing device (300) can perform conversion into a data format including information on the number of digits of a significant bit representing a bit value of 1 for each of the differential windows. In addition, the neural network processing device (300) can preprocess the differential windows according to a booth algorithm for reducing the number of significant bits as a preprocessing for converting the data format. The neural network processing device (300) can calculate the result of a convolution operation between the differential window and the kernel based on the result of a bit-shift operation based on the information on the number of digits of a significant bit of each of the converted differential windows.

FIG. 5 is a diagram illustrating an input feature map (500) according to one embodiment.

In FIG. 5, an embodiment is illustrated in which the input feature map (500) is two-dimensional array data, but the input feature map (500) may be three-dimensional block data or other various types of data and is not limited to the present embodiment.

Referring to Fig. 5, the input feature map (500) is a two-dimensional array data having a 7x6 element size in the row and column directions. The input feature map (500) is composed of 7X6=42 elements. If the input feature map (500) is about image data, each element may correspond to a pixel.

In Fig. 5, each element of the input feature map (500) is expressed as a combination of an index representing a row and an index representing a column. For example, an element of 3 rows and 2 columns of the input feature map (500) is expressed as X ₃₂ .

In FIG. 5, a sliding window (510) for determining the low windows of the input feature map (500) is illustrated. According to the sliding window method, the sliding window (510) slides the input feature map (500) at a predetermined number of element intervals and extracts the low windows. In the present embodiments, unless otherwise referred to as a differential window, the window represents a low window that is distinct from a differential window.

In Fig. 5, the sliding window (510) has a 3X3 element size in the row and column directions. Accordingly, each of the windows of the input feature map (500) is a two-dimensional array data of a 3X3 element size equal to the size of the sliding window.

The embodiments of FIGS. 6 and 7 below relate to windows determined in response to the input feature map (500) and sliding window (510) of FIG. 5. The sliding window (510) moves the input feature map by one element in the column direction and determines windows belonging to the same row. In addition, the sliding window (510) moves by one element in the row direction and determines windows belonging to the next row. In this manner, 5X4=20 windows are determined from the input feature map.

FIG. 6 is a diagram for explaining a differential group and a differential window according to one embodiment.

Figure 6 illustrates four windows belonging to the same row (window 11, window 12, window 13, window 14) and three difference groups (difference group 11, difference group 12, difference group 13) each consisting of two adjacent windows among the four windows.

According to the sliding window sliding direction illustrated in the above Fig. 5, window 11 and window 12, window 12 and window 13, and window 13 and window 14 are mutually adjacent windows and each constitutes a different difference group.

Among the difference groups, the window preceding the difference direction is the difference reference window, and the window following the difference is the difference target window. The difference window is created by differentiating the difference reference window from the difference target window.

For example, in differential group 12, differential window 13 is generated by differentiating the preceding window 12 from the trailing window 13.

At this time, the difference operation between windows is performed on an element basis. For example, the difference window 13 is generated through the difference operation between the elements of the corresponding positions of window 13 and window 12. For example, the element of row 2 and column 2 of the difference window 13 is X ₂₄ -X ₂₃ , which is obtained by differentiating element X 23 of window 12 from element X ₂₄ of row 2 and column ₂ of window 13, which is the corresponding position.

With respect to the correspondence between the low windows, the differential groups and the differential windows, each of the trailing low windows, which are the differential target windows of each of the differential groups, corresponds to each of the differential windows. For example, the low window 12, which is the differential target window of the differential group 11, corresponds to the differential window 12, and the low window 13 of the differential group 12 corresponds to the differential window 13. In addition, as a result of the convolution operation between each of the differential windows and the kernel, different elements of the output feature map are generated. Therefore, the low windows, which are the differential target windows, the differential windows and the elements of the output feature map have a correspondence relationship.

In Fig. 6, the difference window corresponding to the low window, window 11, is not shown. This is because, as explained in Fig. 7, window 11 is a reference window, and the reference element of the corresponding output feature map is obtained by performing a convolution operation between window 11 itself and the kernel without using a separate difference window. Therefore, in grouping the difference group, a difference group that includes window 11, which is a reference window, as a difference target window is not required.

FIG. 7 is a diagram for explaining a cascading method of summation operation according to one embodiment.

In Fig. 7, the kernel is a two-dimensional array data of weights having a 3X3 element size. The kernel performs a convolution operation with each of window 11, differential window 12, differential window 13, and differential window 14.

O ₁₁ , O ₁₂ , O ₁₃ , and O ₁₄ , which represent elements of the output feature map, represent different elements of the output feature map and correspond to window 11 , differential window 12 , differential window 13 , and differential window 14 , respectively. As described above in Fig. 6 , since the differential target window, which is a low window, and the differential window have a corresponding relationship, the elements O ₁₁ , O ₁₂ , O ₁₃ , and O ₁₄ of the output feature map correspond to window 11 , window 12 , window 13 , and window 14 , respectively.

Referring to FIG. 7, the reference element O ₁₁ of the output feature map corresponding to window 11 is directly derived from the convolution result between window 11 and the kernel. In contrast, the remaining elements O ₁₂ , O ₁₃ , and O ₁₄ of the output feature map are derived by adding the convolution results between the reference element O ₁₁ of the output feature map and the differential windows and kernels in a cascading manner. For example, the element O ₁₃ of the output feature map corresponding to differential window 13 is derived by adding dO ₁₂ , which is the convolution result between differential window 12 and the kernel, to the reference element O ₁₁ , and then adding dO ₁₃ , which is the convolution result between differential window 13 and the kernel, to the result of the addition in a cascading manner.

FIG. 8 is a diagram for explaining a convolution operation using a differential window according to one embodiment.

Window0, Window1, and Window2 represent low windows, and DifferentialWindow1 represents a differential window of Window1 that differentiates Window0 from Window1, and DifferentialWindow2 represents a differential window of Window2 that differentiates Window1 from Window2.

In Figure 8, the row windows Window0, Window1, and Window2, the differential windows Differential Window1, Differential Window2, and the kernel are all two-dimensional array data with 2x2 element size.

In a general convolution operation processing, a convolution operation is performed between each of the row windows and the kernel, and each element of the output feature map is produced. For example, the multiplication results 47x2, 47x1, 49x3, 50x2 between the elements 47, 47, 49, and 50 of window 1 and the elements 2, 1, 3, and 2 of the corresponding positions of the kernel are added to produce element 388 of the output feature map corresponding to window 1.

In contrast, in the convolution method using the differential window, only the reference window, window 0, among the low windows is directly convolved with the kernel. For example, in order to produce an element of the output feature map corresponding to window 1, the result of the convolution operation between differential window 1 and the kernel is added to 373 (903), which is the result of the convolution operation between window 0 and the kernel.

First, the difference window 1 is obtained by differentiating the elements 45, 47, 46, 49 of the adjacent window 0 from the elements 47, 47, 49, 50 of the window 1, respectively, to obtain the elements 2, 0, 3, 1 of the difference window 1, respectively. In addition, the multiplication results 2x2, 0x1, 3x3, 1x2 between the elements 2, 0, 3, 1 of the difference window 1, respectively, and the elements 2, 1, 3, 2 of the corresponding positions of the kernel, respectively, are added to obtain the convolution operation result 15 (904) between the difference window 1 and the kernel. In the same manner, by differentiating each of the elements at corresponding positions in the adjacent difference window 1 from the elements in the difference window 2, the elements 0, -1, 1, -2 of the difference window 2 are produced, and the result of the convolution operation between the difference window 2 and the kernel is -2.

The element of the output feature map corresponding to window 1 is calculated as 388 (905) by adding 15 (904) of the convolution operation result between the differential window 1 and the kernel to element 373 (903) of the output feature map corresponding to window 1, which is the reference window. This is identical to element 388 (902) of the output feature map corresponding to window 1 calculated using a general convolution operation.

In addition, the element of the output feature map corresponding to window 2 is calculated as 386, which is the result of a cascading method that sequentially adds -2, which is the result of the convolution operation between the differential window 2 and the kernel, to the result of adding 15 (904) of the convolution operation between the differential window 1 and the kernel and the element 373 (903) of the output feature map corresponding to the reference window 1.

FIG. 9 is a diagram illustrating an implementation example of a neural network processing device according to one embodiment.

In FIG. 9, the neural network processing device may include a plurality of input feature map memories (1200, 1201, 1215) in which input feature maps are stored, a weight memory (1300) in which kernel weights are stored, a plurality of convolution units (CUs) (1100, 1115) that perform convolution operations, and differential reconstruction units (DUs) that perform a cascading-style summation operation to derive elements of an output feature map from a result of a convolution operation using a differential window.

Each of the multiple input feature map memories (1200, 1201, 1215) may store different row windows or differential windows.

In addition, a predetermined number of CUs (1100, 1115) may be grouped into one column. Each of the different Columns may correspond to each of the different columns of the output feature map. For example, if the size of the column direction of the output feature map is 16, Column0, Column1, and Column15 may correspond to each of the 16 columns of the output feature map. However, Column0 to Column15 may receive input data ABin from IM0 (1200), IM1 (1201) to IM15 (1215), and may be grouped in various ways to process data in parallel, and are not limited to the grouping method according to the present embodiment. In Fig. 9, in order to indicate a correspondence relationship, the index of IM, the index of Column, and the second index of the parentheses index of CU are all indicated identically.

Each of the columns includes 16 CUs. Different CUs belonging to the same column can correspond to each of multiple channels of the input feature map and kernel. For example, CU(0, 0)(1100) can process the 1st channel of the window input from IM0(1200), and CU(15,0)(1115) can process the 16th channel. Therefore, 16 channels can be processed in parallel using 16 CUs belonging to the same column.

Each of the Columns may include a buffer. For example, an output feature map generated from a window input to each of the Columns may be stored in the ABout buffer. An element of an output feature map generated from a currently input low window or differential window is stored in a corresponding buffer (Curr), and as an element of an output feature map is generated from a next low window or differential window, the element stored in the buffer (Curr) is moved to and stored in the buffer (Prev), and an element of an output feature map generated from the next low window or differential window is stored in the buffer (Curr). An element of an output feature map stored in the buffer (Prev) may be cascaded and added with a result of a convolution operation between a differential window and a kernel of another Column. For example, an element of an output feature map stored in the buffer (Prev) of Column15 may be added with a result of a convolution operation between a differential window and a kernel of Column0 to output an element of an output feature map corresponding to Column0.

The convolution operation result produced from each of the plurality of CUs can be input to the corresponding DU. If the convolution operation result produced from the CU is the convolution operation result between the low window and the kernel, the convolution operation result is output through the multiplexer, and if the convolution operation result produced from the CU is the convolution operation result between the differential window and the kernel, the result added with the elements of the output feature map corresponding to another column is output through the multiplexer. For example, if ABin input from IM1 (1201) is the low window, the result (1003) input to the DU through CU (0,1) is output as is through the multiplexer (1005). If the ABin input from IM1 (1201) is a differential window, the result (1004) of the convolution operation result (1003) input from CU (0,1) and the element (1002) of the output feature map stored in the buffer (Curr) of Column0 are added in a cascaded manner and output through the multiplexer (1005). In addition, the elements of the output feature map produced from each of the plurality of columns can be stored in IM as an input feature map for the next layer of the current layer.

FIG. 10 is a drawing for explaining a differential window output unit (1130) according to one embodiment. FIG. 10 is for explaining a differential window output unit (1130) added to the implementation example described in FIG. 9, and the embodiment described in FIG. 9 can be applied to the embodiment of FIG. 10.

The differential window output section may include a multiplexer (1150). For example, the multiplexer (1150) outputs an element of an output feature map corresponding to a Column specified by CS (Column Select) that specifies one Column among Column0 to Column16. At this time, the element optionally output from the multiplexer may be converted into an activation value through an activation converter (1110). The element of the output feature map output from the multiplexer may be stored in a buffer (1120). At this time, it is input to a differentiator (1140) that performs a difference operation between the element of the output feature map stored in the buffer and the element of the output feature map of the currently selected Column. The differentiator (1140) performs a difference operation between the input elements and stores the difference operation result in the IM. Therefore, by generating differential windows used in the next layer in the current layer without generating a separate differential window in the next layer, the efficiency of computational processing can be increased.

Figure 11 is a graph for explaining the improvement in the operation processing speed of the operation processing method according to the present disclosure.

The horizontal axis of Fig. 11 represents neural network models to which the conventional neural network operation processing method PRA and the operation processing method Diffy according to the present embodiments are applied, respectively, and the vertical axis represents the degree of increase in speed compared to the conventional method VAA, which serves as a comparison standard for speed improvement. Referring to the graph, Diffy is improved in speed by about 6.1 times compared to VAA, and is improved in speed by about 1.16 times compared to PRA. In addition, in all neural network models, the method Diffy implemented according to the present disclosure is improved in operation processing speed compared to the conventional PRA method.

Figure 12 is a graph for explaining the improvement in frame rate of the operation processing method according to the present disclosure.

FIG. 12 is a graph comparing FPS (HD Frames per Second) representing the frame rate of VAA, PRA, which are conventional computational processing methods, and Diffy, an implementation example of the present disclosure, in each of the neural network models (DnCNN, FFDNet, IRCNN, JointNet, VDSR, and Geom). As shown in the graph, Diffy significantly increases FPS compared to conventional methods. In addition, in the JointNet model, Diffy shows performance close to 30 FPS. This indicates that Diffy is a method more suitable for image-related applications running on terminals such as smartphones than other comparative examples.

Diffy is a DC-based architecture that improves the performance and energy efficiency of CI-DNN (Computational imaging Deep Neural Network) and other convolutional neural networks (CNNs). By utilizing differential values, Diffy can reduce the required storage capacity of on-chip and off-chip memory and reduce traffic. In addition, when applied to state-of-the-art CI-DNNs, Diffy can perform 1K 16 × 16b multiply-accumulate operations per cycle, which can improve the performance by 7.1x and 1.41x compared to VAA and PRA, respectively. Diffy can process HD frames from 3.9 to 28.5 FPS depending on the target application, which is a significant improvement over VAA's 0.7 to 3.9 FPS and PRA's 2.6 to 18.9 FPS. Diffy can reduce on-chip memory storage by 32% and off-chip traffic by 1.43x compared to dynamically determining precision per group for raw values.

Meanwhile, the embodiments of the present invention described above can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the structure of data used in the embodiments of the present invention described above can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optical reading media (e.g., CD-ROM, DVD, etc.).

The present invention has been described with reference to preferred embodiments thereof. Those skilled in the art will appreciate that the present invention may be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is indicated by the claims, not the foregoing description, and all differences within the scope equivalent thereto should be interpreted as being included in the present invention.

Claims (16)

A method for processing a convolutional neural network in a neural network processing device,
A step of grouping multiple raw windows of an input feature map into multiple differential groups for differential operation;
A step of generating differential windows by performing a differential operation between row windows belonging to each of the above-mentioned plurality of differential groups;
A step of obtaining a reference element of an output feature map corresponding to the reference low window by performing a convolution operation between a reference low window and a kernel among the above low windows; and
A step of obtaining the remaining elements of the output feature map by performing a sum operation with the reference element on the results of the convolution operation between each of the differential windows and the kernel,
The above plurality of low windows are sub-feature maps of the input feature map,
The above difference operation is an element-wise difference operation between two row windows belonging to each of the above plurality of difference groups.
method. In paragraph 1,
The above low windows are determined from the input feature map according to a sliding window fashion,
A method wherein each of the above plurality of differential groups is grouped to include two adjacent row windows in the sliding direction according to the sliding window method. In the second paragraph,
The step of generating the above differential windows is:
A method for generating the difference windows by performing an element-wise difference operation between two adjacent row windows belonging to each of the plurality of difference groups. In the second paragraph,
The steps for obtaining the remaining elements are:
A method for obtaining the remaining elements of the output feature map by performing a sum operation with the reference element on the result of the cascading of the convolution operation results between each of the differential windows and the kernel. In paragraph 4,
The above summation result of the above cascading method is,
A method wherein the result of the convolution operation between the current differential window and the kernel is the result of the sum of the convolution operation results corresponding to one or more previous differential windows preceding the current differential window along the sliding direction. In paragraph 1,
If each of the above differential windows is in a bit data format, the method further comprises a step of converting the data into a data format including information on the number of digits of a significant bit representing a bit value of 1 for each of the above differential windows.
The results of the convolution operation between each of the above differential windows and the kernel are
A method for convolution operation results between each of the differential windows converted into the above data format and the kernel. In paragraph 6,
The results of the convolution operation between each of the differential windows converted to the above data format and the above kernel are
A method for producing results according to a bit-shift operation result based on information about the number of digits of the above valid bits. In paragraph 7,
A method further comprising the step of preprocessing the differential windows according to a booth algorithm that reduces the number of valid bits as a preprocessing for converting the above data format. In a neural network processing device,
Memory where the input feature map is stored; and
A neural network processor for processing a convolutional neural network using the input feature map stored in the memory,
The above neural network processor,
Group multiple raw windows of the input feature map into multiple differential groups for differential operation,
Differential windows are generated by performing a differential operation between the row windows belonging to each of the above-mentioned multiple differential groups,
By performing a convolution operation between the reference low window and the kernel among the above low windows, a reference element of the output feature map corresponding to the reference low window is obtained,
By performing a sum operation with the reference element on the results of the convolution operation between each of the above differential windows and the kernel, the remaining elements of the output feature map are obtained.
The above plurality of low windows are sub-feature maps of the input feature map,
The above difference operation is an element-wise difference operation between two row windows belonging to each of the above plurality of difference groups.
device. In Article 9,
The above low windows are determined from the input feature map according to a sliding window fashion,
A device wherein each of the plurality of differential groups is grouped to include two adjacent row windows in the sliding direction according to the sliding window method. In Article 10,
The above neural network processor,
A device for generating the difference windows by performing an element-wise difference operation between two adjacent row windows belonging to each of the plurality of difference groups. In Article 10,
The above neural network processor,
A device for obtaining the remaining elements of the output feature map by performing a sum operation with the reference element on the result of the cascading of the convolution operation results between each of the differential windows and the kernel. In Article 12,
The above summation result of the above cascading method is,
A device, wherein the result of the convolution operation between the current differential window and the kernel is the result of the sum of the convolution operation results corresponding to one or more previous differential windows preceding the current differential window along the sliding direction. In Article 9,
The above neural network processor,
If each of the above differential windows is in a bit data format, convert it into a data format that includes information on the number of digits of significant bits representing a bit value of 1 for each of the above differential windows,
The results of the convolution operation between each of the above differential windows and the kernel are
A device, which is the result of a convolution operation between each of the differential windows converted into the above data format and the above kernel. In Article 14,
The results of the convolution operation between each of the differential windows converted to the above data format and the above kernel are
A device which produces results according to the result of a bit-shift operation based on information about the number of digits of the above valid bits. In Article 15,
The above neural network processor,
A device that preprocesses the differential windows according to a booth algorithm that reduces the number of valid bits as a preprocessing for converting the above data format.