WO2020029181A1 - Three-dimensional convolutional neural network-based computation device and related product - Google Patents

Abstract

A three-dimensional convolutional neural network-based computation device and a related product. The computation device comprises: a score processing module, an exponential kernel, a three-dimensional convolution kernel (303), a frame superposition module (304), a channel superposition module (305), and an output cache (306). The computation device achieves a high computation speed and favorable accuracy.

Description

Three-dimensional convolutional neural network computing device and related products Technical field

The present application relates to the field of computers and artificial intelligence technologies, and in particular, to a three-dimensional convolutional neural network computing device and related products.

Background technique

With the rapid development of machine learning, machine learning algorithms have achieved great success in many applications. Among them, deep learning algorithms, especially three-dimensional convolutional neural network algorithms, have been widely used in many applications such as behavior detection, medical image analysis, and terrain analysis. Unlike 2D convolution, 3D convolution not only extracts features in the 2D image dimension, but also adds feature extraction and analysis in the 3rd dimension, such as time or the 3D space dimension. For three-dimensional information processing, 3D convolutional neural networks are more accurate than traditional 2D convolutional neural networks in 3D data processing, such as analysis of video or 3D topographic maps. Because applications such as behavior detection and terrain geological analysis are deployed on mobile terminals, real-time processing speed is required. Therefore, how to implement 3D convolutional neural network quickly and efficiently and apply it to actual scenes is one of the current research hotspots.

The existing 3D convolutional neural network cannot achieve real-time processing speed, and the calculation speed is slow.

Application content

The embodiments of the present application provide a three-dimensional convolutional neural network computing device and related products, which realize fast processing of the three-dimensional convolutional neural network through floating-point blocks and improve calculation speed.

In a first aspect, an embodiment of the present application provides a three-dimensional convolutional neural network computing device, which is characterized in that the computing device includes: a fraction processing block, an exponential kernel, a three-dimensional convolution kernel, a frame superposition block, a channel superposition block, and an output. Cache

The score processing block is used to receive the score data Lm of the three-dimensional picture data block, divide the score data Lm into three two-dimensional score data, and input the three two-dimensional score data into the three-dimensional convolution kernel;

The exponential kernel is used to receive the shared exponential data Le of the three-dimensional image data block, and input the exponential data Le to the three-dimensional convolution kernel;

A three-dimensional convolution kernel, which is used to perform three-dimensional convolution operations on three two-dimensional fractional data to obtain three fractional convolution operation results, divide the index data Le into three two-dimensional index data, and perform three-dimensional convolution operations to obtain three. Exponential convolution operation results, inputting three exponential convolution operation results and three fractional convolution operation results into the frame superposition block;

Frame superposition block, which is used to perform frame superposition processing on three exponential convolution operation results and three fractional convolution operation results to obtain superimposed data, and input the superimposed processing data to the channel superposition block;

The channel superposition block is used to obtain the preliminary convolution result after performing the channel superposition processing on the data after the superposition processing, and input the preliminary processing result to the output buffer.

According to a second aspect, a method for performing a three-dimensional convolution operation by using the device of the first aspect is provided. The method includes:

Receive the score data Lm of the three-dimensional picture data block, divide the score data Lm into three two-dimensional score data, and input the three two-dimensional score data into the three-dimensional convolution kernel;

Receive the shared index data Le of the three-dimensional picture data block, and input the index data Le to the three-dimensional convolution kernel;

The three two-dimensional fraction data is respectively subjected to a two-dimensional convolution operation to obtain three fractional convolution operation results. The index data Le is divided into three two-dimensional index data and the two-dimensional convolution operation is performed to obtain three exponential convolution operation results. Three exponential convolution operation results and three fractional convolution operation results;

Perform frame superposition processing on three exponential convolution operation results and three fractional convolution operation results to obtain superimposed data;

After performing the channel superposition processing on the data after the superposition processing, a preliminary convolution result is obtained, and the preliminary processing result is input to the output buffer.

In a third aspect, a computer-readable storage medium is provided that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the method as provided in the second aspect.

In a fourth aspect, a computer program product is provided. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the method provided in the second aspect.

The implementation of the embodiments of the present application has the following beneficial effects:

It can be seen that the technical solution provided by this application uses the characteristics of fast calculation of FPGA and overcomes the shortcomings of small FPGA cache through floating point blocks, thereby realizing fast processing of 3D convolutional neural network, which has the advantage of fast calculation speed.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application. Those of ordinary skill in the art can obtain other drawings according to the drawings without paying creative labor.

FIG. 1A is a schematic diagram of a two-dimensional convolution.

FIG. 1B is a schematic diagram of a three-dimensional convolution.

FIG. 2A is a schematic diagram of data representation of a floating-point block.

FIG. 2B is a schematic diagram of data representation of another floating point block.

FIG. 3 is a schematic structural diagram of a three-dimensional convolutional neural network computing device according to the present application.

FIG. 4 is a flowchart of a method for implementing a three-dimensional convolution operation provided by the present application.

detailed description

In the following, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms "including" and "having" and any variations thereof in the specification and claims of this application and the drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device containing a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

Reference to "an embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they independent or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The electronic device in this application may include: a server, a smart camera device, a smart phone (such as an Android phone, an iOS phone, a Windows Phone phone, etc.), a tablet computer, a handheld computer, a laptop computer, and a mobile Internet device (MID, Mobile Internet Devices) Or wearable devices, etc., the above electronic devices are merely examples, not exhaustive, and include but are not limited to the above electronic devices. For convenience of description, the above electronic devices are referred to as user equipment (UE), Terminal or electronic device. Of course, in practical applications, the above-mentioned user equipment is not limited to the above-mentioned realization form, and may include, for example, a smart vehicle terminal, a computer device, and the like.

In the apparatus provided by the first aspect, the three-dimensional convolution kernel includes: three two-dimensional fractional convolution kernels and three two-dimensional exponential convolution kernels, wherein:

The two-dimensional fractional convolution kernel is used for a two-dimensional convolution operation of two-dimensional fraction data,

The two-dimensional exponential convolution kernel is used to perform a two-dimensional convolution operation of two-dimensional exponential data.

In the apparatus provided by the first aspect, the apparatus further includes: a pooling processing module and an output module;

The pooling processing module is configured to perform pooling processing on the preliminary convolution result to obtain a final convolution result, and output the final convolution result through an output module.

In the apparatus provided by the first aspect, the score processing block includes a cache, a frame first cache, a frame second cache, a first score cache, a second score cache, and a third score cache; wherein,

The buffer is connected to the frame first buffer and the first score buffer, the frame first buffer is connected to the frame second buffer, the frame first buffer is also connected to the second score buffer, and the frame second buffer is connected to the third score buffer. The first score buffer, the second score buffer, and the third score buffer respectively output two-dimensional score data.

In the device provided by the first aspect, the index core includes a cache and an index cache,

Buffer for receiving three-dimensional score data,

The index buffer is used for buffering three-dimensional score data and dividing the three-dimensional score data into three two-dimensional score data for output.

In the method provided by the second aspect, the method further includes:

The preliminary convolution result is pooled to obtain the final convolution result, and the final convolution result is output through the output module.

In the method provided in the second aspect, splitting the score data Lm into three two-dimensional score data, and inputting the three two-dimensional score data into the three-dimensional convolution kernel specifically include:

The score data Lm is stored in three frame buffers, and the three frame stores respectively input the two-dimensional score data to the three-dimensional convolution kernel.

Referring to FIG. 1A, FIG. 1A is a schematic diagram of a two-dimensional convolution. For a two-dimensional convolution, a picture of each frame has two channels of data, that is, a frame 1 transparent frame represents data of channel 1 of frame 1, frame 1 The gray box represents the data of channel 2 of frame 1. For the calculated data, the data of the two channels of frame 1 are combined into one data frame. For FIG. 1B, FIG. 1B is a schematic diagram of a three-dimensional convolution. Different from the two-dimensional convolution, different frames are also merged. As shown in FIG. 1B, the merged data frame has two channels of frame 1. Data and data of 2 channels of frame 2.

Referring to FIG. 2A, FIG. 2A is a representation of floating point data. Referring to FIG. 2A, FIG. 2A is data of one frame in one channel, as shown in FIG. 2A, and one line of data represents data of one pixel, as shown in FIG. 2A. Indicates that N indicates the data type indicator bit, Le indicates the exponent bit data of the floating point data, and Lm indicates the fractional bit data of the floating point data.

Referring to FIG. 2B, FIG. 2B is a representation of floating point data of the present application. Comparing FIG. 2B with FIG. 2A, it can be seen that for data sharing Le of one channel and one channel of FIG. 2B, relative to each pixel Both have non-shared Le. The floating-point block data representation of Figure 2B can greatly reduce the data storage space, which can match the FPGA cache, avoiding the problem of small caches that cannot effectively store calculation data, and operations between different blocks. In this case, the number of bits required for the fractional operation is reduced, which greatly reduces the use of on-chip computing resources.

Referring to FIG. 3, FIG. 3 provides a three-dimensional convolutional neural network computing device. As shown in FIG. 3, the computing device includes a score processing block 301, an exponential kernel 302, a three-dimensional convolution kernel 303, a frame superimposing block 304, and a channel. Overlay block 305 and output buffer 306;

The score processing block 301 is configured to receive the score data Lm of the three-dimensional picture data block, divide the score data Lm into three two-dimensional score data, and input the three two-dimensional score data into the three-dimensional convolution kernel 303;

The index kernel 302 is configured to receive the shared index data Le of the three-dimensional picture data block, and input the index data Le to the three-dimensional convolution kernel 303;

The three-dimensional convolution kernel 303 is configured to perform three-dimensional convolution operations on three two-dimensional fraction data to obtain three score convolution operation results, divide the index data Le into three two-dimensional index data, and perform two-dimensional convolution operations to obtain three. Exponential convolution operation results, input three exponential convolution operation results and three fractional convolution operation results to the frame superposition block 304;

The frame superposition block 304 is configured to perform frame superposition processing on three exponential convolution operation results and three fractional convolution operation results to obtain superimposed processing data, and input the superimposed processing data to the channel superposition block 305;

A channel superimposing block 305 is configured to obtain a preliminary convolution result after performing channel superposition processing on the data after the superposition processing, and input the preliminary processing result to the output buffer 306.

The technical solution provided in this application first only deals with floating-point block data, and when processing the data of the floating-point block, the index data of the floating-point block needs to be shared index data, so that the three-dimensional convolution operation is divided into three A two-dimensional convolution operation, and then a frame superposition and a channel superposition are performed to implement a three-dimensional convolution operation. Since the floating point block of the technical solution provided by the present application has a small amount of data, its memory space is small, so FPGA The small cache can adapt to the above structure. In addition, its calculation uses three parallel two-dimensional convolution operations. Parallel operations can save calculation time, so it has the advantage of short calculation time.

Optionally, the three-dimensional convolution kernel includes: three two-dimensional fractional convolution kernels and three two-dimensional exponential convolution kernels. The two-dimensional fractional convolution kernel is used for a two-dimensional convolution operation of two-dimensional fraction data. Two-dimensional exponential convolution kernel, used to perform a two-dimensional convolution operation on two-dimensional exponential data.

Optionally, the above computing device may further include a pooling processing module and an output module. The pooling processing module is configured to perform pooling processing on the preliminary convolution result to obtain a final convolution result, and pass the final convolution result through the output module. Output.

Optionally, the score processing block 301 may include: a cache 3011, a frame first cache 3012, a frame second cache 3013, a first score cache 3014, a second score cache 3015, and a third score cache 3016; among them, the cache 3011 and The first frame buffer 3012 and the first score buffer 3014 are connected, the first frame buffer 3012 is connected to the second frame buffer 3013, the first frame buffer 3012 is also connected to the second score buffer 3015, and the second frame buffer 3013 is connected to the third score The buffer 3016 is connected, and the first score buffer 3014, the second score buffer 3015, and the third score buffer 3016 respectively output two-dimensional score data.

Optionally, the index core 302 may include: a cache 3021 and an index cache 3022, a cache 3021 for receiving three-dimensional score data, and an index cache 3022 for buffering three-dimensional score data and dividing the three-dimensional score data into three two-dimensional score data and output .

The technical solution provided by this application is based on block floating point operation, which reduces the bit width and computational complexity, thereby reducing the use of on-chip memory resources and hardware computing resources; based on the new 3D convolutional neural network hardware architecture, it has faster processing Speed and higher accuracy meet the requirements of real-time processing; compared with existing CPU and GPU implementations, it can make behavioral judgments with lower power consumption, and at the same time ensure correct results.

Referring to FIG. 4, FIG. 4 provides a method for a three-dimensional convolution operation. The method includes:

Step S401: Receive score data Lm of a three-dimensional picture data block, divide the score data Lm into three two-dimensional score data, and input the three two-dimensional score data into a three-dimensional convolution kernel;

Step S402: Receive the shared index data Le of the three-dimensional picture data block, and input the index data Le to the three-dimensional convolution kernel;

Step S403: Perform three-dimensional convolution operations on the three two-dimensional score data to obtain three score convolution operation results, divide the index data Le into three two-dimensional index data, and perform two-dimensional convolution operations to obtain three exponential convolution operations. As a result, three exponential convolution operation results and three fractional convolution operation results are obtained;

Step S404: Perform frame superposition processing on the three exponential convolution operation results and the three fractional convolution operation results to obtain data after the superposition processing;

Step S405: Perform the channel superposition processing on the data after the superposition processing to obtain a preliminary convolution result, and input the preliminary processing result to the output buffer.

An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute any one of the three-dimensional convolution operations described in the foregoing method embodiments. Part or all of the steps of a method.

An embodiment of the present application further provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the operations described in the foregoing method embodiments. Part or all of the steps of any method of 3D convolution operation.

It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all described as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action order. Because according to the present application, certain steps may be performed in another order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for this application.

In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not described in detail in one embodiment, reference may be made to related descriptions in other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are merely schematic

In addition, the processors and chips in the various embodiments of the present application may be integrated in one processing unit, or may exist separately physically, or two or more pieces of hardware may be integrated in one unit. The computer-readable storage medium or computer-readable program may be stored in a computer-readable memory. Based on such an understanding, the technical solution of the present application essentially or part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory, Several instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. The foregoing memories include: U disks, Read-Only Memory (ROM), Random Access Memory (RAM), mobile hard disks, magnetic disks, or optical disks and other media that can store program codes.

A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the foregoing embodiments may be completed by a program instructing related hardware. The program may be stored in a computer-readable memory, and the memory may include a flash disk. , Read-only memory (English: Read-Only Memory, referred to as ROM), random access device (English: Random Access Memory, referred to as RAM), magnetic disks or optical disks, etc.

The embodiments of the present application have been described in detail above. Specific examples have been used in this document to explain the principles and implementation of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. Persons of ordinary skill in the art may change the specific implementation and application scope according to the idea of the present application. In summary, the content of this description should not be construed as a limitation on the present application.

Claims (10)

A three-dimensional convolutional neural network computing device, characterized in that the computing device includes: a fraction processing block, an exponential kernel, a three-dimensional convolution kernel, a frame superposition block, a channel superposition block, and an output buffer; The score processing block is used to receive the score data Lm of the three-dimensional picture data block, divide the score data Lm into three two-dimensional score data, and input the three two-dimensional score data into the three-dimensional convolution kernel; The exponential kernel is used to receive the shared exponential data Le of the three-dimensional image data block, and input the exponential data Le to the three-dimensional convolution kernel; A three-dimensional convolution kernel, which is used to perform three-dimensional convolution operations on three two-dimensional fractional data to obtain three fractional convolution operation results, divide the index data Le into three two-dimensional index data, and perform three-dimensional convolution operations to obtain three. Exponential convolution operation results, inputting three exponential convolution operation results and three fractional convolution operation results into the frame superposition block; Frame superposition block, which is used to perform frame superposition processing on three exponential convolution operation results and three fractional convolution operation results to obtain superimposed data, and input the superimposed processing data to the channel superposition block; The channel superposition block is used to obtain the preliminary convolution result after performing the channel superposition processing on the data after the superposition processing, and input the preliminary processing result to the output buffer. The apparatus according to claim 1, wherein the three-dimensional convolution kernel comprises: three two-dimensional fractional convolution kernels and three two-dimensional exponential convolution kernels, wherein: The two-dimensional fractional convolution kernel is used for a two-dimensional convolution operation of two-dimensional fraction data, The two-dimensional exponential convolution kernel is used to perform a two-dimensional convolution operation of two-dimensional exponential data. The device according to claim 1 or 2, further comprising: a pooling processing module and an output module; The pooling processing module is configured to perform pooling processing on the preliminary convolution result to obtain a final convolution result, and output the final convolution result through an output module. The device according to claim 1 or 2, wherein the score processing block comprises: a cache, a frame first cache, a frame second cache, a first score cache, a second score cache, and a third score cache; wherein , The buffer is connected to the frame first buffer and the first score buffer, the frame first buffer is connected to the frame second buffer, the frame first buffer is also connected to the second score buffer, and the frame second buffer is connected to the third score buffer. The first score buffer, the second score buffer, and the third score buffer respectively output two-dimensional score data. The apparatus according to claim 1 or 2, wherein the exponential core comprises: a cache and an exponential cache, Buffer for receiving three-dimensional score data, The index buffer is used for buffering three-dimensional score data and dividing the three-dimensional score data into three two-dimensional score data for output. A method for performing a three-dimensional convolution operation by using the device according to any one of claims 1-5, wherein the method includes: Receive the score data Lm of the three-dimensional picture data block, divide the score data Lm into three two-dimensional score data, and input the three two-dimensional score data into the three-dimensional convolution kernel; Receive the shared index data Le of the three-dimensional picture data block, and input the index data Le to the three-dimensional convolution kernel; The three two-dimensional fraction data is respectively subjected to a two-dimensional convolution operation to obtain three fractional convolution operation results. The index data Le is divided into three two-dimensional index data and the two-dimensional convolution operation is performed to obtain three exponential convolution operation results. Three exponential convolution operation results and three fractional convolution operation results; Perform frame superposition processing on three exponential convolution operation results and three fractional convolution operation results to obtain superimposed data; After performing the channel superposition processing on the data after the superposition processing, a preliminary convolution result is obtained, and the preliminary processing result is input to the output buffer. The method according to claim 6, further comprising: The preliminary convolution result is pooled to obtain the final convolution result, and the final convolution result is output through the output module. The method according to claim 6 or 7, wherein splitting the score data Lm into three two-dimensional score data, and inputting the three two-dimensional score data into the three-dimensional convolution kernel specifically include: The score data Lm is stored in three frame buffers, and the three frame stores respectively input the two-dimensional score data to the three-dimensional convolution kernel. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute the computer program according to any one of claims 1-4. Methods. A computer program product, characterized in that the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute any one of claims 1-4 The method described.

Images