CN112464930A - Target detection network construction method, target detection method, device and storage medium - Google Patents

Abstract

The present application provides a construction method of a target detection network, a target detection method, an apparatus and a computer-readable storage medium. It involves the field of artificial intelligence, specifically the field of computer vision. The construction method includes: determining a search space of the target detection network; determining an initial network architecture of the target detection network according to the search space of the target detection network, and updating the initial network architecture of the target detection network according to the search space of the target detection network Iterate until the target detection network that meets the preset requirements is obtained. Wherein, the optional connection relationship of the feature fusion layer in the target detection network includes any node of one layer of neural network and any node of another layer of neural network between any two adjacent layers of neural networks in the feature fusion layer Connection. The present application can simplify the complexity of the target detection network.

Description

Object detection network construction method, object detection method, device and storage medium

technical field

The present application relates to the field of artificial intelligence, and more particularly, to a method for constructing a target detection network, a method for target detection, an apparatus and a storage medium.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that responds in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

With the rapid development of artificial intelligence technology, neural networks (eg, deep neural networks) have made great achievements in the processing and analysis of various media signals such as images, videos, and speech in recent years. A well-performing neural network often has a delicate network structure, which requires a lot of effort by human experts with high skills and rich experience to construct. In order to better construct a neural network, a neural network structure search (NAS) method is proposed to build a neural network, and a neural network structure with excellent performance can be obtained by automatically searching the neural network structure.

Object detection is one of the basic tasks in the field of computer vision. Object detection generally locates the target object in the image and assigns the corresponding label to the target object. The current mainstream object detection system is mainly composed of a backbone network, a feature fusion layer, a region proposal network (RPN) and a region convolutional neural network (RCNN) head.

The traditional solution is that the target detection network is manually designed by experts according to a certain strategy, which requires a lot of labor cost and time cost, and the performance of the designed target detection network is also relatively general.

SUMMARY OF THE INVENTION

The present application provides a target detection network construction method, target detection method, device and storage medium, so as to construct a target detection network with lower complexity.

In a first aspect, a method for constructing a target detection network is provided, the target detection network includes a backbone network, a feature fusion layer, a regional candidate network RPN and a regional convolutional neural network RCNN, the method includes: determining a search space of the target detection network , wherein the search space of the target detection network includes the search space of the feature fusion layer; the initial network architecture of the target detection network is determined according to the search space of the target detection network; the initial network architecture of the target detection network is determined according to the search space of the target detection network. Iteratively update until the target detection network that meets the preset requirements is obtained.

Wherein, the search space of the above target detection network includes the search space of the feature fusion layer.

The search space of the above-mentioned feature fusion layer includes the optional connection relationship of the feature fusion layer, and the optional connection relationship of the feature fusion layer specifically includes any node of a layer of neural networks in two adjacent layers of neural networks in a multi-layer neural network. A connection to any node in another layer of neural network.

In addition, the feature fusion layer in the initial network architecture of the above target detection network is determined according to the search space of the feature fusion layer. That is, when the initial network architecture of the target detection network is determined according to the search space of the target detection network, the network architecture of the feature fusion layer in the initial network architecture of the target detection network can be specifically determined according to the search space of the feature fusion layer.

It should be understood that when the initial network architecture of the target detection network is determined in the search space of the target detection network, the number of network layers of the target detection network and the number of nodes included in the target detection network may be predetermined. The application requirements of the network or the requirements of target detection performance determine the number of network layers of the target detection network and the number of nodes contained in the target detection network.

For example, when the target detection performance of the target detection network is required to be high, the number of network layers of the target detection network and the number of nodes contained in the target detection network can be relatively large, and when the target detection speed/complexity of the target detection network is required. When it is higher, the number of network layers of the target detection network and the number of nodes contained in the target detection network can be relatively small.

In this application, since the search space of the feature fusion layer contains more optional connection relationships of the feature fusion layer, this application can more reasonably determine the initial network architecture of the target detection network according to more optional connection relationships The network architecture of the feature fusion layer in the feature fusion layer is updated, and the network architecture of the feature fusion layer is updated, which can simplify the complexity of the final target detection network.

Specifically, in the present application, for the search space of the feature fusion layer, since a search space with a more free optional connection relationship is adopted, compared with the way of manually setting the network architecture, the present application uses feature fusion according to the search space. The search space of the layer determines the network architecture of the feature fusion layer in the initial network architecture of the target detection network, and when the network architecture of the feature fusion layer is updated, a more simplified network structure of the feature fusion layer can be obtained, which can finally simplify the target. The complexity of the detection network is reduced, and the storage space required for the deployment of the target detection network is reduced.

In addition, since the search space of the feature fusion layer contains more optional connection relationships, the network architecture of the feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, and the features The network architecture of the fusion layer is updated, and finally a target detection network with better performance can be constructed.

The method for constructing a target detection network in the first aspect may be an automatic method for constructing a neural network, and the method for constructing a target detection network in the first aspect may be automatically executed by a device for constructing a target detection network.

Optionally, in the above target detection network, the network structure of the backbone network and the network structure of the RPN are predetermined.

That is to say, in the process of updating the initial network architecture of the target detection network, only the feature fusion layer and the RCNN in the initial network architecture of the target detection network may be updated.

In addition, the network architecture of the above-mentioned backbone network and the network architecture of the RPN may also be a network architecture that has not been determined in advance, so that in the process of updating the initial network architecture of the target detection network, the backbone network and features of the target detection network can be updated. The network architecture of fusion layer, RPN and RCNN is updated.

With reference to the first aspect, in some implementations of the first aspect, the above-mentioned initial network architecture of the target detection network is iteratively updated according to the search space of the target detection network, until a target detection network that meets the preset requirements is obtained, including: According to the search space of the target detection network, the initial network architecture of the target detection network is iteratively updated to reduce the value of the loss function corresponding to the target detection network, thereby obtaining a target detection network that meets the preset requirements.

Wherein, the above-mentioned loss function includes the target detection error of the target detection network and/or the complexity of the target detection network.

It should be understood that in the above iterative update process, the initial network architecture of the target detection network can be iteratively updated according to the value of the loss function corresponding to the target detection network, so that the value of the loss function corresponding to the target detection network is as small as possible. , until the target detection network that meets the preset requirements is obtained.

Specifically, when the initial network architecture of the target detection network is iteratively updated, the network structure of the target detection network (the connection relationship between different nodes in the network) can be adjusted, and the corresponding target detection network is calculated after each adjustment. Then, according to the value of the loss function corresponding to the target detection network, the network structure of the target detection network is updated, and so on, until the target detection network that meets the preset requirements is obtained.

In the above iterative update process, the value of the loss function corresponding to the target detection network can be calculated after each update of the network architecture of the target detection network. If the value of the loss function meets the requirements, the update of the target detection network architecture is stopped. Update, the target detection network architecture obtained at this time is the target detection network that meets the preset requirements. If the value of the loss function does not meet the requirements, you can continue to update the network parameters of the target detection network according to the value of the loss function, until A target detection network that meets the preset requirements is obtained.

With reference to the first aspect, in some implementations of the first aspect, the search space of the feature fusion layer further includes an optional operation type of the feature fusion layer, and the optional operation type of the feature fusion layer includes a phase in a multi-layer neural network. The convolution operation corresponding to the connection between any node of one layer of neural network in the adjacent two-layer neural network and any node of the other layer of neural network.

Wherein, the above-mentioned convolution operation includes a hole convolution operation.

In this application, when the optional operation type in the feature fusion layer includes a hole convolution operation, it can achieve substantially the same target detection performance with fewer convolution parameters.

In addition, when the optional operation type in the feature fusion layer includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, in the case of the same amount of parameters, using atrous convolution to perform convolution processing can achieve a larger receptive field than traditional convolution, so the final target detection network can have better target detection performance.

With reference to the first aspect, in some implementations of the first aspect, the RCNN includes multiple basic units, each of the multiple basic units is composed of at least two nodes, and the search space of the target detection network further includes the RCNN. Search space, the search space of RCNN includes the search space of each basic unit in the plurality of basic units, the search space of each basic unit includes the optional connection relationship of each basic unit, and the optional connection relationship of each basic unit includes The connection between any two nodes within each basic unit; the RCNN in the initial network architecture of the object detection network is determined according to the search space of the RCNN.

It should be understood that in each of the above basic units, nodes are connected in the direction from input to output.

In this application, in the search space of RCNN, since any two nodes in each basic unit of RCNN can be connected, this application can more reasonably determine the search space of RCNN according to the more relaxed RCNN search space. The initial network structure and updating the initial network structure of RCNN can simplify the complexity of the target detection network.

Further, in this application, when the feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, and the network architecture of the feature fusion layer is updated, the target detection network is determined according to the search space of the RCNN. The network architecture of the RCNN in the initial network architecture, and when the network architecture of the RCNN is updated, the network structure of the feature fusion layer and the network structure of the RCNN can be updated and optimized at the same time, so that the final optimized RCNN network can be obtained. The structure is more matched with the network structure of the feature fusion layer, and a target detection network with better target detection performance can be obtained.

With reference to the first aspect, in some implementations of the first aspect, the search space of each basic unit further includes an optional operation type of each basic unit, and the optional operation type of each basic unit includes The convolution operation corresponding to the connection between any two nodes of , the convolution operation includes the hole convolution operation.

In this application, when the optional operation type in RCNN includes atrous convolution operation, it can achieve substantially the same target detection performance with fewer convolution parameters.

In addition, when the optional operation type of RCNN includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, in the case of the same amount of parameters, using atrous convolution to perform convolution processing can achieve a larger receptive field than traditional convolution, so the final target detection network can have better target detection performance.

With reference to the first aspect, in some implementations of the first aspect, the convolution operation corresponding to the connection between any two nodes in each basic unit includes an atrous convolution operation with an interval of 2.

Since the resolution of the feature map processed in RCNN is generally low, it is more suitable to use a hole convolution operation with a small number of intervals. Therefore, for each basic unit in RCNN, when the optional operation includes the number of intervals is 2 When the atrous convolution is used, the performance of the final RCNN can be improved, thereby improving the performance of the final target detection network.

With reference to the first aspect, in some implementations of the first aspect, at least two basic units in the plurality of basic units are respectively composed of different numbers of nodes.

In this application, when at least two basic units in the multiple basic units in the RCNN can be composed of different numbers of nodes, the composition of the basic units in the RCNN is more free, so that the initial network structure of the RCNN can be determined and the RCNN When the initial network structure is updated, the possibility of increasing the network structure of RCNN is easy to search for a better network structure of RCNN, which makes it more likely to obtain a target detection network with better target detection performance in the end.

In conjunction with the first aspect, in some implementations of the first aspect, the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

When the resolution of the input feature map of each basic unit in RCNN is the same as the resolution of the output feature map of each basic unit, each basic unit in RCNN does not change the resolution of the feature map when processing the feature map, It is convenient to retain the information of the feature map. In addition, it is also beneficial to jump links. Otherwise, each jump link needs to align the output and the size of the feature map to be input, which is inefficient.

With reference to the first aspect, in some implementations of the first aspect, the target detection network that meets the preset requirements satisfies at least one of the following conditions: the detection performance of the target detection network meets the preset performance requirements; The number of updates of the network architecture is greater than or equal to the preset number of times; the complexity of the target detection network is less than or equal to the preset complexity.

With reference to the first aspect, in some implementations of the first aspect, the complexity of the target detection network is based on the number or size of model parameters of the target detection network, the memory access cost MAC of the target detection network, and the floating point of the target detection network. At least one of the number of operations is determined.

In a second aspect, a method for constructing a target detection network is provided. The target detection network includes a backbone network, a feature fusion layer, a regional candidate network RPN and a regional convolutional neural network RCNN. The method includes: determining a search space of the target detection network, The RCNN includes multiple basic units, each of the multiple basic units is composed of at least two nodes, the search space of the target detection network includes the RCNN search space, and the RCNN search space includes each of the multiple basic units. The search space of each basic unit, the search space of each basic unit includes the optional connection relationship of each basic unit, and the optional connection relationship of each basic unit includes the connection between any two nodes in each basic unit; The initial network architecture of the target detection network is determined according to the search space of the target detection network, wherein the RCNN in the initial network architecture of the target detection network is determined according to the search space of the RCNN; The initial network architecture is iteratively updated until a target detection network that meets the preset requirements is obtained.

It should be understood that the search space of the RCNN may consist of the search space of each basic unit in the RCNN.

In this application, in the search space of RCNN, any two nodes in each basic unit of RCNN can be connected. Space to more reasonably determine the initial network structure of RCNN, and update the initial network structure of RCNN, which can simplify the complexity of the target detection network.

In addition, since the search space of RCNN contains more optional connection relationships, the RCNN in the initial network architecture of the target detection network is determined according to the search space of RCNN, and the network architecture of RCNN is updated to finally construct A target detection network with better performance is obtained.

The method for constructing a target detection network in the second aspect may be an automatic method for constructing a neural network, and the method for constructing a target detection network in the second aspect may be automatically executed by a device for constructing a target detection network.

Optionally, the network architecture of the backbone network and the network architecture of the RPN are predetermined.

If the network architecture of the backbone network and the network architecture of the RPN are predetermined, then when updating the initial network architecture of the target detection network, only the feature fusion layer and the feature fusion layer in the initial network architecture of the target detection network can be updated. RCNN is updated.

With reference to the second aspect, in some implementations of the second aspect, the initial network architecture of the target detection network is iteratively updated according to the search space of the target detection network until a target detection network that meets the preset requirements is obtained, including: In the search space of the target detection network, the initial network architecture of the target detection network is iteratively updated to reduce the value of the loss function corresponding to the target detection network, thereby obtaining a target detection network that meets the preset requirements.

Wherein, the above-mentioned loss function includes the target detection error of the target detection network and/or the complexity of the target detection network.

In the above iterative update process, the initial network architecture of the target detection network can be iteratively updated according to the value of the loss function corresponding to the target detection network, so that the value of the loss function corresponding to the target detection network is as small as possible, until the value of the loss function corresponding to the target detection network is obtained. Object detection network that meets preset requirements.

Specifically, in the above iterative update process, the value of the loss function corresponding to the target detection network can be calculated after each update of the network architecture of the target detection network, and if the value of the loss function meets the requirements, the update of the target detection network is stopped. The update of the network architecture. The target detection network architecture obtained at this time is the target detection network that meets the preset requirements. If the value of the loss function does not meet the requirements, the network of the target detection network can be updated according to the value of the loss function. parameters until a target detection network that meets the preset requirements is obtained.

With reference to the second aspect, in some implementations of the second aspect, the search space of each basic unit further includes an optional operation type of each basic unit, and the optional operation type of each basic unit includes The convolution operation corresponding to the connection between any two nodes of , the convolution operation includes the hole convolution operation.

When the optional operation type in RCNN includes atrous convolution operation, it can achieve basically the same object detection performance with fewer convolution parameters.

When the optional operation type of RCNN includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, in the case of the same amount of parameters, using atrous convolution to perform convolution processing can achieve a larger receptive field than traditional convolution, so the final target detection network can have better target detection performance.

With reference to the second aspect, in some implementations of the second aspect, the convolution operation corresponding to the connection between any two nodes in each basic unit includes an atrous convolution operation with an interval of 2.

Since the resolution of the feature map processed in RCNN is generally low, it is more suitable to use a hole convolution operation with a small number of intervals. Therefore, for each basic unit in RCNN, when the optional operation includes the number of intervals is 2 When the atrous convolution is used, the performance of the final RCNN can be improved, thereby improving the performance of the final target detection network.

In conjunction with the second aspect, in some implementations of the second aspect, at least two of the plurality of basic units are respectively composed of different numbers of nodes.

When at least two of the basic units in the RCNN can be composed of different numbers of nodes, the composition of the basic units in the RCNN is more free, so that the initial network structure of the RCNN can be determined and the initial network structure of the RCNN can be determined. When updating, the possibility of increasing the network structure of RCNN is convenient to search for a better network structure of RCNN, which makes it more likely to obtain a target detection network with better target detection performance in the end.

In conjunction with the second aspect, in some implementations of the second aspect, the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

When the resolution of the input feature map of each basic unit in the RCNN is the same as the resolution of the output feature map of each basic unit, it means that each basic unit in the RCNN does not change the resolution of the feature map when processing the feature map , which is convenient to retain the information of the feature map.

In combination with the second aspect, in some implementations of the second aspect, the search space of the target detection network further includes the search space of the feature fusion layer, and the search space of the feature fusion layer includes the optional connection relationship of the feature fusion layer, and the feature fusion layer The optional connection relationship of the multi-layer neural network includes the connection between any node of one layer of neural network in the adjacent two-layer neural network in the multi-layer neural network and any node in the other layer of neural network; in the initial network architecture of the target detection network The feature fusion layer of is determined according to the search space of the feature fusion layer.

In this application, when the feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, and the network architecture of the feature fusion layer is updated, the initial network of the target detection network is determined according to the search space of the RCNN When the RCNN in the architecture is updated, the network structure of the feature fusion layer and the network structure of the RCNN can be updated and optimized at the same time, so that the final optimized RCNN network structure and feature fusion layer. The network structure is more matched, and the target detection network with better target detection performance can be obtained.

With reference to the second aspect, in some implementations of the second aspect, the search space of the feature fusion layer further includes optional operation types of the feature fusion layer, and the optional operation types of the feature fusion layer include adjacent ones in the multi-layer neural network. The convolution operation corresponding to the connection between any node of one layer of neural network in the two-layer neural network and any node of the other layer of neural network, wherein, the adjacent two layers of neural network in the multi-layer neural network The convolution operation corresponding to the connection between any node of one layer of neural network and any node of another layer of neural network includes atrous convolution operation.

When the optional operation types in the feature fusion layer include atrous convolution operations, substantially the same object detection performance can be achieved with fewer convolution parameters.

In addition, when the optional operation type in the feature fusion layer includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, in the case of the same amount of parameters, using atrous convolution to perform convolution processing can achieve a larger receptive field than traditional convolution, so the final target detection network can have better target detection performance.

With reference to the second aspect, in some implementations of the second aspect, the target detection network that meets the preset requirements satisfies at least one of the following conditions: the detection performance of the target detection network meets the preset performance requirements; The update times of the network architecture are greater than or equal to the preset times; the complexity of the target detection network is less than or equal to the preset complexity.

With reference to the second aspect, in some implementations of the second aspect, the complexity of the target detection network is based on the number or size of model parameters of the target detection network, the memory access cost MAC of the target detection network, and the floating point of the target detection network. At least one of the number of operations is determined.

In a third aspect, a target detection method is provided, the method includes: acquiring an image; using a target detection network to process the image to obtain a target detection result of the image, where the target detection result includes the location of the detection target in the image and the detection result. The classification result of the target; the target detection network includes a backbone network, a feature fusion layer, a regional candidate network RPN, and a regional convolutional neural network RCNN. The target detection network is a detection network that meets preset requirements, and the target detection network is based on target detection. The search space of the network is obtained by iteratively updating the initial network architecture of the target detection network. The initial network architecture of the target detection network is determined according to the search space of the target detection network; the search space of the target detection network includes the search space of the feature fusion layer, The feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer. The search space of the feature fusion layer includes the optional connection relationship of the feature fusion layer, and the optional connection relationship of the feature fusion layer includes multiple layers. The connection between any node of one layer of neural network in the adjacent two-layer neural network in the neural network and any node of the other layer of neural network.

Since the target detection network adopted by the above target detection method contains more optional connection relationships of the feature fusion layer in the search space of the feature fusion layer used in the construction process, the feature fusion layer determined according to the search space of the feature fusion layer contains more optional connection relationships. The layer can perform better feature fusion, so that the final target detection network has better performance in target detection.

With reference to the third aspect, in some implementations of the third aspect, the search space of the feature fusion layer further includes optional operation types of the feature fusion layer, and the optional operation types of the feature fusion layer include adjacent ones in the multi-layer neural network. A convolution operation corresponding to the connection between any node of one layer of neural network in the two-layer neural network and any node of another layer of neural network, wherein the convolution operation includes a hole convolution operation.

When the optional operation types in the feature fusion layer include atrous convolution operations, substantially the same object detection performance can be achieved with fewer convolution parameters. In addition, when the convolution parameters have the same amount of parameters, the convolution processing using atrous convolution can obtain a larger receptive field than the traditional convolution, so the final target detection network can have better target detection. performance.

With reference to the third aspect, in some implementations of the third aspect, the RCNN includes multiple basic units, each of the multiple basic units is composed of at least two nodes, and the search space of the target detection network also includes the RCNN. Search space, the search space of RCNN includes the search space of each basic unit in the plurality of basic units, the search space of each basic unit includes the optional connection relationship of each basic unit, and the optional connection relationship of each basic unit includes The connection between any two nodes within each basic unit; the RCNN in the initial network architecture of the object detection network is determined according to the search space of the RCNN.

With reference to the third aspect, in some implementations of the third aspect, the search space of each basic unit further includes an optional operation type of each basic unit, and the optional operation type of each basic unit includes The convolution operation corresponding to the connection between any two nodes of , and the convolution operation corresponding to the connection between any two nodes in each basic unit includes the hole convolution operation.

When the optional operation type in RCNN includes atrous convolution operation, it can achieve basically the same object detection performance with fewer convolution parameters. In addition, when the optional operation type of RCNN includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, when the convolution parameters have the same amount of parameters, the convolution processing using atrous convolution can achieve a larger receptive field than the traditional convolution. Therefore, using the above target detection network has better target detection performance.

With reference to the third aspect, in some implementations of the third aspect, the convolution operation corresponding to the connection between any two nodes in each basic unit includes an atrous convolution operation with an interval of 2.

Since the resolution of the feature map processed in RCNN is generally low, it is more suitable to use a hole convolution operation with a small number of intervals. For each basic unit in RCNN, when the optional operation includes holes with a number of intervals of 2 When convolution, it can finally improve the target detection performance of the target detection network.

With reference to the third aspect, in some implementations of the third aspect, at least two basic units in the plurality of basic units are respectively constituted by different numbers of nodes.

In conjunction with the third aspect, in some implementations of the third aspect, the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

When the resolution of the input feature map of each basic unit in RCNN is the same as the resolution of the output feature map of each basic unit, each basic unit in RCNN does not change the resolution of the feature map when processing the feature map, It is convenient to retain the information of the feature map, thereby ensuring the target detection performance of the target detection network.

With reference to the third aspect, in some implementations of the third aspect, the above-mentioned target detection network satisfies at least one of the following conditions: the detection performance of the target detection network meets the preset performance requirements; the update of the network architecture of the target detection network The number of times is greater than or equal to the preset number of times; the complexity of the target detection network is less than or equal to the preset complexity.

With reference to the third aspect, in some implementations of the third aspect, the complexity of the above target detection network is based on the number or size of model parameters of the target detection network, the memory access cost MAC of the target detection network, and the floating point of the target detection network. At least one of the number of point operations is determined.

In a fourth aspect, a target detection method is provided, the method comprising: acquiring an image; using a target detection network to process the image to obtain a target detection result of the image, where the target detection result includes the position and detection result of the detection target in the image The classification result of the target; the target detection network includes a backbone network, a feature fusion layer, a regional candidate network RPN, and a regional convolutional neural network RCNN. The target detection network is a detection network that meets preset requirements, and the target detection network is based on target detection. The search space of the network is obtained by iteratively updating the initial network architecture of the target detection network. The initial network architecture of the target detection network is determined according to the search space of the target detection network; RCNN includes multiple basic units, and each of the multiple basic units. Each basic unit consists of at least two nodes, the search space of the target detection network includes the search space of RCNN, the search space of RCNN includes the search space of each basic unit in the plurality of basic units, and the search space of each basic unit includes each The optional connection relationship of each basic unit, the optional connection relationship of each basic unit includes the connection between any two nodes in each basic unit, the RCNN in the initial network architecture of the target detection network is based on the search space of RCNN definite.

Since the target detection network adopted by the above target detection method contains more optional connection relationships of RCNN in the search space of RCNN adopted in the construction process, the RCNN determined according to the search space of RCNN can better perform feature fusion. , so that the final target detection network has better performance in target detection. In addition, since the network structure of the RCNN in the initial network architecture of the target detection network can be more reasonably determined according to more optional connection relationships and the network architecture of the RCNN can be updated, the complexity of the final target detection network can be simplified. .

Specifically, in the present application, for the search space of RCNN, since the search space with optional connection relationship is more free, compared with the way of manually setting the network architecture, the present application uses the search space based on RCNN. Determine the RCNN in the initial network architecture of the target detection network, and when the network architecture of the RCNN is updated, a more simplified network structure of the RCNN can be obtained, which can finally simplify the complexity of the target detection network and reduce the target detection network. occupied storage space.

With reference to the fourth aspect, in some implementations of the fourth aspect, the search space of each basic unit further includes an optional operation type of each basic unit, and the optional operation type of each basic unit includes The convolution operation corresponding to the connection between any two nodes of , and the convolution operation corresponding to the connection between any two nodes in each basic unit includes the hole convolution operation.

When the optional operation types in RCNN include atrous convolution operations, the object detection network is able to achieve essentially the same object detection performance with fewer convolution parameters. When the optional operation type of RCNN includes atrous convolution operation, the above target detection network can be used for target detection with better detection performance under the condition that the amount of convolution parameters is basically the same.

With reference to the fourth aspect, in some implementations of the fourth aspect, the convolution operation corresponding to the connection between any two nodes in each basic unit includes an atrous convolution operation with an interval of 2.

Since the resolution of the feature map processed in RCNN is generally low, it is more suitable to use a hole convolution operation with a small number of intervals. Therefore, for each basic unit in RCNN, when the optional operation includes the number of intervals is 2 When the atrous convolution is used, the performance of the final RCNN can be improved, thereby improving the performance of the final target detection network.

With reference to the fourth aspect, in some implementations of the fourth aspect, at least two basic units in the plurality of basic units are respectively composed of different numbers of nodes.

In conjunction with the fourth aspect, in some implementations of the fourth aspect, the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

When the resolution of the input feature map of each basic unit in the RCNN is the same as the resolution of the output feature map of each basic unit, it means that each basic unit in the RCNN does not change the resolution of the feature map when processing the feature map , which is convenient to retain the information of the feature map.

In combination with the fourth aspect, in some implementations of the fourth aspect, the search space of the target detection network further includes the search space of the feature fusion layer, and the search space of the feature fusion layer includes the optional connection relationship of the feature fusion layer, and the feature fusion layer The optional connection relationship of the multi-layer neural network includes the connection between any node of one layer of the neural network in the two adjacent layers of the multi-layer neural network and any node in the other layer of the neural network; in the initial network architecture of the target detection network The feature fusion layer of is determined according to the search space of the feature fusion layer.

Since the target detection network adopted by the above target detection method contains more optional connection relationships of the feature fusion layer in the search space of the feature fusion layer used in the construction process, the feature fusion layer determined according to the search space of the feature fusion layer contains more optional connection relationships. The layer can perform better feature fusion, so that the final target detection network has better performance in target detection.

With reference to the fourth aspect, in some implementations of the fourth aspect, the search space of the feature fusion layer further includes optional operation types of the feature fusion layer, and the optional operation types of the feature fusion layer include adjacent ones in the multi-layer neural network. The convolution operation corresponding to the connection between any node of one layer of neural network in the two-layer neural network and any node in the other layer of neural network, wherein, the adjacent two-layer neural network in the multi-layer neural network is the convolution operation. The convolution operation corresponding to the connection between any node of one layer of neural network and any node of another layer of neural network includes atrous convolution operation.

When the optional operation types in the feature fusion layer include atrous convolution operations, substantially the same object detection performance can be achieved with fewer convolution parameters. In addition, when the convolution parameters have the same amount of parameters, the convolution processing using atrous convolution can obtain a larger receptive field than the traditional convolution, so the final target detection network can have better target detection. performance.

With reference to the fourth aspect, in some implementations of the fourth aspect, the target detection network satisfies at least one of the following conditions: the detection performance of the target detection network meets the preset performance requirements; the number of updates to the network architecture of the target detection network Greater than or equal to the preset number of times; the complexity of the target detection network is less than or equal to the preset complexity.

With reference to the fourth aspect, in some implementations of the fourth aspect, the complexity of the target detection network is based on the number or size of model parameters of the target detection network, the memory access cost MAC of the target detection network, and the floating point of the target detection network. At least one of the number of operations is determined.

In a fifth aspect, an apparatus for constructing a target detection network is provided, where the apparatus includes a module for executing the method in any one of the implementation manners of the first aspect or the second aspect.

In a fifth aspect, a target detection apparatus is provided. The apparatus includes a module for executing the method in any one of the implementation manners of the third aspect or the fourth aspect.

In a sixth aspect, a device for constructing a target detection network is provided, the device comprising: a memory for storing a program; a processor for executing the program stored in the memory, when the program stored in the memory is executed, The processor is configured to execute the method in any one of the implementation manners of the first aspect or the second aspect.

In a seventh aspect, a target detection device is provided, the device comprising: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processing The device is configured to execute the method in any one of the implementation manners of the third aspect or the fourth aspect.

In an eighth aspect, a computer-readable medium is provided, where the computer-readable medium stores program codes for device execution, the program codes including methods for executing any one of the implementation manners of the first aspect to the fourth aspect .

In a ninth aspect, there is provided a computer program product containing instructions, which, when the computer program product is run on a computer, causes the computer to execute the method in any one of the implementation manners of the first aspect to the fourth aspect.

A tenth aspect provides a chip, the chip includes a processor and a data interface, the processor reads an instruction stored in a memory through the data interface, and executes any one of the first to fourth aspects above method in the implementation.

Optionally, as an implementation manner, the chip may further include a memory, in which instructions are stored, the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the The processor is configured to execute the method in any one of the implementation manners of the first aspect to the fourth aspect.

Description of drawings

1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application;

2 is a schematic diagram of target detection using the convolutional neural network model provided by an embodiment of the present application;

3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

4 is a schematic diagram of a system architecture provided by an embodiment of the present application;

5 is a schematic diagram of a target detection system according to an embodiment of the present application;

FIG. 6 is a schematic diagram of the application of the target detection system in the field of automatic driving;

7 is a schematic flowchart of a method for constructing a target detection network according to an embodiment of the present application;

Fig. 8 is a possible structural schematic diagram of the feature fusion layer;

FIG. 9 is a schematic diagram of a possible structure of RCNN;

FIG. 10 is a schematic diagram of a possible structure of the feature fusion layer;

Figure 11 is a schematic diagram of a possible structure of RCNN;

12 is a schematic flowchart of a method for constructing a target detection network according to an embodiment of the present application;

13 is a schematic flowchart of a target detection method according to an embodiment of the present application;

14 is a schematic block diagram of an apparatus for constructing a target detection network according to an embodiment of the present application;

15 is a schematic block diagram of a target detection apparatus according to an embodiment of the present application;

16 is a schematic block diagram of an apparatus for constructing a target detection network according to an embodiment of the present application;

FIG. 17 is a schematic block diagram of a target detection apparatus according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

The solution of the present application can be applied in the fields of assisted driving, automatic driving, safe city, intelligent terminal and other fields that need to perform target detection (for example, detection of pedestrians in images). The two most commonly used application scenarios are briefly introduced below.

Application Scenario 1: Assisted/Autonomous Driving System

In advanced driving assistant system (ADAS) and autonomous driving system (ADS), it is necessary to detect and avoid pedestrians or obstacles on the road, especially to avoid collision with pedestrians, which requires more Accurate object detection.

Application Scenario 2: Safe City/Video Surveillance System

In the safe city system and video surveillance system, the target detection (detection of pedestrians or vehicles) is carried out in real time, the detection results are marked, and the detection results can be used in the analysis unit of the system to find criminal suspects, missing persons and specific vehicles, etc. .

The solution of the present application involves the construction of a neural network and the use of a neural network for target detection. In order to better understand the solution of the present application, the related terms and concepts of the neural network are first introduced below.

(1) Neural network

A neural network can be composed of neural units, and a neural unit can refer to an operation unit that takes x _s and an intercept 1 as inputs, and the output of the operation unit can be shown in formula (1):

Among them, s=1, 2, ... n, n is a natural number greater than 1, W _s is the weight of x _s , and b is the bias of the neural unit. f is an activation function of the neural unit, and the activation function is used to perform nonlinear transformation on the features in the neural network, so as to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting a plurality of the above single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be an area composed of several neural units.

(2) Deep neural network

Deep neural network (deep neural network, DNN), also known as multi-layer neural network, can be understood as a neural network with multiple hidden layers. The DNN is divided according to the positions of different layers. The neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

其中，

是输入向量，

是输出向量，

是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量

经过如此简单的操作得到输出向量

由于DNN层数多，系数W和偏移向量

的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例，假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为

上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。Although DNN looks complicated, it is not complicated in terms of the work of each layer. In short, it is the following linear relationship expression:

in,

is the input vector,

is the output vector,

is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is just an input vector

After such a simple operation to get the output vector

Due to the large number of DNN layers, the coefficient W and offset vector

The number is also higher. The definitions of these parameters in DNN are as follows: Taking the coefficient W as an example, suppose that in a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as

The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

To sum up, the coefficient from the kth neuron in the L-1 layer to the jth neuron in the Lth layer is defined as

It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better capture the complexities of the real world. In theory, a model with more parameters is more complex and has a larger "capacity", which means that it can complete more complex learning tasks. Training the deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vectors W of many layers).

(3) Convolutional Neural Network

Convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. The convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal. In a convolutional layer of a convolutional neural network, a neuron can only be connected to some of its neighbors. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangle. Neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of location. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights by learning during the training process of the convolutional neural network. In addition, the immediate benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

(4) Residual network

Residual network is a deep convolutional network proposed in 2015. Compared with traditional convolutional neural network, residual network is easier to optimize and can improve the accuracy by increasing a considerable depth. The core of the residual network is to solve the side effect (degeneration problem) of increasing the depth, which can improve the network performance by simply increasing the network depth. Residual networks generally contain many sub-modules with the same structure. Usually, a residual network (ResNet) is used to connect a number to indicate the number of repetitions of the sub-modules. For example, ResNet50 indicates that there are 50 sub-modules in the residual network.

(6) Classifier

Many neural network architectures end up with a classifier that classifies objects in an image. The classifier is generally composed of a fully connected layer and a softmax function (which can be called a normalized exponential function), which can output the probability of different categories according to the input.

(7) Loss function

In the process of training a deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then based on the difference between the two to update the weight vector of each layer of neural network (of course, there is usually an initialization process before the first update, that is, to pre-configure parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make the prediction lower, and keep adjusting until the deep neural network can predict the real desired target value or a value very close to the real desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is used to measure the difference between the predicted value and the target value. important equation. Among them, taking the loss function as an example, the higher the output value of the loss function (loss), the greater the difference, then the training of the deep neural network becomes the process of reducing the loss as much as possible.

(8) Back propagation algorithm

The neural network can use the error back propagation (BP) algorithm to correct the values of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the input signal is passed forward until the output will generate error loss, and the parameters in the initial neural network model are updated by back-propagating the error loss information, so that the error loss converges. The back-propagation algorithm is a back-propagation movement dominated by error loss, aiming to obtain the parameters of the optimal neural network model, such as the weight matrix.

Some basic contents of neural networks are briefly introduced above. The following is an introduction to some specific neural networks that may be used in image data processing.

The following describes the system architecture of the embodiment of the present application in detail with reference to FIG. 1 .

FIG. 1 is a schematic diagram of a system architecture of an embodiment of the present application. As shown in FIG. 1 , the system architecture 100 includes an execution device 110 , a training device 120 , a database 130 , a client device 140 , a data storage system 150 , and a data acquisition system 160 .

In addition, the execution device 110 includes a calculation module 111 , an I/O interface 112 , a preprocessing module 113 and a preprocessing module 114 . The calculation module 111 may include the target model/rule 101, and the preprocessing module 113 and the preprocessing module 114 are optional.

The data collection device 160 is used to collect training data. For the target detection method according to the embodiment of the present application, the training data may include a training image (including pedestrians in the training image) and labeling data, wherein the labeling data provides a bounding box of pedestrians in the training picture. )coordinate of. After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model/rule 101 by training based on the training data maintained in the database 130 .

The following describes the target model/rule 101 obtained by the training device 120 based on the training data. The training device 120 performs object detection on the input training image, and outputs the target detection result (the bounding box of objects such as pedestrians and vehicles in the image and the bounding box of the bounding box). The confidence level) is compared with the labeling result until the difference between the target detection result of the object output by the training device 120 and the pre-labeled result is less than a certain threshold, so that the training of the target model/rule 101 is completed.

The above target model/rule 101 can be used to implement the target detection method of the embodiment of the present application, that is, input the image to be processed (after relevant preprocessing) into the target model/rule 101, and then the target detection result of the image to be processed can be obtained . The target model/rule 101 in this embodiment of the present application may specifically be a neural network. It should be noted that, in practical applications, the training data maintained in the database 130 does not necessarily come from the collection of the data collection device 160, and may also be received from other devices. In addition, it should be noted that the training device 120 may not necessarily train the target model/rule 101 completely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application Limitations of Examples.

The target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. Notebook computers, augmented reality (AR)/virtual reality (VR), in-vehicle terminals, etc., may also be servers or the cloud. In FIG. 1, the execution device 110 is configured with an input/output (I/O) interface 112 for data interaction with external devices, and the user can input data to the I/O interface 112 through the client device 140, the In this embodiment of the present application, the input data may include: an image to be processed input by the client device. The client device 140 here may specifically be a terminal device.

The preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112. In this embodiment of the present application, the preprocessing module 113 and the preprocessing module 114 may be absent. Or just a preprocessing module. When the preprocessing module 113 and the preprocessing module 114 do not exist, the calculation module 111 can be directly used to process the input data.

When the execution device 110 preprocesses the input data, or the calculation module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, codes, etc. in the data storage system 150 for corresponding processing , the data and instructions obtained by corresponding processing may also be stored in the data storage system 150 .

Finally, the I/O interface 112 presents the processing result, such as the target detection result calculated by the target model/rule 101, to the client device 140, thereby providing it to the user.

Specifically, the target detection result obtained through the processing of the target model/rule 101 in the calculation module 111 can be processed by the preprocessing module 113 (and the processing by the preprocessing module 114 can also be added), and then the processing result can be sent to the I/O The O interface, and then the I/O interface sends the processing result to the client device 140 for display.

It should be understood that when the preprocessing module 113 and the preprocessing module 114 do not exist in the above-mentioned system architecture 100, the computing module 111 can also transmit the processed target detection result to the I/O interface, and then the I/O interface will process the result. The results are sent to the client device 140 for display.

It is worth noting that the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above task, thus providing the user with the desired result.

In FIG. 1 , the user can manually specify input data, which can be operated through the interface provided by the I/O interface 112 . In another case, the client device 140 can automatically send the input data to the I/O interface 112 . If the user's authorization is required to request the client device 140 to automatically send the input data, the user can set the corresponding permission in the client device 140 . The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form can be a specific manner such as display, sound, and action. The client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data as shown in the figure, and store them in the database 130 . Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly uses the input data input into the I/O interface 112 and the output result of the output I/O interface 112 as shown in the figure as a new sample The data is stored in database 130 .

It is worth noting that FIG. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 1 , the data The storage system 150 is an external memory relative to the execution device 110 , and in other cases, the data storage system 150 may also be placed in the execution device 110 .

As shown in FIG. 1 , the target model/rule 101 obtained by training the training device 120 may be a neural network in the embodiment of the present application. Specifically, the neural network provided in the embodiment of the present application may be a CNN and a deep convolutional neural network ( deep convolutional neural networks, DCNN) and so on.

Since CNN is a very common neural network, the structure of CNN will be introduced in detail in conjunction with Figure 2 below. As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. learning at multiple levels of abstraction. As a deep learning architecture, CNN is a feed-forward artificial neural network in which individual neurons can respond to images fed into it.

As shown in FIG. 2 , a convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional/pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230 . The relevant contents of these layers are described in detail below.

Convolutional layer/pooling layer 220:

Convolutional layer:

As shown in FIG. 2, the convolutional/pooling layer 220 may include layers 221-226 as examples, for example: in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, and layer 223 is a convolutional layer Layer 224 is a pooling layer, 225 is a convolutional layer, and 226 is a pooling layer; in another implementation, 221 and 222 are convolutional layers, 223 are pooling layers, and 224 and 225 are convolutional layers. layer, 226 is the pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or it can be used as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 221 as an example to introduce the inner working principle of a convolutional layer.

The convolution layer 221 may include many convolution operators. The convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially Can be a weight matrix, which is usually pre-defined, usually one pixel by one pixel (or two pixels by two pixels) along the horizontal direction on the input image during the convolution operation on the image. ...It depends on the value of the stride step) to process, so as to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will be extended to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will result in a single depth dimension of the convolutional output, but in most cases a single weight matrix is not used, but multiple weight matrices of the same size (row × column) are applied, That is, multiple isotype matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" described above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to extract unwanted noise in the image. Blur, etc. The multiple weight matrices have the same size (row×column), and the size of the convolution feature maps extracted from the multiple weight matrices with the same size is also the same, and then the multiple extracted convolution feature maps with the same size are combined to form The output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained by training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions .

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (eg, 221 ) often extracts more general features, which can also be called low-level features; with the convolutional neural network As the depth of the network 200 deepens, the features extracted by the later convolutional layers (eg, 226) become more and more complex, such as features such as high-level semantics. Features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the layers 221-226 as shown in 220 in Figure 2, it can be a convolutional layer followed by a layer. The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. During image processing, the only purpose of pooling layers is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a max pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a certain range to produce an average value as the result of average pooling. The max pooling operator can take the pixel with the largest value within a specific range as the result of max pooling. Also, just as the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the output image after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Fully connected layer 230:

After being processed by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not sufficient to output the required output information. Because as mentioned before, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other relevant information), the convolutional neural network 200 needs to utilize the fully connected layer 230 to generate one or a set of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 2) and the output layer 240, and the parameters contained in the multiple hidden layers may be based on specific task types The relevant training data is pre-trained, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.

After the multi-layer hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, the output layer 240 has a loss function similar to the classification cross entropy, and is specifically used to calculate the prediction error, Once the forward propagation of the entire convolutional neural network 200 (as shown in Figure 2, the propagation from the direction 210 to 240 is forward propagation) is completed, the back propagation (as shown in Figure 2, the propagation from the 240 to 210 direction is the back propagation) will Start to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network, and in a specific application, the convolutional neural network may also exist in the form of other network models.

It should be understood that a convolutional neural network (CNN) 200 shown in FIG. 2 can be used to perform the target detection method of the embodiment of the present application. As shown in FIG. 2 , the image to be processed passes through the input layer 210, the convolution layer/pooling layer 220 After processing with the fully connected layer 230, the detection result of the image to be processed (the bounding box of pedestrians in the to-be-processed image and the confidence level of the bounding box of pedestrians in the image) can be obtained.

FIG. 3 is a hardware structure of a chip provided by an embodiment of the application, and the chip includes a neural network processor 50 . The chip can be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111 . The chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101 . The algorithms of each layer in the convolutional neural network shown in Figure 2 can be implemented in the chip shown in Figure 3.

A neural-network processing unit (NPU) 50 is mounted on a main central processing unit (central processing unit, CPU) (host CPU) as a co-processor, and tasks are allocated by the main CPU. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract the data in the memory (weight memory or input memory) and perform operations.

In some implementations, the arithmetic circuit 503 includes multiple processing units (process engines, PEs). In some implementations, arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 503 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 503 fetches the data corresponding to the matrix B from the weight memory 502 and buffers it on each PE in the operation circuit 503 . The operation circuit 503 takes the data of the matrix A from the input memory 501 and performs the matrix operation on the matrix B, and stores the partial result or the final result of the matrix in the accumulator 508 .

The vector calculation unit 507 can further process the output of the operation circuit 503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. For example, the vector computing unit 507 can be used for network computation of non-convolutional/non-FC layers in the neural network, such as pooling, batch normalization, local response normalization, etc. .

In some implementations, the vector computation unit can 507 store the processed output vectors to the unified buffer 506 . For example, the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the vector of processed outputs can be used as activation input to the arithmetic circuit 503, eg, for use in subsequent layers in a neural network.

Unified memory 506 is used to store input data and output data.

The weight data directly transfers the input data in the external memory to the input memory 501 and/or the unified memory 506 through the storage unit access controller 505 (direct memory access controller, DMAC), stores the weight data in the external memory into the weight memory 502, and The data in unified memory 506 is stored in external memory.

A bus interface unit (bus interface unit, BIU) 510 is used to realize the interaction between the main CPU, the DMAC and the instruction fetch memory 509 through the bus.

an instruction fetch buffer 509 connected to the controller 504 for storing instructions used by the controller 504;

The controller 504 is used for invoking the instructions cached in the memory 509 to control the working process of the operation accelerator.

Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is the memory outside the NPU, and the external memory can be double data rate synchronous dynamic random access. Memory (double data rate synchronous dynamic random access memory, referred to as DDR SDRAM), high bandwidth memory (high bandwidth memory, HBM) or other readable and writable memory.

In addition, in this application, the operation of each layer in the convolutional neural network shown in FIG. 2 may be performed by the operation circuit 503 or the vector calculation unit 507 .

As shown in FIG. 4 , an embodiment of the present application provides a system architecture 300 . The system architecture includes a local device 301, a local device 302, an execution device 210 and a data storage system 250, wherein the local device 301 and the local device 302 are connected with the execution device 210 through a communication network.

The execution device 210 may be implemented by one or more servers. Optionally, the execution device 210 may be used in conjunction with other computing devices, such as data storage, routers, load balancers and other devices. The execution device 210 may be arranged on one physical site, or distributed across multiple physical sites. The execution device 210 may use the data in the data storage system 250 or call the program code in the data storage system 250 to implement the method for searching a neural network structure in this embodiment of the present application.

Specifically, the execution device 210 may perform the following processes: determine the search space of the target detection network; determine the initial network architecture of the target detection network according to the search space of the target detection network; The architecture is iteratively updated until the target detection network that meets the preset requirements is obtained.

Through the above process execution device 210, a target neural network can be built, and the target neural network can be used for target detection.

A user may operate respective user devices (eg, local device 301 and local device 302 ) to interact with execution device 210 . Each local device may represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, gaming console, etc.

Each user's local device can interact with the execution device 210 through any communication mechanism/standard communication network, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

In an implementation manner, the local device 301 and the local device 302 obtain the relevant parameters of the target neural network from the execution device 210, deploy the target neural network on the local device 301 and the local device 302, and use the target neural network to perform target detection. .

In another implementation, the target neural network can be directly deployed on the execution device 210, and the execution device 210 obtains the image to be processed from the local device 301 and the local device 302 (the local device 301 and the local device 302 can upload the image to be processed to the Execute the device 210), and perform target detection on the image to be processed according to the target neural network, and send the target detection result to the local device 301 and the local device 302.

The above execution device 210 may also be referred to as a cloud device, and in this case, the execution device 210 is generally deployed in the cloud.

The target detection system is described in detail below with reference to FIG. 5 .

FIG. 5 is a schematic diagram of a target detection system according to an embodiment of the present application.

As shown in Figure 5, the target detection system includes a backbone network (backbone), a feature fusion layer, a region proposal network (RPN) and a region convolutional neural network (region CNN, RCNN). Called the backbone network, the following describes the four network structures in the target detection network in detail.

Backbone network:

The backbone network is used to extract the underlying image information and is a common structure of vision-based deep neural network models. In practice, the backbone network is usually fine-tuned based on the architecture of general deep convolutional neural networks. For example, the backbone network can be fine-tuned based on the architecture of the Visual Geometry Group (VGG) network, a network proposed by the Visual Geometry Group at Oxford University. For another example, the backbone network can also be fine-tuned based on the architecture of a deep residual network (ResNet).

As shown in Figure 5, the backbone network can perform feature extraction on the image to be detected to obtain image features of the image to be detected.

Feature fusion layer:

The feature fusion layer is used to filter and fuse the multi-scale and multi-level features extracted by the backbone network to generate a more compact and expressive feature vector, which is easy to input into the classifier for further processing. Feature fusion layers are widely used in neural network design based on multi-scale and multi-level features. In practical applications, on the one hand, the pyramid structure can be used to adjust the size, shape and weight of features at different scales, and the results can be added and fused into a single feature vector. More expressive multi-level features.

As shown in FIG. 5 , the feature fusion layer performs fusion processing on the image features of the image to be detected extracted by the backbone network to obtain multi-level features of the image to be detected.

RPN:

RPN is a fast regression classifier for generating coarse object location and class label information (class label information). In practical applications, RPN can use a binary classifier and bounding box regression (the bounding box regression is a regression model for target detection, looking for a target location near the sliding window that is closer to the real window, and the loss function value is smaller. The regression window) is implemented by a two-layer simple network.

As shown in Figure 5, RPN processes the multi-level features of the image to be detected obtained by the feature fusion layer, and obtains the preliminary classification and detection results of the target.

RCNN:

RCNN can also be called RCNN head. RCNN is a unique part of the target detection network, which is used to further optimize the preliminary classification detection results obtained by RPN. In practical applications, RCNN is generally implemented by a multi-layer network that is more complex than RPN. Through the combination of RCNN and RPN, the target detection system can quickly remove a large number of invalid image areas, and can concentrate on detecting more potential image areas in detail, thereby improving the effect of target detection.

As shown in Figure 5, RCNN further processes the preliminary classification detection results obtained by RPN to obtain the classification of the target and the bounding box of the target.

The target detection system obtained by using the method for constructing a target detection network in the embodiment of the present application can be applied in an automatic driving scenario. In the target recognition scene of automatic driving, it is necessary to accurately identify the targets such as vehicles and pedestrians on the road. The automatic driving system needs to respond to emergencies on the road in real time, so the target detection system is required to obtain efficient and accurate target recognition results on limited hardware resources. Since the target detection system constructed by the method for constructing the target detection network in the embodiment of the present application can perform target detection more effectively, the target detection system constructed by the method for constructing the target detection network in the embodiment of the present application can improve the target detection effect , to improve the safety performance of autonomous driving.

Figure 6 is a schematic diagram of the application of the target detection system in the field of automatic driving.

As shown in FIG. 6 , video data can be obtained through the on-board camera, and the video data is composed of a series of road pictures. The video data is processed by the target detection module, and the obstacles and the positions of the obstacles existing in the road picture can be detected. Next, the target detection module can input the detection results (obstacles in the road picture and the positions of the obstacles) to the action detection module, so that the action detection module can generate a driving operation signal according to the detection result, and send the driving operation signal To the corresponding automatic driving executive equipment, so as to realize the automatic control of the vehicle.

In addition, the target detection system in the target detection module of FIG. 6 may be a neural network (model) obtained according to the construction method of the target detection network in the embodiment of the present application.

FIG. 7 is a schematic flowchart of a method for constructing a target detection network according to an embodiment of the present application. The method shown in FIG. 7 may be executed by the apparatus for constructing a target detection network in this embodiment of the present application (for example, the method shown in FIG. 7 may be executed by the apparatus shown in FIG. 14 or FIG. 16 below), as shown in FIG. 7 The method includes steps 1001 to 1003, and these steps are described in detail below.

1001. Determine a search space of the target detection network.

The search space of the above target detection network includes a search space that can feature fusion layers.

For the search space of the feature fusion layer, the optional connection relationship of the feature fusion layer can be included.

Specifically, the optional connection relationship of the feature fusion layer may include any node of one layer of neural network in two adjacent layers of neural network in the multi-layer neural network of the feature fusion layer and any node of another layer of neural network Connection.

For example, as shown in Fig. 8, the feature fusion layer includes A-layer neural network, B-layer neural network and C-layer neural network. Among them, the constituent nodes of each layer of neural network are as follows.

A-layer neural network: P_{0}_1, P_{0}_2, P_{0}_3, and P_{0}_4.

B-layer neural network: P_{1}_1, P_{2}_2, P_{3}_3, and P_{4}_4.

C-layer neural network: P_{2}_1, P_{2}_2, P_{2}_3, and P_{2}_4.

In Figure 8, any node in the A layer neural network can be connected with any node in the B layer neural network, and any node in the B layer neural network can be connected with any node in the C layer neural network.

Since the A-layer neural network to the B-layer neural network, and from the B-layer neural network to the C-layer neural network, considering the jump links between all layers, a feature fusion layer with a better network structure can be constructed.

For convenience of display, FIG. 8 only shows the connection relationship between a node in the layer A neural network and the node in the layer B neural network, and the connection relationship between a node in the layer B and the node in the layer C neural network.

Specifically, as shown in Fig. 8, the node P_{0}_2 in the A layer neural network can be connected with the nodes P_{1}_1, P_{1}_2, P_{1}_3 and P_{ in the B layer neural network 1}_4 connection; the node P_{1}_1 in the B layer neural network can be connected with the nodes P_{2}_1, P_{2}_2, P_{2}_3 and P_{2}_4 in the C layer neural network . It should be understood that FIG. 8 only shows a specific situation of the feature fusion layer, and the present application does not limit the number of neural network layers included in the feature fusion layer and the number of nodes included in each layer of the neural network.

In addition, the search space of the above target detection network may also include the search space of RCNN.

Wherein, the RCNN of the above-mentioned target detection network includes a plurality of basic units, each basic unit in the plurality of basic units is composed of at least two nodes, and the search space of the above-mentioned RCNN includes the search of each basic unit in the plurality of basic units space, the search space of each basic unit includes the optional connection relationship of each basic unit, and the optional connection relationship of each basic unit includes the connection between any two nodes in each basic unit.

For example, as shown in FIG. 9 , the RCNN includes the basic unit 1 at the top of FIG. 9 and the basic unit 2 at the bottom of FIG. 9 , and the basic unit 1 and the basic unit 2 are respectively introduced below.

Basic Unit 1:

Basic unit 1 consists of 7 nodes, including 2 input nodes (H_{0}, H_{0}), 4 intermediate nodes (0, 1, 2, 3) and 1 output node (H_{1} ).

In basic unit 1, except that the two input nodes cannot be directly connected, any two remaining nodes can be directly connected.

Basic Unit 2:

Basic unit 2 consists of 7 nodes, including 2 input nodes (H_{0}, H_{1}), 4 intermediate nodes (0, 1, 2, 3) and 1 output node (H_{2} ).

In basic unit 1, except that the two input nodes cannot be directly connected, any two remaining nodes can be directly connected.

The above-mentioned FIG. 9 only shows one possible connection relationship between the base unit 1 and the base unit 2 .

For the above-mentioned basic unit 1 and basic unit 2, the nodes inside the basic unit 1 or the basic unit 2 are connected according to the direction from input to output when connecting.

1002. Determine an initial network architecture of the target detection network according to the search space of the target detection network.

The initial network architecture of the above target detection network includes the network architecture of the feature fusion layer, and the network architecture of the feature fusion layer is determined according to the search space of the feature fusion layer, that is, in step 1002, it may be based on the feature fusion layer. The search space of layers is used to determine the network architecture of the feature fusion layer in the initial network architecture of the object detection network.

In addition, the initial network architecture of the above-mentioned target detection network may also include the network architecture of the RCNN, and the network architecture of the RCNN is determined according to the search space of the RCNN, that is, in step 1002, it may be determined according to the search space of the RCNN. The network architecture of RCNN.

It should be understood that when the initial network architecture of the target detection network is determined in the above step 1002, the number of network layers of the target detection network and the number of nodes included in the target detection network may be predetermined.

Specifically, before constructing the target detection network, the number of network layers of the target detection network and the number of nodes included in the target detection network may be determined according to the application requirements or target detection performance requirements of the target detection network to be constructed.

For example, when the target detection performance of the target detection network is required to be high, the number of network layers of the target detection network can be relatively large, and the number of nodes contained in the target detection network can also be relatively large. When the requirements are low, the number of network layers of the target detection network can be smaller, and the number of nodes included in the target detection network can also be relatively small.

1003. According to the search space of the target detection network, iteratively update the initial network architecture of the target detection network until a target detection network that meets the preset requirements is obtained.

It should be understood that in step 1003, when the initial network architecture of the target detection network is iteratively updated, both the network architecture of the feature fusion layer and the network architecture of the RCNN in the initial network architecture of the target detection network in the target detection network can be updated. Any one of them can be updated, and the network architecture of the feature fusion layer and the network architecture of RCNN in the initial network architecture of the target detection network can also be updated at the same time.

Specifically, in step 1003, after each iterative update of the network architecture of the target detection network, it is possible to confirm whether the target detection network of the updated network meets the requirements, and if it does not meet the preset requirements, continue to update the target detection network of the target detection network. Network architecture until a target detection network that meets the preset requirements is obtained.

In the above target detection network, the network structure of the backbone network and the network structure of the RPN may be predetermined. In this way, in the process of updating the initial network architecture of the target detection network, only the network architecture of the feature fusion layer in the initial network architecture of the target detection network or the network architecture of the RCNN in the initial network architecture of the target detection network can be updated. renew.

In addition, the network architecture of the above-mentioned backbone network and the network architecture of the RPN may also be a network architecture that has not been determined in advance, so that in the process of updating the initial network architecture of the target detection network, the backbone network, features of the target detection network can be updated. The network architectures of the fusion layer, RPN and RCNN are all updated.

In this application, since the search space of the feature fusion layer contains more optional connection relationships of the feature fusion layer, this application can more reasonably determine the initial network architecture of the target detection network according to more optional connection relationships The feature fusion layer in the feature fusion layer, and the network architecture of the feature fusion layer is updated, which can simplify the complexity of the final target detection network.

Specifically, in this application, for the search space of the feature fusion layer, since a search space with a more free optional connection relationship is adopted, the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer. When the network architecture of the feature fusion layer is updated, a more simplified network architecture of the feature fusion layer can be obtained, which can ultimately simplify the complexity of the target detection network and reduce the storage required for the deployment of the target detection network. space.

In addition, since the search space of the feature fusion layer contains more optional connection relationships, the feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, and the The network architecture is updated, and finally a better-performing target detection network can be constructed.

In this application, in the search space of RCNN, any two nodes in each basic unit of RCNN can be connected. Therefore, this application can more reasonably determine the initial value of RCNN according to the more relaxed RCNN search space. The network structure is updated, and the initial network structure of RCNN is updated, which can simplify the complexity of the target detection network.

In addition, since the search space of RCNN contains more optional connection relationships, the network architecture of RCNN in the initial network architecture of the target detection network is determined according to the search space of RCNN, and the network architecture of RCNN is updated. Finally, a target detection network with better performance can be constructed.

The method shown in FIG. 7 may be an automatic construction method of a neural network, and the method shown in FIG. 7 may be automatically executed by the apparatus for constructing a target detection network according to an embodiment of the present application.

When the network architecture of the feature fusion layer of the initial network architecture of the target detection network in the target detection network and the network architecture of the RCNN are updated at the same time, the resulting feature fusion layer can be more matched with the structure of the RCNN, thereby improving the performance of the RCNN. The performance of the final object detection network.

Optionally, the above-mentioned step 1003 specifically includes: according to the search space of the target detection network, iteratively update the initial network architecture of the target detection network, so as to reduce the value of the loss function corresponding to the target detection network, so as to meet the preset requirements. object detection network.

Wherein, the above-mentioned loss function may include the target detection error of the target detection network and/or the complexity of the target detection network.

When the initial network architecture of the target detection network is iteratively updated, the network structure of the target detection network (the connection relationship between different nodes in the network) can be adjusted, and the loss function corresponding to the target detection network can be calculated after each adjustment. The value of , and then update the network structure of the target detection network according to the value of the loss function corresponding to the target detection network.

Specifically, in the above process, the value of the loss function corresponding to the target detection network (also referred to as the function value) can be determined after each update of the network architecture of the target detection network. If the loss function corresponding to the target detection network The size of the value of is less than the preset threshold, you can stop updating the network architecture of the target detection network. The target detection network obtained at this time is the target detection network that meets the preset requirements. If the loss function corresponding to the target detection network The value of the target detection network is not less than the preset threshold, then, it can be determined whether to continue to update the network architecture of the target detection network according to the value of the loss function corresponding to the target detection network, and so on and so on until the preset requirements are met. object detection network.

Optionally, the above-mentioned target detection network that meets the preset requirements satisfies at least one of the following conditions (1) to (3):

(1) The detection performance of the target detection network meets the preset performance requirements;

(2) The number of updates to the network architecture of the target detection network is greater than or equal to a preset number of times;

(3) The complexity of the target detection network is less than or equal to the preset complexity.

The complexity of the target detection network (also referred to as the complexity of the network structure of the target detection network) may be based on the number or size of model parameters of the target detection network, the memory access cost MAC of the target detection network, and the target detection network. at least one of the number of floating-point operations is determined.

For the search space of the feature fusion layer, in addition to the optional connection relationship of the feature fusion layer, it can also include optional operation types of the feature fusion layer.

Specifically, the optional operation type of the feature fusion layer may include the connection corresponding to any node of one layer of neural network in two adjacent layers of neural network in the multi-layer neural network and any node of the other layer of neural network. convolution operation.

Wherein, the convolution operation corresponding to the connection between any node of one layer of neural network in the two adjacent layers of neural network in the above-mentioned multi-layer neural network and any node of another layer of neural network includes hole convolution operation.

Optional operations for the feature fusion layer can include any of the following operations:

(1) No connection;

(2) 5×5 hole convolution, the number of intervals is 2;

(3) skip connection (identity mapping);

(4) 5×5 hole convolution, the number of intervals is 3;

(5) 3×3 hole convolution, the number of intervals is 2;

(6) 3×3 depthwise separable convolution;

(7) 3×3 hole convolution, the number of intervals is 3;

(8) 5×5 depthwise separable convolution.

Compared with the traditional convolution operator, the atrous convolution operator has a larger receptive field than the atrous convolution operator with the same amount of learnable parameters, so that the atrous convolution operator can be used to extract the image from the image. Features of greater visual range. Alternatively, in order to extract features of the same visual range, the use of atrous convolution operators can reduce the number and size of network parameters in the feature fusion layer. Therefore, when the optional operation types in the feature fusion layer include atrous convolution operations, it is possible to achieve substantially the same object detection performance with fewer convolution parameters.

In addition, when the optional operation type in the feature fusion layer includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, in the case of the same amount of parameters, using atrous convolution to perform convolution processing can achieve a larger receptive field than traditional convolution, so the final target detection network can have better target detection performance.

The network structure of the feature fusion layer of the finally obtained target detection network will be described below with reference to the accompanying drawings.

As shown in FIG. 10 , the number of layers of the neural network included in the feature fusion layer and the nodes in each layer of the neural network are the same as the feature fusion layer shown in FIG. 8 . Figure 10 shows a possible connection mode of the finally obtained feature fusion layer, specifically, in Figure 10, specifically, in Figure 8, P_{1}_1 and P_{ 1}_4 receives the output of all nodes (4 nodes) of the first group, P_{1}_2 and P_{1}_3 in the B layer neural network receive the output of 3 nodes in the A layer neural network; C layer P_{2}_4 in the neural network receives the output of all nodes (4 nodes) in the B-layer neural network, and P_{2}_1, P_{2}_2, P_{2}_3 in the C-layer neural network are all Receive the outputs of 3 nodes in the A layer neural network.

Table 1 shows the input nodes corresponding to each node in the B-layer neural network.

Table 1

B layer neural network corresponding input node P_{1}_1 P_{0}_1, P_{0}_2, P_{0}_3, and P_{0}_4 P_{1}_2 P_{0}_2, P_{0}_3, and P_{0}_4 P_{1}_3 P_{0}_2, P_{0}_3, and P_{0}_4 P_{1}_4 P_{0}_1, P_{0}_2, P_{0}_3, and P_{0}_4

Table 2 shows the input nodes corresponding to each node in the B-layer neural network.

Table 2

C layer neural network corresponding input node P_{2}_1 P_{0}_1, P_{0}_2, P_{0}_3, and P_{0}_4 P_{2}_2 P_{0}_2, P_{0}_3, and P_{0}_4 P_{2}_3 P_{0}_2, P_{0}_3, and P_{0}_4 P_{2}_4 P_{0}_1, P_{0}_2, P_{0}_3, and P_{0}_4

In addition, Fig. 10 also shows a possible selection method of the operation space between nodes. Specifically, the operations between some nodes between the A-layer neural network and the B-layer neural network are shown in Table 3, except that the table In addition to the operation relationship shown in 3, the corresponding operations between the A-layer neural network and the second node are all identity.

table 3

B layer neural network corresponding input node corresponding operation P_{1}_1 P_{0}_3 dil_conv_5×5_r3 P_{1}_2 P_{0}_3 dil_conv_5×5_r3 P_{1}_4 P_{0}_4 dil_conv_5×5_r2

The operations between some nodes between the B-layer neural network and the C-layer neural network are shown in Table 4. In addition to the operation relationship shown in Table 4, the corresponding operations between the B-layer neural network and the C-layer neural network Both are identities.

Table 4

C layer neural network corresponding input node corresponding operation P_{2}_2 P_{1}_1 dil_conv_5×5_r3 P_{2}_4 P_{1}_4 dil_conv_5×5_r3

The search space of the target detection network in the above step 1002 may also include the search space of the RCNN, and the RCNN may include a plurality of basic units, each of which is composed of at least two nodes, and the search space of the RCNN includes: The search space of each basic unit in the plurality of basic units, the search space of each basic unit includes the optional connection relationship of each basic unit, and the optional connection relationship of each basic unit includes any two in each basic unit. connections between nodes.

When the search space of the target detection network in the above step 1002 includes the search space of the RCNN, the RCNN in the initial network architecture of the target detection network in the above-mentioned target detection network may be determined according to the search space of the RCNN. That is, in step 1002, the network architecture of the RCNN in the initial network architecture of the target detection network can be determined according to the search space of the RCNN.

In this application, in the search space of RCNN, since any two nodes in each basic unit of RCNN can be connected, this application can more reasonably determine the search space of RCNN according to the more relaxed RCNN search space. The initial network structure and updating the initial network structure of RCNN can simplify the complexity of the target detection network.

Further, in this application, when the feature fusion layer of the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, and the network architecture of the feature fusion layer is updated, compared with the way of manually setting the network architecture. In this application, the RCNN of the initial network architecture of the target detection network is determined according to the search space of the RCNN, and when the network architecture of the RCNN is updated, the network structure of the feature fusion layer and the network structure of the RCNN can be updated and optimized at the same time, thereby The network structure of the final optimized RCNN is more matched with the network structure of the feature fusion layer, and a target detection network with better target detection performance can be obtained.

Optionally, in the search space of the above-mentioned RCNN, the search space of each basic unit also includes the optional operation type of each basic unit, and the optional operation type of each basic unit includes any two in each basic unit. The convolution operation corresponding to the connection between nodes, the convolution operation includes the hole convolution operation.

In this application, when the optional operation type in RCNN includes atrous convolution operation, it can achieve substantially the same target detection performance with fewer convolution parameters.

In addition, when the optional operation type of RCNN includes atrous convolution operation, a target detection network with better target detection performance can be obtained under the condition that the amount of convolution parameters is basically the same.

Specifically, in the case of the same amount of parameters, using atrous convolution to perform convolution processing can achieve a larger receptive field than traditional convolution, so the final target detection network can have better target detection performance.

Optionally, the convolution operation corresponding to the connection between any two nodes in each basic unit includes a hole convolution operation with an interval of 2.

Since the resolution of the feature map processed in RCNN is generally low, it is more suitable to use a hole convolution operation with a small number of intervals. Therefore, for each basic unit in RCNN, when the optional operation includes the number of intervals is 2 When the atrous convolution is used, the performance of the final RCNN can be improved, thereby improving the performance of the final target detection network.

Optionally, the basic units in the above-mentioned RCNN may be composed of different numbers of nodes.

In the traditional scheme, the basic units in RCNN are generally composed of the same number of nodes, which is not flexible enough to optimize the RCNN network structure. The basic units in the RCNN in this application can be composed of different numbers of nodes, which makes the composition of the basic units in the RCNN more free, so that the initial network structure of the RCNN can be determined and the initial network structure of the RCNN can be updated. The possibility of network structure is convenient to search for a better RCNN network structure, making it more likely to obtain a target detection network with better target detection performance in the end.

In addition, the number of nodes constituting each basic unit in RCNN can be predetermined before constructing the object detection network.

Optionally, the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

When the resolution of the input feature map of each basic unit in RCNN is the same as the resolution of the output feature map of each basic unit, each basic unit in RCNN does not change the resolution of the feature map when processing the feature map, It is convenient to retain the information of the feature map.

The following describes the network structure of the RCNN of the final target detection network with reference to the accompanying drawings.

As shown in Figure 11, the basic units included in the RCNN and the number of nodes contained in each basic unit are the same as the RCNN shown in Figure 9. The network architecture shown in the RCNN shown in Figure 11 can be the final result. The network architecture of RCNN.

Specifically, as shown in Figure 11, for the basic unit 1 of RCNN, the feature map input by the two identical input nodes (H_{0}) and the input nodes (H_{0}) passes through the intermediate nodes 0 to 3 After processing, weighted summation is performed at the output node to obtain the output of the first convolution unit (H_{1}). Among them, the operations from the input node to the intermediate node and the intermediate node to the output node are all 5×5 convolutions (conv_5×5).

For the basic unit 2 of RCNN, the feature maps input by input nodes H_{0} and H_{1} are processed by intermediate nodes 0 to 3, and then weighted and summed at the output node to obtain the first convolution unit. the output of (H_{2}). Among them, the operations from the input node to the intermediate node and the intermediate node to the output node are all 5×5 convolutions (conv_5×5).

The process of the method for constructing a target detection network in this embodiment of the present application will be described in more detail below with reference to FIG. 12 .

FIG. 12 is a flowchart of a method for constructing a target detection network according to an embodiment of the present application. The method shown in FIG. 12 may be executed by the apparatus for constructing a target detection network according to an embodiment of the present application. The method shown in FIG. 12 includes steps 2001 to 2006 , which will be described in detail below.

2001. Initialize the network architecture of the target detection network.

In step 2001, the initial network architecture of the target detection network may be determined first.

Specifically, since the target detection network includes a backbone network, a feature fusion layer, RPN and RCNN, in step 2001, to determine the initial network architecture of the target detection network, it is necessary to determine the backbone network, feature fusion layer, Network architecture of RPN and RCNN.

Among them, the backbone network and RPN can be pre-built networks, therefore, initializing the network structure of the target detection network is equivalent to initializing the network structure of the feature fusion layer and RCNN.

After step 2001, the initial network structure of the feature fusion layer and the initial network structure of the RCNN can be determined.

2002a, perform performance evaluation on the feature fusion layer.

Specifically, in step 2002a, the performance of the feature fusion layer and the resources occupied by the feature fusion layer can be evaluated, so that the network architecture of the feature fusion layer can be updated subsequently according to the performance of the feature fusion layer and the evaluation of the resources occupied.

2002b, Evaluate the performance of RCNN.

Similar to step 2002a, in step 2002b, the performance and resource occupancy of the RCNN can also be evaluated, so that the network architecture of the RCNN can be updated subsequently according to the performance and resource occupancy of the RCNN.

The above steps 2002a and 2002b may occur simultaneously or sequentially, and the present application does not limit the sequence in which the steps 2002a and 2002b occur.

2003a, update the network architecture of the feature fusion layer.

After obtaining the performance and resource occupation of the feature fusion layer through step 2002a, the network architecture of the feature fusion layer may be updated according to the performance and resource occupation of the feature fusion layer.

Specifically, when the performance of the feature fusion layer does not meet the requirements, the complexity of the feature fusion layer can be appropriately increased when the feature fusion layer is updated, so that the updated feature fusion layer has better performance; When there are too many resources, the complexity of the feature fusion layer can be reduced when the feature fusion layer is updated, so that the updated feature fusion layer has a simpler structure, thereby reducing the resources occupied by the updated feature fusion layer.

2003b, update the network architecture of RCNN.

After obtaining the performance and resource occupancy of the RCNN through step 2002b, the network architecture of the RCNN can be updated according to the performance and resource occupancy of the RCNN.

Specifically, when the performance of the RCNN does not meet the requirements, the complexity of the RCNN can be increased when the RCNN is updated, so that the updated RCNN has better performance; and when the RCNN occupies too many resources, it can be updated when the RCNN is updated. The complexity of RCNN is reduced, so that the updated RCNN has a simpler structure, so that the resources occupied by the updated RCNN are reduced.

It should be understood that the above steps 2003a and 2003b may occur simultaneously or sequentially, and the present application does not limit the sequence of steps 2003a and 2003b.

2004. Update the network parameters of the target detection network.

After updating the network architecture of the feature fusion layer and RCNN, the network parameters of the target detection network can be updated. The network parameters of the target detection network here include the backbone network, the feature fusion layer, the network parameters of RPN and RCNN (the network parameters Specifically, convolution parameters can be included).

2005. Determine whether the target detection network satisfies a preset condition.

Specifically, in step 2005, it can be determined whether the target detection network satisfies the following conditions:

(1) The detection performance of the target detection network meets the preset requirements;

(2) Whether the number of network architecture updates of the target detection network reaches a preset number of times;

(3) The complexity of the target detection network is less than or equal to the preset complexity.

When the target network satisfies any one of the above-mentioned conditions (1) to (3), it can be determined that the target detection network satisfies the preset requirements.

The performance of the above-mentioned target detection network satisfies the preset requirement may specifically be that the accuracy of the target detection performed by the target detection network is greater than a certain accuracy threshold. For example, when the target detection accuracy rate of the target detection network is greater than 60%, it is determined that the performance of the target detection network meets the preset requirements.

When it is determined in step 2005 that the target detection network satisfies the preset condition, step 2006 is continued to output the target detection network. When it is determined in step 2005 that the target detection network does not meet the preset conditions, steps 2002a and 2002b are continued to update the network architecture of the target detection network.

2006. Output target detection network.

After it is determined in step 2005 that the target detection network satisfies the preset conditions, the construction of the target neural network is completed, and the target detection network can be output.

FIG. 13 is a schematic flowchart of a target detection method according to an embodiment of the present application. The method shown in FIG. 13 may be performed by the target detection apparatus in this embodiment of the present application. For example, the method shown in FIG. 13 may be performed by the apparatus shown in FIG. 15 or FIG. 17 .

The method shown in FIG. 13 includes steps 3001 and 3002, and the two steps and related contents will be described in detail below.

3001. Obtain an image.

3002. Use a target detection network to process the above image to obtain a target detection result of the image.

The target detection result of the image obtained in the above step 3002 includes the position of the detection target in the image and the classification result to which the detection target belongs.

The target detection network used in the above step 3002 may be constructed according to the construction method of the target detection network in the embodiment of the present application. The above definitions and explanations on the target detection network when introducing the embodiments of the present application are also applicable to the target detection network in step 3002 .

Specifically, the target detection network used in the above step 3002 may be constructed according to the method shown in FIG. 7 or FIG. 12 .

The target detection network used in the target detection method in this application contains more optional connection relationships of the feature fusion layer in the search space of the feature fusion layer used in the construction process. Therefore, in the search space determined according to the feature fusion layer The feature fusion layer can perform better feature fusion, so that the final target detection network has better performance in target detection.

The initial network architecture of the target detection network in the method shown in FIG. 13 above is determined according to the search space of the target detection network, and the final network architecture of the target detection network can be the initial network architecture of the target detection network according to the search space of the target detection network. The network architecture is iteratively updated.

When acquiring the target detection network used in the method shown in FIG. 13 , the search space of the target detection network may include the search space of the feature fusion layer and/or the search space of the RCNN.

When the search space of the above target detection network includes the search space of the feature fusion layer, the feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, and the final feature fusion layer in the target detection network is determined. The network architecture is obtained by iteratively updating the network architecture of the feature fusion layer in the initial network architecture of the target detection network according to the search space of the feature fusion layer.

When the search space of the above target detection network includes the search space of the RCNN, the RCNN in the initial network architecture of the target detection network is determined according to the search space of the RCNN, and the final network architecture of the RCNN in the target detection network is determined according to the search space of the RCNN It is obtained by iteratively updating the RCNN network architecture in the initial network architecture of the target detection network.

Optionally, the search space of the above-mentioned target detection network includes the search space of the feature fusion layer, the search space of the feature fusion layer includes the optional connection relationship of the feature fusion layer, and the optional connection relationship of the feature fusion layer includes multiple layers of the feature fusion layer. The connection between any node of one layer of neural network in the adjacent two-layer neural network in the neural network and any node of the other layer of neural network.

Optionally, the search space of the above-mentioned feature fusion layer also includes the optional operation type of the feature fusion layer, and the optional operation type of the feature fusion layer includes the operation type of one layer of the adjacent two-layer neural network in the multi-layer neural network. A convolution operation corresponding to the connection between any node and any node in another layer of neural network, wherein the convolution operation includes an atrous convolution operation.

When the optional operation types in the feature fusion layer include atrous convolution operations, substantially the same object detection performance can be achieved with fewer convolution parameters. In addition, when the convolution parameters have the same amount of parameters, the convolution processing using atrous convolution can obtain a larger receptive field than the traditional convolution, so it can achieve better target detection when using the target detection network. Object detection performance.

Optionally, the above-mentioned RCNN includes a plurality of basic units, and each basic unit in the plurality of basic units is composed of at least two nodes, and the search space of the target detection network also includes the search space of the RCNN, and the search space of the RCNN includes a plurality of basic units. The search space of each basic unit in the unit, the search space of each basic unit includes the optional connection relationship of each basic unit, and the optional connection relationship of each basic unit includes any two nodes in each basic unit. The connection between the two; the RCNN in the initial network architecture of the target detection network is determined according to the search space of the RCNN.

Optionally, the search space of each basic unit above also includes the optional operation type of each basic unit, and the optional operation type of each basic unit includes the corresponding connection between any two nodes in each basic unit. The convolution operation includes atrous convolution operation.

When the optional operation type in RCNN includes atrous convolution operation, it can achieve basically the same object detection performance with fewer convolution parameters. In addition, when the optional operation type of RCNN includes atrous convolution operation, better target detection performance can be achieved under the condition that the amount of convolution parameters is basically the same.

Specifically, when the convolution parameters have the same amount of parameters, convolution processing using atrous convolution can achieve a larger receptive field than traditional convolution. Therefore, using the above target detection network for target detection has better performance. Object detection performance.

Optionally, the convolution operation corresponding to the connection between any two nodes in each basic unit includes an atrous convolution operation with an interval of 2.

Since the resolution of the feature map processed in RCNN is generally low, it is more suitable to use a hole convolution operation with a small number of intervals. For each basic unit in RCNN, when the optional operation includes holes with a number of intervals of 2 When convolution, it can finally improve the target detection performance of the target detection network.

Optionally, at least two basic units in the above-mentioned plurality of basic units are respectively constituted by nodes of different numbers.

Optionally, the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

When the resolution of the input feature map of each basic unit in RCNN is the same as the resolution of the output feature map of each basic unit, each basic unit in RCNN does not change the resolution of the feature map when processing the feature map, It is convenient to retain the information of the feature map, thereby ensuring a better target detection effect when the target detection network is used for target detection.

Optionally, the above-mentioned target detection network satisfies at least one of the following conditions: the detection performance of the target detection network meets the preset performance requirements; the number of updates to the network architecture of the target detection network is greater than or equal to the preset number of times; The complexity is less than or equal to the preset complexity.

Optionally, the complexity of the target detection network is determined according to at least one of the number or size of model parameters of the target detection network, the memory access cost MAC of the target detection network, and the number of floating point operations of the target detection network.

In order to illustrate the effect of the construction method of the target detection network of the embodiment of the present application, the complexity of the neural network obtained by the construction method of the target detection network of the embodiment of the present application and the complexity of the neural network obtained by the construction method of the target detection network of the embodiment of the present application are described below with reference to the specific test results. The accuracy of target detection by the neural network obtained by the construction method is analyzed.

table 5

Table 5 shows the complexity of object detection networks obtained with different schemes.

Among them, the specific schemes of the existing scheme 1 to the existing scheme 6 are as follows:

Existing solution 1: guided anchoring faster RCNN (guided anchoring faster RCNN);

Existing scheme 2: path aggregation network published in CVPR in 2018;

Existing solution 3: Towards real-time object detection with region proposal networks, published in NIPS in 2015;

Existing scheme 4: feature pyramid network, published in CVPR in 2017;

Existing solution 5: relation networks for object detection, CVPR, published in CVPR in 2018;

Existing scheme 6: focal loss for dense object detection, published in ICCV in 2017.

It can be seen from Table 5 that when the data set COCO is tested, when the backbone networks are ResNet-50 and ResNet-101 respectively, the average precision (mean average precision, mAP) of target detection using the neural network obtained by the solution of this application is higher than The average accuracy of target detection using the neural network obtained by the existing scheme.

Specifically, when the backbone network is ResNet-50, the average accuracy of target detection using the neural network obtained by the solution of the present application is 40.5, while the average accuracy of target detection using the neural network obtained by the existing solution 1 or the existing solution 2 is 40.5 Both are 39.8. Compared with the existing solution 1 and the existing solution 2, the average accuracy of target detection by the neural network obtained by the solution of the present application is improved by 40.5-39.8=0.7.

When the backbone network is ResNet-101, the average accuracy of target detection using the neural network obtained by the solution of this application is 42.5, while the average accuracy of target detection using the neural network obtained by using the existing solution 3 to the existing solution 6 is the highest. 39.1 , compared with the existing scheme 3 to the existing scheme 6, the average accuracy of target detection using the neural network obtained by the proposed scheme is improved by at least 42.5-39.1=3.4 (compared with the existing scheme 3, the corresponding The average accuracy of object detection is even improved by 42.5-34.9=7.6).

Table 6

Table 6 shows the amount of parameters included in the target detection network constructed by the existing solution and the solution of the present application, and the average precision of target detection when the target detection network constructed by the existing solution and the solution of the present application is used for target detection. As shown in Table 6, the total number of parameters of the target detection network constructed on the three data sets (VOC data set, COCO data set and BDD data set) of the proposed scheme is lower than the target detection obtained by the existing scheme. The total number of parameters of the network. In addition, the target detection network constructed by the proposed scheme on the three datasets respectively has higher mAP for target detection on the test dataset (PASCALVOC dataset) than the target detection network constructed by the existing scheme in the test dataset ( mAP for object detection on the PASCALVOC dataset).

Table 7

Similar to Table 6, Table 7 also shows the amount of parameters contained in the target detection network constructed by the existing scheme and the proposed scheme, and the average of the target detection network constructed by the existing scheme and the proposed scheme for target detection. precision.

As shown in Table 7, the total number of parameters of the target detection network constructed on the three data sets (BDD data set, COCO data set and VOC data set) of the proposed scheme is lower than the target detection obtained by the existing scheme. The total number of parameters of the network. In addition, the target detection network constructed on the three data sets in the proposed scheme has higher mAP for target detection on the test data set (BDD data set) than the target detection network constructed by the existing scheme in the test data set (BDD data set). mAP for object detection on BDD dataset).

The construction method of the target detection network according to the embodiment of the present application and the target detection method according to the embodiment of the present application are described in detail above with reference to the accompanying drawings. The following describes the construction device and target detection device of the target detection network according to the embodiment of the present application with reference to the accompanying drawings. describe.

It should be understood that the device for constructing a target detection network described below can perform each step of the method for constructing a target detection network in the embodiment of the present application, and the target detection device described below can perform each step in the target detection method in the embodiment of the present application. , and the repeated description is appropriately omitted in the description of the apparatus for constructing a target detection network and the apparatus for target detection according to the embodiments of the present application.

FIG. 14 is a schematic block diagram of an apparatus for constructing a target detection network according to an embodiment of the present application. The apparatus 5000 shown in FIG. 14 includes a determination unit 5001 and a construction unit 5002 .

The apparatus 5000 may execute each step of the method for constructing a target detection network in this embodiment of the present application. Specifically, the apparatus 5000 may execute the method shown in FIG. 7 or the method shown in FIG. 12 .

Specifically, when the apparatus 5000 executes the method shown in FIG. 7 , the determining unit 5001 may be specifically configured to execute steps 1001 and 1002 , and the constructing unit 5002 may be configured to execute step 1003 .

When the apparatus 5000 executes the method shown in FIG. 12, the determining unit 5001 can be specifically configured to execute step 2001, and the constructing unit 5002 can be configured to execute steps 2002 to 2006, wherein step 2002 includes steps 2002a and 2002b, and step 2003 includes step 2003a and 2003b.

The determining unit 5001 and the constructing unit 5002 in the above apparatus 5000 are equivalent to the processor 7002 in the apparatus 7000 shown in FIG. 16 below.

FIG. 15 is a schematic block diagram of a target detection apparatus according to an embodiment of the present application. The apparatus 6000 shown in FIG. 15 includes an acquisition unit 6001 and a detection unit 6002 .

The apparatus 6000 may execute each step of the target detection method of the embodiment of the present application. Specifically, the apparatus 6000 may execute the method shown in FIG. 7 or the method shown in FIG. 12 .

Specifically, when the apparatus 5000 executes the method shown in FIG. 7 , the determining unit 5001 may be specifically configured to execute steps 1001 and 1002 , and the constructing unit 5002 may be configured to execute step 1003 .

When the apparatus 5000 executes the method shown in FIG. 12, the determining unit 5001 can be specifically configured to execute step 2001, and the constructing unit 5002 can be configured to execute steps 2002 to 2006, wherein step 2002 includes steps 2002a and 2002b, and step 2003 includes step 2003a and 2003b.

The determining unit 5001 and the constructing unit 5002 in the above apparatus 5000 are equivalent to the processor 7002 in the apparatus 7000 shown in FIG. 16 below.

FIG. 16 is a schematic diagram of a hardware structure of a neural network structure search apparatus provided by an embodiment of the present application. The neural network structure search apparatus 3000 shown in FIG. 16 (the apparatus 3000 may specifically be a computer device) includes a memory 3001 , a processor 3002 , a communication interface 3003 and a bus 3004 . The memory 3001 , the processor 3002 , and the communication interface 3003 are connected to each other through the bus 3004 for communication.

The memory 3001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program, and when the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 is configured to execute each step of the method for constructing a target detection network in this embodiment of the present application.

The processor 3002 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more The integrated circuit is used for executing the relevant program, so as to realize the construction method of the target detection network in the embodiment of the present application.

The processor 3002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the method for constructing a target detection network of the present application may be completed by an integrated logic circuit of hardware in the processor 3002 or instructions in the form of software.

The above-mentioned processor 3002 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates Or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 3001, and the processor 3002 reads the information in the memory 3001 and, in combination with its hardware, completes the functions required to be performed by the units included in the neural network structure search apparatus, or performs the construction of the target detection network of the embodiment of the present application method.

The communication interface 3003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 3000 and other devices or a communication network. For example, the information of the neural network to be constructed and the training data required in the process of constructing the neural network can be acquired through the communication interface 3003 .

The bus 3004 may include a pathway for transferring information between the various components of the device 3000 (eg, the memory 3001, the processor 3002, the communication interface 3003).

FIG. 17 is a schematic diagram of a hardware structure of a target detection apparatus according to an embodiment of the present application. The target detection apparatus 4000 shown in FIG. 17 includes a memory 4001 , a processor 4002 , a communication interface 4003 , and a bus 4004 . The memory 4001 , the processor 4002 , and the communication interface 4003 are connected to each other through the bus 4004 for communication.

The memory 4001 may be ROM, static storage device and RAM. The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the target detection method of the embodiment of the present application.

The processor 4002 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, and is used to execute a related program, so as to realize the function required to be performed by the unit in the target detection apparatus of the embodiment of the present application, Or execute the target detection method of the embodiment of the present application.

The processor 4002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the target detection method in the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 4002 or an instruction in the form of software.

The above-mentioned processor 4002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001 and, in combination with its hardware, completes the functions required to be performed by the units included in the target detection apparatus of the embodiment of the present application, or executes the target detection method of the embodiment of the present application .

The communication interface 4003 implements communication between the device 4000 and other devices or a communication network using a transceiver device such as, but not limited to, a transceiver. For example, the image to be processed can be acquired through the communication interface 4003 .

Bus 4004 may include a pathway for communicating information between various components of device 4000 (eg, memory 4001, processor 4002, communication interface 4003).

It should be noted that although the above-mentioned apparatus 7000 and apparatus 5000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 7000 and the apparatus 8000 may also include necessary components for normal operation. of other devices. Meanwhile, according to specific needs, those skilled in the art should understand that the apparatus 7000 and the apparatus 8000 may further include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the apparatus 7000 and the apparatus 8000 may also only include the necessary devices for implementing the embodiments of the present application, and do not necessarily include all the devices shown in FIG. 16 and FIG. 17 .

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution, and the computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (38)

1. A method for constructing an object detection network, wherein the object detection network comprises a backbone network, a feature fusion layer, a regional candidate network RPN and a regional convolutional neural network RCNN, and the method is characterized by comprising the following steps:

determining a search space of the target detection network, wherein the search space of the target detection network comprises a search space of the feature fusion layer, the search space of the feature fusion layer comprises a selectable connection relation of the feature fusion layer, and the selectable connection relation of the feature fusion layer comprises a connection of any node of one layer of neural network and any node of the other layer of neural network in two adjacent layers of neural networks of the feature fusion layer;

determining an initial network architecture of the target detection network according to a search space of the target detection network, wherein a feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer;

and according to the search space of the target detection network, iteratively updating the initial network architecture of the target detection network until the target detection network meeting the preset requirement is obtained.

2. The method according to claim 1, wherein the iteratively updating the initial network architecture of the target detection network according to the search space of the target detection network until the target detection network satisfying a preset requirement is obtained includes:

and according to the search space of the target detection network, iteratively updating the initial network architecture of the target detection network to reduce the value of a loss function corresponding to the target detection network, so as to obtain the target detection network meeting the preset requirement, wherein the loss function comprises a target detection error of the target detection network and/or the complexity of the target detection network.

3. The construction method according to claim 1 or 2, wherein the search space of the feature fusion layer further includes a selectable operation type of the feature fusion layer, the selectable operation type of the feature fusion layer includes a convolution operation corresponding to connection of any one node of one layer of the neural network and any one node of the other layer of the neural network in two adjacent layers of the multilayer neural network, and the convolution operation includes a hole convolution operation.

4. The building method according to any one of claims 1-3, wherein the RCNN comprises a plurality of basic units, each basic unit of the plurality of basic units is composed of at least two nodes, the search space of the object detection network further comprises the search space of the RCNN, the search space of the RCNN comprises the search space of each basic unit of the plurality of basic units, the search space of each basic unit comprises the optional connection relation of each basic unit, and the optional connection relation of each basic unit comprises the connection between any two nodes in each basic unit;

the RCNN in an initial network architecture of the target detection network is determined according to a search space of the RCNN.

5. The construction method according to claim 4, wherein the search space of each basic unit further includes a selectable operation type of each basic unit, the selectable operation type of each basic unit includes a convolution operation corresponding to a connection between any two nodes in each basic unit, and the convolution operation includes a hole convolution operation.

6. The construction method according to claim 5, wherein the hole convolution operation corresponding to the connection between any two nodes in each basic unit comprises a hole convolution operation with an interval number of 2.

7. The construction method according to any one of claims 4 to 6, wherein at least two basic units of the plurality of basic units are respectively constituted by different numbers of nodes.

8. The construction method according to any one of claims 4 to 7, wherein the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

9. The construction method according to any one of claims 2 to 8, wherein the target detection network satisfying preset requirements satisfies at least one of the following conditions:

the detection performance of the target detection network meets the preset performance requirement;

updating the network architecture of the target detection network for more than or equal to a preset number;

the complexity of the target detection network is less than or equal to a preset complexity.

10. The method of claim 9, wherein the complexity of the target detection network is determined according to at least one of a number or a size of model parameters of the target detection network, a memory access cost MAC of the target detection network, and a number of floating point operations of the target detection network.

11. A method for constructing an object detection network, wherein the object detection network comprises a backbone network, a feature fusion layer, a regional candidate network RPN and a regional convolutional neural network RCNN, and the method is characterized by comprising the following steps:

determining a search space of the target detection network, wherein the RCNN includes a plurality of basic units, each basic unit of the plurality of basic units is composed of at least two nodes, the search space of the target detection network includes the search space of the RCNN, the search space of the RCNN includes the search space of each basic unit of the plurality of basic units, the search space of each basic unit includes a selectable connection relation of each basic unit, and the selectable connection relation of each basic unit includes a connection between any two nodes in each basic unit;

determining an initial network architecture of the target detection network according to the search space of the target detection network, wherein the RCNN in the initial network architecture of the target detection network is determined according to the search space of the RCNN;

and according to the search space of the target detection network, iteratively updating the initial network architecture of the target detection network until the target detection network meeting the preset requirement is obtained.

12. The method according to claim 11, wherein the iteratively updating the initial network architecture of the target detection network according to the search space of the target detection network until the target detection network satisfying a preset requirement is obtained includes:

and according to the search space of the target detection network, iteratively updating the initial network architecture of the target detection network to reduce the value of a loss function corresponding to the target detection network, so as to obtain the target detection network meeting the preset requirement, wherein the loss function comprises a target detection error of the target detection network and/or the complexity of the target detection network.

13. The building method according to claim 11 or 12, wherein the search space of each basic unit further includes a selectable operation type of each basic unit, the selectable operation type of each basic unit includes a convolution operation corresponding to a connection between any two nodes in each basic unit, and the convolution operation includes a hole convolution operation.

14. The construction method according to claim 13, wherein the hole convolution operation includes a hole convolution operation of an interval number of 2.

15. The construction method according to any one of claims 11 to 14, wherein at least two basic units of the plurality of basic units are respectively constituted by different numbers of nodes.

16. The construction method according to any one of claims 11 to 15, wherein the resolution of the input feature map of each basic unit is the same as the resolution of the output feature map of each basic unit.

17. The construction method according to any one of claims 11 to 16, wherein the target detection network satisfying preset requirements satisfies at least one of the following conditions:

the detection performance of the target detection network meets the preset performance requirement;

updating the network architecture of the target detection network for more than or equal to a preset number;

the complexity of the target detection network is less than or equal to a preset complexity.

18. The method of claim 17, wherein the complexity of the target detection network is determined according to at least one of a number or a size of model parameters of the target detection network, a memory access cost MAC of the target detection network, and a number of floating point operations of the target detection network.

19. A method of object detection, comprising:

acquiring an image;

processing the image by adopting a target detection network to obtain a target detection result of the image, wherein the target detection result comprises the position of a detection target in the image and a classification result of the detection target;

the target detection network comprises a backbone network, a feature fusion layer, a regional candidate network RPN and a regional convolutional neural network RCNN, the target detection network meets preset requirements, the target detection network is obtained by iteratively updating an initial network architecture of the target detection network according to a search space of the target detection network, and the initial network architecture of the target detection network is determined according to the search space of the target detection network;

the search space of the target detection network comprises the search space of the feature fusion layer, the feature fusion layer in the initial network architecture of the target detection network is determined according to the search space of the feature fusion layer, the search space of the feature fusion layer comprises the optional connection relation of the feature fusion layer, and the optional connection relation of the feature fusion layer comprises the connection of any node of one layer of neural network in two adjacent layers of neural networks in the multilayer neural network with any node in the other layer of neural network.

20. The object detection method of claim 19, wherein the search space of the feature fusion layer further includes a selectable operation type of the feature fusion layer, the selectable operation type of the feature fusion layer includes a convolution operation corresponding to a connection of any one node of one layer of the neural network with any one node of the other layer of the neural network in two adjacent layers of the multi-layer neural network, wherein the convolution operation includes a hole convolution operation.

21. The object detection method according to claim 19 or 20, wherein the RCNN includes a plurality of basic units, each basic unit of the plurality of basic units is composed of at least two nodes, the search space of the object detection network further includes the search space of the RCNN, the search space of the RCNN includes the search space of each basic unit of the plurality of basic units, the search space of each basic unit includes the selectable connection relation of each basic unit, and the selectable connection relation of each basic unit includes a connection between any two nodes within each basic unit;

the RCNN in an initial network architecture of the target detection network is determined according to a search space of the RCNN.

22. The object detection method of claim 21, wherein the search space of each basic unit further includes a selectable operation type of the each basic unit, the selectable operation type of the each basic unit includes a convolution operation corresponding to a connection between any two nodes in the each basic unit, and the convolution operation includes a hole convolution operation.

23. The object detection method of claim 22, wherein the hole convolution operation includes a hole convolution operation of an interval number of 2.

24. The object detection method of any one of claims 21-23, wherein at least two basic units of the plurality of basic units are respectively composed of different numbers of nodes.

25. The object detection method of any one of claims 21-24, wherein the resolution of the input feature map of each elementary unit is the same as the resolution of the output feature map of each elementary unit.

26. The object detection method according to any of claims 19-25, wherein the object detection network fulfils at least one of the following conditions:

the detection performance of the target detection network meets the preset performance requirement;

updating the network architecture of the target detection network for more than or equal to a preset number;

the complexity of the target detection network is less than or equal to a preset complexity.

27. The object detection method of claim 26, wherein the complexity of the object detection network is determined according to at least one of the number or size of model parameters of the object detection network, a memory access cost MAC of the object detection network, and the number of floating point operations of the object detection network.

28. A method of object detection, comprising:

acquiring an image;

processing the image by adopting a target detection network to obtain a target detection result of the image, wherein the target detection result comprises the position of a detection target in the image and a classification result of the detection target;

the target detection network comprises a backbone network, a feature fusion layer, a regional candidate network RPN and a regional convolutional neural network RCNN, the target detection network meets preset requirements, the target detection network is obtained by iteratively updating an initial network architecture of the target detection network according to a search space of the target detection network, and the initial network architecture of the target detection network is determined according to the search space of the target detection network;

the RCNN includes a plurality of basic units, each basic unit of the plurality of basic units is composed of at least two nodes, the search space of the target detection network includes the search space of the RCNN, the search space of the RCNN includes the search space of each basic unit of the plurality of basic units, the search space of each basic unit includes the optional connection relationship of each basic unit, the optional connection relationship of each basic unit includes the connection between any two nodes in each basic unit, and the RCNN in the initial network architecture of the target detection network is determined according to the search space of the RCNN.

29. The object detection method of claim 28, wherein the search space of each basic unit further includes a selectable operation type of the each basic unit, the selectable operation type of the each basic unit includes a convolution operation corresponding to a connection between any two nodes in the each basic unit, and the convolution operation includes a hole convolution operation.

30. The object detection method of claim 29, wherein the hole convolution operation includes a hole convolution operation of an interval number of 2.

31. The object detection method of any one of claims 28-30, wherein at least two elementary units of the plurality of elementary units are each constituted by a different number of nodes.

32. The object detection method of any one of claims 28-31, wherein the resolution of the input feature map of each elementary unit is the same as the resolution of the output feature map of each elementary unit.

33. The object detection method according to any of claims 28-32, wherein the object detection network satisfies at least one of the following conditions:

the detection performance of the target detection network meets the preset performance requirement;

updating the network architecture of the target detection network for more than or equal to a preset number;

the complexity of the target detection network is less than or equal to a preset complexity.

34. The object detection method of claim 33, wherein the complexity of the object detection network is determined according to at least one of the number or size of model parameters of the object detection network, a memory access cost MAC of the object detection network, and the number of floating point operations of the object detection network.

35. An apparatus for constructing an object detection network, comprising:

a memory for storing a program;

a processor for executing the memory-stored program, the processor for performing the construction method of any one of claims 1-10 or 11-18 when the memory-stored program is executed.

36. An object detection device, comprising:

a memory for storing a program;

a processor for executing the memory-stored program, the processor for performing the object detection method of any one of claims 19-27 or 28-34 when the memory-stored program is executed.

37. A computer-readable storage medium, characterized in that the computer-readable medium stores program code for execution by a device, the program code comprising instructions for performing the method of any of claims 1-10 or 11-18.

38. A computer-readable storage medium, characterized in that the computer-readable medium stores program code for execution by a device, the program code comprising instructions for performing the method of any of claims 19-27 or 28-34.

Images