CN110245267B - Multi-user video stream deep learning sharing calculation multiplexing method - Google Patents

Abstract

The invention relates to a computer and video processing, in order to realize the service of providing video inquiry through technologies such as target recognition and target detection in a multi-user scene, the invention discloses a multi-user video stream deep learning sharing calculation multiplexing method, firstly, when a request with detection operation or recognition operation comes, the request is combined according to the relevance of space dimension; then, according to the relevance of the time dimension, inquiring whether proper data are available for multiplexing, and calling a deep learning model with configured parameters for analysis; for the non-reusable part, firstly, finding the most suitable parameter configuration in the analysis process according to the balance relation of the speed precision, then calling a difference detector and a deep learning model to perform video analysis according to the parameter configuration, finally outputting an analysis result and storing the analysis result into a data warehouse, and a lifting module is used for lifting the original result precision in the database so as to facilitate multiplexing the query request with high precision. The invention is mainly applied to video processing occasions.

Description

Multi-user video stream deep learning shared computing multiplexing method

technical field

The invention relates to computer and video processing, in particular to a multi-user video stream deep learning shared computing multiplexing method.

Background technique

At present, deep learning has become an important engine to promote the application and popularization of artificial intelligence. Especially in the field of computer vision, the rapid development of deep learning has brought profound changes to the field, the most representative of which is the continuous development of image analysis technologies such as target detection and target recognition. Object detection can be used to classify objects, such as identifying whether there is a dog, a person, or a table in the image, but the name of the person cannot be recognized; object recognition can identify the specific identity of the person. For an image, applying the object detection model can quickly identify all the objects in the image, which brings a huge change to the video analysis process.

In the traditional video analysis process, query services are provided to users mainly through manual labeling. With the help of the development of deep learning, video content can be automatically analyzed through target recognition, target detection and other technologies to provide more high-quality and richer query services. For example, when a user edits a video, the target recognition algorithm can automatically find all the clips in which someone appears in the video, which brings great convenience to the user. Therefore, it has gradually become a trend to use deep learning-based video stream analysis to replace traditional manual tagged video query methods. However, due to the high resource requirements and long training time of deep learning models, resource sharing is usually used to allow multiple users to share a computing platform, thereby improving the utilization of computing resources in deep learning systems and reducing enterprise costs.

Disadvantages of existing technology

Most of the existing video analysis technologies based on deep learning have two shortcomings: one is that the query results are single, and the other is that they do not solve the locality of multi-user queries under the shared platform. The singleness of query results is especially evident in systems such as noscope and chameleon. In general, object detection models can recognize thousands of different object categories, but the recognition speed is slow. Therefore, in order to solve the problem that the deep learning model processes video data too slowly, noscope proposes to recognize vehicles by training a proprietary model with fewer network layers, which greatly improves the recognition speed. However, this method gives up the versatility of the deep learning model, and it cannot meet the needs of users when they need to query multiple targets.

On the other hand, models such as noscope, chameleon, and videostorm do not solve the limitations of multi-user queries under a shared platform. The limitations are reflected in the following aspects:

Locality in the time dimension. The locality of the time dimension is manifested in the fact that queries in different time periods are repetitive in terms of query data. When a new query arrives, some users have already paid attention to the same content. This is especially evident on some popular videos. The locality of the time dimension is also reflected in the similarity in the content of consecutive frames. Due to the unique characteristics of the video data itself, there is a high similarity between consecutive frames to maintain the coherence of the video. In addition, in some surveillance videos, such as camera surveillance in a park, the content of the video changes very little during part of the time (such as at night). When the target in the video frame is recognized by the deep learning model, the recognition results between these consecutive frames will be basically the same, so these consecutive frames bring a lot of repeated calculations.

Locality in spatial dimensions. The main performance is that the same piece of data will be queried and processed by multiple reasoning requests arriving at the same time. These queries often have the same and similarities. For these query calculations, it is often possible to perform pruning and merging operations to avoid unnecessary redundant calculations.

Locality between data result logic. There is a possibility of logical reuse of the data results reflected in different models. For example, object detection and object recognition often have a logical multiplexing relationship with each other. For example, for a certain frame of a video, when the object detection shows that there is no person, it is impossible for the object recognition algorithm to recognize Zhang San, thereby avoiding the calculation of the object recognition person. Conversely, the results of object recognition can also be directly reused in object detection under certain circumstances. If the object recognition algorithm shows that there is Zhang San, it indicates that there is a person, which directly avoids object detection for people.

Contents of the invention

In order to overcome the deficiencies of the prior art, the present invention aims to propose a multi-user video stream deep learning shared computing multiplexing method. Realize the provision of video query services through target recognition and target detection technologies in multi-user scenarios. And optimize the locality of multi-user video query in the shared computing environment, so that multiple users can share the results of deep learning reasoning, thereby improving the running speed and solving the problem that the speed of deep learning models is too low when processing video data. At the same time, solve the balance between speed and precision brought about by sharing. For this reason, the technical solution adopted by the present invention is a multi-user video stream deep learning shared computing multiplexing method. First, when a request with a detection operation or a recognition operation arrives, the requests are merged according to the relevance of the spatial dimension. Cut out the overlapping parts of the request; then, according to the relevance of the time dimension, request to search in the object detection database or object recognition database, and check whether there is suitable data available for reuse. When processing data, according to the correlation between logics, the data in the object recognition database or object detection database is used to filter the detection operation or recognition operation to reduce invalid calculations, and then call the deep learning model with configured parameters for analysis; for non-reusable parts , first find the most suitable parameter configuration in the analysis process according to the balance between speed and accuracy, then call the difference detector and deep learning model to analyze the video, and finally output the analysis results and store them in the data warehouse. The promotion module is used to Improve the accuracy of the original results in the database so that they can be reused for high-precision query requests.

It is necessary to reasonably configure related parameters such as resolution, selection of deep learning models, and frame skipping rate.

Specific steps for multiplexing the time dimension:

1) Use the method of difference detection to detect the similarity between consecutive frames. The difference detector obtains the histogram of consecutive frames and calculates the histogram distance, further calculates the similarity, and then judges whether the data can be reused according to the similarity;

2) When a new request comes, the request is first split into two parts, the reusable part and the non-reusable part. The data of the reusable part already exists in the database, and can be directly queried through the database with faster running time The operation obtains the result; for the part that cannot be reused, it is still processed through the deep learning model to obtain the request result, and then the requested result is fed back to the database for reuse in subsequent queries.

Multiplexing of spatial dimensions: Merge requests by cutting and merging requests, cut out overlapping parts, and reduce repeated requests.

Reuse of logical dimensions: establish associations between object detection and object recognition data, and find data that is correlated between different models to filter each other.

Specific steps for reuse of logical dimensions:

A video contains m frames of images. There are l types of objects that can be detected and k people can be identified. Two matrices Dl*m and Rk*m are used to store the data. Dl*m represents the matrix of objects, and Rk*m is Represents the character matrix. When a new query comes, first check whether there is corresponding data in Dl*m and Rk*m. If there is, use it directly. Dij is 1, which means there is j object in the i-th frame of the picture, and 0 means the object does not exist;

Theorem 1: If D1j is 0, then when 0<j<=m, R*j is also 0;

Theorem 2: If Rij is 1, then when 0<j<=m, D1j is also 1;

Theorem 1 indicates that when no one is detected in the jth frame, the target recognition model will not detect anyone in the jth frame; Theorem 2 indicates that when the person i is recognized in the jth frame, the target detection matrix will automatically Update the person in the j-th frame. Therefore, when performing logic multiplexing, dynamically update the database data according to Theorem 1 and Theorem 2, and establish associations for target detection and target recognition.

The specific steps to balance the accuracy and speed of queries:

Step 1. Fit the relationship between accuracy and speed of different models under different parameters, obtain the corresponding relationship between the speed and accuracy of each model, and select the optimal parameters for video analysis when the query arrives, so as to meet user needs;

Step 2. According to the Markov chain rule, the accuracy of the next frame is predicted from the accuracy of the previous frame and the difference between the two frames, plus the corresponding adjustment parameters, and the adjustment parameters are continuously corrected through verification experiments. value, so that accurate accuracy evaluation can be performed. Specifically, δdiff is used to represent the difference between two frames, A(fi-1) is used to represent the accuracy of i-1 frame, and k is used as an adjustment parameter, then according to the Markov chain rule , the accuracy of the i-th frame is:

A(fi-1)＝k*δdiff*A(fi-1).

Finally, the multiplexing of results between reasoning requests is conditional multiplexing;

Step 3. Retain the results of the deep learning model, re-detect part of the results of the difference detector using a high-precision model, and fully reuse the calculation results of the difference detector on the similarity between two frames, and re-evaluate the difference between consecutive frames. Accuracy, thereby improving the detection accuracy as a whole.

Conditional reuse specifically includes the use of increased frequency of use of deep learning models.

Features and beneficial effects of the present invention are:

The present invention builds a variety of deep learning models, so it can provide video query services through technologies such as target recognition and target detection in a multi-user scenario. It also optimizes the limitations of multi-user video query in a shared computing environment, so that multiple users can share the results of deep learning reasoning, thereby improving the running speed and solving the problem that the speed of deep learning models is too low when processing video data. At the same time, solve the balance between speed and precision brought about by sharing.

Description of drawings:

Figure 1: Multiplexing of request results for the same type of data.

Figure 2: Merge processing of queries.

Figure 3: Multiplexing of logical dimensions.

Figure 4: The overall architecture of multiple inference request result multiplexing.

Detailed ways

The invention relates to the fields of computer vision and high-performance computing, and proposes a multi-user video stream deep learning shared computing multiplexing method. The method realizes the provision of video query service through technologies such as target recognition and target detection in a multi-user scenario. And optimize the locality of multi-user video query in the shared computing environment, so that multiple users can share the results of deep learning reasoning, thereby improving the running speed and solving the problem that the speed of deep learning models is too low when processing video data. At the same time, solve the balance between speed and precision brought about by sharing.

Aiming at the limitations of existing technologies, we propose a new video stream analysis method, which supports data sharing in a multi-user sharing environment to improve analysis speed. In a shared computing environment, the inference request submitted by the user includes several aspects: request type (object detection, object recognition, object tracking, etc.), requested data and range, required accuracy, etc. In order to meet the diverse content of reasoning requests from users, it is necessary to build a variety of deep learning models in a shared computing environment, and dynamically select the appropriate model to meet user needs when the request comes. In the multiplexing of the results of multiple inference requests, the focus of research is to find the connection between multiple inference requests among the diverse request content, and build the correlation relationship between multiple requests for reuse to speed up the running time. At the same time, it solves the problems of accuracy and other issues caused by multiplexing, and achieves the effect of meeting user needs.

This method mainly includes two modules, one is to build the correlation between requests, and the other is to solve the problem of balance between precision and speed caused by multiplexing.

Build associations between requests

Step 1, the multiplexing of the time dimension. For video data, the relevance of the time dimension is reflected in two aspects: first, in video analysis, there is a high degree of similarity in content between consecutive frames; second, for popular data, there will be a large number of queries at different time periods that will It performs search analysis. These two aspects bring a lot of possibilities for reuse. First of all, for the similarity of consecutive frames, since the inference request time of the deep learning model is relatively long, for two highly similar frames, the result of the previous frame (called the reference frame) can be multiplexed to the next frame to avoid The detection of the next frame by the deep learning model is improved, and the speed is improved. To this end, we use the method of difference detection to detect the similarity between consecutive frames. The difference detector obtains the histogram of consecutive frames and calculates the histogram distance, and further calculates the similarity. Then judge whether the data can be reused according to the similarity.

The repetition of query data in different periods of time also brings a lot of redundant calculations to the system. Since there is no correlation between queries, each query will request and analyze the video data independently, resulting in a huge waste of resources. To solve this problem, the request results are stored, and when a new query arrives, duplicate parts are found and reused directly. As shown in Figure 1, when a new request arrives, the request is first split into two parts, the reusable part and the non-reusable part. The reusable part of the data already exists in the database, and the results can be obtained directly through the database query operation with faster running time. For the non-reusable parts, the request results are still processed through the deep learning model. Then the requested results are fed back to the database for reuse in subsequent queries. Assuming that a video contains m frames of images, there are a total of l types of objects that can be detected and k people can be identified. We use two matrices Dl*m and Rk*m to store the data. When a new query comes, first query Dl Whether there is corresponding data in *m and Rk*m, if there is, use it directly, avoiding double calculation. If Dij is 1, it means that there is an object j in the i-th frame of the picture, and if it is 0, it means that the object does not exist.

Step 2, multiplexing of spatial dimensions. In a shared computing environment, there are a large number of concurrent query operations. When these concurrent query requests are interested in the same part of data, a large number of repeated calculations are caused. The present invention merges the requests by cutting and merging the requests, cuts out overlapping parts, and reduces repeated requests. as shown in picture 2. There is an overlap between two simultaneous requests for query Q1 and Q2 for part of the data. Through query merging, two requests are reasonably merged into one request Q1' to avoid double calculation of heavy parts. The same is true when multiple queries arrive at the same time. Multiple queries are combined to maximize the overlap between multiple requests to reduce duplicate detection.

Step 3, reuse of logical dimensions. The video analysis system provides a variety of analysis methods, including object detection, object recognition, object tracking and other aspects of data in the video, which requires a variety of deep learning models to provide different detection capabilities. However, there is also the possibility of reuse between these different models. The most intuitive point is that if the object detection cannot detect people in some frames, the process of recognizing people in these frames can be directly eliminated by the object recognition model. Conversely, if the object recognition has identified a person, the object detection can also avoid the detection of the person in these frames. As shown in Figure 3, we establish associations between object detection and object recognition data, and find data that is correlated between different models to filter each other. For example, when the object recognition model recognizes a person, while updating the object recognition database, the data will also be fed back to the object detection database for updating, and the detection result will be marked as a person. When a new inference request queries whether there is a person, it can be filtered directly based on this result. We use D1j to record whether there is someone in the jth frame, use Rij to record whether there is person i in the jth frame, and R*j means all the characters in the jth frame. According to these properties, we conclude the following two theorems.

Theorem 1: If D1j is 0, then when 0<j<=m, R*j is also 0.

Theorem 2: If Rij is 1, then D1j is also 1 when 0<j<=m.

Theorem 1 indicates that when no one is detected in the jth frame, then the target recognition model will not detect anyone in the jth frame. Theorem 2 states that when a person i is recognized in the jth frame, the target detection matrix will automatically update the person in the jth frame. Therefore, when performing logic reuse, we can dynamically update the database data according to Theorem 1 and Theorem 2, and establish associations for target detection and target recognition.

Balance query precision and speed

In the current deep learning model, although the analysis accuracy is gradually improving, the resource consumption brought about by the accuracy improvement is also increasing, which requires a longer running time. In general, models with higher accuracy run slower, and models with lower accuracy run faster. On the other hand, the image resolution in the video analysis process and the number of skipped frames of the difference detector will also affect the accuracy and speed. This poses a challenge for parameter configuration during analysis. Because in practical applications, different inference requests have different requirements for accuracy and speed. Some requests, such as the detection of kidnapping and robbery, require high accuracy to maintain correctness, for which part of the speed can be sacrificed; while requests such as traffic light detection require high timeliness, and the accuracy can only reach an appropriate level. Therefore, how to reasonably select parameters such as model, resolution, and number of skipped frames in the process of video analysis is the primary problem.

Step 1: Fit the relationship between accuracy and speed of different models under different parameters to obtain the corresponding relationship between speed and accuracy of each model. When the query comes, the optimal parameters can be selected for video analysis to meet user needs.

On the other hand, difference detectors introduce uncertainty into the measure of accuracy. The difference detector detects the similarity between the current frame and the previous frame (reference frame), and directly reuses the analysis results of the previous frame in the case of high similarity, thereby avoiding costly inference calculations. However, the direct multiplexing leads to the deviation between the multiplexing result and the actual result, which reduces the detection accuracy. Therefore, it is necessary to perform an effective accuracy evaluation on the difference detector, so as to be able to calculate the overall effective accuracy and accurately meet the user's accuracy requirements. Since the difference detector detects two consecutive frames, it is obvious that the accuracy of the latter frame is closely related to the accuracy of the previous frame and the degree of difference between the two frames. This property between consecutive frames follows the Markov chain rule.

Step 2. According to the Markov chain rule, the accuracy of the next frame is predicted from the accuracy of the previous frame and the difference between the two frames, plus the corresponding adjustment parameters. Through a large number of verification experiments to continuously correct the value of the adjustment parameters, so that accurate accuracy evaluation can be carried out. We use δdiff to represent the difference between two frames, use A(fi-1) to represent the accuracy of frame i-1, and use k as an adjustment parameter, then according to the Markov chain rule, the accuracy of frame i is:

A(fi-1)＝k*δdiff*A(fi-1).

Finally, the multiplexing of results between reasoning requests is conditional multiplexing, not complete multiplexing without consideration, and the impact of precision on multiplexing is particularly important. For example, when a request for high-precision results arrives, the existing low-precision results obviously cannot be reused for high-precision requests. Therefore, there is a problem that data with different precisions cannot be reused. Obviously, the most intuitive method is to discard low-precision results and use high-precision models for re-detection, but this will undoubtedly bring a lot of overhead. Considering that the multiplexed data is the result of both the deep learning model and the difference detector, for example, the deep learning model detects the first frame, and the difference detector is used to process the last four frames. The accuracy of the results brought by the deep learning model is compared with The accuracy of the results brought by the difference detector is higher. For example, for the same piece of video data, it is assumed that the query request Q1 requires 90% accuracy. According to this requirement, the accuracy and speed balance part selects a frequency of 5 frame skipping steps for detection. That is, recognition is performed every 5 frames through the deep learning model, and the difference degree is obtained through the difference detector for the skipped 5 frames. After a period of time, the new query request Q2 arrives and requires 95% accuracy. Obviously, the results of Q1 cannot be directly used for Q2. At this time, the accuracy requirement can be met when the number of frame skipping steps is 2 through calculation. First, reuse the data in Q1, that is, reuse the detected frames. For the 5 frames skipped by the difference detector, use the deep learning model to detect the third frame, so that the requirement of 2 frame skipping steps can be achieved. . Finally, the effect of conditional reuse is achieved.

Step 3: Keep the results of the deep learning model, and re-detect part of the results of the difference detector (such as a frame in the last four frames) using a high-precision model. And fully reuse the calculation results of the difference detector about the similarity between two frames, and re-evaluate the accuracy between consecutive frames, so as to improve the detection accuracy as a whole.

Combining the above parts, the overall architecture of the data reuse model for multiple reasoning requests is shown in Figure 4. The system provides a variety of video analysis requests. There are two main types of deep learning models used in these requests: object recognition models and object detection models. first. When a request with a detection operation (or recognition operation) comes, according to the correlation of the spatial dimensions, the system first merges the requests and cuts out the overlapping parts of the requests. Then, in the multiplexing module part in the figure, according to the relevance of the time dimension, request to search in the object detection database (or object recognition database) to check whether there is suitable data available for reuse. In addition, according to the correlation between logics, the data in the object recognition database (or object detection database) is used to filter the detection operation (or recognition operation) to reduce invalid calculations. When there is no reusable data, the deep learning model with configured parameters is invoked for analysis. Before invoking the deep learning model, first of all, according to the user's demand for accuracy and speed, the relevant parameters must be configured through the speed-accuracy balance part, so as to achieve the highest possible speed while meeting the user's precision demand. In the inference part of the figure, it is necessary to reasonably configure relevant parameters such as resolution, deep learning model selection, frame skip rate, etc., then call the difference detector and deep learning model for video analysis, and finally output the analysis results and store them in the data storehouse. The precision improvement module continuously updates the result precision in the database so that high-precision query requests can be reused.

Claims (3)

1. A multi-user video stream deep learning shared computing multiplexing method is characterized in that, first, when a request with a detection operation or a recognition operation arrives, the requests are merged according to the relevance of the spatial dimension, and the request is cut out There are overlapping parts; then, according to the relevance of the time dimension, request to search in the object detection database or object recognition database, query whether there is existing data available for reuse, when there is no reusable data, according to the logic Relevance between objects, use the data in the object recognition database or object detection database to filter the detection operation or recognition operation to reduce invalid calculations, and then call the deep learning model with configured parameters for analysis; for non-reusable parts, firstly, according to the speed accuracy The balance relationship, find the parameter configuration in the analysis process, and then call the difference detector and deep learning model to analyze the video, and finally output the analysis results and store them in the data warehouse. The improvement module is used to improve the accuracy of the original results in the database. It is suitable for multiplexing high-precision query requests; it is necessary to reasonably configure relevant parameters including resolution, selection of deep learning models, and frame skipping rate; specific steps to balance query accuracy and speed: Step 1. Fit the relationship between accuracy and speed of different models under different parameters, obtain the corresponding relationship between the speed and accuracy of each model, and select the optimal parameters for video analysis when the query arrives, so as to meet user needs; Step 2. According to the Markov chain rule, the accuracy of the next frame is predicted from the accuracy of the previous frame and the difference between the two frames, plus the corresponding adjustment parameters, and the adjustment parameters are continuously corrected through verification experiments. value, so that accurate accuracy evaluation can be performed. Specifically, δdiff is used to represent the difference between two frames, A(fi-1) is used to represent the accuracy of i-1 frame, and k is used as an adjustment parameter, then according to the Markov chain rule , the accuracy of the i-th frame is: A(fi-1)＝k*δdiff*A(fi-1). Finally, the multiplexing of results between inference requests is conditional multiplexing, that is, the frequency of using the deep learning model is increased; step 3, the results of the deep learning model are retained, and part of the results of the difference detector are re-detected using a high-precision model , and fully reuse the calculation results of the difference detector about the similarity between two frames, and re-evaluate the accuracy between consecutive frames, thereby improving the detection accuracy as a whole; Among them, the reuse of logical dimensions: establish associations between object detection and object recognition data, and find data that is correlated between different models to filter each other. Specific steps for the reuse of logical dimensions: A video contains m frames of images. There are l types of objects that can be detected and k people can be identified. Two matrices Dl*m and Rk*m are used to store the data. Dl*m represents the matrix of objects, and Rk*m is Represents the character matrix. When a new query comes, first check whether there is corresponding data in Dl*m and Rk*m. If there is, use it directly. D _ij is 1, which means that there is the i-th object in the j-th frame of the picture, and it is 0 It means that the i-th object does not exist; if R _ij is 1, it means that the i-th person exists in the j-th frame of the picture, and if it is 0, it means that the i-th person does not exist; D _1j records whether there is someone in the j-th frame; R _*j means that the j-th All characters in the frame; Theorem 1: If D1j is 0, then when 0<j<=m, R*j is also 0; Theorem 2: If Rij is 1, then when 0<j<=m, D1j is also 1; Theorem 1 indicates that when no one is detected in the jth frame, the target recognition model will not detect anyone in the jth frame; Theorem 2 indicates that when the person i is recognized in the jth frame, the target detection matrix will automatically Update the person in the j-th frame. Therefore, when performing logic multiplexing, dynamically update the database data according to Theorem 1 and Theorem 2, and establish associations for target detection and target recognition. 2. The multi-user video stream deep learning sharing calculation multiplexing method as claimed in claim 1, is characterized in that, the multiplexing concrete steps of time dimension: 1) Use the method of difference detection to detect the similarity between consecutive frames. The difference detector obtains the histogram of consecutive frames and calculates the histogram distance, further calculates the similarity, and then judges whether the data can be reused according to the similarity; 2) When a new request comes, the request is first split into two parts, the reusable part and the non-reusable part. The data of the reusable part already exists in the database, and can be directly queried through the database with faster running time The operation obtains the result; for the part that cannot be reused, it is still processed through the deep learning model to obtain the request result, and then the requested result is fed back to the database for reuse in subsequent queries. 3. The multi-user video stream deep learning shared computing multiplexing method according to claim 1, characterized in that the multiplexing of spatial dimensions: the request is merged by cutting and merging requests, and overlapping parts are cut out to reduce Multiple repeated requests.