CN110969072B

CN110969072B - Model optimization method, device and image analysis system

Info

Publication number: CN110969072B
Application number: CN201910556196.5A
Authority: CN
Inventors: 谭晶晶; 户军; 许毅; 张记伟
Original assignee: Hangzhou Hikvision System Technology Co Ltd
Current assignee: Hangzhou Hikvision System Technology Co Ltd
Priority date: 2018-09-30
Filing date: 2019-06-25
Publication date: 2023-05-02
Anticipated expiration: 2039-06-25
Also published as: CN110969072A

Abstract

The invention discloses a model optimization method, model optimization equipment and an image analysis system, and belongs to the field of image monitoring. The method comprises the following steps: acquiring alternative images from a plurality of images acquired by a first image acquisition device, selecting training samples from the alternative images, and training a detection model of the first user side device based on the training samples, wherein the detection model of the first user side device is used for detecting the images acquired by the first image acquisition device; and when the trained detection model is better than the detection model, updating the detection model of the first user terminal equipment by adopting the trained detection model. The problem that the detection model obtained through training cannot be matched with the image analysis scene corresponding to the user side equipment is solved, and the reliability of the optimized detection model is improved.

Description

Model optimization method, device and image analysis system

The present application claims priority from chinese patent application No. 201811161087.5 entitled "model optimization method, apparatus, and image analysis system," filed on 9/30/2018, the entire contents of which are incorporated herein by reference.

Technical Field

The present invention relates to the field of video monitoring, and in particular, to a method and apparatus for model optimization and an image analysis system.

Background

The image analysis system is widely applied to various fields of security, traffic, scientific research, entertainment and the like, and the image analysis system based on big data can comprise a central management server (English: central Management Server; abbreviated: CMS) and a plurality of user side devices, wherein each user side device manages a plurality of image acquisition devices in an image analysis scene. The image acquisition equipment can send the acquired image to corresponding user equipment, and the user equipment automatically analyzes the key information for processing by image monitoring personnel.

In the related art, the CMS establishes a detection model according to a pre-established image library, and sends the detection model to each piece of user equipment, and each piece of user equipment analyzes an image acquired by an image acquisition device managed by the user equipment based on the detection model. The image library may be updated continuously based on images provided by the plurality of client devices. When a certain user terminal device puts forward a model optimization requirement, the CMS can acquire a plurality of training samples from the image library, train the detection model based on the plurality of training samples, and update the detection model in the user terminal device by taking the trained detection model as an optimized model.

However, the detection model obtained by training in the related art cannot be adapted to the image analysis scene corresponding to the user terminal device, so that the reliability of the optimized model is low.

Disclosure of Invention

The embodiment of the invention provides a model optimization method, model optimization equipment and an image analysis system, which can solve the problem of lower reliability of an optimized model in the related technology. The technical scheme is as follows:

according to a first aspect of the present invention, there is provided a model optimization method, the method comprising:

acquiring an alternative image from a plurality of images acquired by first image acquisition equipment, wherein the first image acquisition equipment is image acquisition equipment managed by first user equipment;

selecting a training sample from the alternative images, and adding the training sample to a training sample library corresponding to the first user terminal equipment;

training the detection model of the first user side equipment based on samples in the training sample library to obtain a trained detection model, wherein the detection model of the first user side equipment is used for detecting images acquired by the first image acquisition equipment;

and when the trained detection model is better than the detection model, updating the detection model of the first user terminal equipment by adopting the trained detection model.

Optionally, the acquiring the candidate image from the plurality of images acquired by the first image acquisition device includes:

inputting the plurality of images into the detection model respectively;

receiving a first analysis result of each image output by the detection model;

receiving an audit instruction input by a user at the first user terminal equipment, wherein the audit instruction is used for indicating whether the first analysis result is accurate or not;

and when the auditing instruction indicates that the first analysis result of any image is accurate, determining any image as an alternative image, and storing the first analysis result.

Optionally, the acquiring the candidate image from the plurality of images acquired by the first image acquisition device further includes:

when the auditing instruction indicates that the first analysis result is inaccurate, a first analysis result updating instruction input by a user at the first user terminal equipment is received, the first analysis result updating instruction is used for indicating an updated first analysis result,

determining an image containing the updated first analysis result as an alternative image, and storing the updated first analysis result;

or when the auditing instruction indicates that the first analysis result of any image is inaccurate, determining any image as a non-alternative image.

Optionally, before the step of updating the detection model of the first user terminal device with the trained detection model, the method further includes:

selecting a test image from the candidate images;

judging whether the trained detection model is better than the detection model or not based on the test image;

the updating the detection model of the first user equipment by adopting the trained detection model comprises the following steps:

Optionally, the determining, based on the test image, whether the trained detection model is better than the detection model includes:

inputting the test image into the trained detection model to obtain a second analysis result;

detecting whether the accuracy of the trained detection model meets a specified accuracy condition or not based on a second analysis result of the test image and a first analysis result stored in advance;

and when the accuracy of the trained detection model accords with the specified accuracy condition, judging that the trained detection model is better than the detection model.

Optionally, the test image has a plurality of test images, and the detecting whether the accuracy of the trained detection model meets the specified accuracy condition based on the second analysis result of the test image and the pre-stored first analysis result includes:

counting the first quantity of the test images with the same first analysis result and the corresponding second analysis result;

when the ratio of the first quantity to the total number of the test images is greater than a specified proportion threshold, determining that the accuracy of the trained detection model meets a specified accuracy condition;

and when the ratio of the first quantity to the total number of the test images is not greater than a specified proportion threshold, determining that the accuracy of the trained detection model does not meet a specified accuracy condition.

Optionally, the selecting a training sample in the candidate image includes:

selecting images with specified proportions from the alternative images as training samples;

the selecting a test image from the candidate images includes:

and taking images except the training sample in the alternative images as test images.

Optionally, before the training the detection model of the first user terminal device based on the samples in the training sample library, the method further includes:

Backing up the detection model of the first user equipment;

and deleting the backup detection model when the backup detection model accords with the deletion condition.

Optionally, the method is performed by the first user terminal device, and the training sample library is stored locally on the first user terminal device.

According to a second aspect of the present invention, there is provided a model optimizing apparatus characterized by comprising:

the acquisition module is used for acquiring an alternative image from a plurality of images acquired by first image acquisition equipment, wherein the first image acquisition equipment is image acquisition equipment managed by first user equipment;

the adding module is used for selecting training samples from the alternative images and adding the training samples to a training sample library corresponding to the first user terminal equipment;

the training module is used for training the detection model of the first user side equipment based on the samples in the training sample library to obtain a trained detection model, and the detection model of the first user side equipment is used for detecting the image acquired by the first image acquisition equipment;

and the updating module is used for updating the detection model of the first user terminal equipment by adopting the trained detection model when the trained detection model is better than the detection model.

Optionally, the acquiring module is configured to:

inputting the plurality of images into the detection model respectively;

receiving a first analysis result of each image output by the detection model;

Optionally, the acquiring module is further configured to:

Optionally, the apparatus further includes:

a selection module for selecting a test image from the candidate images;

The judging module is used for judging whether the trained detection model is better than the detection model or not based on the test image;

Optionally, the judging module includes:

the input sub-module is used for inputting the test image into the trained detection model to obtain a second analysis result;

the detection sub-module is used for detecting whether the accuracy of the trained detection model meets the specified accuracy condition or not based on the second analysis result of the test image and the first analysis result which is stored in advance;

and the judging sub-module is used for judging that the trained detection model is better than the detection model when the accuracy of the trained detection model accords with the specified accuracy condition.

Optionally, the test images have a plurality of pieces, and the detection submodule is configured to:

Optionally, the apparatus further includes:

and the backup module is used for backing up the detection model of the first user terminal equipment.

Optionally, the apparatus further includes:

and the deleting module is used for deleting the backup detection model when the backup detection model accords with the deleting condition.

Optionally, the model optimizing device is the first user equipment, and the training sample library is stored locally to the first user equipment.

According to a third aspect of the present invention, there is provided an image analysis system comprising, the model optimizing apparatus according to the second aspect;

the image analysis system further comprises at least one user side device, and each user side device manages at least one image acquisition device.

According to a fourth aspect of the present invention, there is provided a computer device comprising a processor and a memory,

wherein the memory is used for storing a computer program;

the processor is configured to execute the program stored in the memory, and implement the model optimization method according to the first aspect.

According to a fifth aspect of the present invention, there is provided a storage medium having stored therein a computer program which, when executed by a processor, implements the model optimization method of the first aspect.

The technical scheme provided by the embodiment of the invention has the beneficial effects that:

the detection model of the first user terminal equipment is trained by selecting a training sample from the candidate images so as to obtain a trained detection model, and because the candidate images are images acquired from a plurality of images acquired by the first image acquisition equipment, and the first image acquisition equipment performs image acquisition aiming at a certain image analysis scene where the first user terminal equipment is located, the trained detection model is the detection model aiming at the first user terminal equipment and can be matched with the certain image analysis scene where the first user terminal equipment is located, the problem that the detection model obtained by training in the related technology cannot be matched with the image analysis scene corresponding to the user terminal equipment is solved, and the reliability of the optimized detection model is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an image analysis system according to an embodiment of the present invention;

FIG. 2 is a flow chart of a model optimization method provided by an embodiment of the present invention;

FIG. 3 is a flow chart of another model optimization method provided by an embodiment of the present invention;

fig. 4 is a flowchart of a method for training a detection model of a first user device based on samples in a training sample library to obtain a trained detection model according to an embodiment of the present invention;

FIG. 5 is a flowchart of a method for determining whether a trained test model is better than a test model according to an embodiment of the present invention;

FIG. 6 is a flowchart of a method for determining whether a backup detection model meets a deletion condition according to an embodiment of the present invention;

FIG. 7 is a method flow diagram of yet another model optimization method provided by an embodiment of the present invention;

FIG. 8 is a block diagram of a model optimizing apparatus provided by an embodiment of the present invention;

FIG. 9 is a block diagram of another model optimization device provided by an embodiment of the present invention;

FIG. 10 is a block diagram of a determination module according to an embodiment of the present invention;

FIG. 11 is a block diagram of yet another model optimization apparatus provided by an embodiment of the present invention;

FIG. 12 is a block diagram of yet another model optimization apparatus provided by an embodiment of the present invention;

FIG. 13 is a block diagram of yet another model optimization apparatus provided by an embodiment of the present invention;

fig. 14 is a block diagram of a configuration of a ue according to an embodiment of the present invention.

Specific embodiments of the present invention have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.

Detailed Description

For the purpose of promoting an understanding of the principles and advantages of the invention, reference will now be made in detail to the embodiments of the invention, some but not all of which are illustrated in the accompanying drawings. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Fig. 1 illustrates an image analysis system 10 related to a model optimization method according to an embodiment of the present invention, where the image analysis system 10 may include at least one user side device 12 and at least one image acquisition device 13, and the image analysis system 10 may be a big data based image analysis system. Wherein each client device 12 manages at least one image acquisition device 13. Fig. 1 illustrates an image analysis system 10 including two client devices 12 and two image capturing devices 13 managed by each client device 12, but is not limited thereto.

The image pickup device 13 may be a device capable of image pickup such as an image pickup device or a photographic device, and may be a double-sided camera or a wide-angle camera, for example. Optionally, the image acquisition device may be a network camera (english: internet Protocol Camera; abbreviated as IPC) such as a rifle bolt or a ball camera, where the network camera may transmit the acquired video to a user terminal device connected to the network through the network. In the embodiment of the present invention, the image capturing device 13 may be installed in a scene where the image analysis system is to perform image analysis, for example, in an image analysis system applied to a metropolitan environment, an object to be subjected to image analysis by the image analysis system may be an event such as a vendor or a store in a city, and the purpose of analysis is to determine an offending event, so that the image capturing device may be installed in a region patrol by a metropolitan area. The client device 12 may comprise a user operated terminal and in an alternative embodiment the client device 12 may further comprise a server connected to the user operated terminal.

Optionally, the image analysis system 10 may further include a central management server 14, where the central management server 14 may be a server, or a server cluster formed by a plurality of servers, or a cloud computing service center, and the central management server 14 may provide a service required by a client device and optimize the service, for example, in the image analysis system provided by the embodiment of the present invention, the central management server 14 may establish an initial detection model according to a scenario where the image analysis system 10 is required to perform image analysis and a required analysis result, so that the client device may use the initial detection model, and optionally, the central management server 14 may further optimize the initial detection model. In other alternative embodiments, the central management server may also be considered a laboratory device for providing services, embodiments of the invention are not limited herein.

In the image analysis system 10, the above-described client devices 12 are installed with a designated client through which the center management server 14 can provide services to the client devices.

Optionally, the image analysis system 10 may further comprise a supervisory platform 15, the supervisory platform 15 may comprise a terminal for operation by a supervisory person, and in an alternative embodiment, the supervisory platform 15 may further comprise a server connected to the terminal for the supervisory person. The supervision platform 15 is used for supervising the scene of image analysis according to the analysis result obtained by the supervision platform 15, and compared with the user terminal device 12, the supervision platform 15 is a user device with higher authority, and the supervision personnel can further process the analysis result according to the supervision platform 15.

Currently, image analysis systems can be applied to different application environments (i.e., uses), and for each application environment, there are multiple image analysis scenarios. In general, in an image analysis system applied to an application environment, a client device may be configured for each image analysis scene, and the client device may manage one or more image capturing devices to perform images in the image analysis scene. Of course, a plurality of client devices may be configured for each image analysis scene. The embodiment of the invention is illustrated by taking a user terminal device corresponding to each image analysis scene as an example. When each image analysis scene corresponds to one user terminal device, the actions performed by each user terminal device refer to the following embodiments.

The image analysis scene is divided differently in different application environments, for example, the image analysis system is applied to a metropolitan environment, which is a metropolitan image analysis system, and the image analysis scene can be divided in a city division manner. For example, china includes a plurality of cities such as Chongqing city, hangzhou city and Tianjin city, etc., and the city management environment of the traffic image analysis system may include Chongqing city image analysis scene, hangzhou city image analysis scene, tianjin city image analysis scene, etc.

In other alternative embodiments, the image analysis system may be applied to a traffic environment, which may be a traffic image analysis system, and the image analysis scene may be divided by adopting a mode of urban internal partition, for example, the western security city includes a plurality of partitions such as a goose tower region, a tombstone forest region, a lotus lake region, etc., and then the traffic environment of the traffic image analysis system may include a goose tower region image analysis scene, a tombstone forest region image analysis scene, a lotus lake region image analysis scene, etc. Of course, the image analysis system described in the embodiments of the present invention may also be applied to other application environments, and embodiments of the present invention are not listed here.

Currently, in an image analysis system corresponding to each application environment, a feature parameter to be analyzed needs to be set, and a detection model is built based on the feature parameter, so that after an image is input into the detection model, the detection model outputs an analysis result matched with the feature parameter, and the analysis result may include a target object matched with the feature parameter and a position of the target object in the image.

For example, the image analysis system is a city management image analysis system for analyzing an offence event in a city, and the object to be subjected to image analysis by the image analysis system is an event such as a vendor or a shop in the city. When a detection model is built, characteristic parameters for representing various illegal event types are required to be set, then a plurality of images of the various illegal event types are used as a training sample set to train the detection model, and the analysis result of the detection model comprises: the type of violation event in the image and/or the location of the violation event is generated.

For another example, the image analysis system is a traffic image analysis system for analyzing a offending vehicle, and the object to be subjected to image analysis by the image analysis system is a vehicle traveling in a road. When a detection model is established, characteristic parameters for representing various types of violating vehicles are required to be set, then a plurality of images of the various types of violating vehicles are used as a training sample set to train the detection model, and the analysis result of the detection model comprises: the type of violation of the violation vehicle in the image, and/or the location of the violation vehicle.

However, in each existing image analysis system, each detection model provided by the CMS is specific to all image analysis scenes in one application environment, that is, for the one application environment, no matter how many image analysis scenes the application environment includes, training samples of the CMS for optimizing the detection model are obtained by selecting in a fixed image library, and finally, the optimized detection model is applied to each analysis scene in the application environment and cannot be specific to the image analysis scene corresponding to the client device, so that reliability of the optimized model is low, for example, a situation that some specific image analysis scenes are over-fitted easily occurs.

FIG. 2 is a flow chart of a model optimization method for use in the image analysis system 10 of FIG. 1, which may include the following steps:

step 201, acquiring an alternative image from a plurality of images acquired by a first image acquisition device.

The first image acquisition equipment is image acquisition equipment managed by first user equipment. For example, the first client device is one of the at least one client devices in the image analysis system 10.

Step 202, a training sample is selected from the candidate images, and the training sample is added to a training sample library corresponding to the first user terminal device.

Step 203, training the detection model of the first user terminal device based on the samples in the training sample library, to obtain a trained detection model.

The detection model of the first user side device is used for detecting the image acquired by the first image acquisition device.

And 204, when the trained detection model is better than the detection model, updating the detection model of the first user side equipment by adopting the trained detection model.

In summary, according to the model optimization method provided by the embodiment of the invention, the training sample is selected from the candidate images to train the detection model of the first user terminal device so as to obtain the trained detection model, and because the candidate images are images acquired from a plurality of images acquired by the first image acquisition device, the first image acquisition device performs image acquisition aiming at a certain image analysis scene where the first user terminal device is located, the trained detection model can be adapted to the certain image analysis scene, thereby solving the problem that the detection model obtained by training in the related technology cannot be adapted to the image analysis scene corresponding to the user terminal device, and improving the reliability of the optimized detection model.

The above model optimization method is further described in an alternative implementation below. If the model optimization method provided by the embodiment of the invention is adopted to optimize the detection model in the national city management image analysis system, taking the national city management environment including three image analysis scenes of Chongqing city image analysis scenes, hangzhou city image analysis scenes and Tianjin city image analysis scenes as an example, the three image analysis scenes are assumed to be in one-to-one correspondence with three user side devices, namely a first user side device, a second user side device and a third user side device, wherein the first user side device can be used for correspondingly managing the first image acquisition device in Chongqing city image analysis scenes, the second user side device can be used for correspondingly managing the second image acquisition device in Hangzhou city image analysis scenes, the third user side device can be used for correspondingly managing the third image acquisition device in Tianjin city image analysis scenes and the like. Taking a Chongqing city image analysis scene as an example, the plurality of images acquired by the first image acquisition equipment are all the plurality of images in the Chongqing city image analysis scene, the candidate images are acquired from the plurality of images, further, a training sample is selected to train a detection model corresponding to the Chongqing city image analysis scene, the trained detection model can be aimed at the Chongqing city image analysis scene, if the first image acquisition equipment continuously inputs the acquired images into the detection model, the analysis result output by the detection model is more suitable for the Chongqing city image analysis scene, so that the detection model has more pertinence and higher reliability, and the detection model aiming at the Hangzhou city image analysis scene and the detection model aiming at the Tianjin city image analysis scene can be trained by analogy.

It should be noted that, the foregoing model optimization method may be executed by any one of the foregoing at least one ue, or executed by the central management server, and the detection model may be an initial detection model sent by the central management server.

In a first alternative implementation manner, the foregoing model optimization method is executed by a first ue, where the first ue is any one of at least one ue of the model optimization system, please refer to fig. 3, where fig. 3 shows a flowchart of another model optimization method provided by an embodiment of the present invention, the model optimization method may include a plurality of model optimization processes that are sequentially executed, an embodiment of the present invention is described by taking a round of model optimization process as an example, and in actual implementation of the embodiment of the present invention, each round of model optimization process may refer to the round of model optimization process, which is not repeated herein, and the model optimization method includes:

step 301, the first user terminal device inputs a plurality of images acquired by the first image acquisition device into a detection model of the first user terminal device respectively.

The first image acquisition device may be an image acquisition device managed by the first user side device. The first image capturing device may capture images of the scene in which the first image capturing device is located at a fixed frequency and transmit the images to the first user terminal device in real time, or when the number of images of the scene in which the first image capturing device is located at the fixed frequency reaches a fixed threshold, transmit the number of images to the first user terminal device. The detection model of the first user side device is used for detecting the image acquired by the first image acquisition device.

The first image acquisition device and the first user terminal device can be connected through a network, the first image acquisition device can be an IPC, and the IPC can transmit acquired video to the first user terminal device through the network. In actual implementation, the first user terminal device may extract a single frame image in the video, and input a plurality of Shan Zhen images into the detection model respectively. Alternatively, the first user terminal device may extract a single frame image in the video stream according to a predetermined frequency (for example, capture one frame image every 5 minutes), which is not limited in the embodiment of the present invention.

When the one-round model optimization process provided by the embodiment of the invention is the first-round model optimization process, the detection model is an initial detection model sent by the central management server. The initial detection model may be established by a central management server and provided to the first client device.

Optionally, in order to ensure that the initial detection model occupies less memory, the installation is convenient, and the central management server may generate the initial detection model by using a small amount of training samples, where the training samples may be images acquired by any image acquisition device, or images acquired from the internet, or images acquired by other means. Alternatively, in order to ensure the validity of the established initial detection model, a part of images with obvious characteristics can be manually selected from the images acquired by any image acquisition device, and then a detection algorithm is adopted to generate the initial detection model. Since in machine vision learning, a detection algorithm is generally referred to as an object detection algorithm (e.g., an object detection algorithm that classifies based on a sliding window), a detection model is generally referred to as an object detection model. Thus, the initial detection model may be established based on a target detection algorithm, which may be various, for example, a single multi-frame detector (English: single Shot Multibox Detector; abbreviated: SSD) target detection algorithm or a single-eye target detection algorithm (English: you Only Look Once; abbreviated: YOLO) target detection algorithm, etc., and embodiments of the present invention are not limited herein.

It should be noted that, because the accuracy of the analysis result output by the image analysis system on the detection model is generally required to be higher, the detection model in the embodiment of the invention can be established based on a supervised learning method.

In addition, when the model optimization process of the present embodiment is the model optimization process of the nth round, n is an integer greater than 1, and the detection model is a model obtained by updating the initial detection model at least once.

Step 302, the first user side device receives a first analysis result of each image output by the detection model.

The first user terminal equipment receives a first analysis result of each image respectively output by the detection model after respectively inputting the plurality of images acquired by the first image acquisition equipment into the detection model, thereby realizing acquisition of the analysis result.

Step 303, the first ue receives an audit command input by a user at the first ue, where the audit command is used to indicate whether the first analysis result is accurate.

The first user terminal device can be a device with a display screen, such as a terminal operated by a user, a tablet personal computer or a smart phone, and the first user terminal device can present a first analysis result of each image for the user to review. The first analysis result output by the detection model is presented, for example, through a User Interface (UI).

The first analysis result is obtained by analyzing each image by using a detection model in the optimization process of the round of models. For example, the first analysis result may include a type of the target object of the image and a region of the target object, and optionally, the type of the target object may be labeled in the form of text in the vicinity of the region of the target object.

Of course, for each image, besides presenting the first analysis result output by the detection model, information such as time and area for acquiring the image may also be presented on each image, which is not limited herein.

In other optional implementation manners, when the first user side device presents the first analysis result output by the detection model, a prompt message may also be sent to prompt the user to audit the first analysis result, where the prompt message may be sent in the form of text, sound and/or light, and the embodiment of the present invention is not limited herein.

After the first user terminal equipment presents the first analysis result output by the detection model through the UI, a user of the first user terminal equipment can review the first analysis result by combining with own experience, and the first user terminal equipment is operated to trigger the first user terminal equipment to generate a review instruction aiming at the first analysis result.

Optionally, the UI of the first user side device may sequentially present a plurality of images, so that a user may sequentially audit a first analysis result of the plurality of images, so that the first user side device may sequentially receive a plurality of audit instructions. Of course, the UI of the first user side device may also present a plurality of images at one time, so that the user may review the first analysis results of the plurality of images, and when the first user side device receives the review instruction of the plurality of images, a plurality of new images are presented on the UI for review by the user.

Optionally, the first ue may receive the audit command in multiple manners, for example, a corresponding button is set in the UI, and the user sends the audit command for the first analysis result to the first ue by triggering the button. The buttons may include an accurate button that indicates that the first analysis result is accurate and/or an inaccurate button that indicates that the first analysis result is inaccurate. Alternatively, in the UI, a corresponding button may be provided for each image.

Step 304, when the auditing instruction indicates that the first analysis result of any image is accurate, the first user side device determines the any image as an alternative image, and stores the first analysis result.

After the first user terminal device receives the auditing instruction indicating that the first analysis result of any image is accurate, the first user terminal device can determine any image as an alternative image and store the first analysis result. The stored first analysis result is an analysis result determined by a user, and the analysis result can be used for training a detection model of the first user terminal equipment and checking the reliability of the trained detection model. For example, when a certain candidate image is used as a training sample, the content recorded in the first analysis result can be used as a labeling result, namely a label, of the training sample.

Step 305, when the audit instruction indicates that the first analysis result is inaccurate, the first ue receives a first analysis result update instruction input by the user at the first ue, where the first analysis result update instruction is used to indicate the updated first analysis result.

When the user checks that the first analysis result is inaccurate, the user can continuously trigger the first analysis result updating instruction after triggering the checking instruction indicating that the first analysis result is inaccurate, and correspondingly, the first user terminal equipment sequentially receives the checking instruction and the first analysis result updating instruction. The manner in which the user triggers the first analysis result update instruction may be various, and the following two ways are exemplified:

In the first manner, the user may directly update the first analysis result, and the UI presented by the first user side device may be provided with a function for the user to select an area in the image, where the update manner may include: and re-checking the area in the image and/or updating the text description of the area, wherein when the user re-checks the area in the image and/or revises the text description of the area, the first user terminal equipment can receive a first analysis result updating instruction.

In the second mode, a button marked with a first analysis result updating is arranged in a UI presented by the first user terminal equipment, after the button is triggered by a user, the first user terminal equipment presents an updating interface for the user to input a new analysis result in the updating interface, and correspondingly, the first user terminal equipment receives a first analysis result updating instruction input by the user at the first user terminal equipment.

In another optional embodiment, when the user determines that the first analysis result is inaccurate, the user may trigger generation of the audit instruction and the first analysis result update instruction through the same operation, that is, the audit instruction and the first analysis result update instruction are the same instruction. Embodiments of the present invention are not limited in this regard as to other alternative implementations.

Step 306, the first user terminal device determines the image containing the updated first analysis result as an alternative image, and stores the updated first analysis result.

Optionally, after the first user terminal device receives the first analysis result updating instruction, an option for prompting the user whether to complete updating of the first analysis result may be presented in the UI, and after the user triggers the option, the first user terminal device may determine an image including the updated first analysis result as an alternative image and store the updated first analysis result.

Step 307, when the auditing instruction indicates that the first analysis result of any image is inaccurate, the first user side device determines the any image as a non-alternative image.

In another alternative embodiment, when the auditing instructions indicate that the first analysis result of any image is inaccurate, the user may not update the first analysis result, but rather determine any image as a non-alternative image that would not be used in subsequent operations. That is, the

steps

305 and 307 may be alternatively performed.

The steps 304 to 307 describe how to determine the alternative image, and it can be seen from the steps that the alternative image is an image audited by the user of the first user terminal, and because the process of determining the alternative image is participated by the user of the first user terminal, the user is more familiar with the image analysis scene corresponding to the first user terminal device due to long-term contact with the image in the image analysis scene, so that the auditing result is more accurate, and therefore, the accuracy of the alternative image is ensured.

Further, after the foregoing

steps

304 and 306, if the image analysis system includes a supervision platform, the first user side device may send the alternative image to the supervision platform (refer to fig. 1) for the supervision personnel to further process the alternative image, for example, for the metropolitan image analysis system, the supervision platform may be a report platform, and after the alternative image is sent to the report platform, the supervision personnel may process the type of the violation event and generate the case event from the report platform.

By way of example, it is assumed that the detection model is applied in an image analysis system of a metropolitan environment. The UI presents an image containing a first analysis result, where the first analysis result includes an area where the violation event is located and a type of the violation event, and a user (e.g., a metropolitan area manager) of the first user terminal device may audit the first analysis result.

When the user considers that the first analysis result is accurate, a 'complete audit' button in the UI can be clicked, the first user terminal equipment receives an audit instruction corresponding to the button, the audit instruction can indicate that the first analysis result presented in the UI is accurate, and the first user terminal equipment can determine the image as an alternative image and store the first analysis result; when the user considers that at least one of the area of the first analysis result where the violation event is located or the type of the violation event is inaccurate, the first analysis result may be updated, and the updating process may include: when the area where the violating event is located is inaccurate, the user can re-select the area where the violating event is located in the image, when the type of the violating event is inaccurate, the user can re-select the violating type in the UI, when the user re-selects the area where the violating event is located or the type of the violating event is re-selected, the auditing instruction and the first analysis result updating instruction in the steps are triggered at the same time, when the user considers that the updated first analysis result is accurate, the user can click a button of completing auditing of the UI, the first user terminal device determines the image containing the updated first analysis result as an alternative image, stores the updated first analysis result, and when the area where the violating event is located and the type are both inaccurate, the two updating processes can be referred to, and details are omitted.

Optionally, when the user considers that the first analysis result is accurate, or after the user updates the first analysis result, a "report case" button under the UI may be triggered to report the candidate image to the supervision platform.

In addition, the UI may further include a number (e.g., camera 01) of an image capturing device for capturing the image, time information (e.g., alarm time 10:00) for capturing the image, processing comments (e.g., confirm violations, not enough to be set up), remark information, and the like.

Step 308, the first user terminal device selects a training sample from the candidate images, and adds the training sample to a training sample library corresponding to the first user terminal device.

When the first user terminal device selects the test image from the candidate images, the image with the specified proportion may be selected as the training sample, or of course, a specified number of images may be selected from the candidate images as the training sample.

The user of the first user terminal equipment carries out auditing on the first analysis result output by the detection model according to the self relevant experience, updates the first analysis result which is considered to be inaccurate by the user, ensures the accuracy of the alternative image, and further ensures the accuracy of the training sample.

For example, for a city management image analysis system, the city management image analysis system adopts a city division manner to divide an image analysis scene, and since the city management of each city has different judgment standards for violations: for the same behavior, urban management of some cities may consider the behavior as not illegal, and urban management of some cities may consider the behavior as calculating illegal; if the same kind of illegal act is considered by the urban management of some cities to belong to the lane occupation operation, the illegal act may be considered by the urban management of some cities to belong to the out-of-store operation. If a unified detection model is established for the urban management image analysis system by adopting the CMS in the related technology, so that the detection model cannot effectively analyze scenes for each urban image, in the model optimization method provided by the embodiment of the invention, the urban management managing the fixed city is used as a user, and the acquired image of the city is checked based on the illegal judgment standard, so that the trained detection model is more in accordance with the cognition and habit of the urban management of the city, and the pertinence of the trained detection model is improved. For example, a city management of Chongqing city may review images collected in Chongqing city image analysis scenes based on the determination criteria of the Chongqing city for violations, a city management of Hangzhou city may review images collected in Hangzhou city image analysis scenes based on the determination criteria of the Hangzhou city for violations, and a city management of Tianjin city may review images collected in Tianjin city image analysis scenes based on the determination criteria of the Tianjin city for violations, such that each detection model may be trained for each image analysis scene, respectively.

Optionally, the training sample library may be a local sample library of the first user terminal device, so that leakage of image resources is avoided, and confidentiality of images is achieved. Especially when the image acquired by the first image acquisition device is security image or confidential image such as the image of confidential place, the first user side device can perform model training based on a local training sample library, and the security of the image can be effectively ensured without the participation of other external devices. The local sample library can be updated in real time along with the images acquired by the first image acquisition equipment managed by the first user equipment, so that the timeliness is high, and the updated detection model can be ensured to be more adaptive to the continuously-changed specific image analysis scene of the first user equipment.

Step 309, the first ue backs up the detection model of the first ue.

In the model optimization method provided by the embodiment of the invention, even if the detection model trained later is optimized compared with the previous detection model, the situation that the use requirement in the actual operation of the user of the first user terminal equipment cannot be met may occur. Therefore, in order to ensure that the rollback of the detection model can be realized in this case, the first ue may backup the detection model of the first ue (i.e. the detection model that has not currently performed the training process).

Furthermore, the first user equipment may also backup each of the trained detection models, so as to provide a reference for a subsequent model optimization process.

Since the backup detection model is in an idle state, in order to avoid the leakage of the backup detection model, which leads to the damage of the security of the image analysis system, the backup detection model can be encrypted.

Step 310, the first ue trains the detection model of the first ue based on the samples in the training sample library, to obtain a trained detection model.

As shown in fig. 4, step 310 may include the steps of:

in step 3101, when the number of samples in the training sample library is greater than the specified sample number threshold, the first user equipment performs at least one training process on the detection model until the detection model meets a training stop condition.

The first ue may actively trigger the performing, by the user, at least one training process on the detection model of the first ue, or may automatically perform, by the first ue, at least one training process on the detection model when a specified condition is satisfied.

Optionally, the specific condition may be that the number of samples in the training sample library is greater than a specific sample number threshold, and when the number of samples in the training sample library is greater than the specific sample number threshold, the first user terminal device automatically performs at least one training process on the detection model until the detection model meets a training stop condition.

By setting the specified conditions, the efficiency of the model optimization method can be improved, and the system performance of the first user terminal equipment is prevented from being consumed in transition.

In order not to influence normal use of the first user terminal equipment by a user, when the first user terminal equipment meets the specified condition, the first user terminal equipment can detect whether the first user terminal equipment is in an idle state at the moment, and when the first user terminal equipment is in the idle state, at least one training process is automatically executed on the detection model.

Alternatively, the training stop condition may include: the number of exercises reaches a specified number of exercises threshold and/or the exercise error converges within a specified range. There are various ways to determine whether the training error converges within the specified range, for example, calculating the training error by using a loss function, and minimizing the training error by using a gradient descent algorithm. Of course, the training stop condition may also include other conditions, and embodiments of the present invention are not limited herein.

Step 3102, the first ue acquires the trained detection model and empties the training sample library.

Because the memory space of the first user equipment is limited, the first user equipment can empty the training sample library after acquiring the trained detection model. When a new training sample is added to the training sample library, the number of samples in the training sample library is counted again to re-perform step 3101 described above.

Step 311, the first user equipment selects a test image from the candidate images.

The test image may be used to determine whether the test model trained by the training sample is superior to the test model. The first ue may select a training sample from the candidate images, and may select a test image from the candidate images, where the image serving as the training sample and the image serving as the test image may be the same image, but if the image serving as the training sample is reused as the test image, the test image may not accurately test the reliability of the detection model because the detection model has already been adapted to the image.

Therefore, in order to ensure the accuracy of the judgment result, the test image is different from the training sample. The first user terminal device may use the images other than the training sample in the candidate images as the test image.

Step 312, the first ue determines whether the trained detection model is better than the detection model based on the test image.

Optionally, as shown in fig. 5, the step of the first ue determining whether the trained detection model is better than the detection model (i.e. the detection model that does not perform the training process) may include:

In step 3121, the first ue inputs the test image into the trained detection model, to obtain a second analysis result.

Similar to the first analysis result, the second analysis result is an analysis result obtained by analyzing each test by using the trained detection model in the optimization process of the model of the present round. For example, the second analysis result may include a type of the target object of the image and a region of the target object.

In step 3122, the first ue detects whether the accuracy of the trained detection model meets the specified accuracy condition based on the second analysis result of the test image and the first analysis result stored in advance.

Referring to step 304, the first analysis result stored in advance is an analysis result checked by the user, and the second analysis result based on the test image is an analysis result output by the trained detection model, and by comparing the two sets of analysis results, it can be detected whether the accuracy of the trained detection model meets the specified accuracy condition.

Optionally, step 3122 may include:

and step X1, the first user terminal equipment counts the first number of the test images with the same first analysis result as the corresponding second analysis result.

Optionally, determining whether the first analysis result is the same as the corresponding second analysis result may include: judging whether the similarity between the first analysis result and the corresponding second analysis result is larger than a preset threshold value, and determining that the first analysis result is identical to the corresponding second analysis result when the similarity is larger than the preset threshold value.

The corresponding first analysis result and second analysis result refer to analysis results of the same image, for example, for the same Zhang Bei selection image, the first analysis result of the candidate image is stored in step 304, and in step 3121, the candidate image is input into the trained detection model to obtain the second analysis result, where the first analysis result corresponds to the second analysis result.

And X2, when the ratio of the first quantity to the total number of the test images is greater than a specified proportion threshold, the first user terminal equipment determines that the accuracy of the trained detection model meets a specified accuracy condition.

The specified proportion threshold value can be preset by a user and can be adjusted in a later period to meet the personalized requirements of the user.

And X3, when the ratio of the first quantity to the total number of the test images is not greater than a specified proportion threshold, the first user terminal equipment determines that the accuracy of the trained detection model does not meet a specified accuracy condition.

And step X4, when the accuracy of the trained detection model accords with the specified accuracy condition, the first user side equipment judges that the trained detection model is superior to the detection model.

It should be noted that, when the scenes described in the steps X1 to X4 are a plurality of test images, and when there are 1 test image, the process of detecting whether the accuracy of the trained detection model meets the specified accuracy condition may include: when the first analysis result and the second analysis result of the test image are the same, the first user terminal equipment determines that the accuracy of the trained detection model accords with a specified accuracy condition; when the first analysis result and the second analysis result of the test image are different, the first user side device determines that the accuracy of the trained detection model does not meet the specified accuracy condition.

Step 313, when the trained detection model is better than the detection model, the first ue updates the detection model of the first ue by using the trained detection model.

It should be noted that, in the process of performing the model optimization method, various intermediate files may be generated, for example, an xml format annotation file (generated when the user updates the first analysis result) may be deleted after the first user side device updates the detection model of the first user side device with the trained detection model, so as to save the memory occupation amount.

In step 314, when the backed-up detection model meets the deletion condition, the first ue deletes the backed-up detection model.

In order to ensure that the memory occupation amount of the image analysis system is as small as possible, in the embodiment of the present invention, when the backup detection model meets the deletion condition, the backup detection side model may be deleted, and optionally, the step of determining whether the backup detection side model meets the deletion condition occurs in a new round of model optimization process, as shown in fig. 6, the step may include the following steps:

in step 3141, the first user terminal device acquires a plurality of new images acquired by the first image acquisition device.

In step 3142, the first user terminal device inputs the multiple new images into the trained detection model respectively.

The related process of step 3142 may refer to step 301, and the description of the embodiment of the present invention is omitted here.

Step 3143, the first ue receives a third analysis result of each updated image output by the detection model.

The related process of step 3143 may refer to step 302 described above, and the description of the embodiment of the present invention is omitted here.

And the third analysis result is similar to the first analysis result and the second analysis result, and in the process of optimizing the model of the round, the analysis result obtained by each test is analyzed by using the detection model in the first user terminal. For example, the third analysis result may include a type of the target object of the image and a region of the target object.

In step 3144, the first ue receives an audit instruction for the updated image, which is input by the user at the first ue, where the audit instruction is used to indicate whether the third analysis result is accurate.

The related process of step 3144 may refer to step 303, and the description of the embodiment of the present invention is omitted here.

Step 3145, based on the audit instructions corresponding to the multiple new images, the first user terminal device counts the proportion of accurate third analysis results of the multiple new images in all third analysis results.

For example, the number of audit instructions indicating that the third analysis result is accurate is 80, and the ratio is 8/10 if the third analysis result is 100. Of course, the proportion of inaccurate third analysis results of the plurality of new images in all the third analysis results may also be counted, and the embodiment of the present invention is not limited herein.

In step 3146, when the proportion is greater than the specified proportion threshold, the first ue determines that the backup detection model meets the deletion condition, and deletes the backup detection model.

For example, if the specified proportion threshold is 1/2, when the proportion of the accurate third analysis results of the multiple new images in all the third analysis results is 8/10, it may be determined that the backup detection model meets the deletion condition, and the backup detection model is deleted. Of course, in other possible embodiments, the backup detection model may not be deleted, but rather saved in a file.

If the proportion is not greater than the specified proportion threshold, the first user equipment can determine that the backed-up detection model does not meet the deletion condition, and the backed-up detection model can be reinstalled in the first user equipment to replace the detection model of the first user equipment at the moment.

Therefore, in the model optimization method provided by the embodiment of the invention, various methods (refer to the steps 309 and 314) capable of backing the version of the detection model are provided, so that the flexibility of the use of the image analysis system is ensured, and the use demands of multiple aspects of users can be met.

It should be noted that, when a plurality of pieces of user equipment are configured in each image analysis scene, the plurality of pieces of user equipment are in the same image analysis scene, so that the same detection model can be shared, the model optimization method provided in the steps 301 to 314 can be performed in one of the pieces of user equipment, and then the detection model finally reserved in the piece of user equipment is replaced with the detection model in the other piece of user equipment, so that the plurality of pieces of user equipment can analyze the images in the image analysis scene in a targeted manner.

In the related art, a detection model is built by the CMS according to a pre-built image library, the detection model is sent to each piece of user equipment, when a certain piece of user equipment puts forward a model optimization requirement, the CMS can acquire a plurality of training samples from the image library, then train the detection model based on the plurality of training samples, and update the detection model in the user equipment by taking the trained detection model as an optimized model.

Because each piece of user equipment uses the detection model established by the CMS to carry out image analysis, when the CMS establishes the detection model, a large number of images are required to be used as training samples in order to cater to the use scene of each piece of user equipment, so that the universality of the trained detection model is improved.

Before training the detection model by using the training sample, the training sample needs to be labeled according to the analysis result of the detection model, for example, the type of the target event in the labeled image and the region of the target event. When a large number of images are used as training samples, a large amount of manpower is consumed in the process of labeling the images by the staff of the CMS.

In the model optimization method provided by the embodiment of the invention, the user of the first user terminal equipment carries out the auditing of the first analysis result of the image, and as the auditing process of the first analysis result is a step which is usually carried out by the user in order to ensure the validity of the analysis result output by the detection model, the auditing process is combined in the model optimization process, which is equivalent to issuing the image labeling work to each user to realize the marginalization of the model optimization. Especially in big data scene, can effectively simplify the mark flow of CMS side, show the human cost of reduction.

In addition, in the related art, for some specific image analysis systems, for example, for a city management image analysis system, since the images used for training are often only from some specific image analysis scenes, for example, the images used for training are mainly from Chongqing image analysis scenes and Hangzhou image analysis scenes, when a large number of images in Chongqing image analysis scenes and Hangzhou image analysis scenes are used as training samples, and the detection model trained based on the training samples is applied to each city image analysis scene in the whole country, the situation that the images are overfitted for a certain image analysis scene occurs, which results in lower reliability of the detection model.

According to the model optimization method provided by the embodiment of the invention, the training sample is selected from the alternative images to train the detection model of the first user terminal equipment so as to obtain the trained detection model, and because the alternative images are images acquired from a plurality of images acquired by the first image acquisition equipment, the first image acquisition equipment performs image acquisition aiming at a certain image analysis scene where the first user terminal equipment is located, the detection model aiming at the first user terminal equipment can be trained, the trained detection model can be matched with the certain image analysis scene, the problem that in the related art, the finally trained detection model cannot be matched with the image analysis scene corresponding to the user terminal equipment due to the fact that the training sample is randomly selected from the image library is solved, and the reliability of the optimized detection model is improved.

In addition, in the related art, if the detection model established by the CMS is installed in a new ue, the new ue refers to that the image acquired by the image acquisition component managed by the ue is not added to the image library of the CMS. Because the new client device corresponds to the new image analysis scene, if the detection model established by the CMS is used to analyze the image under the new image analysis scene, the accuracy of the analysis result output by the detection model is low, and at this time, the CMS needs to increase the image under the image analysis scene in the image library, and select a training sample based on the image to retrain the detection model. If the new user terminal devices are more, the frequency of the CMS optimization detection model is higher; if the image in the image analysis scene cannot be sent to the CMS due to secret, the accuracy of the analysis result output by the detection model for the image analysis scene is low.

The model optimization method provided by the embodiment of the invention avoids the problem of higher frequency of optimizing and detecting the model by the CMS due to the model optimization performed on the user side equipment, and also avoids the problem of lower accuracy of analysis results caused by the fact that the CMS image library cannot add images in the image analysis scene.

In a second alternative implementation manner, the model optimization method is executed by a central management server, and assuming that the central management server performs model optimization of a first ue based on a requirement of the first ue, where the first ue is any ue in the image analysis system 10 shown in fig. 1, please refer to fig. 7, which shows a flowchart of still another model optimization method provided by an embodiment of the present invention, the method includes:

step 701, the central management server acquires an alternative image from a plurality of images acquired by the first image acquisition device.

The first image acquisition equipment is image acquisition equipment managed by first user equipment.

The first image acquisition device can acquire an image analysis scene of the first user terminal device, so that a plurality of images acquired by the first image acquisition device are images in the image analysis scene, and candidate images acquired from the plurality of images are images in the image analysis scene. The central management server can establish a plurality of image libraries, each image library corresponds to an image analysis scene where the corresponding user terminal equipment is located, and is used for storing a plurality of images under the image analysis scene where the corresponding user terminal equipment is located. The center management server acquires an alternative image from the image library of the first user terminal device to form a training sample library.

Alternatively, there may be various ways for the central management server to obtain the alternative image, and the following two cases are taken as examples in the embodiment of the present invention:

first, as described in the above steps 301 to 307, the user of the first client device checks the first analysis result of the image through the first client device, uses the checked image as an alternative image, and sends the alternative image to the central management server through the first client device.

And secondly, manually labeling the images in the image library provided by the first user terminal equipment, so that the labeled images are used as candidate images.

In an alternative embodiment, if the image analysis system includes a supervision platform, the staff on the supervision platform can audit the analysis result of the image output by the detection model, and then the staff can send the audited image to the central management server as an alternative image.

It should be noted that, if the image analysis scene where the first user terminal device is located is a secret-related scene, and the image collected by the first image collecting device is a security image or a secret-related image such as an image of a secret-related place, the process of obtaining the alternative image by the central management server needs authorization of the first user terminal device so as to avoid secret leakage. In the first case, the candidate image is sent by the first user terminal device, and in the second case, the image library is provided by the first user terminal device. This ensures that the alternative image is an image that the first user terminal device allows the central management server to acquire.

Step 702, the central management server selects a training sample from the candidate images, and adds the training sample to a training sample library corresponding to the first user terminal device.

Because the alternative image is built based on the image acquired by the first image acquisition equipment, the trained detection model is suitable for the model analysis scene where the first image acquisition equipment is located, so that the trained detection model has pertinence, the image characteristics of the scene can be extracted more accurately, the characteristic parameters related to the analysis result can be extracted more conveniently and accurately, the trained detection model can be matched with the scene corresponding to the first user equipment accurately, and the reliability of the optimized model is improved.

In step 703, the central management server trains the detection model of the first user terminal device based on the samples in the training sample library, so as to obtain a trained detection model.

In step 703, the central management server trains the detection model of the first ue based on the samples in the training sample library, and the process of obtaining the trained detection model may refer to the related process in step 310, which is not described in detail in the embodiment of the present invention.

And step 704, when the trained detection model is better than the detection model, the central management server updates the detection model of the first user terminal equipment by adopting the trained detection model.

Before the central management server updates the detection model of the first user terminal device by adopting the trained detection model, the central management server can also select a test image from the alternative images to judge whether the detection model trained by the training sample is better than the detection model, and when the trained detection model is better than the detection model, the detection model of the first user terminal device is updated by using the trained detection model.

Optionally, in step 704, the process of selecting the test image from the candidate images by the central management server to determine whether the test model trained by the training sample is better than the test model may refer to the related processes in steps 311 to 313, which are not described herein.

Optionally, in order to ensure that the detection model can fall back to the detection model obtained by the previous optimization, the central tube server may backup the detection model (including the initial detection model and the model obtained by updating the initial detection model at least once) sent to the first user terminal device. In order to ensure that the memory occupation amount of the image analysis system is as small as possible, the backup detection side model can be deleted when the backup detection model accords with the deletion condition. The process of determining whether the detection model meets the deletion condition may refer to step 314 described above. Of course, in other optional embodiments, the first ue may also backup the detection model sent by the central management server, and then delete the backup detection side model when the backup detection model meets the deletion condition, where the embodiment of the present invention is not limited herein.

It should be noted that, in the second alternative implementation manner, when the application is performed, since the multiple images acquired by the first image acquisition device may be related, the first user terminal device generally cannot provide the images to the central management server, and when the first alternative implementation manner is adopted to perform model optimization, the first user terminal device may locally perform model optimization, so that an alternative image may be directly acquired from the multiple images, and a training sample is selected from the alternative image to train the detection model of the first user terminal device, thereby further improving pertinence of the trained detection model to an image analysis scene where the first user terminal device is located.

Further, in the two alternative implementations, the users that label the training samples may be the same or different. In the two implementations, the user for marking is the user of the first user terminal device; or in the first alternative implementation manner, the user performing the labeling is a user of the first user terminal device, and in the second alternative implementation manner, the user performing the labeling is a staff of the CMS. When the user of the first user terminal equipment directly carries out auditing on the first analysis result, the user contacts the image in the image analysis scene corresponding to the first user terminal equipment for a long time, so that the image analysis scene is more familiar, and the auditing result is more accurate, namely, the accuracy of the result of labeling the training sample is higher, and the accuracy of the trained detection model is further ensured. In addition, as the user can audit the first analysis result, the participation of the user is improved, the user experience is enhanced, and as the detection model is continuously optimized, the process of auditing the first analysis result by the user is easier and easier, and the user experience is further improved.

It should be noted that, the foregoing model optimization method may also be performed by the first ue and the central management server in cooperation, for example, the foregoing

steps

301, 302, 308 to 314 may be performed by the central management server, and the steps 303 to 307 are performed by the first ue. That is, the central management server is used for training and updating the detection model of the first user terminal device; the first user side device is used for determining the alternative image, and related information in the first user side device and the central management server interact through a wired or wireless network between the first user side device and the central management server. The first user terminal device and the central management server can also cooperate to execute the model optimization method in other modes, so long as training of the candidate images acquired in the images acquired by the first image acquisition device is ensured to acquire a detection model for the first image acquisition device. The embodiment of the present invention is not limited thereto.

It should be noted that, for different image analysis systems, the image capturing device and the user device that are disposed therein may be the same device. For example, the image acquisition device arranged in the image analysis scene of the western security tombstone forest area can be used as a metropolitan management image analysis system, the image acquisition device in the western security city image analysis scene is used for acquiring illegal events of the western security city. Accordingly, the same user terminal device can be in different image analysis scenes, so that different detection models corresponding to the different image analysis scenes one by one can be installed in the same user terminal device, and a user can select and use the different detection models in the first user terminal device according to the different image analysis scenes. The foregoing embodiment is described taking an optimization method of one detection model of the first ue device as an example, and the optimization method of the other detection models of the first ue device may refer to the foregoing optimization method.

An embodiment of the present invention provides a model optimization apparatus 80, which is applied to an image analysis system, where the image analysis system includes at least one client device, and each client device manages at least one image acquisition device, as shown in fig. 8, and the apparatus 80 includes:

an acquiring module 801, configured to acquire an alternative image from a plurality of images acquired by a first image acquisition device, where the first image acquisition device is an image acquisition device managed by a first user terminal device;

an adding module 802, configured to select a training sample from the candidate image, and add the training sample to a training sample library corresponding to the first user terminal device;

the training module 803 is configured to train the detection model of the first user side device based on the samples in the training sample library, to obtain a trained detection model, where the detection model of the first user side device is used to detect the image acquired by the first image acquisition device;

and the updating module 804 is configured to update the detection model of the first ue device by using the trained detection model when the trained detection model is better than the detection model.

In summary, in the model optimizing apparatus provided in the embodiment of the present invention, the training sample is selected from the candidate images to train the detection model of the first user terminal device, so as to obtain a trained detection model, and since the candidate images are images obtained from a plurality of images acquired by the first image acquisition device, the first image acquisition device performs image acquisition for a certain image analysis scene where the first user terminal device is located, the trained detection model can be adapted to the certain image analysis scene, thereby solving the problem that the detection model obtained by training in the related art cannot be adapted to the image analysis scene corresponding to the user terminal device, and improving the reliability of the optimized detection model.

Optionally, the obtaining module 801 is configured to:

inputting the plurality of images into the detection model respectively;

receiving a first analysis result of each image output by the detection model;

Optionally, the acquiring module 801 is further configured to:

Optionally, as shown in fig. 9, the apparatus 80 further includes:

a selection module 805 configured to select a test image from the candidate images;

a determining module 806, configured to determine, based on the test image, whether the trained detection model is better than the detection model;

Optionally, as shown in fig. 10, the determining module 806 includes:

an input submodule 8061, configured to input the test image into the trained detection model, and obtain a second analysis result;

A detection submodule 8062, configured to detect whether the accuracy of the trained detection model meets a specified accuracy condition based on the second analysis result of the test image and the first analysis result stored in advance;

and the judging submodule 8063 is used for judging that the trained detection model is better than the detection model when the accuracy of the trained detection model accords with the appointed accuracy condition.

Optionally, there are multiple test images, and the detection submodule 8062 is configured to:

Optionally, an adding module 802 is configured to: selecting images with specified proportions from the alternative images as training samples;

a selection module 805 for: and taking images except the training sample in the alternative images as test images.

Optionally, the training module 803 is configured to:

when the number of samples in the training sample library is greater than a specified sample number threshold, executing at least one training process on the detection model until the detection model meets a training stop condition;

and acquiring a trained detection model, and emptying the training sample library.

Optionally, as shown in fig. 11, the apparatus 80 further includes:

a backup module 807, configured to backup the detection model of the first user equipment;

optionally, as shown in fig. 12, the apparatus 80 further includes:

and the deleting module 808 is configured to delete the backup detection model when the backup detection model meets a deletion condition.

Optionally, the deleting module 808 is configured to:

acquiring a plurality of new images acquired by a first image acquisition device;

respectively inputting the plurality of new images into the trained detection model;

receiving a third analysis result of each new image output by the trained detection model;

receiving an audit instruction of a third analysis result of the new image, which is input by a user at first user terminal equipment, wherein the audit instruction is used for indicating whether the third analysis result is accurate or not;

Based on auditing instructions corresponding to the plurality of new images, counting the proportion of accurate third analysis results of the plurality of new images in all the third analysis results;

and when the proportion is larger than a specified proportion threshold, determining that the backup detection model accords with a deletion condition, and deleting the backup detection model.

Optionally, the image analysis system further includes a supervisory platform, as shown in fig. 13, and the apparatus 80 further includes:

and the sending module 809 is configured to send the candidate image and the saved first analysis result to the supervision platform.

Optionally, the image analysis system is a metropolitan image analysis system, and the analysis result output by the detection model includes: the type of the offending event in the image and/or the location of the offending event is generated.

Optionally, the image analysis system further includes: the detection model is an initial detection model sent by the central management server, or a model obtained by updating the initial detection model at least once by the first user terminal equipment.

An embodiment of the present invention provides a storage medium, which may be a non-volatile computer readable storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements any of the model optimization methods provided in the foregoing embodiments.

The embodiment of the invention also provides a computer program product containing instructions, which when run on a computer, cause the computer to execute the model optimization method provided by the embodiment of the method.

An embodiment of the present invention provides an image analysis system, including: the image analysis system comprises a model optimization device as described in any of fig. 8, 9 and 11 to 13. Other structures and architectures of the image analysis system may refer to the image analysis system shown in fig. 1.

Fig. 14 is a block diagram illustrating a configuration of a client device 900 according to an exemplary embodiment of the present invention. The client device 900 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, motion picture expert compression standard audio plane 3), an MP4 (Moving Picture Experts Group Audio Layer IV, motion picture expert compression standard audio plane 4) player, a notebook computer, or a desktop computer. The user-side device 900 may also be referred to by other names as user device, portable terminal, laptop terminal, desktop terminal, etc.

Generally, the client device 900 includes: a processor 901 and a memory 902.

Processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 901 may also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 901 may be re-integrated with a GPU (Graphics Processing Unit, image processor) for processing computing operations related to machine learning, such as training and analysis of detection models, etc. In some embodiments, the processor 901 may include an AI (Artificial Intelligence ) processor that may have the same functionality as the GPU, i.e., for processing computing operations related to machine learning, such as training and analysis of detection models, and the like.

The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 902 is used to store at least one instruction for execution by processor 901 to implement the model optimization methods provided by the method embodiments in the present application.

In some embodiments, the client device 900 may further optionally include: a peripheral interface 903, and at least one peripheral. The processor 901, memory 902, and peripheral interface 903 may be connected by a bus or signal line. The individual peripheral devices may be connected to the peripheral device interface 903 via buses, signal lines, or circuit boards. Specifically, the peripheral device includes: at least one of radio frequency circuitry 904, a touch display 905, a camera 906, audio circuitry 907, positioning components 908, and a power source 909.

The peripheral interface 903 may be used to connect at least one peripheral device associated with an I/O (Input/Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, memory 902, and peripheral interface 903 are integrated on the same chip or circuit board; in some other embodiments, either or both of the processor 901, the memory 902, and the peripheral interface 903 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 904 is configured to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 904 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 904 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuit 904 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity ) networks. In some embodiments, the radio frequency circuit 904 may also include NFC (Near Field Communication ) related circuits, which are not limited in this application.

The display 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display 905 is a touch display, the display 905 also has the ability to capture touch signals at or above the surface of the display 905. The touch signal may be input as a control signal to the processor 901 for processing. At this time, the display 905 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 905 may be one, and a front panel of the client device 900 is provided; in other embodiments, the display 905 may be at least two, and disposed on different surfaces of the ue 900 or in a folded design; in still other embodiments, the display 905 may be a flexible display disposed on a curved surface or a folded surface of the user side device 900. Even more, the display 905 may be arranged in an irregular pattern other than rectangular, i.e., a shaped screen. The display 905 may be an LCD (Liquid Crystal Display ), OLED (Organic Light-Emitting Diode) display.

The camera assembly 906 is used to capture images or video. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal and the rear camera is disposed on the rear surface of the terminal. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some embodiments, camera assembly 906 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.

The audio circuit 907 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 901 for processing, or inputting the electric signals to the radio frequency circuit 904 for voice communication. For purposes of stereo acquisition or noise reduction, a plurality of microphones may be respectively disposed at different locations of the ue 900. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some embodiments, the audio circuit 907 may also include a headphone jack.

The location component 908 is used to locate the current geographic location of the client device 900 to enable navigation or LBS (Location Based Service, location-based services). The positioning component 908 may be a positioning component based on the United states GPS (Global Positioning System ), the Beidou system of China, the Granati system of Russia, or the Galileo system of the European Union.

The power supply 909 is used to supply power to the various components in the client device 900. The power supply 909 may be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the client device 900 also includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: acceleration sensor 911, gyroscope sensor 912, pressure sensor 913, fingerprint sensor 914, optical sensor 915, and proximity sensor 916.

The acceleration sensor 911 can detect the magnitudes of accelerations on three coordinate axes of the coordinate system established by the client device 900. For example, the acceleration sensor 911 may be used to detect components of gravitational acceleration in three coordinate axes. The processor 901 may control the touch display 905 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signal acquired by the acceleration sensor 911. The acceleration sensor 911 may also be used for the acquisition of motion data of a game or a user.

The gyro sensor 912 may detect the body direction and the rotation angle of the user terminal device 900, and the gyro sensor 912 may cooperate with the acceleration sensor 911 to collect the 3D motion of the user on the user terminal device 900. The processor 901 may implement the following functions according to the data collected by the gyro sensor 912: motion sensing (e.g., changing UI according to a tilting operation by a user), image stabilization at shooting, game control, and inertial navigation.

The pressure sensor 913 may be disposed on a side frame of the user side device 900 and/or on an underlying layer of the touch display 905. When the pressure sensor 913 is disposed on the side frame of the user terminal device 900, a holding signal of the user terminal device 900 by the user can be detected, and the processor 901 performs the left-right hand recognition or the shortcut operation according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed at the lower layer of the touch display 905, the processor 901 performs control of the operability control on the UI interface according to the pressure operation of the user on the touch display 905. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

The fingerprint sensor 914 is used for collecting the fingerprint of the user, and the processor 901 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the identity of the user according to the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, the processor 901 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying for and changing settings, etc. The fingerprint sensor 914 may be disposed on the front, back, or side of the client device 900. When a physical key or vendor Logo is provided on the client device 900, the fingerprint sensor 914 may be integrated with the physical key or vendor Logo.

The optical sensor 915 is used to collect the intensity of ambient light. In one embodiment, the processor 901 may control the display brightness of the touch display 905 based on the intensity of ambient light collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the touch display 905 is turned up; when the ambient light intensity is low, the display brightness of the touch display panel 905 is turned down. In another embodiment, the processor 901 may also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 915.

A proximity sensor 916, also referred to as a distance sensor, is typically provided on the front panel of the client device 900. The proximity sensor 916 is used to capture the distance between the user and the front of the client device 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the user side device 900 gradually decreases, the processor 901 controls the touch display 905 to switch from the bright screen state to the off screen state; when the proximity sensor 916 detects that the distance between the user and the front surface of the user side device 900 gradually increases, the processor 901 controls the touch display 905 to switch from the off-screen state to the on-screen state.

Those skilled in the art will appreciate that the architecture shown in fig. 14 is not limiting and that more or fewer components than shown may be included or certain components may be combined or a different arrangement of components may be employed.

It should be noted that the client device may be a server.

The term "and/or" in the present invention is merely an association relation describing the association object, and indicates that three kinds of relations may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

In the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The term "plurality" refers to two or more, unless explicitly defined otherwise.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims

1. A method of model optimization, the method comprising:

2. The model optimization method according to claim 1, wherein the acquiring the candidate image from the plurality of images acquired by the first image acquisition device includes:

Inputting the plurality of images into the detection model respectively;

receiving a first analysis result of each image output by the detection model;

3. The model optimization method according to claim 2, wherein the acquiring the candidate image from the plurality of images acquired by the first image acquisition device further comprises:

4. A method of model optimization according to claim 2 or 3, characterized in that before said updating the detection model of the first user side device with the trained detection model, the method further comprises:

selecting a test image from the candidate images;

5. The model optimization method of claim 4, wherein the determining whether the trained test model is better than the test model based on the test image comprises:

6. The model optimization method according to claim 5, wherein the test image has a plurality of test images, and the detecting whether the accuracy of the trained detection model meets a specified accuracy condition based on the second analysis result of the test image and the first analysis result stored in advance comprises:

7. The model optimization method of claim 1, wherein prior to said training the detection model of the first client device based on the samples in the training sample library, the method further comprises:

Backing up the detection model of the first user equipment;

8. The model optimization method according to any one of claims 1 to 7, wherein the method is performed by the first client device, and the training sample library is stored locally to the first client device.

9. A model optimization apparatus, the apparatus comprising:

10. The model optimization device of claim 9, wherein the acquisition module is configured to:

inputting the plurality of images into the detection model respectively;

receiving a first analysis result of each image output by the detection model;

11. The model optimization device of claim 10, wherein the acquisition module is further configured to:

12. Model optimization device according to claim 10 or 11, characterized in that the device further comprises:

a selection module for selecting a test image from the candidate images;

13. The model optimization device of claim 12, wherein the determining module comprises:

14. The model optimization apparatus of claim 13, wherein the test image has a plurality of test images, and the detection sub-module is configured to:

15. The model optimization device of claim 9, wherein the device further comprises:

the backup module is used for backing up the detection model of the first user terminal equipment;

16. The model optimization apparatus according to any one of claims 9 to 15, wherein the model optimization apparatus is the first client device, and the training sample library is stored locally to the first client device.

17. An image analysis system, characterized in that the image analysis system comprises the model optimization device according to any one of claims 9 to 16;

18. A computer device, comprising a processor and a memory,

wherein the memory is used for storing a computer program;

the processor is configured to execute a program stored in the memory, and implement the model optimization method according to any one of claims 1 to 8.

19. A storage medium having stored therein a computer program which, when executed by a processor, implements the model optimization method of any one of claims 1 to 8.