CN111563174A - Image processing method, image processing apparatus, electronic device, and storage medium - Google Patents

Abstract

The present application discloses an image processing method, an image processing apparatus, an electronic device and a storage medium. The image processing method is implemented by an image processing framework, and the image processing framework includes a view access layer, a view processing layer, a storage layer, and a task delivery layer; the view access layer is used to obtain at least one first to-be-received layer. processing an image; the task delivery layer is used to acquire at least one image processing task to be executed; the view processing layer is used to process the at least one first image to be processed to obtain at least one first processing result ; the storage layer is used to store the at least one first processing result; the task delivery layer is used to store the at least one image processing task stored in the storage layer according to the at least one image processing task to be executed. A first processing result is processed to obtain a second processing result.

Description

Image processing method, image processing device, electronic device, and storage medium

technical field

The present application relates to the field of computer technology, and in particular, to an image processing method, an image processing apparatus, an electronic device, and a storage medium.

Background technique

With the rapid development of computer vision technology, various computer vision models with different functions have emerged. For example, face recognition models can be used for face recognition, object detection models can be used to detect objects, and motion monitoring models can be used to monitor whether A specific action occurs.

In order to use the computer vision model to perform image processing tasks, software for implementing the computer vision model can be written, and the image processing tasks can be performed by running the software through a device. In order to further improve the development efficiency, the above software development can be implemented based on a software framework, but the development efficiency based on the traditional software framework is low, resulting in low image processing efficiency.

SUMMARY OF THE INVENTION

The present application provides an image processing method, an image processing apparatus, an electronic device and a storage medium.

A first aspect provides an image processing method, the image processing method is implemented by an image processing framework, and the image processing framework includes a view access layer, a view processing layer, a storage layer, and a task delivery layer;

the view access layer, used to acquire at least one first image to be processed;

The task delivery layer is used to acquire at least one image processing task to be executed;

the view processing layer, configured to process the at least one first image to be processed to obtain at least one first processing result;

the storage layer, configured to store the at least one first processing result;

The task delivery layer is configured to process the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

Since the business logic is determined by the image processing task to be executed, and in this aspect, the image processing task to be executed is obtained through the task delivery layer, the business logic and the computer vision model in the view processing layer are independent of each other. In this way, after the image processing software is developed based on the image processing framework, different to-be-executed image processing tasks are input to the task delivery layer of the image processing software, and/or, by updating (the updating here includes at least one of the following operations) : replace, delete, add) the computer vision model in the software, so that the updated software can be used to realize different image processing functions, thereby improving the versatility of the software, the development efficiency of developers, and the efficiency of image processing.

With reference to any embodiment of the present application, the at least one first processing result includes image data and non-image data; the storage layer includes an image data storage module and a non-image data storage module;

The storage layer is also used to:

storing the image data in the image data storage module, and storing the non-image data in the non-image data storage module;

An index is established between the image data and the non-image data by using the non-image data as an index entry for the image data.

In this embodiment, the image data and the non-image data in the at least one first processing result are stored separately, which facilitates the management of the at least one first processing result. By establishing an index between the non-image data and the image data, the image data can be searched according to the non-image data, thereby improving the efficiency of retrieving the image data from the at least one first processing result.

With reference to any embodiment of the present application, it is characterized in that the view processing layer includes a picture processing module and a video processing module; the view processing layer is further used for:

In the case where the number of the first images to be processed is greater than or equal to 2, and an instruction to perform image processing and an instruction to perform video processing are received, according to the instruction to perform image processing, from at least two of the first images to be processed Select at least one second image to be processed from the

using the image processing module to process the at least one second image to be processed to obtain at least one third processing result;

Selecting at least two third images to be processed from the at least two first images to be processed according to the executing video processing instruction;

Using the video processing module to process the at least two third images to be processed to obtain at least one fourth processing result;

The at least one first processing result is obtained according to the at least one third processing result and the at least one fourth processing result.

In this embodiment, the picture processing module and the video processing module are two independent modules, and the software developed based on the image processing framework can be used to perform both picture processing tasks, video processing tasks, and Perform image processing tasks and video processing tasks at the same time.

With reference to any embodiment of the present application, the use of the image processing module to process the at least one second image to be processed to obtain at least one third processing result includes:

Using the image processing module to perform object detection processing on the fourth image to be processed, to obtain the position of the first object in the fourth image to be processed; the fourth image to be processed belongs to the at least one second image to be processed an image; the first object belongs to the fourth image to be processed;

determining a pixel area including the first object according to the position;

Intercept the pixel area from the fourth to-be-processed image to obtain an object image;

Use the image processing module to perform attribute extraction processing on the object image to obtain the attribute of the first object.

In this embodiment, since the amount of data contained in the object image is greater than the amount of data contained in the fourth image to be processed, the amount of data processing brought by the attribute extraction processing on the object image is smaller than that brought by the attribute extraction processing on the fourth image to be processed. Therefore, the attribute of the first object can be obtained by performing attribute extraction processing on the object image, which can reduce the data processing amount and improve the processing speed.

With reference to any embodiment of the present application, the image data includes the at least one first image to be processed and the object image, and the non-image data includes attributes of the first object.

Combined with any embodiment of the present application, the view access layer is further used for:

Acquire at least one fifth image to be processed;

If the communication protocol of the at least one fifth image to be processed is different from the communication protocol used by the image processing framework, converting the communication protocol of the at least one fifth image to be processed into the image processing The communication protocol used by the framework to obtain the at least one first image to be processed.

In this embodiment, in the case where the communication protocol of the image acquired from the imaging device is different from the communication protocol used by the image processing framework, by converting the communication protocol of the acquired image to the communication protocol used by the image processing framework The protocol enables software developed based on the image processing framework to process images obtained from any imaging device.

In combination with any embodiment of the present application, the image processing framework further includes an information index layer;

The information index layer is used to obtain retrieval information;

The at least one first processing result is retrieved using the retrieval information to obtain a fifth processing result.

In a second aspect, another image processing method is provided, the method comprising:

Acquiring at least one first image to be processed and at least one image processing task to be performed;

Perform picture processing and/or video processing on the at least one first image to be processed to obtain at least one first processing result;

storing the at least one first processing result;

According to the at least one image processing task to be executed, the at least one first processing result stored in the storage layer is processed to obtain a second processing result.

In combination with any embodiment of the present application, the image processing method is implemented by an image processing framework, and the construction process of the image processing framework includes:

Obtain the at least one image processing task to be executed and a frame to be processed; the frame to be processed includes a view processing layer, a storage layer, and a task delivery layer;

In the case where the at least one image processing task to be executed is obtained through the task delivery layer, obtain a picture processing module and/or a video processing module matching the at least one image processing task to be executed; the picture processing The module is used to realize image processing; the video processing module is used to realize video processing;

The picture processing module and/or the video processing module are added to the view processing layer of the to-be-processed image processing frame to obtain the image processing frame.

In a third aspect, an image processing apparatus is provided, and the image processing apparatus includes:

a view access unit, configured to acquire at least one first image to be processed;

a task issuing unit, used for acquiring at least one image processing task to be executed;

a view processing unit, configured to process the at least one first image to be processed to obtain at least one first processing result;

a storage unit for storing the at least one first processing result;

A task issuing unit, configured to process the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

With reference to any embodiment of the present application, the at least one first processing result includes image data and non-image data; the storage unit includes an image data storage subunit and a non-image data storage subunit;

The storage unit is used for:

storing the image data in the image data storage subunit, and storing the non-image data in the non-image data storage subunit;

An index is established between the image data and the non-image data by using the non-image data as an index entry for the image data.

With reference to any embodiment of the present application, the view processing unit includes a picture processing subunit and a video processing subunit; the view processing unit is used for:

In the case where the number of the first images to be processed is greater than or equal to 2, and an instruction to perform image processing and an instruction to perform video processing are received, according to the instruction to perform image processing, from at least two of the first images to be processed Select at least one second image to be processed from the

Using the picture processing subunit to process the at least one second image to be processed to obtain at least one third processing result;

Selecting at least two third images to be processed from the at least two first images to be processed according to the executing video processing instruction;

using the video processing subunit to process the at least two third images to be processed to obtain at least one fourth processing result;

The at least one first processing result is obtained according to the at least one third processing result and the at least one fourth processing result.

With reference to any embodiment of the present application, the view processing unit is used for:

Using the image processing subunit to perform object detection processing on the fourth image to be processed to obtain the position of the first object in the fourth image to be processed; the fourth image to be processed belongs to the at least one second image to be processed processing an image; the first object belongs to the fourth image to be processed;

determining a pixel area including the first object according to the position;

Intercept the pixel area from the fourth to-be-processed image to obtain an object image;

Using the image processing subunit to perform attribute extraction processing on the object image to obtain the attribute of the first object.

With reference to any embodiment of the present application, the image data includes the at least one first image to be processed and the object image, and the non-image data includes attributes of the first object.

With reference to any embodiment of the present application, the view access unit is used for:

Acquire at least one fifth image to be processed;

If the communication protocol of the at least one fifth image to be processed is different from the communication protocol used by the image processing framework, converting the communication protocol of the at least one fifth image to be processed into the image processing The communication protocol used by the framework to obtain the at least one first image to be processed.

With reference to any embodiment of the present application, the image processing apparatus further includes: an information indexing unit;

The information indexing unit is configured to acquire retrieval information, and use the retrieval information to retrieve the at least one first processing result to obtain a fifth processing result.

In a fourth aspect, another image processing apparatus is provided, and the image processing apparatus includes:

an acquisition unit, configured to acquire at least one first image to be processed and at least one image processing task to be performed;

a first processing unit, configured to perform picture processing and/or video processing on the at least one first image to be processed to obtain at least one first processing result;

a storage unit for storing the at least one first processing result;

The second processing unit is configured to process the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

With reference to any embodiment of the present application, the method executed by the image processing apparatus is implemented by an image processing framework, and the acquiring unit is further configured to acquire the at least one image processing task to be executed and the frame to be processed; the to-be-processed The framework includes a view processing layer, a storage layer, and a task delivery layer;

The acquiring unit is further configured to acquire a picture processing module and/or an image processing module matching the at least one image processing task to be performed when the at least one image processing task to be performed is acquired through the task delivery layer a video processing module; the picture processing module is used for realizing picture processing; the video processing module is used for realizing video processing;

The image processing device further includes:

The adding unit is configured to add the picture processing module and/or the video processing module to the view processing layer of the to-be-processed image processing frame to obtain the image processing frame.

In a fifth aspect, a processor is provided, and the processor is configured to execute the method in the above-mentioned first aspect and any one of its embodiments, or the processor is configured to execute the above-mentioned second aspect and any one of its embodiments. method in .

In a sixth aspect, an electronic device is provided, comprising: a processor, a sending device, an input device, an output device, and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, in the processing When the computer executes the computer instructions, the electronic device operates as the method in the first aspect and any one of its embodiments, or the electronic device operates as the method in the second aspect and any one of its embodiments. .

In a seventh aspect, a computer-readable storage medium is provided, where a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions that, when the program instructions are executed by a processor, cause all The processor is caused to perform the method as in the first aspect and any embodiment thereof, or the processor is caused to perform the method as in the second aspect and any embodiment thereof.

In an eighth aspect, a computer program product is provided, the computer program product includes a computer program or an instruction, and when the computer program or instruction is run on a computer, the computer is made to run the first aspect and any one thereof The method of an embodiment, or causing the computer to run the method of the second aspect and any one of its embodiments.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the accompanying drawings required in the embodiments or the background technology of the present application will be described below.

The accompanying drawings, which are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, serve to explain the technical solutions of the present application.

FIG. 1 is a schematic structural diagram of an image processing framework provided by an embodiment of the present application;

2 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

3 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

4 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

FIG. 5 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

6 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a data flow provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of another data flow provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of another image processing apparatus provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of a hardware structure of an image processing apparatus according to an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

It should be understood that in this application, "at least one (item)" refers to one or more, "multiple" refers to two or more, and "at least two (item)" refers to two or three And three or more, "and/or" is used to describe the association relationship of related objects, indicating that three kinds of relationships can exist, for example, "A and/or B" can mean: only A exists, only B exists, and A exists at the same time and B three cases, where A, B can be singular or plural. The character "/" generally indicates that the associated objects are an "or" relationship. "At least one item(s) below" or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a) of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" ", where a, b, c can be single or multiple.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

With the rapid development of computational vision technology, various computer vision models with different functions have emerged. For example, face recognition models can be used for face recognition, object detection models can be used to detect objects, and motion monitoring models can be used to monitor whether A specific action occurs.

The image is used as the input of the computer vision model, and the computer vision model can output the processing result obtained by processing the image. For example, a face image is input into a face recognition model, and the face image is processed by the face recognition model, and the output processing results include: a feature vector carrying the semantic information of the face image, the position of the key points of the face, the Face attributes (eg gender, age, race). For another example, an image containing an object is input into the object detection model, the image is processed by the object detection model, and the output processing result includes: a feature vector carrying the semantic information of the image, the position of the object in the image, and the category of the object. (such as whether the object is a table or a chair), object attributes (such as color, shape).

The device (including: mobile phone, computer, processor, tablet computer) processes the processing result output by the computer vision model according to the business logic matching the image processing task, and can complete the image processing task. For example (Example 1), assuming that the device is an access control device of Company A, and the image processing task is face recognition, the task specifically includes: determining the person (hereinafter referred to as the face image to be recognized) in the collected face image (hereinafter referred to as the face image to be recognized). will be referred to as the person to be identified) is an employee of Company A. The face image database of the access control device stores the face images of all employees of company A. The process for the access control device to perform the image processing task is as follows: determine the similarity between the face image to be recognized and the image in the face image database, and obtain a similarity set. If the maximum value in the similarity set is greater than or equal to the similarity threshold, determine that the person to be identified is an employee of company A, and if the maximum value in the similarity set is less than the similarity threshold, determine that the person to be identified is not from company A Staff. According to the above process, it can be determined that the business logic matching the image processing task is: collecting the face image to be recognized through the access control device, and using the face recognition model to determine the similarity between the face image to be recognized and the image in the face image library. , obtain a similarity set, and determine whether the person in the face image to be recognized is an employee of company A according to the magnitude relationship between the maximum value in the similarity set and the similarity threshold.

As can be seen from Example 1, to complete the image processing task, the image processing and business logic of the computer vision model are indispensable. Through software development, a software including computer vision model and business logic can be used to process images to complete image processing tasks.

In order to improve development efficiency, software framework can be used for development. In the process of development using a traditional software framework, by integrating the code for implementing business logic (hereinafter referred to as the business logic layer) and the code for implementing the computer vision model (hereinafter referred to as the model engine layer) in the traditional In the software framework, software for completing image processing tasks is available. Since in the traditional software framework, the business logic layer, the model engine layer, and the traditional software framework are bound together, when performing new image processing tasks, it is necessary to develop new image processing tasks based on the traditional software framework. software. Continue with Example 1 (Example 2), integrate the business logic layer and model engine layer in Example 1 into the traditional software framework, and obtain a software that can be used to perform the face recognition in Example 1 (hereinafter referred to as face recognition software). If company A's business includes machining, a large number of spare parts for machining, such as screws and bolts, are placed in company A's workshop. Sorting of screws and bolts is now required due to the large number of them being placed together. For the sake of saving labor costs, Company A hopes to complete the sorting of screws and bolts through computational vision technology. Specifically, the sorting process is as follows: the sorting machine collects images of the parts to be identified through the camera, and uses the object detection model to treat them. The image of the identified spare parts is processed to obtain the category of the spare parts to be identified in the image of the spare parts to be identified. The sorter places the parts to be identified in the area that matches the category of the parts to be identified. According to the sorting process, the business logic of sorting can be determined as follows: the sorter collects the image of the spare parts to be identified, obtains the image of the spare parts to be identified, and the sorter inputs the image of the spare parts to be identified into the object detection model (the model). It can be used to identify whether the object is a screw or a bolt). The object detection model processes the image of the spare parts to be identified, outputs the category of the spare parts to be identified, and places the spare parts to be identified in the area that matches the category of the spare parts to be identified. Since the business logic in the face recognition software is different from the business logic in the sorting process, it is impossible to obtain software that can be used for sorting spare parts by replacing the face recognition model in the face recognition software with an object detection model. Therefore, it is necessary to re-develop a software that can be used to sort spare parts based on the traditional software framework, that is, to integrate the code used to implement the business logic of sorting and the code used to implement the object monitoring model into the traditional software framework to obtain the code used for sorting. software for spare parts.

Obviously, the software developed based on the traditional software framework has low generality, which in turn leads to low development efficiency and low image processing efficiency. In order to improve development efficiency and image processing efficiency, the embodiments of the present application provide an image processing framework, which can obtain software for performing different image processing tasks by replacing computer vision models, thereby improving development efficiency and image processing efficiency.

FIG. 1 shows an architecture diagram of an image processing framework provided by an embodiment of the present application. As shown in FIG. 1 , the image processing framework includes a view access layer, a view processing layer, and a task delivery layer.

The view access layer is configured to acquire at least one first image to be processed, and transmit the at least one first image to be processed to the view processing layer. The view processing layer is configured to process the at least one first image to be processed to obtain at least one first processing result. The storage layer stores at least one first processing result obtained by the view processing layer. The task delivery layer is used to obtain at least one image processing task to be executed, and according to the at least one image processing task to be executed, process at least one first processing result stored in the storage layer to obtain a second processing result, thereby completing image processing tasks.

The device running the image processing framework provided by this application may be an image processing apparatus. Optionally, the image processing apparatus may be one of the following: a mobile phone, a computer, a server, a processor, and a tablet computer. The following describes the process of using the image processing framework to perform image processing tasks with reference to the accompanying drawings in the embodiments of the present application.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of an image processing task performed by using an image processing framework provided by an embodiment of the present application.

201. The view access layer acquires at least one first image to be processed.

In this embodiment of the present application, the first image to be processed may contain any content. For example, the first image to be processed may include a road. As another example, the first image to be processed may include roads and vehicles. As another example, the first image to be processed may include a person. The present application does not limit the content of the first image to be processed.

In an implementation manner of acquiring at least one first image to be processed, the view access layer receives at least one first image to be processed input by the user through the input component. The above input components include: keyboard, mouse, touch screen, touch pad, audio input and so on.

In another implementation manner of acquiring at least one first image to be processed, the view access layer receives at least one first image to be processed sent by the first terminal. Optionally, the first terminal may be any one of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device.

In yet another implementation manner of acquiring at least one first image to be processed, a communication connection is provided between the image processing device and the surveillance camera, and the view access layer can receive at least one first image to be processed sent by the surveillance camera through the communication connection. Process images. Optionally, the surveillance camera is deployed on the road or indoors.

In another implementation manner of acquiring at least one first image to be processed, a communication connection is provided between the image processing device and the surveillance camera, and the view access layer can receive the video stream sent by the surveillance camera through the communication connection, and convert the video stream to the surveillance camera. At least one image in is used as at least one first image to be processed. Optionally, the surveillance camera is deployed on the road or indoors.

In another implementation manner of acquiring the first image to be processed, the view access layer may directly acquire the image to be processed through its own image acquisition component, such as a camera.

202. The task delivery layer acquires at least one image processing task to be executed.

In the embodiment of the present application, the image processing task to be performed is a goal to be achieved based on a computer vision model. For example, the image processing task to be performed may be based on face recognition to realize the door opening without feeling. As another example, the image processing task to be performed may be the classification of spare parts. For another example, the image processing task to be performed may be to monitor whether a pedestrian falls, whether a safety helmet is worn, whether there is smoke, or whether there is an open flame.

Obviously, the business logic required to perform different image processing tasks to be performed is different. For example, the business logic required to implement a sensorless door opening is different from the business logic that implements the classification of spare parts.

In this embodiment of the present application, when the number of image processing tasks to be executed is greater than or equal to 2, any two image processing tasks to be executed are different, and accordingly, the business logic required to execute any two image processing tasks to be executed are different. For example, the acquired at least one to-be-executed image processing task includes: to-be-executed image processing task A and to-be-executed image processing task B, wherein the business logic required to execute to-be-executed image processing task A and the to-be-executed image processing task B The required business logic is different.

In an implementation manner of acquiring at least one to-be-executed image processing task, the task delivery layer receives at least one to-be-executed image processing task input by a user through an input component. The above input components include: keyboard, mouse, touch screen, touch pad, audio input and so on.

In another implementation manner of acquiring at least one to-be-executed image processing task, the task delivery layer receives at least one to-be-executed image processing task sent by the second terminal. The second terminal includes a mobile phone, a computer, a tablet computer, a server, and a wearable device.

In yet another implementation manner of acquiring at least one image processing task to be executed, the task delivery layer receives a selection instruction input by the user, and according to the selection instruction, selects at least one image processing task library stored in the task delivery layer. An image processing task, as at least one image processing task to be executed.

In yet another implementation manner of acquiring at least one image processing task to be executed, the task delivery layer, according to preset triggering rules, such as timing triggering or triggering when a target object is detected, etc., from the image stored in the task delivery layer At least one image processing task corresponding to the selection rule in the processing task library is used as the at least one image processing task to be executed.

203. The view processing layer performs image processing on the at least one first image to be processed to obtain at least one first processing result.

In this embodiment of the present application, the view processing layer includes a computer vision model, and the computer vision model includes at least one of the following: a face recognition model, an object detection model, an action recognition model, a pedestrian re-identification model, and an image quality detection model.

In this embodiment of the present application, the view processing layer may obtain a first processing result by performing image processing on a first image to be processed. The view processing layer may also obtain a first processing result by performing image processing on at least two first images to be processed. For example, a face recognition model is used to process a first image to be processed, and a first processing result is obtained, where the first processing result includes: a feature vector carrying semantic information of the first image to be processed, a Location, face attributes (eg gender, age, race). For another example, use the fall detection model to process at least two first images to be processed to obtain a first processing result, where the first processing result includes: whether the characters included in the at least two first images to be processed have fallen, The position of the person in each of the first images to be processed.

204. The storage layer stores the at least one first processing result.

The traditional software framework uses the computer vision model to process the image, and after obtaining the intermediate processing result, according to the business logic, the intermediate processing result is processed to obtain the target processing result, and the image processing task can be completed. For example (Example 3), if a face recognition software is developed based on a traditional software framework to complete the image processing task in Example 1. Using the face recognition model to process the face image to be recognized, the obtained intermediate processing results include: the feature vector of the face image to be recognized, the character attributes in the face image to be recognized (including: gender, age, face shape, skin color, hairstyle ). Since the similarity set can be obtained by calculating the feature vector of the face image to be recognized and the feature vector of the image in the face image database, and in the process of obtaining the similarity set, there is no need to use the character attributes, so, according to the business logic, The process of processing the intermediate processing result to obtain the target processing result is as follows: based on the similarity set and the similarity threshold, it is determined whether the person in the face image to be identified is an employee of company A.

In the process of performing image processing tasks, the intermediate processing results obtained by the computer vision model contain a variety of data, but since the business logic of the traditional software framework is fixed, the data required to complete the business logic in the intermediate processing results (hereinafter referred to as required data) will be stored and used, and the data other than the required data in the intermediate processing result will be discarded. Continuing with Example 3 (Example 4), after obtaining the feature vector and person attribute through the face recognition model, the feature vector will be selected from the intermediate processing result to realize the business logic without storing the person attribute. This will lead to poor generality of software developed based on traditional software frameworks, resulting in low efficiency in developing software based on traditional software frameworks. Continuing with example 4, although the face recognition model can extract the character attributes of the face image to be recognized, the face recognition software can only obtain whether the person in the face image to be recognized is an employee of company A, the face to be recognized or not. The feature vector and similarity set of the image are used for example. The face recognition software processes the face image to be recognized, and cannot output the character attributes of the face image to be recognized. If it is necessary to obtain the character attributes of the face image to be recognized through the face recognition model, it is necessary to re-develop a character attribute extraction software based on the traditional software framework.

In the image processing framework provided by the present application, after obtaining at least one first processing result through the view processing layer, the image processing apparatus stores the at least one first processing result in the storage layer, that is, the image processing apparatus stores the at least one first processing result in the storage layer. All data is stored. In this way, in the subsequent processing, different data in at least one processing result can be used according to different image processing tasks to be executed, so that different image processing tasks to be executed can be completed.

205. The above-mentioned task issuing layer processes the above-mentioned at least one first processing result according to the above-mentioned at least one to-be-executed image processing task to obtain a second processing result.

At least one business logic may be determined based on at least one image processing task to be performed. According to the at least one business logic, the data in the at least one first processing result can be processed to obtain a second processing result, and the at least one image processing task to be executed can be completed.

For example (Example 5), the first processing result stored in the storage layer is obtained by using the face recognition model to process the face image a, and the first processing result includes: character attributes, the feature vector of the face image a (below will be called eigenvector b). At least one image processing task to be performed includes: face recognition. According to the business logic determined by face recognition, calculate the similarity between the feature vector b and the feature vector of the image in the face image database to obtain a similarity set, and determine the face image a based on the similarity set and the similarity threshold identities of the characters in .

Since the business logic is obtained through the image processing task to be executed, in the image processing framework provided by this application, the image processing task to be executed is obtained through the service delivery layer, that is to say, the business logic and the computer vision model are independent of each other. This increases the versatility of software developed based on the image processing framework. Continuing with example 5, it is assumed that the face recognition model in example 5 belongs to the character attribute extraction software, and the character attribute extraction software is developed based on the image processing framework. Since the character attribute extraction software can not only obtain the feature vector of the face image and the identity of the character in the face image, but also obtain the character attribute in the face image, so it can also obtain the character attribute of the face image. The character attribute extraction software is used to extract the character attributes of the face image.

In addition, development based on the image processing framework provided by this application can also improve development efficiency. For example, in Example 2, when the face recognition software is developed based on the traditional software framework, Company A needs to re-develop a software for sorting spare parts. In the case of developing a face recognition software based on the image processing framework of the present application, by replacing the face recognition model in the face recognition software with an object detection model, the software for sorting spare parts can be obtained, and there is no need to develop a new one. software to improve development efficiency.

Since the business logic is determined by the image processing task to be executed, in the image processing framework provided by the embodiment of the present application, the image processing task to be executed is obtained through the task delivery layer, and the business logic and the computer vision model are independent of each other. In this way, after the image processing software is developed based on the image processing framework, different to-be-executed image processing tasks are input to the task delivery layer of the image processing software, and/or, by updating (the updating here includes at least one of the following operations) : replace, delete, add) the computer vision model in the software, so that the updated software can be used to realize different functions, thereby improving the versatility of the software and the development efficiency of developers, thereby improving the efficiency and flexibility of image processing .

As an optional implementation manner, the above-mentioned image processing framework further includes an information index layer. The information index layer is used for acquiring retrieval information, and using the retrieval information to retrieve the at least one first processing result to obtain a fifth processing result.

The retrieval information can be obtained by parsing the image processing tasks to be executed. For example, the information index layer can obtain the image processing tasks to be executed from the task delivery layer for analysis, and then the retrieval information of the retrieval target can be obtained.

For example, the image task to be executed is: determining an image matching the attribute of the reference pedestrian from at least one first image to be processed. The retrieval information obtained by the information index layer is the reference pedestrian attribute, and the reference pedestrian attribute is: white shirt, black pants, and white hat. The information indexing layer retrieves at least one first processing result using the reference pedestrian attribute, and obtains a fifth processing result matching the reference pedestrian attribute as the second processing result of the image task to be executed.

For another example, the at least one first image to be processed includes an image in area A, and the image task to be executed is: determining an image matching the attribute of the reference pedestrian from the image in area A. The reference pedestrian attributes obtained by the information index layer are: white shirt, black pants, and white hat. According to the to-be-executed image task obtained by the task delivery layer, the processing result obtained according to the image of the A area may be selected from at least one first processing result to obtain the processing result to be selected. The information index layer uses the reference pedestrian attribute to retrieve the candidate processing result, and obtains a fifth processing result matching the reference pedestrian attribute as the second processing result of the to-be-executed image task.

Please refer to FIG. 3 , which is a schematic flowchart of a possible implementation manner of step 204 provided in this embodiment of the present application.

301. Store image data in an image data storage module, and store non-image data in a non-image data storage module.

In this embodiment of the present application, the storage layer includes an image data storage module and a non-image data storage module; at least one first processing result includes image data and non-image data. Image data includes pictures and video streams. The non-image data includes data other than image data, and the non-image data includes at least one of the following: character attributes, vehicle attributes, object attributes, and the recognition result of the object in the first image to be processed (including: whether the vehicle is parked illegally, whether the vehicle is worn safety helmet, whether a pedestrian has fallen, whether there is smoke, whether there is an open flame), the feature vector of the first image to be processed, and the position of the person in the first image to be processed.

By processing at least one first image to be processed, non-image data can be obtained. In practical applications, not only non-image data but also image data may be used. For example, a character attribute is developed based on the image processing framework. Extraction software that can be used to extract character attributes. Using the human attribute extraction software to process the image obtained from the surveillance camera in place A, the human attribute in the image of place A can be obtained. There is a robbery in place A. According to the information provided by witnesses at the scene, the suspect was wearing a black shirt, black pants, and white shoes. The suspect wore a mask and had short black hair. After the police know the attributes of the criminal suspect (i.e. black shirt, black pants, white shoes, wearing a mask, short black hair), they can use the attributes of the criminal suspect to retrieve the attributes of the characters in the image of location A, and get the attributes that are related to the criminal suspect. The image of the attributes of the person whose attributes match, as the target image. When the police determine that the person in the target image is the criminal suspect, the whereabouts of the criminal suspect can be determined according to the collection time of the target image and the collection location of the target image.

Therefore, not only non-image data but also image data need to be stored. In the embodiment of the present application, the storage layer includes a non-image data storage module and an image data storage module, wherein the non-image data storage module is used to store non-image data, and the image data storage module is used to store image data. Optionally, the non-image data storage module includes at least one of the following components: an information storage component (kafka), a file storage component (seaweeds), a distributed storage component (minio), a local file storage component (local-FS), a database storage component Components (cassandra). The image data storage module includes: an image storage component (open scene graph, OSG).

For example, the first image to be processed contains the first vehicle, and the vehicle attribute extraction process is performed on the first image to be processed by using the vehicle attribute extraction model to obtain the attributes of the first vehicle (eg, white, sedan), and the first processing result includes The attributes of the first vehicle and the first image to be processed. In the process of storing the first processing result, the attributes of the first vehicle may be stored in the non-image data storage module, and the first image to be processed may be stored in the image data storage module.

For another example, assuming that at least one first image to be processed includes image a and image b, both image a and image b include the second vehicle, and the illegal parking monitoring model is used to process image a and image b, the obtained recognition result is: : the second vehicle is illegally parked, then the first processing result obtained based on the image a and the image b includes: the image a, the image b, and the second vehicle illegally parked. In the process of storing the first processing result, the illegal parking of the second vehicle may be stored in the non-image data storage module, and the image a and the image b may be stored in the image data storage module.

By storing the image data and the non-image data separately, the management of the at least one first processing result and the subsequent processing of the at least one first processing result are facilitated.

302. Establish an index between the above-mentioned image data and the above-mentioned non-image data by using the above-mentioned non-image data as an index entry of the above-mentioned image data.

By taking the above non-image data as the index entry of the above image data, and establishing an index between the above image data and the above non-image data, the image in the image data storage module can be quickly searched through the non-image data in the non-image data storage module .

As described above, the computer vision model is used to process at least one first image to be processed, and the obtained first processing result includes at least one of the following data: character attributes, feature vectors, and action recognition results (such as whether to fall). For computer vision models, the number of images required to obtain different data is different. For example, using the character attribute extraction model to process a first image to be processed can obtain character attributes, and using the action recognition model to obtain the action recognition result. At least two first images to be processed are required. Optionally, computer vision models can be divided into image processing models and video processing models according to the number of required images, and image processing tasks can also be divided into image processing tasks and video processing tasks according to the number of required images. , the number of images required by the image processing model and the image processing task is greater than or equal to 1, and the number of images required by the video processing model and the video processing task is greater than or equal to 2. For example, using a face recognition model to process a face image can obtain the character attributes of the face image, then the face recognition model used to obtain the character attributes is a picture processing model, and the character attribute extraction is a picture processing task. Use the fall detection model to process at least two images to determine whether the person in the at least two images falls down, then the fall detection model used to monitor whether the person falls is a video processing model, and the fall detection is video processing Task.

In practical applications, the image processing tasks to be performed may include both picture processing tasks and video processing tasks. Therefore, it is of great significance to perform corresponding processing on the first acquired image to be processed according to the category of the view processing task. . Please refer to FIG. 4. FIG. 4 is a schematic flowchart of a possible implementation manner of step 203 provided by an embodiment of the present application.

401. In the case where the number of the above-mentioned first images to be processed is greater than or equal to 2, and the execution of the picture processing instruction and the execution of the video processing instruction are received, according to the above-mentioned execution of the picture processing instruction, from the at least two first images to be processed. Select at least one second image to be processed.

In this embodiment of the present application, the view processing layer includes a picture processing module and a video processing module, wherein the picture processing module includes a picture processing model, and the video processing module includes a video processing model, that is, using the picture processing module to process an image can obtain For a processing result, a processing result can be obtained by using the video processing module to process at least two images.

The above picture processing instruction is used to instruct the image processing apparatus to perform the picture processing task. Optionally, the picture processing instruction carries the following information: the picture processing task to be executed and the image to be processed. For example, the at least one first image to be processed obtained by the image processing apparatus includes: image a and image b, and the information carried by the image processing instruction includes performing a character attribute extraction process on image a. For another example, there is a communication connection between the image processing device and the camera A, and there is also a communication connection between the image processing device and the camera B, and the image obtained from the camera A and the image obtained from the camera B are the first to be Process images. The information carried by the image processing instruction includes performing human attribute extraction processing on the image obtained from the camera A.

The above video processing instructions are used to instruct the image processing apparatus to perform video processing tasks. Optionally, the video processing instruction carries the following information: the video processing task to be executed, and the image to be processed. For example, the at least one first image to be processed obtained by the image processing apparatus includes: image a, image b, and image c, and the information carried by the image processing instruction includes performing fall monitoring processing on image a and image b. For another example, there is a communication connection between the image processing device and the camera A, and there is also a communication connection between the image processing device and the camera B, and the image obtained from the camera A and the image obtained from the camera B are the first to be Process images. The information carried by the image processing instruction includes performing fall detection processing on at least two images acquired from the camera A.

The image processing apparatus receives the picture processing instruction and the video processing instruction, indicating that the image processing apparatus needs to perform both the picture processing task and the video processing task. Before performing the picture processing task, it is necessary to determine the images required to perform the picture processing task from at least two first images to be processed. In the image processing framework, at least one image is selected from the at least two first images to be processed according to the image processing instruction, as at least one second image to be processed, for performing the image processing task.

402. Use the image processing module to process the at least one second image to be processed to obtain at least one third processing result.

Using the image processing module to process a second image to be processed, a third processing result can be obtained.

For example, it is assumed that at least one second image to be processed includes: image a and image b, and the image processing module includes a character attribute extraction model. The image a is processed by the image processing module to obtain a third processing result A, wherein the third processing result A includes: a white shirt, black pants, and long yellow hair. The image b is processed by the image processing module to obtain a third processing result B, wherein the third processing result B includes: a black top, blue pants, and long yellow hair.

403. Select at least two third images to be processed from the at least two first images to be processed according to the above execution video processing instruction.

Before performing the picture processing task, the images required to perform the video processing task need to be determined from at least two first images to be processed. In the image processing framework, at least two images are selected from the at least two first images to be processed according to the video processing instructions, as at least two third images to be processed, for performing video processing tasks.

It is understandable that one of the at least two first images to be processed may be either the second image to be processed or the third image to be processed. For example, the at least two first images to be processed include: image a, image b, and image c. At least one second to-be-processed image determined from the at least two first to-be-processed images according to the picture processing task instruction includes image a, and at least two second to-be-processed images determined from the at least two first to-be-processed images according to the video processing task instruction The three images to be processed include image a and image b. At this time, image a is both the second image to be processed and the third image to be processed.

404. Use the video processing module to process the at least two third images to be processed to obtain at least one fourth processing result.

A fourth processing result can be obtained by processing at least two third images to be processed by using the video processing module.

For example, it is assumed that at least one third image to be processed includes: image a, image b, and image c, and the video processing module includes a fall monitoring model. The image a and the image b are processed by the video processing module, and a fourth processing result A is obtained, wherein the fourth processing result A includes: the characters included in the image a and the image b have not fallen. The image b and the image c are processed by the video processing module, and a fourth processing result B is obtained, wherein the fourth processing result B includes: the person included in the image b and the image c has fallen. Use the video processing module to process the image a, the image b and the image c to obtain a fourth processing result C, wherein the image a, the image b and the image c all contain the same person, and the fourth processing result C includes: the person falls .

405. Obtain the at least one first processing result according to the at least one third processing result and the at least one fourth processing result.

In the embodiment of the present application, the processing of the at least one second image to be processed by the picture processing module and the processing of the at least two third images to be processed by the video processing module are two independent processes. At least one third processing result obtained by the picture processing module and at least one fourth processing result obtained by the video processing module are both processing results obtained by the view processing layer, that is, at least one first processing result includes at least one third processing result and at least one fourth processing result.

In a possible implementation manner, at least one third processing result and at least one fourth processing result are used as at least one first processing result. For example, if at least one third processing result includes third processing result a and third processing result b, and at least one fourth processing result includes fourth processing result c and fourth processing result d, then at least one first processing result includes: Three processing results a, third processing results b, fourth processing results c, and fourth processing results d.

In another possible implementation manner, at least one third processing result and at least one fourth processing result are divided to obtain at least one first processing result. For example, at least one third processing result includes a third processing result a and a third processing result b, and at least one fourth processing result includes a fourth processing result c and a fourth processing result d, wherein the timestamp of the third processing result a The timestamp of the fourth processing result c is the same as that of the fourth processing result c, and the timestamp of the third processing result b is the same as the timestamp of the fourth processing result d. The third processing result a and the fourth processing result c are taken as the first processing result e. The third processing result b and the fourth processing result d are taken as the first processing result f. The at least one first processing result includes: a first processing result e and a first processing result f.

In the image processing framework, the processing of at least one second image to be processed by the image processing module and the processing of at least two third images to be processed by the video processing module are two independent processes, developed based on the image processing framework. The software can be used to perform both picture processing tasks and video processing tasks, and can also be used to perform picture processing tasks and video processing tasks at the same time.

For example, Company A has developed a human attribute extraction software for extracting human attributes in images. Now Company A wants to develop a fall detection software to detect whether a pedestrian falls at the escalator. Assuming that the character attribute extraction software is developed based on the image processing framework, in the character attribute extraction software, the character attribute extraction model is located in the image processing module. From the image input of the camera used to monitor the escalator to the fall detection model, a software can be obtained that can not only be used to extract human attributes, but also can be used to detect whether a pedestrian falls at the escalator.

For another example, Company A has developed a software (hereinafter referred to as the first software) that can be used not only for extracting person attributes, but also for monitoring whether a pedestrian falls at the escalator. Now Company B wants to purchase a software from Company A for detecting whether a pedestrian falls at the escalator. Although the first software can be used to monitor whether a pedestrian falls at the escalator, the first software also includes a character attribute extraction model required for extracting character attributes, and the functional modules required for extracting character attributes are very important to Company B. The language is redundant, so it is necessary to remove the character attribute extraction model in the first software. Assuming that the first software is developed based on the image processing framework, in the first software, the human attribute extraction model is located in the image processing module, and the fall detection model is located in the video processing module. Delete it, and you can get a software for monitoring whether a pedestrian falls at the escalator.

As can be seen from the above two examples, since the picture processing module and the video processing module are two independent modules, the image processing framework can also improve the development efficiency of developers.

Please refer to FIG. 5 , which is a schematic flowchart of a possible implementation manner of step 402 provided by an embodiment of the present application.

501. Acquire a fourth image to be processed.

The fourth image to be processed in the embodiment of the present application is one image in the at least one second image to be processed. For an implementation manner of acquiring the fourth image to be processed, reference may be made to step 201, which will not be repeated here.

502. Use the image processing module to perform object detection processing on the fourth image to be processed, to obtain the position of the first object in the fourth image to be processed.

In this embodiment of the present application, the first object belongs to the fourth image to be processed. For example, if the fourth image to be processed includes a car, the car is the first object. For another example, if the fourth image to be processed includes a person, the person is the first object. For another example, the fourth image to be processed includes a person and a car, and the first object may be one of the person and the car.

In this embodiment of the present application, by performing object detection processing on the fourth image to be processed, it can be determined whether the fourth image to be processed contains an object. When the fourth image to be processed contains an object, the position of the object in the fourth image to be processed can be obtained. The above position may be the coordinates of any pair of diagonal corners of the object frame containing the object in the pixel coordinate system, or the coordinates of the center of the object frame containing the object and one of the diagonal corners in the pixel coordinate system.

In a possible implementation manner, the image processing module includes a convolutional neural network, and the object detection processing on the fourth image to be processed may be implemented by a convolutional neural network. By using multiple images with annotation information as training data, the convolutional neural network is trained, so that the trained convolutional neural network can complete the object detection processing on the image. The annotation information of the images in the training data is the object and the position of the object. In the process of using the training data to train the convolutional neural network, the convolutional neural network extracts the characteristic data of the image from the image, and determines whether there is an object in the image according to the characteristic data. The feature data of the image gets the position of the object. The annotation information is used as the supervision information to supervise the results obtained by the convolutional neural network in the training process, and the parameters of the convolutional neural network are updated to complete the training of the convolutional neural network. In this way, the trained convolutional neural network can be used to process the fourth image to be processed to obtain the position of the first object in the fourth image to be processed.

In another possible implementation manner, the image processing module includes an object detection processing algorithm, and the object detection processing can be implemented by the object detection algorithm. Optionally, the above-mentioned object detection processing can be implemented by one of the following algorithms: you only look once algorithm (you only look once, YOLO), target detection algorithm (deformable part model, DMP), single image multi-target detection algorithm ( single shot multiBox detector, SSD), Faster-RCNN algorithm, etc. This application does not specifically limit the object detection algorithm for implementing object detection processing.

503. Intercept the pixel area determined by the foregoing position from the foregoing fourth image to be processed to obtain an object image.

In a possible implementation manner, according to the position of the first object in the fourth image to be processed, a corresponding pixel area is determined in the fourth image to be processed, and the pixel area is intercepted from the fourth image to be processed to obtain An object image containing the first object. Optionally, the shape of the above-mentioned pixel area may be any shape, such as a rectangle, a circle, and a rhombus, and the present application does not limit the shape of the above-mentioned pixel area.

It should be understood that each object image contains only one object. In the case where the fourth image to be processed contains multiple objects, multiple object images will be obtained. For example, the fourth image to be processed contains person A and person B, By processing the fourth image to be processed, an object image including person A and an object image including person B can be obtained.

504. Use an image processing module to perform attribute extraction processing on the object image to obtain the attribute of the first object.

The image processing module includes an attribute extraction model, and the attribute extraction process is performed on the object image by using the attribute extraction model to obtain the attribute of the first object. For example, the attribute extraction model may be a character attribute extraction model, and the attribute extraction process is performed on the object image by using the character attribute extraction model to obtain the character attribute of the first object. For another example, the attribute extraction model may be a vehicle attribute extraction model, and the vehicle attribute of the first object can be obtained by performing attribute extraction processing on the object image by using the vehicle attribute extraction model.

Since the amount of data contained in the object image is smaller than the amount of data contained in the fourth image to be processed, the amount of data processing brought by the attribute extraction processing on the object image is smaller than the amount of data processing caused by the attribute extraction processing on the fourth image to be processed. The attributes of the first object are obtained by performing attribute extraction processing on the object image, which can reduce the amount of data processing and improve the processing speed.

In order to improve the ability of social security management and control and maintain a good social security environment, surveillance cameras are installed in more and more places. When the relevant personnel need to find the target object, they can collect the data from the surveillance cameras arranged in different locations according to the attributes of the target object. Determine the whereabouts of the target object in the image. For example, there was a robbery in place A, and the information about the suspect provided to the police by witnesses at the crime scene included: wearing a black shirt, black pants, a mask, and short black hair. The police can use the target attributes determined based on the suspect’s information (i.e. black shirt, black pants, face mask, black short hair) to compare the database of place A (the database includes: images collected through surveillance cameras in place A, The attributes of the people in the image), get the image containing the target object, as the target image. According to the acquisition time and/or acquisition location of the target image, the whereabouts of the target object can be determined.

Since the first image to be processed may contain multiple objects, in order to facilitate the user (here, the user who points to the image processing device and input the target attribute) to identify whether the object with the attribute matching the target attribute is the target The object image of the attribute whose target attribute matches, as the target object image. When the user determines that it is a high probability event that the object in the target object image is the target object, the first to-be-processed image including the target object image may be further displayed. Optionally, the at least one first processing result includes an object image, that is, the image data includes the aforementioned object image and at least one first image to be processed. As an optional implementation manner, step 301 includes the following steps:

31. Store the attributes of the first object in the non-image data storage module, and store the fourth to-be-processed image and the object image in the image data storage module.

As an optional implementation manner, step 302 includes the following steps:

32. By taking the attribute of the first object as the index entry of the fourth image to be processed, and the attribute of the first object as the index entry of the object image, between the image data and the non-image data Build an index.

By using the attribute of the first object as the index entry of the fourth image to be processed, and using the attribute of the first object as the index entry of the object image, an index is established between the above-mentioned image data and the above-mentioned non-image data, which can be stored by storing Based on the attributes of the first object in the non-image data module, the object image and the fourth to-be-processed image stored in the image data storage module are quickly searched.

As an optional implementation manner, step 201 specifically includes the following steps:

21. The view access layer acquires at least one fifth image to be processed.

In a possible implementation manner, the image processing apparatus is communicatively connected with at least one imaging device (including at least one of the following: a camera, a mobile phone, and a camera), and the view access layer obtains at least one image from the at least one imaging device through the communication connection. An image as the at least one fifth image to be processed.

22. If the communication protocol of the at least one fifth image to be processed is different from the communication protocol used by the image processing framework, convert the communication protocol of the at least one fifth image to be processed into the communication protocol of the image processing framework. Using the communication protocol used, the at least one first image to be processed is obtained.

In the case where the communication protocol used by the imaging device is different from that used by the image processing framework, the image processing framework cannot parse the image acquired from the imaging device, and thus cannot process the image. Therefore, when the communication protocol used by the imaging device is different from the communication protocol used by the image processing framework, it is necessary to convert the communication protocol of the image obtained from the imaging device into the communication protocol used by the image processing framework to obtain at least one The converted images are used as at least one first image to be processed.

For example, company A developed a pedestrian attribute extraction software for company B based on the image processing framework. The communication protocol used by the pedestrian attribute extraction software is the same as that used by company B's surveillance cameras. Now Company C also wants Company A to develop a pedestrian attribute extraction software for it, but the communication protocol used by Company C's surveillance cameras is different from that used by Company B's surveillance cameras. Although the communication protocol used by the surveillance cameras of Company C is different from that used by the pedestrian attribute extraction software, in the process of using the pedestrian attribute extraction software to process the images obtained from the surveillance cameras of Company C, the pedestrian attribute extraction The software can process the images obtained from the surveillance cameras of Company C by converting the communication protocol of the images obtained by the surveillance cameras of Company C into the communication protocols used by the surveillance cameras of Company B.

In the case where the communication protocol of the image acquired from the imaging device is different from the communication protocol used by the image processing framework, by converting the communication protocol of the acquired image into the communication protocol used by the image processing framework, make the image processing framework based The developed software can process images acquired from any imaging device.

The embodiments of the present application also provide an implementation manner of performing image processing tasks by using software developed based on the above-mentioned image processing framework. Please refer to FIG. 6, which is a schematic flowchart of another image processing method provided by an embodiment of the present application.

601. Acquire at least one first image to be processed and at least one image processing task to be executed.

At least one first image to be processed in this step is the same as at least one first image to be processed in step 201 , and at least one image processing task to be executed in this step is the same as at least one image processing task to be executed in step 202 same.

Refer to step 201 for the implementation manner of the image processing apparatus acquiring at least one first image to be processed, and for the implementation manner of acquiring at least one image processing task to be performed by the image processing apparatus, referring to step 202, which will not be repeated here.

602. Perform picture processing and/or video processing on the at least one first image to be processed to obtain at least one first processing result.

In the embodiments of the present application, picture processing refers to obtaining a processing result by processing at least one image, and video processing refers to obtaining a processing result by processing at least two images.

For the implementation of this step, reference may be made to steps 401 to 405, which will not be repeated here.

603. Store the at least one first processing result.

For the implementation of this step, reference may be made to step 204, which will not be repeated here.

604. Process the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

For the implementation of this step, reference may be made to step 205, which will not be repeated here.

Optionally, an embodiment of the present application further provides a method for constructing an image processing framework, the method comprising the following steps:

61. Obtain the at least one image processing task to be executed and a frame to be processed, wherein the frame to be processed includes a view processing layer, a storage layer, and a task delivery layer.

In this embodiment of the present application, the architecture of the frame to be processed is the same as the architecture of the image processing frame. Optionally, refer to FIG. 1 for the architecture of the frame to be processed.

62. When the at least one image processing task to be executed is obtained through the task delivery layer, obtain a picture processing module and/or video processing module matching the at least one image processing task to be executed.

The above picture processing module is used for image processing, and the above video processing module is used for video processing. The picture processing module and/or video processing module matched with the at least one image processing task to be executed means that the picture processing module and/or the video processing module can be used to execute the at least one image processing task to be executed. For example, the at least one image processing task to be performed includes: face recognition. The picture processing module and/or video processing module matched with at least one image processing task to be performed includes: a face recognition model. For another example, the at least one image processing task to be performed includes: fall detection. The picture processing module and/or video processing module matched with at least one image processing task to be executed includes: a fall detection model. For another example, the at least one image processing task to be performed includes: face recognition and fall detection. The picture processing module and/or video processing module matched with at least one image processing task to be performed includes: a face recognition model and a fall detection model.

In a possible implementation manner, when the image processing apparatus obtains at least one image processing task to be executed through the task delivery layer, the image processing apparatus outputs the information of the at least one image processing task to be executed, so that the user can rely on the at least one image processing task. For the information of the image processing task to be executed, select the picture processing module and/or video processing module matching with at least one image processing task to be executed. The image processing apparatus takes the picture processing module and/or video processing module input by the user for the information of at least one image processing task to be performed as the picture processing module and/or video processing module matching the at least one image processing task to be performed.

In another possible implementation manner, when the image processing apparatus obtains at least one image processing task to be executed through the task delivery layer, the image processing apparatus selects from the model library the image processing task that matches the at least one image processing task to be executed. Picture processing module and/or video processing module.

63. Add the above-mentioned picture processing module and/or the above-mentioned video processing module to the view processing layer of the to-be-processed image processing framework to obtain the above-mentioned image processing framework.

The image processing apparatus adds a picture processing module and/or a video processing module matching at least one image processing task to be performed to the view processing layer, so as to obtain the above-mentioned image processing framework.

Based on the image processing framework provided by the embodiments of the present application, the embodiments of the present application further provide several possible application scenarios.

Scenario 1: With the rapid growth of the number of cameras in public places, how to effectively determine the attributes of the characters in the video stream through massive video streams, and determine the whereabouts of the characters based on the attributes of the characters is of great significance.

When theft occurs in the smart park, the managers of the smart park hope to use the monitoring software to monitor the video and/or images collected by the surveillance cameras in the smart park (hereinafter, the videos and/or images collected by the surveillance cameras will be referred to as the first original image) to determine the whereabouts of the suspect in the robbery case, the monitoring software is developed based on the image processing framework provided by this application. The monitoring software runs on the server, and there is a communication connection between the server and the monitoring cameras in the smart park.

FIG. 7 shows the data flow in the process of processing the first original image by the monitoring software. As shown in FIG. 7 , the view access layer and the surveillance camera are connected in communication, and through the communication connection, the view access layer can obtain the first original image from the surveillance camera. The view access layer can input the acquired first original image to the view processing layer, and can also display the first original image through the view preview module. For example, the user inputs an instruction to preview the first original image to the server through the view preview module. , the server displays the first original image through the view preview module when receiving the instruction to preview the first original image.

The image storage component, information storage component, index storage component, spatiotemporal data storage component, and target attribute storage component shown in FIG. 7 all belong to the storage layer, wherein the image storage component is used to store image data, and the information storage component, index storage component, Both the spatiotemporal data storage component and the target attribute storage component are used to store non-image data. In FIG. 7 , by processing the first original image through the view processing layer, at least one first processing result can be obtained, the image data in the at least one first processing result is stored in the image storage component, and the non-image data is stored in the image storage component. The feature vectors and attributes are stored in the information storage component, and the temporal information and spatial information in the non-image data are stored in the spatiotemporal data storage component. For example (Example 6), the first original image includes: image a and image b, wherein the timestamp of image a is t ₁ , the timestamp of image b is t ₂ , image a is captured by surveillance camera No. 1, and image b is captured by Captured by surveillance camera No. 2. The first original image is processed by the view processing layer, and it is determined that the image a contains a person and the image b does not contain a person. Further, by processing the image a, the attributes of the person in the image a, the feature vector of the image a are obtained, and the person object image including the person is intercepted from the image a. Store the image a and the human object image in the OSG, store t ₁ , t ₂ , the position of the surveillance camera No. 1, and the location of the surveillance camera No. 2 in the TFD, and store the attributes of the person in the image a and the feature vector of the image a in the Kafka.

By using the non-image data as an index entry of the image data, an index is established between the non-image data and the image data, and the index entry is stored in the information storage component. Create an index between non-image data and spatiotemporal information, and store the index entries to the index storage component. Continue with example 6, take the attributes of the person in image a, the feature vector of image a, as the index entry of image a (hereinafter referred to as index entry 1), and take the position of camera t ₁ and No. 1 as the image The index entry of the feature vector of a (hereinafter referred to as index entry 2). The attributes of the person in the image b and the feature vector of the image b are used as the index entry of the image b (hereinafter referred to as index entry 3), and the position of the camera t ₂ and No. 2 are used as the index entry of the feature vector of the image b (hereinafter referred to as index entry 4). The index entry 1 and the index entry 3 are stored in the information storage component, and the index entry 2 and the index entry 4 are stored in the index storage component.

In addition, the second original image obtained by processing the first original image by the view processing layer may be stored in the view storage component. For example, the first original image is processed by the view processing layer, and a character frame is added to the first original image to obtain the second original image, wherein the character in the character frame is a suspect. In this way, the manager of the smart park can view the second original image through the view preview module, which can more efficiently determine the whereabouts of the suspect.

Monitoring tasks can be obtained through the task issuing layer in FIG. 7 , and the monitoring tasks carry characteristics of suspects (hereinafter referred to as target characteristics). For example, the smart park enters the command to the server to perform the task of tracking the suspect. The target characteristics carried by the command include: white shirt, black pants, long black hair, and appearing at the position of the No. 3 surveillance camera. After acquiring the monitoring task, the server uses the spatiotemporal information in the monitoring task (ie, the No. 3 surveillance camera) to retrieve the data in the index storage component, and obtains an index entry that matches the spatiotemporal information as the first target index entry. According to the first index entry, the first non-image data is determined from the information storage component, and the first non-image data is extracted to the target feature storage component. The first non-image data in the target feature storage component is retrieved using the features other than the spatiotemporal information in the target features (hereinafter referred to as target attributes), and the second non-image data matching the target attribute is obtained as the retrieval result. Push the retrieval result to the information storage component, so that the information storage component outputs the second non-image data, and according to the second non-image data and the index entry stored in the information storage component, determine from the image storage component the difference between the second non-image data and the second non-image data. Image data corresponding to the image data (hereinafter will be referred to as an image containing a suspect). The image and the second non-image data containing the suspect are output through the task delivery layer, so that the administrator of the smart park can know the whereabouts of the suspect.

Optionally, in a specific example of scenario 1, the above image processing component is OSG, the above information storage component is Kafka, and the above index storage component is keyspaces.

Before the image processing device uses the software developed based on the image processing framework to perform image processing tasks, the image processing device builds a picture storage space (bucket) in OSG, builds a data topic (kafka-topic) in Kafka, and builds a key value in Cassandra Storage space (keyspaces).

Input the address of the camera to the view access layer via the image processing device. The view access layer can further generate a "communication protocol" according to the address, where the "communication protocol" is a message code corresponding to a communication protocol that can be parsed by the image processing framework. This "communication protocol" is encoded in a way to represent binary data (base64) based on 64 printable characters, eg, post/engine/video-ingress/v1/generate_rtsp_address.

The image processing device obtains the video stream from the camera according to the "communication protocol" and the address of the camera, and pushes the video stream to the view preview module and the view processing layer. For example, the address of the camera is: vis's address, and the "communication protocol" is post/engine/video-ingress/v1/generate_rtsp_address. The "vis's address + 'communication protocol'" is called the target address, and the image processing device can obtain the video stream from the camera according to the target address.

After the image processing apparatus acquires the video stream, the video stream can be transmitted to the view preview module to display the content of the video stream. The image processing apparatus may also use the picture processing module and/or the video processing module in the view processing layer to process the video stream to obtain image data and non-image data. Store image data in bucket, and store non-image data in kafka-topic. Create an index between image data and non-image data (hereinafter referred to as an image index entry), and store the image index entry in kafka-topic.

The image processing apparatus stores the spatiotemporal information of the non-image data in the spatiotemporal library in the storage layer. The image processing apparatus establishes an index between non-image data and spatiotemporal information (hereinafter referred to as spatiotemporal index entry), and stores the spatiotemporal index entry in keyspaces.

In the process of creating an image processing task to be executed by the image processing apparatus, a target library is constructed in a static library (ie, the target feature storage component in FIG. 7 ), and the non-image data of the target object is stored in the target library.

After completing the creation of the image processing task to be executed by the image processing device, the image processing device determines the non-image data matching the spatiotemporal information of the image processing task to be executed from the kafka-topic according to the spatiotemporal information of the image processing task to be executed, and Extract the non-image data matching the spatiotemporal information of the image processing task to be performed to the target library. The image processing device uses the non-image data of the target object to retrieve the non-image data in the static library that matches the spatiotemporal information of the image processing task to be executed to obtain the retrieval result, and pushes the retrieval result to kafka-topic.

Scenario 2: In the case of a large flow of people, if someone falls at the escalator, it will bring a great safety hazard. Therefore, the managers of the smart park hope to use monitoring software to monitor whether there is a pedestrian falling at the escalator, so as to reduce the occurrence of safety accidents.

In Scenario 1, the computer vision model included in the view processing layer is the character attribute extraction model. Since the human attribute extraction model cannot realize the pedestrian fall detection, the monitoring software cannot realize the pedestrian fall detection. The monitoring software is obtained by replacing the character attribute extraction model in the monitoring software with the pedestrian fall monitoring model. By processing the video stream collected by the surveillance camera at the escalator (hereinafter, the video stream collected by the surveillance camera at the escalator is called the first original video stream), it is possible to monitor whether there are pedestrians at the escalator. fall.

FIG. 8 shows the data flow in the process of processing the first original video stream by the monitoring software. As shown in FIG. 8 , the view access layer and the surveillance camera are connected in communication, and through the communication connection, the view access layer can obtain the first original video stream from the surveillance camera. The view access layer can input the acquired first original video stream to the view processing layer, and can also display the first original video stream through the view preview module. For example, the user inputs the preview first original video to the server through the view preview module. Stream instruction, when the server receives the instruction to preview the first original video stream, display the first original video stream through the view preview module.

The network video recorder, the information storage component, the cloud storage component, and the monitoring result storage component shown in FIG. 8 all belong to the storage layer. After the first original video stream is processed by the view processing layer, the non-image data is stored in the information storage component, and in the case that the non-image data includes the monitoring result of pedestrian falling, the information of the falling pedestrian is stored in the monitoring result. A storage component, so that managers can obtain the information of the fallen pedestrian (including: person attribute, falling position, and falling time) stored in the monitoring result storage module through the service delivery layer for viewing. In the case that the non-image data includes the monitoring result of the pedestrian falling, the information storage component sends a video editing instruction to a network video recorder (NVR) through the editing module, so as to instruct the NVR to edit the pedestrian from the first original video stream. Fall down video stream, get fall down video stream. The NVR uploads the fall video stream to the cloud storage component through the editing module, so that managers can view the fall video stream stored in the cloud storage component through the service delivery layer. For example, when the first original video stream is processed by the view processing layer, and it is determined that a pedestrian has fallen, the monitoring software can output alarm information through the service delivery layer to prompt the management personnel that a pedestrian has fallen. The administrator can view the video stream of the falling pedestrian stored in the cloud storage component through the service delivery layer, and/or view the information of the falling pedestrian stored in the monitoring result storage component.

In addition, the second original video stream obtained by processing the first original video stream by the view processing layer may be stored in the view storage component. For example, the first original image is processed by the view processing layer, and a character frame is added to the first original video stream to obtain the second original video stream, wherein the character in the character frame is a fallen pedestrian. In this way, the managers of the smart park can more efficiently determine the location of the fallen pedestrian by viewing the second original video stream.

It can be seen from Scenario 1 and Scenario 2 that by replacing the character attribute extraction model in the monitoring software with the fall monitoring model, and adjusting the storage components in the storage layer, the monitoring system for detecting whether a pedestrian falls can be obtained. software.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

The methods of the embodiments of the present application are described in detail above, and the apparatuses of the embodiments of the present application are provided below.

Please refer to FIG. 9 , which is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application. The image processing apparatus 1 includes: a view access unit 11 , a task issuing unit 12 , a view processing unit 13 , and a storage unit 14 , the information indexing unit 15, wherein:

a view access unit, configured to acquire at least one first image to be processed;

a task issuing unit, used for acquiring at least one image processing task to be executed;

a view processing unit, configured to process the at least one first image to be processed to obtain at least one first processing result;

a storage unit for storing the at least one first processing result;

A task issuing unit, configured to process the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

With reference to any embodiment of the present application, the at least one first processing result includes image data and non-image data; the storage unit includes an image data storage subunit and a non-image data storage subunit;

The storage unit is used for:

storing the image data in the image data storage subunit, and storing the non-image data in the non-image data storage subunit;

An index is established between the image data and the non-image data by using the non-image data as an index entry for the image data.

With reference to any embodiment of the present application, the view processing unit includes a picture processing subunit and a video processing subunit; the view processing unit is used for:

In the case where the number of the first images to be processed is greater than or equal to 2, and an instruction to perform image processing and an instruction to perform video processing are received, according to the instruction to perform image processing, from at least two of the first images to be processed Select at least one second image to be processed from the

Using the picture processing subunit to process the at least one second image to be processed to obtain at least one third processing result;

Selecting at least two third images to be processed from the at least two first images to be processed according to the executing video processing instruction;

using the video processing subunit to process the at least two third images to be processed to obtain at least one fourth processing result;

The at least one first processing result is obtained according to the at least one third processing result and the at least one fourth processing result.

With reference to any embodiment of the present application, the view processing unit is used for:

Using the image processing subunit to perform object detection processing on the fourth image to be processed to obtain the position of the first object in the fourth image to be processed; the fourth image to be processed belongs to the at least one second image to be processed processing an image; the first object belongs to the fourth image to be processed;

determining a pixel area including the first object according to the position;

Intercept the pixel area from the fourth to-be-processed image to obtain an object image;

Using the image processing subunit to perform attribute extraction processing on the object image to obtain the attribute of the first object.

With reference to any embodiment of the present application, the image data includes the at least one first image to be processed and the object image, and the non-image data includes attributes of the first object.

With reference to any embodiment of the present application, the view access unit is used for:

Acquire at least one fifth image to be processed;

If the communication protocol of the at least one fifth image to be processed is different from the communication protocol used by the image processing framework, converting the communication protocol of the at least one fifth image to be processed into the image processing The communication protocol used by the framework to obtain the at least one first image to be processed.

With reference to any embodiment of the present application, the image processing apparatus further includes: an information indexing unit;

The information indexing unit is configured to acquire retrieval information, and use the retrieval information to retrieve the at least one first processing result to obtain a fifth processing result.

Please refer to FIG. 10 . FIG. 10 is a schematic structural diagram of another image processing apparatus provided by an embodiment of the present application. The image processing apparatus 2 includes: an acquisition unit 21 , a first processing unit 22 , a storage unit 23 , and a second processing unit 24 , adding unit 25, wherein:

The image processing device includes:

an acquisition unit, configured to acquire at least one first image to be processed and at least one image processing task to be performed;

a first processing unit, configured to perform picture processing and/or video processing on the at least one first image to be processed to obtain at least one first processing result;

a storage unit for storing the at least one first processing result;

The second processing unit is configured to process the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

With reference to any embodiment of the present application, the method executed by the image processing apparatus is implemented by an image processing framework, and the acquiring unit is further configured to acquire the at least one image processing task to be executed and the frame to be processed; the to-be-processed The framework includes a view processing layer, a storage layer, and a task delivery layer;

The acquiring unit is further configured to acquire a picture processing module and/or an image processing module matching the at least one image processing task to be performed when the at least one image processing task to be performed is acquired through the task delivery layer a video processing module; the picture processing module is used for realizing picture processing; the video processing module is used for realizing video processing;

The image processing apparatus further includes: an adding unit, configured to add the picture processing module and/or the video processing module to the view processing layer of the to-be-processed image processing frame to obtain the image processing frame.

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present application may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the above method embodiments. For brevity, here No longer.

FIG. 11 is a schematic diagram of a hardware structure of an image processing apparatus according to an embodiment of the present application. The image processing device 3 includes a processor 31 , a memory 32 , an input device 33 , and an output device 34 . The processor 31, the memory 32, the input device 33, and the output device 34 are coupled through a connector, and the connector includes various types of interfaces, transmission lines, or buses, which are not limited in this embodiment of the present application. It should be understood that, in various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, such as various interfaces, transmission lines, and buses.

The processor 31 may be one or more graphics processing units (graphics processing units, GPUs). In the case where the processor 31 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 31 may be a processor group composed of multiple GPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor may also be another type of processor, etc., which is not limited in this embodiment of the present application.

The memory 32 may be used to store computer program instructions and various types of computer program codes including program codes for executing the solutions of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), Or portable read-only memory (compact disc read-only memory, CD-ROM), which is used for related instructions and data.

The input device 33 is used for inputting data and/or signals, and the output device 34 is used for outputting data and/or signals. The input device 33 and the output device 34 may be independent devices or may be an integral device.

It can be understood that, in this embodiment of the present application, the memory 32 can be used not only to store related instructions, but also to store related data. For example, the memory 32 can be used to store at least one first image to be processed obtained through the input device 33 , or the The memory 32 may also be used to store at least one first processing result obtained by the processor 31, etc., and the data stored in the memory is not limited in this embodiment of the present application.

It can be understood that FIG. 11 only shows a simplified design of an image processing apparatus. In practical applications, the image processing apparatus may also include other necessary elements, including but not limited to any number of input/output devices, processors, memories, etc., and all image processing apparatuses that can implement the embodiments of the present application are included in this application. within the scope of protection of the application.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here. Those skilled in the art can also clearly understand that the description of each embodiment of the present application has its own emphasis. For the convenience and brevity of the description, the same or similar parts may not be repeated in different embodiments. Therefore, in a certain embodiment For the parts that are not described or not described in detail, reference may be made to the descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions can be sent from one website site, computer, server, or data center to another via wired (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) means. A website site, computer, server or data center for transmission. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, digital versatile disc (DVD)), or semiconductor media (eg, solid state disk (SSD)) Wait.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented. The process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed , which may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: read-only memory (ROM) or random access memory (RAM), magnetic disk or optical disk and other mediums that can store program codes.

Claims (13)

1. An image processing method is characterized in that the image processing method is realized through an image processing framework, and the image processing framework comprises a view access layer, a view processing layer, a storage layer and a task issuing layer;

the view access layer is used for acquiring at least one first image to be processed;

the task issuing layer is used for acquiring at least one image processing task to be executed;

the view processing layer is used for processing the at least one first image to be processed to obtain at least one first processing result;

the storage layer is used for storing the at least one first processing result;

and the task issuing layer is used for processing the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

2. The method of claim 1, wherein the at least one first processing result comprises image data and non-image data; the storage layer comprises an image data storage module and a non-image data storage module;

the storage layer is further to:

storing the image data in the image data storage module, and storing the non-image data in the non-image data storage module;

and establishing an index between the image data and the non-image data by using the non-image data as an index entry of the image data.

3. The method according to claim 1 or 2, wherein the view processing layer comprises a picture processing module and a video processing module; the view processing layer is further to:

under the condition that the number of the first images to be processed is greater than or equal to 2 and an execution image processing instruction and an execution video processing instruction are received, selecting at least one second image to be processed from at least two first images to be processed according to the execution image processing instruction;

processing the at least one second image to be processed by using the image processing module to obtain at least one third processing result;

selecting at least two third images to be processed from the at least two first images to be processed according to the executing video processing instruction;

processing the at least two third images to be processed by using the video processing module to obtain at least one fourth processing result;

and obtaining the at least one first processing result according to the at least one third processing result and the at least one fourth processing result.

4. The method of claim 3, wherein the view processing layer is further configured to:

using the image processing module to perform object detection processing on a fourth image to be processed to obtain the position of a first object in the fourth image to be processed; the fourth image to be processed belongs to the at least one second image to be processed; the first object belongs to the fourth image to be processed;

determining a pixel region containing the first object according to the position;

intercepting the pixel area from the fourth image to be processed to obtain an object image;

and performing attribute extraction processing on the object image by using the image processing module to obtain the attribute of the first object.

5. The method according to claim 4, characterized in that the image data comprise the at least one first image to be processed and the object image, and the non-image data comprise properties of the first object.

6. The method of any of claims 1 to 5, wherein the view access layer is further configured to:

acquiring at least one fifth image to be processed;

and under the condition that the communication protocol of the at least one fifth image to be processed is different from the communication protocol used by the image processing framework, converting the communication protocol of the at least one fifth image to be processed into the communication protocol used by the image processing framework to obtain the at least one first image to be processed.

7. The method of any of claims 1 to 6, wherein the image processing framework further comprises an information indexing layer;

the information index layer is used for acquiring retrieval information;

and retrieving the at least one first processing result by using the retrieval information to obtain a fifth processing result.

8. An image processing method, characterized in that the method comprises:

acquiring at least one first image to be processed and at least one image processing task to be executed;

performing picture processing and/or video processing on the at least one first image to be processed to obtain at least one first processing result;

storing the at least one first processing result;

and processing the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

9. The method according to claim 8, wherein the image processing method is implemented by an image processing framework, and the construction process of the image processing framework comprises:

acquiring the at least one image processing task to be executed and a frame to be processed; the frame to be processed comprises a view processing layer, a storage layer and a task issuing layer;

under the condition that the at least one image processing task to be executed is obtained through the task issuing layer, obtaining an image processing module and/or a video processing module matched with the at least one image processing task to be executed; the picture processing module is used for realizing picture processing; the video processing module is used for realizing video processing;

and adding the picture processing module and/or the video processing module to a view processing layer of an image processing frame to be processed to obtain the image processing frame.

10. An image processing apparatus characterized by comprising:

the view access unit is used for acquiring at least one first image to be processed;

the task issuing unit is used for acquiring at least one image processing task to be executed;

the view processing unit is used for processing the at least one first image to be processed to obtain at least one first processing result;

a storage unit for storing the at least one first processing result;

and the task issuing unit is used for processing the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

11. An image processing apparatus characterized by comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring at least one first image to be processed and at least one image processing task to be executed;

the first processing unit is used for carrying out picture processing and/or video processing on the at least one first image to be processed to obtain at least one first processing result;

a storage unit for storing the at least one first processing result;

and the second processing unit is used for processing the at least one first processing result stored in the storage layer according to the at least one image processing task to be executed to obtain a second processing result.

12. An electronic device, comprising: a processor and a memory for storing computer program code comprising computer instructions which, when executed by the processor, the electronic device executes the image processing framework of any of claims 1 to 7 or the electronic device executes the image processing method of claim 8 or 9.

13. A computer-readable storage medium, in which a computer program is stored, which computer program comprises program instructions which, when executed by a processor, cause the processor to execute an image processing framework according to any one of claims 1 to 7, or cause the processor to execute an image processing method according to claim 8 or 9.

Images