CN111539992A - Image processing method, device, electronic device and storage medium - Google Patents

Abstract

The embodiment of the disclosure discloses an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a plurality of frame images; performing limb key point detection processing on a target object in a first image in the multi-frame images to obtain first key point information corresponding to part of limbs of the target object; determining second key point information corresponding to the part of the limb of the target object in a second image based on the first key point information; the first image is any one of the multi-frame images; the second image is a frame of image after the first image.

Description

Image processing method, device, electronic device and storage medium

technical field

The present disclosure relates to the technical field of computer vision, and in particular, to an image processing method, apparatus, electronic device and storage medium.

Background technique

The target tracking technology is usually based on the limb detection algorithm and the limb key point detection algorithm, using the human body detected by the limb detection algorithm and the key points of the human body detected by the limb key point detection algorithm to achieve target tracking. However, the current limb detection algorithm and limb key point detection algorithm cannot adapt to the scene with only the upper body limbs, so that the target with only the upper body limbs cannot be tracked.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide an image processing method, apparatus, electronic device, and storage medium.

An embodiment of the present disclosure provides an image processing method, the method includes: obtaining multiple frames of images; performing limb key point detection processing on a target object in a first image in the multiple frame images, and obtaining an image of the target object. first key point information corresponding to a part of the limb; determining second key point information corresponding to the part of the limb of the target object in the second image based on the first key point information; wherein the first image is the any one frame of images in the multiple frames of images; the second image is a frame of images after the first image.

In the above solution, performing limb key point detection processing on the target object in the first image to obtain first key point information corresponding to part of the limbs of the target object includes: detecting the target object in the first image The object is subjected to limb detection processing to determine the first region of the target object; the first region includes the region where part of the limbs of the target object are located; the pixel points corresponding to the first region are subjected to limb key point detection processing to obtain The first key point information corresponding to the part of the limb of the target object.

In the above solution, the determining, based on the first key point information, the second key point information corresponding to the part of the limb of the target object in the second image includes: based on the first key point information, in the A second area is determined in the first image; the second area is larger than the first area of the target object; the first area includes the area where part of the body of the target object is located; according to the second area or the first area a key point information, determine a third area in the second image corresponding to the position range of the second area; perform limb key point detection processing on the pixels in the third area in the second image , to obtain the second key point information corresponding to the part of the limb.

In the above solution, the performing limb detection processing on the target object in the first image includes: using a limb detection network to perform limb detection processing on the target object in the first image; wherein, the The limb detection network is obtained by training the first-type sample images; the first-type sample images are marked with a detection frame of the target object; the marked range of the detection frame includes the region where part of the target object's limbs are located.

In the above solution, performing limb key point detection processing on the pixels corresponding to the first area includes: using a limb key point detection network to perform limb key point detection processing on the pixels corresponding to the first area; wherein, The limb key point detection network is obtained by training the second type of sample images; the second type of sample images are marked with key points including part of the limbs of the target object.

In the above solution, the partial limbs of the target object include at least one of the following: head, neck, shoulder, chest, waist, hip, arm, hand; the first key point information and the second The key point information includes contour key point information and/or bone key point information of at least one limb among the head, neck, shoulder, chest, waist, hip, arm, and hand.

In the above solution, the method further includes: in response to obtaining the first key point information corresponding to part of the limbs of the target object, assigning a tracking identifier to the target object; The number of the assigned tracking identifiers determines the number of target objects in the multi-frame images.

In the above solution, the method further includes: determining the gesture of the target object based on the second key point information; determining an interaction instruction corresponding to the target object based on the gesture of the target object.

The embodiment of the present disclosure further provides an image processing apparatus, the apparatus includes: an acquisition unit, a detection unit, and a tracking determination unit; wherein, the acquisition unit is used for acquiring multiple frames of images; the detection unit is used for Performing limb key point detection processing on the target object in the first image of the multi-frame images to obtain first key point information corresponding to part of the limbs of the target object; The key point information determines the second key point information corresponding to the part of the limb of the target object in the second image; wherein, the first image is any one frame of images in the multi-frame images; the second image The image is a frame of image after the first image.

In the above solution, the detection unit includes: a limb detection module and a limb key point detection module; wherein, the limb detection module is configured to perform limb detection processing on the target object in the first image, and determine the The first area of the target object; the first area includes the area where part of the limbs of the target object are located; the limb key point detection module is configured to perform limb key point detection processing on the pixel points corresponding to the first area, Obtain first key point information corresponding to the part of the limb of the target object.

In the above solution, the tracking determination unit is configured to determine a second area in the first image based on the first key point information; the second area is larger than the first area of the target object; the first area An area includes the area where part of the limb of the target object is located; according to the second area or the first key point information, determine a third area in the second image corresponding to the position range of the second area; Perform limb key point detection processing on the pixels in the third area in the second image to obtain second key point information corresponding to the part of the limb.

In the above solution, the limb detection module is used to perform limb detection processing on the target object in the first image by using a limb detection network; wherein, the limb detection network is obtained by training the first type of sample images; A detection frame of the target object is marked in the sample image of the first type; the marked range of the detection frame includes the area where part of the limb of the target object is located.

In the above solution, the limb key point detection module is configured to use the limb key point detection network to perform limb key point detection processing on the pixel points corresponding to the first area; wherein, the limb key point detection network adopts the second type. The sample images are obtained through training; the second type of sample images are marked with key points including part of the limbs of the target object.

In the above solution, the partial limbs of the target object include at least one of the following: head, neck, shoulder, chest, waist, hip, arm, hand; the first key point information and the second The key point information includes contour key point information and/or bone key point information of at least one limb among the head, neck, shoulder, chest, waist, hip, arm, and hand.

In the above solution, the device further includes an allocation unit and a statistics unit; wherein, the allocation unit is configured to respond to the situation that the detection unit obtains the first key point information corresponding to the partial limbs of the target object. The target object is assigned a tracking identifier; the statistics unit is configured to determine the number of target objects in the multi-frame image based on the number of the tracking identifiers assigned during the processing of the multi-frame image.

In the above solution, the apparatus further includes a determination unit, configured to determine the gesture of the target object based on the second key point information; and determine the interaction instruction corresponding to the target object based on the gesture of the target object.

The embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, implements the steps of the image processing method described in the embodiments of the present disclosure.

Embodiments of the present disclosure also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the methods described in the embodiments of the present disclosure when the processor executes the program. The steps of an image processing method.

The image processing method, device, electronic device, and storage medium provided by the embodiments of the present disclosure identify the key points of the partial limbs of the target object in the first image of the multi-frame images to be processed, and identify the partial limbs based on the identified partial limbs. The key points of the target object determine the key points of the partial limbs of the target object in the second image, so as to realize the target tracking in the scene with the partial limbs (eg upper body) of the target object in the image.

Description of drawings

FIG. 1 is a schematic flowchart 1 of an image processing method according to an embodiment of the present disclosure;

2 is a schematic flowchart of a method for detecting and processing limb key points in an image processing method according to an embodiment of the present disclosure;

3 is a schematic flowchart of a method for tracking limb key points in an image processing method according to an embodiment of the present disclosure;

4 is a second schematic flowchart of an image processing method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram 1 of a composition structure of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 6 is a second schematic diagram of the composition and structure of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram 3 of a composition structure of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 8 is a fourth schematic diagram of the composition and structure of an image processing apparatus according to an embodiment of the disclosure;

FIG. 9 is a schematic structural diagram of a hardware composition of an electronic device according to an embodiment of the disclosure.

Detailed ways

The present disclosure will be described in further detail below with reference to the accompanying drawings and specific embodiments.

In the following description, for purposes of illustration and not limitation, specific details such as specific system structures, interfaces, techniques, etc. are set forth in order to provide a thorough understanding of the present application.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship. Also, "multiple" herein means two or more than two.

Embodiments of the present disclosure provide an image processing method. FIG. 1 is a schematic flowchart 1 of an image processing method according to an embodiment of the present disclosure; as shown in FIG. 1 , the method includes:

Step 101: obtain multiple frames of images;

Step 102: Perform limb key point detection processing on the target object in the first image in the multi-frame images to obtain first key point information corresponding to part of the limbs of the target object;

Step 103: Determine the second key point information corresponding to the part of the limb of the target object in the second image based on the first key point information; wherein, the first image is any one frame of images in the multi-frame images; the second image is the first image A frame of image after the image.

The image processing method in this embodiment can be applied to an image processing apparatus, and the image processing apparatus can be set in an electronic device with processing functions such as a personal computer, a server, or the like, or implemented by a processor executing a computer program.

In this embodiment, the above-mentioned multi-frame images may be continuous videos collected by a built-in or external camera device of the electronic device, or may also be received videos transmitted by other electronic devices, or the like. In some application scenarios, the above-mentioned multi-frame images may be surveillance videos collected by surveillance cameras, so as to track each target object in the surveillance videos. In other application scenarios, the above-mentioned multi-frame images may also be videos stored locally or in other video libraries, so as to track each target object in the video. In still other application scenarios, the image processing method of this embodiment may be applied to application scenarios such as virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR), or somatosensory games; the above-mentioned multi-frame images may also be The image of the operator collected in the virtual reality or augmented reality scene can control the action of the virtual object in the virtual reality scene or the augmented reality scene by recognizing the gesture of the operator in the image; The collected images of target objects (such as multiple users) participating in the game, etc.

In some application scenarios, the image processing apparatus may establish a communication connection with one or more surveillance cameras, and obtain surveillance video captured by the surveillance cameras in real time as multi-frame images to be processed. In other application scenarios, the image processing apparatus can also obtain video from the video stored by itself as the multi-frame image to be processed, or can also obtain the video from the video stored by other electronic devices as the multi-frame image to be processed, etc. Wait. In still other application scenarios, the image processing device can also be placed in the game device, and the output and displayed image is taken as the multi-frame image to be processed during the execution of the computer program by the processor of the game device to realize the operation of the game operator. The target object in the image (the target object corresponds to the game operator) is tracked.

In this embodiment, the multi-frame images to be processed may include target objects, and the target objects may be one or more; in some application scenarios, the target objects may be real people; in other application scenarios, the target objects may also be It is other objects determined according to the actual tracking needs, such as avatars or other virtual objects.

In this embodiment, each frame of images in the multi-frame images may be called a frame image, which is the smallest unit that constitutes a video (ie, an image to be processed). The acquisition time of the frame images forms the above-mentioned multi-frame images, and the time parameters corresponding to each frame image are continuous.

Exemplarily, taking the target object as a real person as an example, in the case where the target object is included in the multi-frame image, the time range corresponding to the multi-frame image may include one or more target objects, or the multi-frame image may include one or more target objects. A part of the time range within the time range includes one or more target objects, which is not limited in this embodiment.

In this embodiment, the above-mentioned first image is any one frame of images among the multiple frames of images, and the second image is an image of one frame after the first image. Wherein, in some optional embodiments, the second image may be a subsequent frame of images temporally continuous with the first image. For example, the multi-frame images include 10 frames of images, the above-mentioned first image is the second frame of images in the 10-frame images, and the above-mentioned second image is the third frame of images. In some other optional embodiments, the second image may also be a frame of images after the first image and separated from the first image by a preset number of frames of images. For example, the multi-frame images include 20 frames of images, and the above-mentioned first image is the second frame of images in the 20 frames of images. Assuming that the preset number of frames of images is 3 frames of images, the above-mentioned second image may be the 6th image in the 20 frames of images. frame image. Wherein, the above-mentioned preset number can be preset according to the actual situation, for example, the preset number can be preset according to the moving speed of the target object. This embodiment can effectively reduce the amount of data processing, thereby reducing the consumption of the image processing device.

In this embodiment, the image processing apparatus may perform limb key point detection processing on the target object in the first image through the limb key point detection network, and obtain first key point information corresponding to some limbs of the target object. In this embodiment, the partial limbs of the target object include at least one of the following: a head, a neck, a shoulder, a chest, a waist, a hip, an arm, and a hand. Correspondingly, the first key point information and the second key point information corresponding to part of the limbs of the target object include at least one limb among the head, neck, shoulder, chest, waist, hip, arm, and hand of the target object. The contour key point information and/or bone key point information of .

Exemplarily, part of the limbs of the target object in this embodiment are the upper body limbs of the target object, so that the target object with the upper body in the multi-frame images can be identified, so as to realize the tracking of the target object with only the upper body or the whole body.

Exemplarily, the key points corresponding to the first key point information and the second key point information may include: at least one key point of the head, at least one key point of the shoulder, at least one key point of the arm, and at least one key point of the chest. Key points, at least one key point of the hip, and at least one key point of the waist; optionally, the key points corresponding to the first key point information and the second key point information may also include at least one key point of the hand. Whether the image processing device can obtain the key points of the hand depends on whether the key points of the hand are marked in the sample image used for training the limb key point detection network; in the case where the key points of the hand are marked in the sample image, Then the key points of the hand can be detected by the limb key point detection network.

In some optional embodiments, when part of the limbs of the target object includes the head, the first key point information and the second key point information may include key point information of at least one organ, and key point information of at least one organ The information may include at least one of the following: nose key point information, eyebrow key point information, and mouth key point information.

In some optional embodiments, in the case that the part of the limb of the target object includes an arm, the first key point information and the second key point information may include elbow key point information.

In some optional embodiments, when part of the limbs of the target object include hands, the first key point information and the second key point information may include wrist key point information. Optionally, the first key point information and the second key point information may further include contour key point information of the hand.

In some optional embodiments, when part of the limbs of the target object includes a hip, the first key point information and the second key point information may include left hip key point information and right hip key point information. Optionally, the first key point information and the second key point information may further include spine root key point information.

The above-mentioned first key point information may specifically include the coordinates of the key points. The above-mentioned first key point information may include the coordinates of contour key points and/or the coordinates of bone key points. It can be understood that the contour edges of the corresponding partial limbs can be formed by the coordinates of the contour key points; the bones of the corresponding partial limbs can be formed by the coordinates of the bone key points.

FIG. 2 is a schematic flowchart of a method for detecting and processing limb key points in an image processing method according to an embodiment of the present disclosure; in some optional embodiments, step 102 may refer to FIG. 2 , including:

Step 1021: Perform limb detection processing on the target object in the first image to determine a first region of the target object; the first region includes the region where part of the limbs of the target object are located;

Step 1022 : Perform limb key point detection processing on the pixels corresponding to the first area to obtain first key point information corresponding to part of the limbs of the target object.

In this embodiment, limb detection is first performed on each target object in the first image, and the first area of each target object is determined. For example, the first area corresponding to the upper body of each target object or the first area corresponding to the whole body of each target object can be determined. an area. In practical applications, the first region corresponding to a part of the limb may be represented by a detection frame (eg, a rectangular frame) that identifies the target object, for example, the upper body of each person in the first image is identified by each rectangular frame.

In some optional embodiments, performing limb detection processing on the target object in the first image includes: using a limb detection network to perform limb detection processing on the target object in the first image; wherein the limb detection network uses the first limb detection network. One class of sample images is obtained by training; the first class of sample images is marked with a detection frame of the target object; the labeling range of the detection frame includes the area where part of the target object's limbs are located; and the part of the target object's limbs may be the upper body limbs of the target object.

In this embodiment, a pre-trained limb detection network can be used to perform limb detection on the first image to determine the first region of the target object, that is, to obtain detection frames of each target object in the first image. The above-mentioned detection frame can identify part or all of the limbs of the target object, that is, all limbs or upper body limbs of the target object can be detected and obtained through the limb detection network. The above-mentioned limb detection network may adopt any network structure capable of detecting limbs of the target object, which is not limited in this embodiment.

Exemplarily, taking the detection frame of the partial limbs of the target object detected by the limb detection network as an example, the limb detection network can be used to perform feature extraction on the first image, and the characteristics of each target object in the first image can be determined based on the extracted features. The center point of the partial limb and the height and width of the detection frame corresponding to the partial limb of each target object. Based on the center point and the corresponding height and width of the partial limb of each target object, the detection frame of the partial limb of each target object can be determined. .

In this embodiment, the limb detection network can be obtained by training the first type of sample images marked with the detection frame of the target object; wherein, the marked range of the detection frame includes part of the limbs of the target object. It can be understood that the first type of sample images may be The detection frame marked only with part of the limbs of the target object (for example, the upper body limbs of the target object) may also be marked with the detection frame of the complete limbs of the target object. Exemplarily, taking the labeling range of the detection frame as the part of the limb of the target object as an example, the limb detection network can be used to extract the feature data of the first type of sample images, and based on the feature data, the partial limbs of each target object in the first type of sample images can be determined. Based on the predicted center point of the partial limbs and the height and width of the predicted detection frame of the corresponding part of the limb, the predicted detection frame corresponding to each partial limb is determined based on the predicted center point of the above-mentioned partial limb and the corresponding height and width; According to the predicted detection frame and the marked part of the limb The detection frame determines the loss, and the network parameters of the limb detection network are adjusted based on the loss.

In some optional embodiments, performing limb key point detection processing on the pixels corresponding to the first area includes: using a limb key point detection network to perform limb key point detection processing on the pixels corresponding to the first area; wherein, The above-mentioned limb key point detection network is obtained by training the second type of sample images; the second type of sample images are marked with key points of the target object; the marked range of the above-mentioned key points includes part of the limbs of the target object.

In this embodiment, a pre-trained limb key point detection network may be used to perform limb key point detection on the pixel points corresponding to the first area to determine the first key point information of part of the limbs of each target object. Exemplarily, the above-mentioned first area may include part of the limbs of the target object, and the pixel points corresponding to the detection frame of each target object may be input into the limb key point detection network to obtain the first key point information corresponding to the partial limbs of each target object. . The above-mentioned limb key point detection network may adopt any network structure capable of detecting limb key points, which is not limited in this embodiment.

In this embodiment, the limb key point detection network can be obtained by training the second type of sample images marked with the key points of the target object, wherein the marked range of the key points includes part of the limbs of the target object. Only the key points of part of the limbs of the target object (for example, the upper body limbs of the target object) can be marked, or the key points of the complete limbs of the target object can be marked. Exemplarily, taking the key points of part of the limbs of the target object marked in the second type of sample image as an example, the limb key point detection network can be used to extract the feature data of the second type of sample image, and the second type of sample image can be determined based on the feature data. The prediction key points of some limbs of each target object in the above; determine the loss based on the above predicted key points and the marked key points, and adjust the network parameters of the limb key point detection network based on the loss.

FIG. 3 is a schematic flowchart of a method for tracking limb key points in an image processing method according to an embodiment of the present disclosure; in some optional embodiments, step 103 may refer to FIG. 3 , and the method includes:

Step 1031: Determine a second area in the first image based on the first key point information; the second area is larger than the first area of the target object; the first area includes the area where part of the limbs of the target object are located;

Step 1032: Determine a third area in the second image corresponding to the position range of the second area according to the second area or the first key point information;

Step 1033: Perform limb key point detection processing on the pixels in the third area in the second image to obtain second key point information corresponding to some limbs.

In this embodiment, for a target object in the first image, an area is determined based on the first key point information of the partial limb of the target object, and the area may be the smallest area including all the key points of the partial limb of the target object . Exemplarily, if the area is a rectangular area, the rectangular area is the smallest area including all the key points of the partial limbs of the target object. Then the above-mentioned second area is an area obtained by enlarging the first area in the first image.

Exemplarily, if the first area is a rectangle as an example, assuming that the height of the first area is H and the width is W, the center point of the area can be the center, and the four sides of the area can be extended in the direction away from the center point, For example, in the height direction, extending H/4 in the direction away from the center point, and in the width direction, extending W/4 in the direction away from the center point, the second area can pass through the center in the first image. The point is the center, and it is represented by a rectangular area with a height of 3H/2 and a width of 3W/2.

In this embodiment, the third region in the second image corresponding to the above-mentioned position range may be determined according to the position range of the second region in the first image, or the position range of the first key point information in the first image. .

In some optional embodiments, determining a third area in the second image corresponding to the position range of the second area according to the second area may further include: performing limb key point detection processing on the pixels corresponding to the second area, Obtaining third key point information; determining a position range of the third key point information in the first image, and determining a third area in the second image corresponding to the position range based on the position range.

Exemplarily, in this embodiment, the limb key point detection network is still used to perform limb key point detection processing on the pixels corresponding to the second area, and the pixels corresponding to the enlarged second area in the first image may be used as The input data of the limb key point detection network outputs third key point information, and the third key point information is used as the predicted key point information of the target object in the second image, that is, the embodiment of the present application uses The area where the target object is located is enlarged (for example, the area where part of the body of the target object is located in the previous frame image is enlarged), and the key points obtained are used as the current frame image by performing limb key point detection on the enlarged area. The predicted key points corresponding to the target object (eg, part of the limbs of the target object) in a frame of images (ie, the second image) following the first image (ie, the first image). Further based on the predicted position range, limb key point detection processing is performed on the pixels corresponding to the third area in the second image, and the detected key point information is the second key point information corresponding to the partial limbs of the target object.

In other optional embodiments, step 103 may further include: based on the first image, the first area of the target object, and the target tracking network, determining the predicted area of the target object in the second image, based on the above-mentioned Perform limb key point detection processing on the pixels in the prediction area to obtain second key point information corresponding to part of the limbs of the target object; wherein, the target tracking network is obtained by training multiple frames of sample images; the multi-frame sample images include at least the first sample The image and the second sample image, the second sample image is a frame image after the first sample image; the position of the target object is marked in the first sample image, and the position of the target object is marked in the second sample image. Exemplarily, a detection frame of the target object is marked in the multi-frame sample images, and the position of the target object in the sample image is represented by the detection frame; the labeling range of the detection frame includes the area where part of the limb of the target object is located; part of the limb of the target object Can be the upper body limb of the target subject.

In this embodiment, the position of the target object in the previous frame of image (ie, the first image) and the target object in the image can be used to determine the position of the target object in the next frame of image (ie, the second image) through a pre-trained target tracking network. Predicted location. Exemplarily, the first image containing the detection frame of the target object can be input into the target tracking network to obtain the predicted position of the target object in the second image; The key point detection process obtains second key point information of part of the limbs of the target object in the second image. The above target tracking network may adopt any network structure capable of implementing target tracking, which is not limited in this embodiment.

In this embodiment, the target tracking network can be obtained by training multiple frames of sample images marked with the position of the target object (for example, a detection frame including the target object, or a detection frame including a part of a limb of the target object). Exemplarily, taking a multi-frame sample image including at least a first image and a second image as an example, the target tracking network can be used to process the first sample image, and the position of the target object is marked in the first sample image. The result is the predicted position of the target object in the second sample image; then the loss can be determined according to the predicted position and the marked position of the target object in the second image, and the network parameters of the target tracking network can be adjusted based on the loss.

It should be noted that, after determining the second key point information corresponding to the partial limb of the target object in the second image based on the first key point information, the second key point corresponding to the partial limb of the target object in the second image may be used. The information further determines the key point information corresponding to the part of the limb of the target object in the subsequent image, and so on, until the key point information corresponding to the part of the limb of the target object cannot be detected in the next frame of image. The above-mentioned target object is not included in the processed multi-frame image, that is, the target object has moved out of the field of view of the multi-frame image to be processed.

In some optional embodiments, the image processing apparatus may also perform limb detection on the target object in each frame of image, to obtain the region where the target object is located in each frame of image. The detected target object is used as the tracking object, so as to determine whether a new target object appears in the current frame image; if a new target object appears in the current frame image, the new target object is used as the tracking object, and the new target object is used as the tracking object. The pixel points in the first area corresponding to the target object are subjected to limb key point detection processing, that is, the processing of step 103 in the embodiment of the present disclosure is performed for a new target object. Exemplarily, the image processing apparatus may perform the limb detection processing of the target object in the image every preset time or every preset number of image frames, so as to detect whether a new target object appears in the image at regular intervals, A new target object is tracked.

In some optional embodiments of the present disclosure, the above method further includes: in response to obtaining the first key point information corresponding to part of the limbs of the target object, assigning a tracking identifier to the target object; The number of assigned tracking identities determines the number of target objects in the multi-frame image.

In this embodiment, when the image processing device detects the target object in the first frame of the multi-frame images to be processed, that is, when obtaining the first key point information corresponding to part of the limbs of the target object, a tracking identifier is allocated to the target object, The tracking identifier is associated with the target object until the target object cannot be tracked during the process of tracking the target object.

In some optional embodiments, the image processing apparatus may also perform limb detection on the target object in each frame of image, obtain the area corresponding to part of the limb of the target object in each frame of image, and use the detected target object as the tracking object. Based on this, the image processing apparatus detects the first frame of the image to be processed, and assigns a tracking identifier to the detected target object. After that, the tracking identifier follows the target object until the target object cannot be tracked. If a new target object is detected in a certain frame of image, a tracking identifier is assigned to the new target object, and the above scheme is repeated. It can be understood that each target object detected at the same time corresponds to different tracking identifiers; the target objects tracked in a continuous time range correspond to the same tracking identifier; the target objects detected respectively in a non-continuous time range Corresponds to different trace identifiers.

For example, if three target objects are detected in a certain frame of image, respectively, a tracking mark is allocated to the three target objects, and each target object corresponds to a tracking mark respectively.

For another example, for a 5-minute multi-frame image, three target objects are detected in the first 1 minute, and a tracking identification is assigned to the three target objects, for example, it can be recorded as identification 1, identification 2 and identification 3; In the second 1 minute, the first target object among the above three target objects disappears, then in the current 1 minute, there are only two target objects, and the corresponding tracking IDs are ID 2 and ID 3; Within 1 minute, the above-mentioned first target object appeared in the image again, that is, compared with the previous image, a new target object was detected, although the target object was the target object that appeared in the first 1 minute (i.e. The first target object), the target object is still assigned identifier 4 as the tracking identifier, and so on.

Based on this, the technical solution of this embodiment can determine the number of target objects that have appeared in the multi-frame images based on the number of corresponding tracking marks in the multi-frame image processing process. Exemplarily, the number of target objects that appear in the multi-frame images refers to the number of times the target objects appear in the time range corresponding to the multi-frame images.

By adopting the technical solutions of the embodiments of the present disclosure, the key points of the partial limbs of the target object in the first image in the multi-frame images to be processed are identified, and the following second limbs are determined based on the identified key points of the partial limbs. The key points of the partial limbs of the target object in the image, so as to realize the target tracking in the scene with only partial limbs (eg upper body) of the target object in the image, that is, the technical solutions of the embodiments of the present disclosure can simultaneously adapt to the complete limbs Scenes and partial limb (eg upper body) scenes, enabling target tracking in images.

The embodiments of the present disclosure also provide an image processing method. FIG. 4 is a second schematic flowchart of an image processing method according to an embodiment of the present disclosure; as shown in FIG. 4 , the method includes:

Step 201: obtain multiple frames of images;

Step 202: Perform limb key point detection processing on the target object in the first image in the multi-frame images, and obtain first key point information corresponding to part of the limbs of the target object;

Step 203: Determine the second key point information corresponding to the part of the limb of the target object in the second image based on the first key point information; wherein, the first image is any one frame of images in the multi-frame images; the second image is the first image A frame of image after the image;

Step 204: Determine the gesture of the target object based on the second key point information; determine the interaction instruction corresponding to the target object based on the gesture of the target object.

For a specific description of steps 201 to 203 in this embodiment, reference may be made to the descriptions of steps 101 to 103, and details are not repeated here.

In this embodiment, the gesture of the target object can be determined based on the tracked target object and then based on the second key point information of the target object, and the interaction instruction corresponding to each gesture can be determined based on the gesture of the target object. After that, respond to the interaction instructions corresponding to each gesture.

This embodiment is applicable to action interaction scenarios. The image processing apparatus can determine the corresponding interaction instruction based on each gesture, and respond to the above interaction instruction; in response to the above interaction instruction, for example, the image processing apparatus itself or the electronic device where the image processing apparatus is located can be turned on or off. Some functions of itself; or, in response to the above-mentioned interactive instructions, the above-mentioned interactive instructions can also be sent to other electronic devices, and other electronic devices receive the above-mentioned interactive instructions, and turn on or off certain functions based on the interactive instructions. In other words, the above-mentioned interactive instructions The interactive instructions can also be used to turn on or turn off corresponding functions of other electronic devices.

This embodiment is also applicable to various application scenarios such as virtual reality, augmented reality, or somatosensory games. The image processing device can perform corresponding processing based on various interactive instructions, including but not limited to controlling virtual reality or augmented reality scenes to perform corresponding actions on virtual objects; controlling somatosensory game scenes to perform corresponding actions on virtual characters corresponding to target objects. Actions. In some examples, if the method is applied to scenes such as augmented reality or virtual reality, the corresponding processing performed by the image processing apparatus based on the interaction instruction may include controlling the virtual target object to perform actions corresponding to the interaction instruction in the real scene or the virtual scene.

By adopting the technical solutions of the embodiments of the present disclosure, on the one hand, target tracking in a scene where only part of the limbs (such as the upper body) of the target object is in the image is realized, that is, the technical solutions of the embodiments of the present disclosure can be adapted to the scene of the complete limbs and the scene of the entire body at the same time. Some limbs (such as the upper body) scene, realize the target tracking in the image; on the other hand, during the target tracking process, the key point information of the tracked target object is detected, and the tracked target object is determined based on the key point information of the target object Based on the gesture of the target object, the corresponding interaction instruction is determined, which realizes the human-computer interaction in specific application scenarios (such as virtual reality scenarios, augmented reality scenarios, somatosensory game scenarios and other interactive scenarios) and improves the user's interactive experience.

Embodiments of the present disclosure also provide an image processing apparatus. FIG. 5 is a schematic diagram 1 of the composition structure of an image processing apparatus according to an embodiment of the present disclosure; as shown in FIG. 5 , the apparatus includes: an acquisition unit 31 , a detection unit 32 and a tracking determination unit 33 ; wherein,

The above-mentioned obtaining unit 31 is used to obtain multiple frames of images;

The above-mentioned detection unit 32 is configured to perform limb key point detection processing on the target object in the first image in the multi-frame images, and obtain first key point information corresponding to part of the limbs of the above-mentioned target object;

The above-mentioned tracking determination unit 33 is configured to determine, based on the above-mentioned first key point information, second key point information corresponding to the above-mentioned part of the limb of the above-mentioned target object in the second image; wherein, the above-mentioned first image is any of the above-mentioned multiple frames of images. A frame of image; the second image is a frame of image after the first image.

In some optional embodiments of the present disclosure, as shown in FIG. 6 , the foregoing detection unit 32 includes: a limb detection module 321 and a limb key point detection module 322; wherein,

The above-mentioned limb detection module 321 is used to perform limb detection processing on the target object in the above-mentioned first image, and determine the first area of the target object; the first area includes the area where part of the limbs of the target object are located;

The limb key point detection module 322 is configured to perform limb key point detection processing on the pixel points corresponding to the first region to obtain first key point information corresponding to the partial limb of the target object.

In some optional embodiments of the present disclosure, the above-mentioned tracking determination unit 33 is configured to determine a second area in the first image based on the above-mentioned first key point information; the above-mentioned second area is larger than the above-mentioned first area of the target object; An area includes the area where part of the limb of the target object is located; according to the second area or the first key point information, determine the third area in the second image corresponding to the position range of the second area; Perform limb key point detection processing on the pixel points of the above-mentioned parts, and obtain the second key point information corresponding to the above-mentioned part of the limbs.

In some optional embodiments of the present disclosure, the above-mentioned limb detection module 321 is configured to use a limb detection network to perform limb detection processing on the above-mentioned target object in the above-mentioned first image; wherein, the above-mentioned limb detection network adopts the first type of sample images Training is obtained; the first type of sample image is marked with a detection frame of the target object; the labelled range of the detection frame includes the area where part of the limbs of the target object are located.

In some optional embodiments of the present disclosure, the above-mentioned limb key point detection module 322 is configured to use a limb key point detection network to perform limb key point detection processing on the pixels corresponding to the above-mentioned first area; wherein, the above-mentioned limb key point detection The network is obtained by training the sample images of the second type; the above-mentioned sample images of the second type are marked with key points including part of the limbs of the target object.

In some optional embodiments of the present disclosure, the partial limbs of the target object include at least one of the following: head, neck, shoulder, chest, waist, hip, arm, and hand; the first key point information And the above-mentioned second key point information includes contour key point information and/or bone key point information of at least one limb among the head, neck, shoulder, chest, waist, hip, arm and hand.

In some optional embodiments of the present disclosure, as shown in FIG. 7 , the above-mentioned apparatus further includes: an allocation unit 34 and a statistics unit 35; wherein,

The above-mentioned allocation unit 34 is configured to allocate a tracking identifier to the target object in response to the situation in which the above-mentioned detection unit 32 obtains the first key point information corresponding to part of the limbs of the target object;

The above-mentioned statistical unit 35 is configured to determine the number of target objects in the multi-frame images based on the number of tracking identifiers allocated in the process of processing the multi-frame images.

In some optional embodiments of the present disclosure, as shown in FIG. 8 , the above-mentioned apparatus further includes a determination unit 36, configured to determine the posture of the target object based on the second key point information; determine the posture corresponding to the target object based on the posture of the target object interactive instructions.

In the embodiment of the present disclosure, the acquisition unit 31 , the detection unit 32 (including the limb detection module 321 and the limb key point detection module 322 ), the tracking determination unit 33 , the allocation unit 34 , the statistics unit 35 and the determination unit 36 in the above-mentioned image processing apparatus , in practical applications, it can be composed of a central processing unit (CPU, Central Processing Unit), a digital signal processor (DSP, Digital Signal Processor), a microcontroller unit (MCU, Microcontroller Unit) or a programmable gate array (FPGA, Field-Programmable). Gate Array) implementation.

It should be noted that: when the image processing apparatus provided in the above-mentioned embodiments performs image processing, only the division of the above-mentioned program modules is used as an example for illustration. That is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the image processing apparatus and the image processing method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.

Embodiments of the present disclosure also provide an electronic device. FIG. 9 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the disclosure; as shown in FIG. 9 , the electronic device 40 may include a memory 42 , a processor 41 and a computer program stored in the memory 42 and running on the processor 41 , when the processor 41 executes the program, the steps of the image processing method in the embodiment of the present disclosure are implemented.

It will be appreciated that the various components in electronic device 40 may be coupled together by bus system 43 . It is understood that the bus system 43 is used to implement the connection communication between these components. In addition to the data bus, the bus system 43 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the various buses are designated as bus system 43 in FIG. 9 .

It will be appreciated that the memory 42 may be either volatile memory or non-volatile memory, and may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-only memory) Only Memory), Electrically Erasable Programmable Read-Only Memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), Magnetic Random Access Memory (FRAM, ferromagnetic random access memory), Flash Memory (Flash Memory), Magnetic Surface Memory , CD-ROM, or Compact Disc Read-Only Memory (CD-ROM, Compact Disc Read-Only Memory); the magnetic surface memory can be a magnetic disk memory or a tape memory. The volatile memory may be Random Access Memory (RAM), which is used as an external cache memory. By way of example and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory Memory (DRAM, Dynamic Random Access Memory), Synchronous Dynamic Random Access Memory (SDRAM, SynchronousDynamic Random Access Memory), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), Synchronous Link Dynamic Random Access Memory (SLDRAM, SyncLink Dynamic Random Access Memory), Direct Memory Bus Random Access Memory (DRRAM, Direct Rambus Random Access Memory) . The memory 42 described in the embodiments of the present disclosure is intended to include, but not be limited to, these and any other suitable types of memory.

The methods disclosed in the above embodiments of the present disclosure may be applied to the processor 41 or implemented by the processor 41 . The processor 41 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by a hardware integrated logic circuit in the processor 41 or an instruction in the form of software. The above-mentioned processor 41 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The processor 41 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present disclosure. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the methods disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, and the storage medium is located in the memory 42, and the processor 41 reads the information in the memory 42, and completes the steps of the foregoing method in combination with its hardware.

In an exemplary embodiment, the electronic device 40 may be implemented by one or more of Application Specific Integrated Circuit (ASIC, Application Specific Integrated Circuit), DSP, Programmable Logic Device (PLD, Programmable Logic Device), Complex Programmable Logic Device (CPLD, Complex Programmable Logic Device), FPGA, general-purpose processor, controller, MCU, microprocessor (Microprocessor), or other electronic components implemented for performing the aforementioned method.

In an exemplary embodiment, the embodiment of the present disclosure further provides a computer-readable storage medium, such as a memory 42 including a computer program, and the computer program can be executed by the processor 41 of the electronic device 40 to complete the steps of the aforementioned method. . The computer-readable storage medium can be memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one or any combination of the above-mentioned memories, such as Mobile phones, computers, tablet devices, personal digital assistants, etc.

The embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, implements the steps of the image processing methods described in the embodiments of the present disclosure.

The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined under the condition of no conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain a new product embodiment.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be all integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute It includes the steps of the above method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk and other media that can store program codes.

Alternatively, if the above-mentioned integrated units of the present disclosure are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present disclosure essentially or the parts that make contributions to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk and other mediums that can store program codes.

The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited to this. should be included within the scope of protection of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims (18)

1. an image processing method, is characterized in that, described method comprises: Get multiple frames of images; Performing limb key point detection processing on the target object in the first image of the multi-frame images to obtain first key point information corresponding to part of the limbs of the target object; The second key point information corresponding to the part of the limb of the target object in the second image is determined based on the first key point information; wherein, the first image is any one frame of images in the multiple frames of images ; the second image is a frame of image after the first image. 2 . The method according to claim 1 , wherein, performing limb key point detection processing on the target object in the first image to obtain first key point information corresponding to part of the limbs of the target object, comprising: 2 . Perform limb detection processing on the target object in the first image, and determine a first area of the target object; the first area includes the area where part of the limbs of the target object are located; Perform limb key point detection processing on the pixel points corresponding to the first area to obtain first key point information corresponding to the part of the limb of the target object. 3 . The method according to claim 1 , wherein the determining, based on the first key point information, the second key point information corresponding to the part of the limb of the target object in the second image comprises: 3 . A second area is determined in the first image based on the first key point information; the second area is larger than the first area of the target object; the first area includes the area where part of the target object's limbs are located ; determining a third area in the second image corresponding to the position range of the second area according to the second area or the first key point information; Perform limb key point detection processing on the pixels in the third area in the second image to obtain second key point information corresponding to the part of the limb. 4. The method according to claim 2 or 3, wherein the performing limb detection processing on the target object in the first image comprises: Use a limb detection network to perform limb detection processing on the target object in the first image; The limb detection network is obtained by training the first type of sample images; the first type of sample images are marked with a detection frame of the target object; the marked range of the detection frame includes the area where part of the limbs of the target object are located. 5. The method according to claim 2 or 3, characterized in that, performing limb key point detection processing on the pixels corresponding to the first area, comprising: Use the limb key point detection network to perform limb key point detection processing on the pixels corresponding to the first area; Wherein, the limb key point detection network is obtained by training the second type of sample images; the second type of sample images are marked with key points including part of the limbs of the target object. 6. The method according to any one of claims 1 to 5, wherein the partial limbs of the target object comprise at least one of the following: head, neck, shoulder, chest, waist, hip, arm ,hand; The first key point information and the second key point information include contour key point information and/or bone key information of at least one limb in the head, neck, shoulder, chest, waist, hip, arm and hand. point information. 7. The method according to any one of claims 1 to 6, wherein the method further comprises: In response to obtaining the first key point information corresponding to the partial limbs of the target object, assigning a tracking identifier to the target object; The number of target objects in the multi-frame images is determined based on the number of the tracking identifiers allocated during the processing of the multi-frame images. 8. The method according to any one of claims 1 to 7, wherein the method further comprises: determining the pose of the target object based on the second key point information; An interaction instruction corresponding to the target object is determined based on the gesture of the target object. 9. An image processing device, characterized in that the device comprises: an acquisition unit, a detection unit, and a tracking determination unit; wherein, The acquisition unit is used to acquire multiple frames of images; The detection unit is configured to perform limb key point detection processing on the target object in the first image of the multi-frame images, and obtain first key point information corresponding to part of the limbs of the target object; The tracking determination unit is configured to determine, based on the first key point information, second key point information corresponding to the part of the limb of the target object in the second image; wherein the first image is the multiple Any frame image in the frame images; the second image is a frame image after the first image. 10. The device according to claim 9, wherein the detection unit comprises: a limb detection module and a limb key point detection module; wherein, The limb detection module is configured to perform limb detection processing on the target object in the first image, and determine a first region of the target object; the first region includes the region where part of the limbs of the target object are located ; The limb key point detection module is configured to perform limb key point detection processing on the pixel points corresponding to the first area to obtain first key point information corresponding to the part of the limb of the target object. 11. The apparatus according to claim 9, wherein the tracking determination unit is configured to determine a second area in the first image based on the first key point information; the second area is larger than the the first area of the target object; the first area includes the area where part of the body of the target object is located; according to the second area or the first key point information, determine the difference between the second image and the first The third area corresponding to the position range of the second area; performing limb key point detection processing on the pixel points in the third area in the second image to obtain second key point information corresponding to the part of the limb. 12. The device according to claim 10 or 11, wherein the limb detection module is configured to perform limb detection processing on the target object in the first image by using a limb detection network; The limb detection network is obtained by training the first type of sample images; the first type of sample images are marked with a detection frame of the target object; the marked range of the detection frame includes the area where part of the limbs of the target object are located. 13. The device according to claim 10 or 11, wherein the limb key point detection module is used to perform limb key point detection processing on the pixels corresponding to the first area by using a limb key point detection network; Wherein, the limb key point detection network is obtained by training the second type of sample images; the second type of sample images are marked with key points including part of the limbs of the target object. 14. The device according to any one of claims 9 to 13, wherein part of the limbs of the target object comprises at least one of the following: head, neck, shoulder, chest, waist, hip, arm ,hand; The first key point information and the second key point information include contour key point information and/or bone key information of at least one limb in the head, neck, shoulder, chest, waist, hip, arm and hand. point information. 15. The device according to any one of claims 9 to 14, wherein the device further comprises an allocation unit and a statistics unit; wherein, the assigning unit, configured to assign a tracking identifier to the target object in response to the detection unit obtaining the first key point information corresponding to part of the limbs of the target object; The statistics unit is configured to determine the number of target objects in the multi-frame images based on the number of the tracking identifiers allocated during the processing of the multi-frame images. 16. The apparatus according to any one of claims 9 to 15, wherein the apparatus further comprises a determination unit, configured to determine the pose of the target object based on the second key point information; based on the target The pose determination of the object corresponds to the interaction instruction of the target object. 17. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented. 18. An electronic device comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements any one of claims 1 to 8 when executing the program the steps of the method.

Images