TWI618031B - Image edge detection method - Google Patents

Abstract

An image edge detection method. Obtain a depth image of the scene containing the target and a color image. A first edge contour of the object located in the depth image and a second edge contour of the object located in the color image are obtained. A third edge profile is obtained from the second edge profile based on the first edge profile. One or more outer edge blocks are obtained based on the second edge contour and the third edge contour, and the second edge mask is obtained by deleting the outer edge block from the first mask generated based on the third edge contour.

Description

Image edge detection method

The present invention relates to an image processing method, and in particular to an image edge detecting method.

Most depth cameras on the market often find that the depth information obtained at the edge of the object is not accurate when acquiring the depth information of the object. For example, when a finger is opened, a click error often occurs between two fingers, as shown in FIG. This is because there are objects just right next to the two fingers, so the depth camera will treat the background and fingers as the same depth when making a judgment. In addition, the edge of the human body may also cause the edge of the body to be not well cut due to the error of the depth camera. FIG. 11 is a schematic diagram showing the results obtained by performing edge detection using only depth images in the prior art. It can be clearly seen in Fig. 11 that the silhouette of the portrait after removal of the background produces a defective block between the fingers that is misjudged as part of the silhouette of the portrait.

In general, the white edges appearing at the edge of the body or the ridges between the fingers can be filtered by taking the edge information in the image or using the erosion and expansion algorithms used in image processing. Except the white side. However, for large-area backgrounds where the fingers are misjudged, the erosion and expansion algorithm is not good for the hand block, and may even cause the image to be broken, or the fine parts such as the fingers are also removed. .

Another way to do this is to find the position of the hand through the skin color detection and eliminate the squat block between the fingers. Skin color detection is based on the RGB color of the pixels in the picture, or the picture is converted to YCbCr or HSV color space, and by setting a color range that may be skin color, to find the face, hand or other in the picture. The parts belonging to the skin color are then eliminated from the background of these blocks. The problem with this approach is that the skin color in the picture often differs depending on the light and darkness of the ambient light, the color, or the actual color of the human race and each person, resulting in a large error in finding the color range of the skin color. In addition, if you want to find the portrait block and go back in time when recording, if you use other more complicated algorithms, it will slow down the execution performance of the whole program, causing the power consumption and performance burden of the machine, and the frame rate of the movie. (frame rate) will also drop together.

The invention provides an image edge detection method, which can accurately remove the background and find the edge contour closest to the target (for example, a portrait).

The image edge detection method of the present invention comprises the following steps: obtaining a depth image of a scene including a target object and a color image. Edge detection is performed on the depth image and the color image, respectively, to obtain a first edge contour of the object located in the depth image and a second edge contour of the object located in the color image. Stacking the first edge contour and the second edge contour, sequentially extracting two adjacent pixels from the first edge contour, and taking out a normal and a second passing through one of the pixels in a connection direction of the two pixels The intersection of the edge contours intersects to form a third edge contour from the plurality of intersections. Stacking the second edge contour and the third edge contour, forming a fourth edge contour according to the overlapping region and the non-overlapping region of the second edge contour and the third edge contour, according to the at least one non-overlapping region of the second edge contour and the third edge contour Forming at least one outer edge block of the fourth edge profile, and deleting the image corresponding to the at least one outer edge block from the first mask generated based on the third edge profile to obtain the second mask.

In an embodiment of the invention, the step of obtaining the second mask further comprises: performing a back-removing action in the color image based on the first mask and the second mask.

In an embodiment of the invention, the step of performing the back-removal operation in the color image based on the first mask and the second mask comprises: performing an erosion expansion operation on the second mask to obtain a plurality of pattern blocks; The centroids of each pattern block are grouped, and the group center of each group is taken as a center point and set as a region of interest; the outer edge region corresponding to the position where the region of interest is deleted from the first mask The image of the block produces a third mask; and a background removal operation is performed in the color image based on the third mask to preserve the image of the object in the color image.

In an embodiment of the invention, the step of extracting the intersection of the normal line of one of the pixels and the intersection of the second edge contour in the direction of the connection of the two pixels further comprises: when connecting the connection direction of the two pixels When the normal of one of the pixels does not intersect with the contour of the second edge, the color image is converted into a grayscale image, and the gradient is detected in the extending direction of the corresponding normal in the grayscale image to obtain the intersection.

In an embodiment of the invention, wherein the stacking the second edge contour and the third edge contour, forming a fourth edge contour according to the at least one overlapping region of the second edge contour and the third edge contour and the at least one non-overlapping region, according to The step of forming at least one outer edge block of the fourth edge contour by the at least one non-overlapping region of the third edge contour and the third edge contour further comprises: expanding the second edge contour and the third edge contour, and then stacking has been expanded The calculated second edge contour and the third edge contour form a fourth edge contour according to the at least one overlapping region of the second edge contour and the third edge contour after the expansion operation and the at least one non-overlapping region, according to the expanded operation The second edge contour and the at least one non-overlapping region of the third edge contour form at least one outer edge block of the fourth edge contour.

In an embodiment of the invention, the step of performing the erosion expansion operation on the second mask to obtain the pattern block comprises: performing an erosion operation on the second mask first, and then performing an expansion operation.

In an embodiment of the invention, the step of obtaining the second mask from the first mask generated by the third edge contour to delete the image corresponding to the at least one outer edge block comprises: generating the second edge contour based on the third edge contour a first mask; stacking the first mask and the fourth edge contour; deleting an image of the at least one outer edge block in the first mask to obtain the second mask.

Based on the above, the edge information of the depth image and the color image is used as a basis for removing the excess background to find the boundary of the target object (for example, a portrait), thereby avoiding the problem of edge misjudgment caused by the difference between the ambient light source and the light and the individual skin color. Separate the target from the background more accurately.

The above described features and advantages of the invention will be apparent from the following description.

1 is a block diagram of an electronic device in accordance with an embodiment of the present invention. Referring to FIG. 1 , the electronic device 100 includes a processor 110 , a storage device 120 , and an imaging device 130 . The processor 110 is coupled to the storage device 120 and the camera device 130.

The processor 110 is, for example, a central processing unit (CPU), an image processing unit (GPU), a physical processing unit (PPU), a programmable microprocessor (Microprocessor), Embedded control chip, Digital Signal Processor (DSP), Application Specific Integrated Circuits (ASIC) or other similar devices.

The storage device 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, and hard memory. Disc or other similar device or a combination of these devices. A plurality of code segments are stored in the storage device 120. After being installed, the code segments are executed by the processor 110 to implement the image edge detection method described below.

The imaging device 130 is, for example, a color depth camera (RGB-D camera). In addition to color images, color depth cameras capture depth images of objects you see.

In this embodiment, the processor 110 executes the code segment in the storage device 120 to process the depth image and the color image of a scene captured by the camera 130, including a target, to find the target and The boundaries of the background, in turn, more accurately remove the background. That is, the edge information of the depth image and the color image is used as a basis for removing the excess background to replace the skin color detection to find the background and the edge of the portrait. This avoids the problem of edge misjudgment caused by ambient light source and light, and the difference in skin color of each person.

The image edge detection method will be further described below.

FIG. 2A is a flowchart of an image edge detection method according to an embodiment of the invention. Referring to FIG. 2A, in step S20, a scene including a target is captured by the imaging device 130 to obtain a depth image and a color image of the scene including the object. In this embodiment the object is for example at least one portrait.

Next, in step S21, edge detection is performed on the depth image and the color image, respectively, to obtain a first edge contour of the object located in the depth image and a second edge contour of the object located in the color image. For the sake of convenience of explanation, only one palm is illustrated for description. However, it should be noted that this embodiment performs edge detection for the entire image. For example, edge detection is performed using a Sobel algorithm, a Prewitt algorithm, or a Canny algorithm.

3 is a schematic diagram of an edge contour of a local portion in a depth image and a color image according to an embodiment of the invention. In Fig. 3, only the edge contour of the local part of the target (i.e., the palm) is shown. Here, the first edge contour of the depth image is represented by a dotted line, and the second edge contour of the color image is represented by a solid line (including a black block).

Next, in step S22, the first edge contour and the second edge contour are stacked, the adjacent two pixels are sequentially taken out from the first edge contour, and the connection direction in the two pixels is taken out through one of the two pixels. An intersection of the normal of the pixel and the intersection of the second edge contour serves as points of the third edge contour, and the intersections form a third edge contour. That is, the first edge contour of the depth image and the second edge contour of the color image are used to obtain an edge contour (third edge contour) closer to the target (ie, the portrait).

For example, Figure 4 is a schematic illustration of the removal of a third edge profile in accordance with an embodiment of the present invention. Referring to FIG. 4, it is assumed that the points A1 to A5 are pixels of the first edge contour, and the points B1 to B7 are pixels of the second edge contour. For example, to find the intersection point of the point A1 in the second contour edge, take the point A1 and the point A2 (two adjacent pixels) from the first edge contour, and connect the line according to the point A1 and the point A2. The direction obtains a straight line L, and further obtains a straight line L passing through the normal N on the point A1, and finds a point B1 on the normal line N that intersects with the second edge contour, that is, the intersection point where the normal line N intersects the second edge contour is Point B1 and point B1 as the point of the third edge contour.

Next, find the point at which point A2 corresponds to the second contour edge. As described above, the adjacent point A2 and the point A3 are taken out from the first edge contour, and the intersection point of the point A2 intersects with the second edge contour by the line A2 and the point A3 in the line connecting direction is found. The intersection of the two edge contours corresponding to the point A2, with the point as the point of the third edge contour. By analogy, the plurality of intersections are obtained to form a third edge contour.

In addition, if the point in the first edge contour overlaps the point of the second edge contour, the above steps may be omitted, and the point of the overlapping second edge contour is directly used as the point of the third edge contour. Moreover, when the normal line and the second edge contour do not intersect, the gradient detection can be used to obtain the intersection point, and more specifically, the color image is converted into a gray scale image, and corresponding to the normal line in the gray scale image. The extension direction uses gradient detection to obtain a point where the grayscale value changes rapidly as an intersection point.

Figure 5 is a schematic illustration of the first, second, and third edge profiles of a partial portion in accordance with the present invention. In FIG. 5, the first edge contour of the depth image is indicated by a dotted line, the second edge contour of the color image is indicated by a solid line (including a black block), and the third edge contour is indicated by a broken line. Since the third edge contour is obtained from the second edge contour, it can be known from FIG. 5 that the point of the third edge contour (dashed line) portion overlaps with the point of the second edge contour (solid line).

After the third edge contour is obtained, in step S23, the second edge contour and the third edge contour are stacked, and the fourth edge is formed according to the at least one overlapping region of the second edge contour and the third edge contour and the at least one non-overlapping region. And contouring, forming at least one outer edge block of the fourth edge contour according to the second edge contour and the at least one non-overlapping region of the third edge contour. In addition, in step S24, a second mask is obtained by deleting an image corresponding to the at least one outer edge block from a first mask generated based on the third edge contour.

6 is a schematic diagram of a fourth edge profile obtained by stacking a second edge profile and a third edge profile, in accordance with an embodiment of the present invention. In this embodiment, the second edge contour and the third edge contour are expanded, and the second edge contour and the third edge contour after the expanded operation are stacked, according to the second edge contour and the expanded edge Forming, by the at least one overlapping region of the three-edge contour, the fourth edge contour with the at least one non-overlapping region, forming at least one outer contour of the fourth edge contour according to the expanded second edge contour and the at least one non-overlapping region of the third edge contour Marginal block. The block 600 (ie, the block filled with the slanted line) is the block obtained after the expansion operation, and the block 600 is also the overlapping area of the second edge contour and the third edge contour after the expansion operation, so the block 600 was not judged as the outer edge block.

Figure 6 is taken as an example to illustrate how to obtain the outer edge block. First, the second edge contour of the expanded operation and the plurality of regions r1 rr10 of the third edge contour are taken out in the fourth edge contour. Next, it is judged whether or not the outermost edge contour of the regions r1 to r10 has a portion where the second edge contour overlaps with the third edge contour. Further, an area of the portion where the outermost edge contour does not have the second edge contour overlapping the third edge contour is determined as the outer edge block.

In Fig. 6, black represents the point of the second edge contour, and dark gray represents the point where only the third edge contour exists. Light gray represents the point where the second edge contour and the third edge contour exist simultaneously. Since the regions r2, r3, r4, r6, r7, r8, r9 do not have a portion where the second edge contour overlaps with the third edge contour, that is, the non-overlapping region, it is determined as the outer edge block, and the regions r5 and r10 are The outermost edge contour has a portion where the second edge contour overlaps with the third edge contour, that is, the overlapping region, and thus does not belong to the outer edge block.

In step S24, the second mask is obtained by deleting the image corresponding to the outer edge block from the first mask generated based on the third edge contour. For example, Figure 7 is a schematic illustration of a first mask in accordance with an embodiment of the present invention. Figure 8 is a schematic illustration of a second mask in accordance with an embodiment of the present invention. Figure 7 shows the first mask M1 and Figure 8 shows the second mask M2. Stacking the first mask M1 and the fourth edge contour, and deleting the regions r2, r3, r4, r6, r7, r8, r9 corresponding to the fourth edge contour shown in FIG. 6 in the first mask M1 (ie, the outer edge The image of the block can be used to obtain the second mask M2. After the second mask M2 is obtained, a backing action can be performed in the color image based on the first mask and the second mask.

In addition, in order to avoid accidentally deleting the outer edge block of different objects, the object located in the color image behind the back is caused to be missing. Therefore, the present invention further proposes how to perform edge detection more accurately, and another embodiment is given below. Description. FIG. 2B is a flowchart of an image edge detection method according to another embodiment of the present invention. Referring to FIG. 2B, in the embodiment, steps S205 to S227 are the same as steps S20 to S25 of FIG. 2A above. Therefore, please refer to the above description.

After obtaining the second mask M2, a back-removing action can be performed on the color image based on the first mask and the second mask. The detailed steps are as follows. In step S230, the second mask M2 is subjected to an erosion expansion operation. Obtain multiple pattern blocks. Here, the second mask M2 is first subjected to an erosion operation, and then the expansion operation is performed. For example, an erosion operation is used to find out what may be a part of a finger. This is because the finger belongs to a relatively elongated portion with respect to other parts of the portrait, so that after the erosion operation, a relatively large deformation occurs (for example, the area of the second mask becomes small, and even breakage occurs). Thereafter, an expansion operation is used to repair the fracture caused by the erosion operation to obtain the pattern block. Steps S225 to S230 are for finding the part of the hand to avoid misidentification of the portion having a small portrait range as the outer edge block.

Next, in step S235, the centroid of each of the pattern blocks is grouped, and the group center of each group is used as a center point and set as a region of interest (ROI). Thereafter, in step S240, the image of the outer edge block corresponding to the position of the region of interest is deleted from the first mask M1 to generate a third mask. In this embodiment, this step may be performed only for a partial part of the portrait (for example, a palm).

The following is an example to illustrate. Figure 9 is a block diagram of a block obtained by performing an erosion expansion operation in accordance with an embodiment of the present invention. 10 is a schematic diagram of a block of interest in accordance with an embodiment of the present invention.

As shown in FIG. 9, the second mask M2 obtains the pattern blocks 901 to 906 after performing the erosion expansion operation. The centroids of the pattern blocks 901 to 905 are c1 to c5. The center of mass of the pattern block 906 is not shown here. Since the above methods are all processed for the entire image, the portion of the block 906 is connected to the body, so the center of mass of the block 906 falls at a position not shown. Here, these centroids can be grouped using a k-means algorithm. Assume that the results after grouping are: centroids c1 and c2 are grouped into group 1, and centroids c3, c4, and c5 are grouped into group 2. The group center of group 1 is G1, and the group center of group 2 is G2.

Next, a region of interest of the same size is set for each group heart. As shown in FIG. 10, the corresponding region of interest R1 is set with the cluster center G1 as a center point, and the corresponding region of interest R2 is set with the cluster center G2 as a center point. After obtaining the regions of interest R1, R2, the image of the outer edge block corresponding to the position of the region of interest R1, R2 is deleted from the first mask M1, that is, the corresponding outer edge blocks r2', r3', r4' are deleted. The image of r6', r7', r8', r9' produces a third mask.

Thereafter, in step S245, the back masking operation is performed on the color image based on the third mask to preserve the image of the object in the color image.

FIG. 11 is a schematic diagram showing the results obtained by performing edge detection using only depth images in the prior art. FIG. 12 is a schematic diagram of results obtained by using an image edge detection method according to an embodiment of the invention. It will be apparent from Figure 12 that the disadvantages of the prior art are significantly improved.

In summary, the depth information of the depth camera is used to find the block mask of the entire portrait, and then the edge detection of the color image is performed to find a more similar image edge, and the outer edge block is found according to the two kinds of information. position. Then, the edge contour of the portrait is eroded. In this process, the portion of the finger is easily cut off by the erosion operation, thereby finding the detail of the portrait or the part of the finger. When the finger cut portion is obtained, the information block of the finger cut portion is found by using the information, and the center point of each pattern block is calculated, and then each center point is subgrouped, and After finding each group heart position, set a ROI block in this group. Finally, it is determined that the outer edge blocks found by the edge information of the depth information and the color image are in the ROI, and the image of the outer edge block falling in the ROI is correspondingly deleted in the color image. Try to remove the plaques that belong to the background between your fingers. According to this, the edge misjudgment caused by the difference of the ambient light source, the light, and the individual skin color can be avoided, and the background can be more accurately separated from the target object (for example, a portrait).

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

100‧‧‧Electronic devices

110‧‧‧ processor

120‧‧‧Storage device

130‧‧‧ camera

S20~S25‧‧‧Image edge detection method steps

S205~S245‧‧‧Image edge detection method steps

600, 901~906‧‧‧ blocks

A1~A5, B1~B8‧‧‧ points

C1~c5‧‧‧ centroid

G1, G2‧‧‧ group heart

L‧‧‧ Straight line

M1‧‧‧ first mask

M2‧‧‧ second mask

N‧‧‧ normal

R1~r10‧‧‧ area

R1, R2‧‧‧ Areas of Interest

1 is a block diagram of an electronic device in accordance with an embodiment of the present invention. 2A and 2B are flowcharts of an image edge detection method according to various embodiments of the present invention. 3 is a schematic diagram of an edge contour of a local portion in a depth image and a color image according to an embodiment of the invention. 4 is a schematic illustration of the removal of a third edge profile in accordance with an embodiment of the present invention. Figure 5 is a schematic illustration of the first, second, and third edge profiles of a stacked partial portion in accordance with the present invention. 6 is a schematic diagram of a fourth edge profile obtained by stacking a second edge profile and a third edge profile, in accordance with an embodiment of the present invention. Figure 7 is a schematic illustration of a first mask in accordance with an embodiment of the present invention. Figure 8 is a schematic illustration of a second mask in accordance with an embodiment of the present invention. FIG. 9 is a block diagram of a block obtained by performing an erosion expansion operation of a second mask according to an embodiment of the invention. 10 is a schematic diagram of a block of interest in accordance with an embodiment of the present invention. FIG. 11 is a schematic diagram showing the results obtained by performing edge detection using only depth images in the prior art. FIG. 12 is a schematic diagram of results obtained by using an image edge detection method according to an embodiment of the invention.

Claims (6)

An image edge detection method includes: obtaining a depth image of a scene including a target object and a color image; performing edge detection on the depth image and the color image respectively to obtain the target object located in the depth image a first edge contour and a second edge contour of the object located in the color image; stacking the first edge contour and the second edge contour, sequentially extracting adjacent two from the first edge contour a pixel, and extracting an intersection point in a direction of a line connecting the two pixels through a normal of one of the pixels and the second edge contour, the intersections forming a third edge contour; when located in the two pixels When the normal direction of one of the pixels does not intersect with the second edge contour, the color image is converted into a grayscale image, and the gradient direction is used for the extension direction of the corresponding normal line in the grayscale image. Obtaining the intersection; stacking the second edge contour and the third edge contour according to at least one overlapping region of the second edge contour and the third edge contour and at least The non-overlapping region forms a fourth edge contour; forming at least one outer edge block of the fourth edge contour according to the second edge contour and the at least one non-overlapping region of the third edge contour; and from the third edge based on the third edge And deleting a corresponding image of the at least one outer edge block in a first mask generated by the contour to obtain a second mask. For example, the image edge detection method described in claim 1 is The step of obtaining the second mask further includes: performing a back-back operation in the color image based on the first mask and the second mask. The image edge detection method of claim 2, wherein the step of performing the back-off operation in the color image based on the first mask and the second mask comprises: performing the second mask Obscuring the expansion operation to obtain a plurality of pattern blocks; by grouping the centroids of the plurality of pattern blocks, taking the group center of each group as a center point and setting it as a region of interest; Deleting the image of the outer edge block corresponding to the position of the region of interest in the first mask to generate a third mask; and performing the back motion in the color image based on the third mask, The image of the target in the color image is retained. The image edge detection method of claim 1, wherein the second edge contour and the third edge contour are stacked according to the at least one overlapping region of the second edge contour and the third edge contour Forming, by the at least one non-overlapping region, the fourth edge contour, the step of forming the at least one outer edge block of the fourth edge contour according to the second edge contour and the at least one non-overlapping region of the third edge contour further comprises: Performing an expansion operation on the second edge contour and the third edge contour, and stacking the second edge contour and the third edge contour after the expansion operation, according to the second edge contour and the third after the expanded operation The at least one overlapping area of the edge contour Forming the fourth edge contour with the at least one non-overlapping region, forming the at least one outer edge of the fourth edge contour according to the second edge contour after the expanded operation and the at least one non-overlapping region of the third edge contour Block. The image edge detection method of claim 3, wherein the step of performing the erosion expansion operation on the second mask to obtain the pattern blocks comprises: first performing an erosion operation on the second mask, and then Perform the expansion operation. The image edge detection method of claim 1, wherein the second mask is obtained by deleting an image corresponding to the at least one outer edge block from the first mask generated by the third edge contour. The step of: forming the first mask based on the third edge contour; stacking the first mask and the fourth edge contour; and deleting an image of the first mask corresponding to at least one outer edge block, Obtain the second mask.