TWI850858B - Object recognition method of three-dimensional space and computing apparatus - Google Patents

Abstract

An object recognition method of three-dimensional (3D) space and a computing apparatus are provided. In the method, multiple sensing points in a 3D space are allocated to multiple areas. The 3D space is established by the sensing points generated by scanning a space. Multiple two-dimensional (2D) images in each area are captured. The 2D images of each area are recognized. One or more objects in the 3D space are determined according to the recognized result of those 2D images of the area. Accordingly, the object in the 3D space could be recognized.

Description

Three-dimensional space object recognition method and computing device

The present invention relates to an object detection technology, and in particular to a three-dimensional object recognition method and computing device.

In order to simulate the real space, the real space can be scanned to generate a simulated space that looks like the real space. The simulated space can be realized in applications such as games, home decoration, robot movement, etc. It is worth noting that although two-dimensional image recognition technology is widely used today, it is difficult for two-dimensional image recognition to fully understand the object recognition and labeling of three-dimensional space.

In view of this, the present invention provides a method and a computing device for object recognition in three-dimensional space, which converts three-dimensional space into two-dimensional images, and realizes object recognition in three-dimensional space by means of two-dimensional image recognition.

The object recognition method in three-dimensional space of the embodiment of the present invention includes (but is not limited to) the following steps: assigning multiple sensing points in three-dimensional space to multiple regions. The three-dimensional space is established by multiple sensing points generated by scanning space. Capturing multiple two-dimensional images of each region. Recognizing those two-dimensional images of each region. Determining one or more objects in the three-dimensional space based on the recognition results of those two-dimensional images of those regions.

The computing device of the embodiment of the present invention includes a memory and a processor. The memory is used to store program codes. The processor is coupled to the memory. The processor loads the program codes to execute: assigning multiple sensing points in the three-dimensional space to multiple regions, capturing multiple two-dimensional images of each region, identifying those two-dimensional images of each region, and determining one or more objects in the three-dimensional space based on the identification results of those two-dimensional images of those regions. This three-dimensional space is established by multiple sensing points generated by scanning the space.

Based on the above, according to the method and computing device for object recognition in three-dimensional space of the embodiment of the present invention, the three-dimensional space is initially divided into multiple regions, and then two-dimensional images are captured for each region, and objects in the three-dimensional space are recognized based on the recognition results of the two-dimensional images. This helps to recognize and understand objects in three-dimensional space.

In order to make the above features and advantages of the present invention more clearly understood, embodiments are specifically cited below and described in detail with reference to the accompanying drawings.

FIG1 is a block diagram of a computing device 10 according to an embodiment of the present invention. Referring to FIG1 , the computing device 10 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a server, or an intelligent assistant device. The computing device 10 includes (but is not limited to) a memory 11 and a processor 12.

The memory 11 can be any type of fixed or removable random access memory (RAM), read only memory (ROM), flash memory, traditional hard disk drive (HDD), solid-state drive (SSD) or similar components. In one embodiment, the memory 11 is used to store program code, software module, data (e.g., sensing point, location information, color information, recognition result or three-dimensional model) or file, and its details will be described in detail in the subsequent embodiments.

The processor 12 is coupled to the memory 11. The processor 12 may be a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), programmable controller, application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC) or other similar components or a combination of the above components. In one embodiment, the processor 12 is used to execute all or part of the operations of the computing device 10, and can load and execute the program code, software module, file and/or data stored in the memory 11. In one embodiment, the processor 12 executes all or part of the operations of the embodiment of the present invention. In some embodiments, the software modules or program codes stored in the memory 11 may also be implemented by physical circuits.

Hereinafter, the method described in the embodiment of the present invention will be described in conjunction with various components in the computing device 10. The various processes of the method can be adjusted according to the implementation situation, and are not limited thereto.

FIG2 is a flow chart of a method for object recognition in three-dimensional space according to an embodiment of the present invention. Referring to FIG2 , the processor 12 allocates a plurality of sensing points in the three-dimensional space to a plurality of regions (step S210). Specifically, the three-dimensional space is established by one or more sensing points generated by scanning the (real or virtual) space. The sensing points can reflect the existence of objects in the three-dimensional space. For example, an optical, radar or acoustic scanning signal is reflected by an object to generate an echo, and the echo can be used to determine the position, depth and/or direction relative to the object. In one embodiment, the three-dimensional space is a point cloud image formed by the sensing points. In other embodiments, the three-dimensional space may also be in other model formats. In addition, depending on different application scenarios, the object may be furniture, home appliances, plants, processing equipment or decorations, or it may be a wall, ceiling or floor, but is not limited thereto.

Region allocation is used to group sensing points. Sensing points in the same group/region have similar features. Features are, for example, related to position, color, or other image features (e.g., rectangle, edge, or corner). Generally speaking, sensing points with similar features are more likely to belong to the same object.

In one embodiment, the processor 12 may group the sensing points in a multidimensional space according to the position information of the sensing points in the three-dimensional space and the color information in the color space to determine the regions. Specifically, the multidimensional space includes dimensions of both the three-dimensional space and the color space. The dimensions in the three-dimensional space are, for example, three axes perpendicular to each other in the space. The dimensions in the color space are, for example, RGB (red, green, and blue), CMYK (cyan, magenta, yellow, and black), or HSV (hue, saturation, and value).

The distance between the position information and color information of the sensing points in each area in the corresponding space is less than the distance threshold. In other words, this distance threshold is used to evaluate whether the position information and/or color information between multiple sensing points are similar. In response to the distance between the position information and color information of two sensing points in the corresponding space being less than the distance threshold, the processor 12 may assign the two sensing points to the same group/area. In response to the distance between the position information and color information of the two sensing points in the corresponding space being not less than the distance threshold, the processor 12 may assign the two sensing points to different groups/areas. The position information may be the coordinates of a three-dimensional space corresponding coordinate system. For example, the distance threshold is a distance of 5 centimeters in three-dimensional space. The color information may be a primary color, a standard color, or the intensity of an attribute. For example, the distance threshold may be a 3rd order intensity in the color space. However, the actual value of the distance threshold still needs to be defined according to actual needs, and the embodiment of the present invention is not limited thereto.

In one embodiment, the processor 12 may utilize a k-means algorithm, a Gaussian mixture model (GMM), a mean-shift algorithm, a hierarchical clustering method, a spectral clustering algorithm, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, or other clustering/clustering algorithms, and set distance parameters (e.g., the aforementioned distance threshold or parameters related to the distance threshold) and point number (or the minimum number of points in a group/region) parameters.

For example, FIG3A is a schematic diagram of sensing points S1 to S8 in a three-dimensional space TS according to an embodiment of the present invention. Referring to FIG3A , it is assumed that after scanning the space, a three-dimensional space TS including eight sensing points S1 to S8 is formed, and the position information of the sensing points S1 to S8 can be defined by the distance relative to the three axes X, Y, and Z. FIG3B is a schematic diagram of area determination according to an embodiment of the present invention. Referring to FIG3B , it is assumed that the distance of sensing points S1 to S3 is within 5 centimeters and all are black, and the distance of sensing points S4 to S8 is within 8 centimeters and all are red. Therefore, the processor 12 assigns sensing points S1 to S3 to area A1, and assigns sensing points S4 to S8 to area A2. The shape of the region is, for example, a geometric three-dimensional shape or a concrete three-dimensional shape formed by the center of gravity of the sensing points in the same region, for example, a sphere, a cube, or an olive sphere. However, the shape of the region may also be an irregular three-dimensional shape or a shape defined by a clustering algorithm.

In other embodiments, if more other features are referenced, a multi-dimensional space with more dimensions may be formed. In addition, the number and position of the sensing points S1-S8 shown in FIG3A and FIG3B are only used for example description, and the number and position of the sensing points still need to be determined according to the actual application scenario.

2 , the processor 12 captures a plurality of two-dimensional images of each region (step S220 ). Specifically, the processor 12 sets the virtual camera at a plurality of viewing positions in the three-dimensional space and shoots toward each region to capture those two-dimensional images.

In one embodiment, the processor 12 determines a reference axis for a certain area. The reference axis is, for example, an imaginary line that passes through the center of the area and is perpendicular to the ground (or parallel to the direction of gravity). However, the angle of the reference axis relative to the ground can still be changed as needed. The processor 12 can circle the area with this reference axis as the axis and at intervals of a certain distance (for example, 10, 15 or 20 centimeters, but the distance may also be dynamically adjusted according to the shape of the area), and capture multiple two-dimensional images corresponding to multiple capture directions toward the area through a virtual camera. For example, the processor 12 defines a capture direction every 20 degrees. Therefore, the two-dimensional image is captured through the virtual camera every 20 degrees of rotation according to the reference axis. However, the capture direction can still be changed according to actual needs.

For example, Fig. 4 is a schematic diagram of image capture according to an embodiment of the present invention. Referring to Fig. 4, according to reference axis RS, a two-dimensional image IM1 can be captured in the capture direction CD1; a two-dimensional image IM2 can be captured in the capture direction CD2; and a two-dimensional image IM3 can be captured in the capture direction CD3.

It should be noted that the image capture is not limited to capturing in a circular manner around the axis, and the application user can determine the capture method according to actual needs.

Referring to FIG. 2 , the processor 12 identifies the two-dimensional images of each region (step S230 ). Specifically, the processor 12 may identify the type of object in each two-dimensional image based on a neural network algorithm (e.g., YOLO (You only look once), Region Based Convolutional Neural Networks (R-CNN), or Fast R-CNN) or a feature matching algorithm (e.g., Histogram of Oriented Gradient (HOG), Scale-Invariant Feature Transform (SIFT), Harr, or Speeded Up Robust Features (SURF) feature matching). In one embodiment, the recognition result of the two-dimensional image includes the type of the object. In one embodiment, the recognition result of the two-dimensional image includes a probability of similarity to one or more object types.

Referring to FIG. 2 , the processor 12 determines one or more objects in the three-dimensional space based on the recognition results of the two-dimensional images of those regions (step S240). Specifically, if the recognition results of more two-dimensional images in the same region are the same, it means that the possibility of the existence of an object of a specific object type obtained by the recognition result in this region is higher. In one embodiment, the processor 12 may determine one or more objects located in the first region of those regions based on the recognition results of the two-dimensional images of the first region in multiple capture directions. For example, if the recognition results of more than a certain number of two-dimensional images in a certain region are the same, the processor 12 determines that the object in the recognition result exists in this region.

In addition, if the recognition results of more two-dimensional images in adjacent areas are the same, it means that the possibility of objects of the specific object type obtained by the recognition results existing in these areas is higher.

In one embodiment, the objects obtained from the recognition result include a first object and a second object. In response to the first object and the second object being detected by at least two two-dimensional images in multiple capture directions of the first area, the processor 12 can determine the probability that the first object and the second object in the first area are the same. For example, a classifier based on a neural network determines the similarity probability of the first object and the second object, or the proportion of the same image features as a specific object. In other words, in the same area, if two or more two-dimensional images detect two objects, the processor 12 will further determine whether the two objects are the same.

The processor 12 compares the probability with the probability threshold to obtain a comparison result. The probability threshold can be updated based on a machine learning algorithm. The comparison result is, for example, that the probability is greater than the probability threshold and that the probability is not greater than the probability threshold.

Then, the processor 12 can determine that the first object and the second object are the same according to the comparison result. If the probability is greater than the probability threshold, the processor 12 determines that the first object and the second object are the same. If the probability is not greater than the probability threshold, the processor 12 determines that the first object and the second object are different. For example, the processor 12 detects the table top and the four table legs from the four two-dimensional images respectively, so the processor 12 can determine that the table top and the four table legs are the same. That is, the table.

In summary, in the three-dimensional space object recognition method and computing device of the present invention, images are captured in the three-dimensional space to obtain two-dimensional images in different capture directions, and objects in the three-dimensional space are determined based on the recognition results of the two-dimensional images. In this way, objects in the three-dimensional space can be understood through two-dimensional image recognition technology.

Although the present invention has been disclosed as above by the embodiments, they are not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the scope of the attached patent application.

10: Computing device 11: Memory 12: Processor S210~S240: Steps TS: Three-dimensional space S1~S8: Sensing points A1, A2: Area X, Y, Z: Axes RS: Reference axis CD1~CD3: Capture direction IM1~IM3: Two-dimensional image

FIG. 1 is a block diagram of a computing device according to an embodiment of the present invention. FIG. 2 is a flow chart of a method for object recognition in a three-dimensional space according to an embodiment of the present invention. FIG. 3A is a schematic diagram of sensing points in a three-dimensional space according to an embodiment of the present invention. FIG. 3B is a schematic diagram of region determination according to an embodiment of the present invention. FIG. 4 is a schematic diagram of image capture according to an embodiment of the present invention.

S210~S240: Steps

Claims (8)

A method for object recognition in three-dimensional space includes: generating a three-dimensional space based on a plurality of sensing points generated by scanning a space through a processor, wherein the three-dimensional space is in the form of a three-dimensional model; grouping the sensing points in the three-dimensional space according to their positions in the three-dimensional space and color information in a color space through the processor, so as to allocate the sensing points to a plurality of regions according to a grouping result, wherein each sensing point determines its position information through an echo generated by an object reflected by a scanning signal, the color information is the intensity in the color space, and the grouping result is the distance between its position information and its color information in the corresponding space. Two sensing points less than a distance threshold are allocated to an area, and each area has a three-dimensional shape in the three-dimensional space; a virtual camera is set in the three-dimensional space and photographed toward each area by the processor to capture multiple two-dimensional images of each area, including: determining a reference axis of the area; and circling around the reference axis and capturing the two-dimensional images corresponding to multiple capture directions toward the area by the virtual camera; identifying the two-dimensional images of each area by the processor; and determining at least one object in the three-dimensional space by the processor according to the recognition results of the two-dimensional images of the areas. The object recognition method for three-dimensional space as described in claim 1, wherein the step of assigning the sensing points in the three-dimensional space to the regions includes: grouping the sensing points in a multi-dimensional space according to the position information of the sensing points in the three-dimensional space and the color information in the color space to determine the regions, wherein the multi-dimensional space includes the dimensions of both the three-dimensional space and the color space, and the distance between the position information and the color information of the sensing points in each region in the corresponding space is less than a distance threshold value. The object recognition method for three-dimensional space as described in claim 1, wherein the step of determining the at least one object in the three-dimensional space based on the recognition results of the two-dimensional images of the regions includes: determining the at least one object located in a first region among the regions based on the recognition results of the two-dimensional images in the capture directions. The object recognition method for three-dimensional space as described in claim 3, wherein the at least one object includes a first object and a second object, and the step of determining the at least one object in the three-dimensional space based on the recognition results of the two-dimensional images of the regions includes: detecting the first object and the second object based on the two-dimensional images of at least two of the capture directions of the first region, determining a probability that the first object and the second object in the first region are the same; comparing the probability with a probability threshold to obtain a comparison result; and judging that the first object and the second object are the same based on the comparison result. A computing device includes: a memory for storing a program code; and a processor coupled to the memory and loaded with the program code to execute: generating a three-dimensional space based on a plurality of sensing points generated by scanning a space, wherein the three-dimensional space is in the form of a three-dimensional model; grouping the sensing points in the three-dimensional space according to their positions in the three-dimensional space and color information in a color space, so as to allocate the sensing points to a plurality of regions according to a grouping result, wherein each sensing point determines its position information through an echo generated by an object reflected by a scanning signal, the color information is the intensity in the color space, and the grouping result is its position information. and the distance between the two sensing points and their color information in the corresponding space is less than a distance threshold, and each of the regions has a three-dimensional shape in the three-dimensional space; a virtual camera is set in the three-dimensional space and shoots toward each of the regions to capture multiple two-dimensional images of each of the regions, wherein the processor is further used to: determine a reference axis of the region; and capture the two-dimensional images corresponding to multiple capture directions toward the region with the virtual camera around the reference axis; identify the two-dimensional images of each of the regions; and determine at least one object in the three-dimensional space according to the recognition results of the two-dimensional images of the regions. The computing device as described in claim 5, wherein the processor is further used to: group the sensing points in a multi-dimensional space according to the position information of the sensing points in the three-dimensional space and the color information in the color space to determine the regions, wherein the multi-dimensional space includes the dimensions of both the three-dimensional space and the color space, and the distance between the position information and the color information of the sensing points in each region in the corresponding space is less than a distance threshold value. The computing device as described in claim 5, wherein the processor is further used to: determine the at least one object located in a first area among the areas according to the recognition result of the two-dimensional image in the capture directions of the first area. The computing device as described in claim 7, wherein the at least one object includes a first object and a second object, and the processor is further used to: detect the first object and the second object based on the two-dimensional images of at least two of the capture directions of the first area, determine a probability that the first object and the second object in the first area are the same; compare the probability with a probability threshold to obtain a comparison result; and determine that the first object and the second object are the same according to the comparison result.

Images