JPH06174839A - Method for generating image of three-dimensional object - Google Patents

Abstract

(57) [Abstract] [Purpose] It enables the recognition of three-dimensional objects with excellent resolution both vertically and horizontally. [Structure] Ultrasonic sensors 11 ₁ to ₁ 1 arranged in a plane
The ultrasonic sensor measures the planar distribution of the distance information to the surface of the object facing these sensors, which is composed of the sensor array 11 of 1n and the television camera 12. Then, a first stereoscopic image of the object is extracted based on the distance information, and a plane image of the object is extracted from the video image by the television camera 12 based on the position information of the first stereoscopic image. And a first stereoscopic image are combined to generate a fine second stereoscopic image. This makes it possible to supplement an ultrasonic image having poor lateral resolution with a video image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object recognition method,
Particularly, the present invention relates to a method for generating a three-dimensional object image with improved resolution by using a distance sensor and a video camera together.

[0002]

2. Description of the Related Art The applicant of the present invention has two types for recognizing a three-dimensional object.
A method using a three-dimensionally arranged ultrasonic sensor and a neural network has already been proposed in Japanese Patent Application No. 3-246558. By this method, it is possible to recognize a three-dimensional object whose front, rear, left, right, and top positions are free even in an outdoor environment in which brightness changes and shadows occur. In this method, as shown in FIG.
As shown in FIG. 3, a plurality of ultrasonic sensors 11 _{1 to} 11n as distance sensors are two-dimensionally arranged to form an ultrasonic sensor array 11, and an ultrasonic imaging method using these sensors is used as an object. The distance distribution to the surface and the background of the object 10 is obtained, the object shape is extracted from the background based on the distance information, and the depth information (measurement data) 19 of the object is obtained.
Layered neural network using error backpropagation 1
3 is input to recognize a three-dimensional object.

Reference numeral a in FIG. 17 indicates each ultrasonic sensor 1.
1 ₁ ～11N distance information to the surface of the opposing object 10, the reference numeral b represents the distance to the reference surface which is placed the object 10, the distance b between the object surface to the reference plane which is placed the object 10 The depth information of the object is obtained from the difference in the distance a up to. The neural network 13 is composed of an input layer 14, an intermediate layer 15 and an output layer 16. Since this method uses a neural network, there is also an advantage that a complicated procedure for extracting feature amounts having different shapes for various objects is unnecessary.

[0004]

However, such a problem
In the method of recognizing a three-dimensional object image, a pulse delay time method using an ultrasonic sensor is used to measure three-dimensional information of the object, and therefore, an excellent resolution of about wavelength can be obtained in the depth direction, but Since the resolution is poor, there is a drawback that it is difficult to recognize a small object. An object of the present invention is to provide a method for solving this problem, which enables recognition of a three-dimensional object having excellent resolution in both the vertical and horizontal directions.

[0005]

To achieve the above object, the present invention provides a plurality of distance sensors arranged in a plane so as to face an object and a video camera for picking up the image of the object. The plane distribution of the distance information to the surface of the object is measured by the distance sensor, the first stereoscopic image of the object is extracted based on the distance information, and the video by the video camera based on the position information of the first stereoscopic image. A plane image of an object is extracted from an image, and the extracted plane image and the first stereoscopic image are combined to generate a fine second stereoscopic image.

[0006]

According to the present invention, a distance sensor and a video camera are used together, and an object image is extracted from the video image based on the distance information from the distance sensor, so that a stereoscopic image having poor lateral resolution is complemented by the video image. It becomes possible to do.

[0007]

EXAMPLES Before describing the examples of the present invention, the outline of the present invention will be described. In order to recognize a three-dimensional object, a shape measuring unit and a shape recognizing unit based on the measured information are required. In the present invention, the object recognition is the final object, but in order to improve the recognition accuracy, it is necessary to improve the measurement accuracy of the object shape. Therefore, a shape measuring means capable of generating a fine three-dimensional object image as compared with the case of using only the conventional ultrasonic sensor is considered, and the outline of each means will be described.

"Shape recognition means": As a recognition means, a layered or perceptron type neural network using the same back propagation method as in Japanese Patent Application No. 3-246558 is used. The structure is shown in FIG. As described above, the neural network 13 is composed of the input layer 14, the intermediate layer 15, and the output layer 16. The intermediate layer 15 does not necessarily have to be one layer. This neural network 1
3, the signal input to the input layer 14 corresponds to the output layer 1
6 can be trained to output a constant signal. Once learning is performed, the learned signal is output for an input signal that is substantially the same as the learned input signal. Therefore, a complicated procedure for extracting the feature amount from the shape information is unnecessary.

"Shape measuring means": The distance to the object surface can be directly measured by irradiating the surface of the object with ultrasonic waves and measuring the time required for reflection from the object surface. Therefore, by arranging a plurality of ultrasonic sensors 11 ₁ to 11 n two-dimensionally as shown in FIG. 3 to form the ultrasonic sensor array 11,
It is possible to obtain the distance information to the surface of the object 10 facing each of the sensors 11 ₁ to 11 n. Further, the depth information of the object can be obtained from the difference between the distance b to the reference plane on which the object 10 is placed and the distance a to the surface of the object (see FIG. 17). With this pulse delay time method, measurement with high distance resolution can be performed in the depth direction.

However, in this method, even if the ultrasonic sensor used has a directivity angle of ± 5 degrees or less, the lateral resolution is limited to about the sensor diameter. To compensate for this drawback, a video camera such as a CCD camera, that is, a small television camera 12 is provided in the sensor device 17 including the ultrasonic sensor array 11. The television camera 12 has a very high lateral resolution. However, the distance information cannot be obtained by the TV camera alone. Moreover, it is difficult to extract shape information from the background based on the difference in brightness in an environment where brightness changes or shadows occur, for example, outdoors. However, if the shape information extracted by the ultrasonic method is used, it is possible to extract the shape which is difficult to obtain by the television camera image alone. This mechanism will be described in detail later. If the video image of the object can be extracted, it is possible to obtain the shape information having much better lateral resolution than the ultrasonic image. Therefore,
By using this shape information together with the previously obtained distance information having excellent resolution in the depth direction, a highly accurate three-dimensional object image can be obtained.

Next, examples of the present invention will be described. FIG. 3 shows the configurations of the ultrasonic sensor array and the television camera according to this embodiment. This is an ultrasonic sensor (MA200 manufactured by Murata Manufacturing Co., Ltd.) for both transmission and reception with a frequency of 200 kHz.
A1) A flat sensor device 17 is formed by arranging a total of 64 pieces of 11 ₁ to 11 n vertically and horizontally at 30 mm intervals (6 pieces are simply shown in the figure) to form an ultrasonic sensor array 11. Make up. Then, for example, a CCD camera as a small television camera 12 is fixed to the center of the sensor array 11. Incidentally, the ultrasonic sensor 11 ₁ ~11n and the television camera 12 as long as the relative positions do not change, the positional relationship is free. Even with highly directional ultrasonic sensor 11 ₁ ~11n of 200kHz, 10 mm approximately equal to the outer diameter of the sensor is the limit of the lateral resolution of the ultrasonic sensor alone. Reference numeral 18 in FIG. 3 is a light projector for illuminating the object 10, and symbols X, Y and Z represent respective three-dimensional axes.

FIG. 1 shows a block diagram of a shape measuring system according to this embodiment. In this embodiment, the ultrasonic sensor array 11 is used.
64 ultrasonic sensors 11 ₁ for both transmission and reception that constitute the
11n is arbitrarily selected based on a command from the personal computer (personal computer) 30, and is switched by the switches 21 _{1 to} 21n of the switch unit 21 corresponding to the 11n. Then, among the selected ultrasonic sensors, the ultrasonic sensor for three wavelengths accompanying the synchronous signal (frequency 200 kHz) of the synchronous oscillation circuit 22 is transmitted from the transmission sensor to the object, and the counter 27 is synchronized with the signal. To drive. Further, the pulse of the received reflected wave is amplified by the amplifier 24, and this pulse is input to the gate 26 of the counter 27 to stop the counter 27.

As a result, the counter 27 counts the clock pulses sent from the crystal oscillation circuit 25 through the gate 26 during that period, outputs the count value to the personal computer 30 via the interface 29, and at the same time, the count value is predetermined. The distance to the object can be measured by multiplying by the proportional constant, and the depth information of the object can be obtained based on the distance information. In this case, in such a configuration, each ultrasonic sensor 11 ₁ is used by using a single distance measuring circuit.
~ 11n is switched. In addition, all the ultrasonic sensors are used for both transmission and reception, and not only the transmitted sensor receives the reflected wave but also the signal transmitted by the adjacent sensor, so that the 8 × 8 sensor has 15 × 15 points. Can be measured. This is Japanese Patent Application No. 2-407217 relating to the same applicant.
This is accomplished using the method previously filed in the issue.

The television camera image is C used for it.
Although it depends on the CD camera 12, for example, a vertical 512, a horizontal 48
It is composed of 8 bits of 5 pixels, that is, brightness information of 256 gradations. Then, this information is converted into digital information by the A / D converter 31 and then output to the personal computer 30 to extract a three-dimensional object from the television camera image based on ultrasonic information described later. There is. In FIG. 1, reference numeral 23 is a power amplifier for transmission, 28 is a control logic for controlling ON / OFF of each of the switches 21 ₁ to 21 n of the switch unit 21, 32 is a control of the diaphragm of the CCD camera 12 by a command from the personal computer 30. It is an aperture controller for doing.

Here, FIG. 4 shows an example of an ultrasonic image 41 of a cylinder having a diameter and a height of 6 cm. In the experiment, the cylinder was left standing on the desk, and the sensor device 1 of FIG.
7 was fixed and the ultrasonic sensor array 11 photographed. The measurement area 42 on the desk is a square with a side of 21 cm. By subtracting the distance to the desk and the distance to the surface of the cube, the horizontal distribution of depth information can be obtained. It is also a feature of the method of this embodiment that it has a very excellent resolution of 2 mm in the depth direction.
Further, it is also a great feature of the method of this embodiment that shape information whose location, size and shape are unknown can be extracted from the background by using the distance information or the depth information.

FIG. 5 (b) shows a plane image obtained by ultrasonic waves in FIG. 5 (a).
Figure 2 shows the two-dimensional images taken by the TV camera. Figure 5 (a)
The measurement area 52 corresponding to the ultrasonic pixel 15 × 15 is shown by a solid line. Since the pixels of the TV camera are very small, the pixel display is omitted in FIG. 5 (b). It is obvious that TV cameras are superior in lateral resolution. Figure 5
Medium code 10 indicates an object, 51 is the center of gravity of the object, 53
Indicates a television camera measurement area, and 54 indicates an area around the object. Hereinafter, the procedure for extracting an object from a television camera image based on an ultrasonic image will be described with reference to FIG.

First, the ultrasonic sensor array is used to measure the distance information to the surface of the object facing the ultrasonic sensor array to measure an ultrasonic image (step 100). This distance information is given as a two-dimensional array corresponding to the top, bottom, left and right. After the distance information is subjected to noise removal by isolation processing (101), the distance difference between the background and the object surface is calculated for each element, and this is newly made into a two-dimensional array (10).
2). This is an array indicating the depth information of the object, and the elements in the background portion are zero, so that the projected shape of the object can be obtained by displaying the plane distribution of the non-zero elements. The center of gravity of the object is obtained from this projected shape (see 10
3).

In this case, since the relative positions of the television camera image and the ultrasonic sensor do not change, the pixels of both images in FIGS. 5 (a) and 5 (b) have a one-to-one correspondence. From this correspondence,
The pixel shown in FIG. 9B corresponding to the object position shown in FIG. Therefore, the object image can be extracted from the video image by calculating the lightness information of the point corresponding to the object in the television camera image based on the ultrasonic image (at 105 to 107). About this,
Will be described in detail.

7 (a) and 7 (b) show a cylindrical object 10 (FIG. 7 (c)) measured by a monochrome television camera and a background X.
The lightness is shown on the axis and the lightness on the Y-axis, and the lightness is different on the object and on the background. However, the hatched portion in FIG. 7 (c) represents the shadow 10a. Therefore, the plane image of the object can be extracted by the difference in brightness. As this automatic extraction method, the present invention uses the position information of the object obtained by ultrasonic waves. As indicated by hatching in FIG. 5, the area 54 around the object in the video image can be estimated based on the ultrasonic information.

FIG. 8 shows a brightness histogram of a grayscale plane image measured by the above video camera, and this histogram is composed of an object image and a background. Here, the inside of the area corresponds to the object portion and the outside of the area corresponds to the periphery of the object. Therefore, centering on the lightness of the center of gravity of the object,
The object image can be extracted from the video image by extracting the distribution range of the histogram. Moreover, you may extract below the minimum brightness value outside the area. The obtained image is far finer than the ultrasonic plane image. In the extraction based on the brightness, the average value, the variance, the maximum value or the minimum value may be used.

As described above, since the brightness information of the point corresponding to the object in the TV camera image can be specified based on the ultrasonic image, the object can be viewed from the TV camera image even in an outdoor environment where the brightness changes or a shadow occurs. There is also an advantage that the image can be extracted. As described above, according to the present invention,
It has become possible to supplement the ultrasonic image, which has poor lateral resolution, with a TV camera image. As a result, by substituting the average height of the ultrasonic image of FIG. 4 into the obtained plane image, as shown in FIG. 9, it is possible to synthesize a precise three-dimensional object image as it is.

Next, an example in which a neural network is used for synthesizing an ultrasonic image and a video plane image will be shown. First, the differential plane image is automatically extracted by the same method as that for automatically extracting the grayscale plane image. For the differentiation, for example, a Laplacian type filter with 4 rows and 4 columns is used. Of the measured ultrasonic information of 225 points and the television camera information of 3600 pixels, as shown in FIG. 10, one point O (called a synthesis center) of ultrasonic measurement points and its front, rear, left and right (A, B, C, D) are displayed. 1 corresponding to ultrasonic information (distance data) 61 at a total of 5 points and ultrasonic information at the point O
The 6-pixel differential image 62 is converted into a neural network (N.
N) to obtain the height corresponding to the differential image according to the learning rule. Then, a synthetic image is created by repeating this processing for all ultrasonic information.

Here, as the neural network, for example, a three-layered layered neural circuit using a sigmoid function is used. The ultrasonic information is standardized by the maximum height of the object, and the differential image is binarized into "1" and "0" and input to the neural network. From this differential image, it is not possible to determine which is higher, the right or left of the line segment. Therefore, the height relationship in the differential image is estimated from the height relationship in the parts A to D adjacent to the front, rear, left, and right.

FIG. 11 shows an example thereof. For simplicity, it is assumed that the rectangular parallelepiped 71 is placed on a plane. When the height changes like a step edge in this way, the edge is detected by the differential image. The ultrasonic information corresponding to the edge portion is either the height of the cube or the height of the background depending on the relative positional relationship between the sensor and the object. However, since the height of the plane portion adjacent to the edge is accurately measured by ultrasonic waves, when there is an object below the edge 64 as shown in FIG. 11, the pixel below the edge has the height of the element C. To
Further, the learning is performed by substituting the element A, that is, the height of the background, into the pixels above the line segment. However, reference numeral 63 in FIG. 11 represents the composite data, and the height relationship in the image is edge 6
The lower pixel is “H (high); high” on the border of 4,
The upper pixel becomes “L (low); low)”. At this time, the height of the pixel O at the synthesis center and the pixels A to D on the front, rear, left, and right sides in the ultrasonic information, that is, the distance data 61 are “H; high” and “L;

Although various curves can be assumed as the line segments in the differential image, the 16 patterns shown in (1) to (16) of FIG. 12 were learned by limiting them to straight lines in order to reduce the learning amount. As a result, the height can be estimated for the case where the edge 64 is vertically, horizontally, or diagonally 45 degrees. When the edge is not included in the synthesis center, the height of the synthesis center O is substituted into the corresponding 16 pixels.

Next, as an example, the height, width, and length are 3, respectively.
6cm diameter and 6cm height in the center of 6,18cm wooden box
The execution result of this invention when the sample which put the wooden cylinder of is used is shown. Since the surface is unpainted, grain can be observed. FIG. 13 shows an ultrasonic image of this sample. Although the outline of the rectangular parallelepiped has been reproduced, it is difficult to recognize that it is a cylinder.

FIG. 14 shows an ultrasonic plane image (solid line, dotted line) superimposed on the differential image (thick line portion) used as input data to the neural network. This figure shows the correspondence between the two. The differential image does not always become a continuous straight line or curve depending on the lighting condition. Figure 1
In the ultrasonic plane image in FIG. 4, the rectangular parallelepiped part is shown by a solid line and the cylindrical part is shown by a dotted line. As a whole, the ultrasonic image is measured slightly larger than the differential image. This seems to be a characteristic of the ultrasonic measurement system. As the characteristics of ultrasonic waves, the vertex portion of the object is apt to be chipped, but in the present embodiment, all four vertexes are measured. Random line segments are observed in places other than edges in the differential image, which correspond to the grain of the sample surface. Human beings are indifferent to the grain of the sample surface, but the camera image faithfully reproduces the grain as well.

FIG. 15 shows a composite image by the neural network. Not only the rectangular parallelepiped of the base but also the curve of the upper cylindrical part was reproduced poorly, and a composite image showing the characteristics of the object shape was obtained. Further improvement can be expected by examining the learning rules of neural networks. There are several noises in response to the grain, but texture information that does not correspond to the ultrasonic image is almost completely ignored. This is a feature of the present invention, and the texture information in the differential image can be extracted by comparing the differential image and the composite image. Further, since the resolution of the synthetic image is improved as compared with the ultrasonic image, it is expected that the automatic recognition accuracy of the object is improved by using the synthetic image.

By inputting the fine three-dimensional object image generated by the above method to the neural network, it is possible to recognize an object whose front, rear, left, right and up and down positions are free. The detailed method is the same as in the above-mentioned Japanese Patent Application No. 3-246558, so the description is omitted and the experimental results are described.

Using the apparatus of FIG. 3, the four types of objects shown in Table 1 are placed in the center of the measurement area respectively to measure the shape, and after learning by a three-layer neural network (NN), recognition is performed. An experiment was conducted. Here, the number of elements of the neural network is the input layer 52 × 52, the intermediate layer 8, and the output layer 4. The number of elements in the input layer can be increased to about 400 × 400 corresponding to the number of pixels of the television camera. The larger the number of elements, the finer the shape can be recognized. However, in this experiment, the size was 52 × 52 due to the limitation of the memory of the computer. Therefore, for a large object, the object image extracted from the TV camera image is compressed into 52 × 52 pixels.
In the experiment, four types of wooden samples were placed in any position in the measurement area on the desk in any direction, and the recognition rate was measured. Three
The shape of a three-dimensional object looks different depending on the observation position and direction. Therefore, the recognition requires a recognition algorithm that does not depend on the position or direction of the object.

[0031]

[Table 1]

FIG. 16 shows an example of the object recognition algorithm used. The extracted shape information, that is, the composite stereoscopic image is translated to the center of the center of gravity in the center of the screen (step 150,
151), each time it is rotated, it is input to the neural network (at 152 and 153), and the shape is determined by the maximum output (at 154). The rotation was performed every 10 degrees up to 350 degrees.
As a result of obtaining the recognition rate 50 times for each of the four types of samples, a recognition rate of 100% could be obtained, and the effectiveness of the present invention could be confirmed. If only ultrasound is used,
As in the above-mentioned Japanese Patent Application No. 3-246558, recognition of an object with a diameter of about 6 cm was limited, but by using a television camera together, it was possible to recognize an object with a diameter of 15 mm.

In the present invention, the ultrasonic sensor is used as the distance sensor as an embodiment, but the present invention is not limited to this, and the present invention can be applied to a laser range sensor using an optical cutting method or an optical phase method. It goes without saying that it is possible.

[0034]

As described above, according to the present invention, a distance sensor such as an ultrasonic sensor and a television camera are used together, and an object image is extracted from a video image based on the distance information from the distance sensor. , It becomes possible to supplement a stereoscopic image having poor lateral resolution with a video image. Therefore, it is possible to automatically generate and recognize a precise three-dimensional image of a small object that cannot be measured only by ultrasonic waves. by this,

(1) A large object and a small object can be recognized. (2) Recognize objects with free distance, position, and orientation. (3) It can be recognized even in an outdoor environment in which brightness changes or shadows occur. If it is applied to a robot, autonomous movement without obstacles, object capture by a manipulator, and various tasks are possible.

[Brief description of drawings]

FIG. 1 is a block diagram of a shape measuring system according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a neural network used in this embodiment.

FIG. 3 is a diagram showing a configuration of an ultrasonic sensor and a television camera according to the present embodiment.

FIG. 4 is a diagram showing an ultrasonic image of a cylinder used in the description of the present embodiment.

5A and 5B are explanatory views of a plane image of an object used in the description of the present embodiment, in which FIG. 5A is an ultrasonic image and FIG. 5B is a television camera image.

FIG. 6 is a diagram showing a procedure of extracting an object from a television camera image in the present embodiment.

FIG. 7 is an X in a video image used for explaining the present embodiment.
It is a figure which shows the brightness on an axis and a Y-axis.

FIG. 8 is a lightness histogram of a grayscale plane image used in the description of the present embodiment.

FIG. 9 is a diagram showing a stereoscopic image of an object (composite ultrasonic stereoscopic image and video plane image) obtained in the present embodiment.

FIG. 10 is an explanatory diagram of a method of synthesizing ultrasonic information and video information used in the description of the present embodiment.

FIG. 11 is an explanatory diagram of a method of synthesizing ultrasonic information and video information, which is also used in the description of the present embodiment.

FIG. 12 is an explanatory diagram of a learning rule of a neural network used in the description of the present embodiment.

FIG. 13 is a diagram showing an ultrasonic image in which a cylinder is placed on a rectangular parallelepiped used in the description of the present embodiment.

FIG. 14 is a diagram showing correspondence between an ultrasonic image and a differential image used in the description of the present embodiment.

FIG. 15 is a diagram showing a synthetic stereoscopic image by a neural network in the present embodiment.

FIG. 16 is a diagram showing an object recognition algorithm used in the description of the present embodiment.

FIG. 17 is an explanatory diagram of shape recognition using only a conventional ultrasonic sensor.

[Explanation of symbols]

10 Three-dimensional object 11 Ultrasonic sensor array 11 _{1 to} 11n Ultrasonic sensor 12 Television camera 13 Neural network 17 Sensor device

Claims (4)

[Claims] 1. A plurality of distance sensors arranged in a plane are provided facing an object and a video camera for imaging the object is provided, and a plane distribution of distance information up to the surface of the object is measured by the distance sensor. Then, the first stereoscopic image of the object is extracted based on the distance information, and the plane image of the object is extracted from the video image by the video camera based on the position information of the first stereoscopic image. And a method for generating a fine second stereoscopic image by synthesizing the first stereoscopic image with the first stereoscopic image. 2. The method for generating a three-dimensional object image according to claim 1, wherein a grayscale plane image or a differential plane image is extracted from the video image based on the brightness of the position of the center of gravity of the first stereoscopic image. 3. The method for generating a three-dimensional object image according to claim 1, wherein a grayscale plane image or a differential plane image is extracted from the video image based on the brightness around the first stereoscopic image. 4. The differential plane image is extracted from the video image based on the center of gravity position of the first stereoscopic image or the brightness of the surroundings, and the differential plane image and the first stereoscopic image are combined using a neural network. Then, the pixel group of the differential plane image corresponding to one image of the first stereoscopic image is combined with the pixel group of the first stereoscopic image surrounding the one image and the distance information of the one image. A method for generating a three-dimensional object image.