JPH0799644A - Image communication device - Google Patents

Abstract

(57) [Summary] [Purpose] Match the line of sight of each other in a video conference. [Configuration] The cameras 1 and 2 capture the same person. The captured image is processed by the video signal processing units 3 and 4, and is applied to the corresponding point extraction unit 5 and the normal vector extraction unit 6. The corresponding point extraction unit 5 extracts the corresponding points of the two images, and the normal vector extraction unit 6 extracts the normal vector. The three-dimensional structure processing unit 7 uses the information extracted by the corresponding point extraction unit 5 and the normal vector extraction unit 6 to calculate approximate three-dimensional position information of the subject. Using the structure information of the subject calculated in this way, the coordinate conversion unit 8 points the cameras 1 and 2 in the directions in which the lines of sight coincide, and the transmission / reception transmission unit 9 transmits the captured image to the communication partner.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image communication device,
More specifically, the present invention relates to an image communication device capable of performing a video conference while keeping the lines of sight of each other matched.

[0002]

2. Description of the Related Art The following is known as a compound-eye image pickup system which uses images taken by two image pickup devices in combination. For example, when displaying an ultra-high-definition image of 4,000 × 4,000 pixels on an ultra-high-definition monitor, high density and high sensitivity in the image pickup system are problems. As a solution in the imaging system, compound eye imaging that captures a common subject using two imaging systems with a small number of pixels and combines two images obtained by each imaging system to obtain one high-definition image The principle of the device has been proposed (Aizawa et al.
"Fundamental Study for Acquisition of Ultra High Definition Images," IEICE Preliminary Report 90-03-04, p. 23-28).

[0003] multi-lens imaging apparatus based on this principle, as shown in FIG. 7, prepared left and imaging system 110 _L and the right imaging system 110 _R, the sampling points in the left side imaging system 110 _L and the right imaging system 110 _R with imaging an object 101 is shifted 1/2 pitch with Soratoi phase, and a right image I _R obtained by the left image I _L and right-side imaging system 110 _R obtained in the left imaging system 110 _L microprocessor ( Hereinafter, by performing a combining process in "CPU") 120, one output image I _OUT having a higher definition than that obtained when the object 101 is imaged by one imaging system is obtained.

FIG. 8 is a left side imaging system 110 _L shown in FIG.
And the basic optical structure of the right imaging system 110 _R.
Left imaging system 110 _L is composed of a left imaging optical system 111 _L and left image sensor 112 _L. Right imaging system 110
_R is the right imaging optical system 111 _R and the right image sensor 11
It consists of 2 _R. Here, the left-side imaging optical system 111 _L and the right-side imaging optical system 111 _R have specifications (or performance) equivalent to each other, and in this case, they are zoom lenses. Also, the left image sensor 112 _L and the right image sensor 112 _R
Have specifications (or performance) equivalent to each other, and are composed of an image pickup tube such as a SATICON or a solid-state image pickup device such as a CCD type image pickup device.

The left-side image pickup system 110 _L and the right-side image pickup system 110 _R have their optical axes L _L and L _R substantially intersect at a point O on the object plane 102, and a normal line OO of the object plane 102. It is arranged so as to be line symmetric with respect to. It should be noted that when the angles formed by the optical axes L _L and L _R and the normal line OO ′ of the subject plane 102 (hereinafter, referred to as “tilt angles”) are respectively θ, 2θ is defined as a vergence angle. To do.

In this compound-eye image pickup device, when the subject distance changes, for example, the left image pickup system 110 _L and the right image pickup system 110 _R are respectively rotated around the X mark shown in FIG. 8 to change the subject distance. The imaging is performed after changing the vergence angle 2θ according to. Next, the distance image will be described. FIG. 9 is an explanatory diagram of triangulation used for obtaining a distance image. In the following description, unless otherwise specified, the image sensors of the right camera and the left camera are placed on the positive surface. According to triangulation, an object (subject) in the three-dimensional space by using two cameras (right camera and the left camera) when captured, the center point of the right camera lens O _R, the left camera lens When the center point of the object is O _L , the projection point P _{R of the} point P on this object on the sensor surface A _SR of the right camera
And the projection point P _L on the sensor plane A _SL of the left camera, the three-dimensional coordinates of one point P on this object can be obtained.

In FIG. 9, the base line B, the base line length L _B , the epipolar plane (line of sight) A _e, and the epipolar lines (line of sight images) L _eR and L _eL are defined as follows.
That is, the base line B is a line connecting the center point O _R of the lens of the right camera and the center point O _L of the lens of the left camera. The base line length L _B means the length of the base line B. The epipolar plane A _e is a plane formed by connecting one point P on the object, the projection point P _R, and the projection point P _L. The epipolar line (line-of-sight image) L _eR is a line of intersection between the epipolar surface A _e and the sensor surface A _SR of the right camera, and the epipolar line L _eL is
The line of intersection between the epipolar surface A _e and the sensor surface A _SL of the left camera.

As shown in FIG. 10, the midpoint of the base line B is the origin O (0,0,0), the x axis is along the base line B, the y axis is perpendicular to the plane of the drawing (not shown), and , The base axis B and the z axis in the direction perpendicular to the y axis, the focal lengths of the lens of the right camera and the lens of the left camera are f, and the coordinates of a point P on the object are (x _P , y _P , z _P ), And the coordinates of the projection point P _R are (x _PR , y _PR , z _PR ) and the coordinates of the projection point P _L are (x _PL , y _PL , z _PL ). At this time, when the optical axes of the right camera and the left camera are respectively perpendicular to the base line B as shown in FIG. 10 (that is, when the two optical axes are parallel to each other), the following formula is established. To do. That is,

[0009]

[Equation 1] (x _PL + L _B / 2) / f = (x _P + L _B / 2) / z _P

[0010]

[Equation 2] (x _PR -L _B / 2) / f = (x _P -L _B / 2) / z _P

[0011]

[Number 3] _{y L / f = y R /} f = y / z P

[0012]

Equation 4] _{(L B + x PL -x PR} ) / f = L B / z P Therefore, the coordinates of a point P on the object (x _P, y _P, z _P) are
It is calculated by the following formula. That is,

[0013]

[Formula 5] x _P = L _B ((x _PL + x _PR ) / 2) / (L _B + x _PL -x _PR )

[0014]

[Equation 6] y _P = L _B ((y _PL + y _PR ) / 2) / (L _B + X _PL -x _PR )

[0015]

## EQU00007 ## z _P = L _B f / (L _B + x _PL -x _PR ) Further, the optical axes of the right camera and the left camera are each at a predetermined angle (convergence) with respect to the base line B as shown in FIG. Corner)
When θ is held, the following formula is established. That is,

[0016]

(Equation 8) (x _PL + L _B / 2) z _PL = (x _P + L _B / 2) / z _P

[0017]

(Formula 9) (x _PR -L _B / 2) / z _PR = (x _P -L _B / 2) / z _P

[0018]

[Equation 10] y _PL / z _PL = y _PR / z _PR = y _P / z _P

[0019]

Equation 11] _{L B / z P = ((} z PL + L B / 2) - (z PL / z
_{PR) (x PR -L B /} 2) / Z PL _{However, | x PR | ≧ | x} PL |

[0020]

[Equation 12] L _B / z _P = (-(x _PR -L _B / 2) + (z _PR / z _PL ) (x _PL + L _B / 2)) / z _PR where | x _PR | <| x _PL ｜

[0021]

[Formula 13] z _PR = (x _PR -L _B / 2) tan (θ) + fcos (θ)

[0022]

[Formula 14] z _PL =-(x _PL + L _B / 2) tan (θ) + fcos (θ)

These

[Equation 1] ~

From the equation (14), the coordinates (x _P , y _P , z
_P ) can be obtained.

By the triangulation described above, the distance to the object (subject) can be obtained from the two images picked up by the compound eye image pickup system including the right side image pickup system and the left side image pickup system. However, triangulation is based on the assumption that the projection point P _R on the sensor surface A _SR of the right camera and the projection point P _L on the sensor surface A _SL of the left camera are projection points of the same point P. Since the distance to is calculated, the projection point P _R on the sensor plane A _SR of the right camera corresponding to the projection point P _L on the sensor plane A _SL of the left camera needs to be extracted. Therefore, how to extract corresponding points in order to obtain distance information using the compound-eye imaging system (corresponding point extraction method)
Is a problem. As a typical corresponding point extraction method, there is a template matching method already applied in factories and the like.

The template matching method will be described.
The template matching method considers a template surrounding an arbitrary point of the left image formed on the sensor surface A _SL of the left camera, and an image in this template is formed on the sensor A _{SR of the} right camera. The corresponding image is compared with the right image and the corresponding point is determined from the similarity. Note that SSDA (Sequential Sim) is used to determine the similarity.
ilarity Detection Algorithm
hm) method and correlation method.

In the SSDA method, as shown in equation 15, for all the pixels on the epipolar line L _eL of the left image and all the pixels on the epipolar line L _eR of the right image, in the template of the left image. The sum E obtained by adding the difference between the pixel value E _L in the image of _P and the pixel value E _R in the right image to be searched for
The coordinates that minimize (x, y) are the coordinates of the corresponding points.

[0027]

[Equation 15]

In the SSDA method, when the sum of the differences in the pixel values being calculated becomes larger than the minimum value at other coordinates calculated up to now, the calculation is stopped and the operation moves to the next coordinate. Since it is good, the calculation time can be shortened by eliminating extra calculation.

In the correlation method, the correlation value P is obtained by taking the cross-correlation between the pixel value E L in the image in the template of the left image and the pixel value E _R in the right image to be searched, as shown in Equation 16.
(X, y) is obtained, and the coordinate at which the obtained correlation value P (x, y) is maximum is used as the coordinate of the corresponding point. In the normalized cross-correlation shown in Expression 16, the maximum value is 1.

[0030]

[Equation 16]

Regarding the technique for matching the lines of sight,
There is a patent application publication No. 196696 in 1992. In the face-to-face communication in which the face image of the communication partner is simultaneously observed and the image of the communication partner who is the observer is captured and transmitted to the communication partner, there is a method for optically achieving line-of-sight matching with the communication partner. Something like is proposed. For example, in order to obtain a good line-of-sight match in communication directly facing a communication partner displayed as a moving image, a configuration is disclosed in which the optical axes of the imaging system (camera) and the display system (display device) are coupled by a half mirror. There is.

As another example, as shown in FIG. 12, a camera is arranged at a position close to the display device. Figure 1
2 (1) is a front view and FIG. 2 (2) is a side view thereof, and the camera 2 is arranged above a display device (display). The camera 2 has an angle θ with respect to the person to be photographed 3 (an angle formed by the camera 2 and a line connecting the eyes E of the person to be photographed 3 gazing at the center point T of the moving image window on the display 1 surface).
This is a device arrangement configuration that does not make the user feel a line of sight mismatch when the distance L between the display surface and the person to be photographed is large.

[0033]

However, in the conventional example,
There was a problem that it felt visually uncomfortable. Further, a large screen display is indispensable in order to enhance the sense of realism, and the screen size of the display device must be increased and the size of the semi-transparent mirror must be increased. As a result, there is a problem that the depth becomes large and the display / imaging device itself becomes large.

It is indispensable to use a wide-angle lens also in the image pickup apparatus in accordance with the increase in screen size, and the price of the semi-transparent mirror becomes high depending on the size. As a result, the entire apparatus becomes expensive. There was a problem that

Further, in the conventional example, there is no effective means for avoiding eyes.

It is an object of the present invention to provide an image communication device that solves such problems.

[0037]

An image communication apparatus according to the present invention is an image communication apparatus for displaying images of communication partners on a mutual display device for conversation, and each communication terminal includes a plurality of image pickup devices. And a line-of-sight matching processing device that processes images captured by the plurality of imaging devices so as to match the line-of-sight of the communication partner.

[0038]

By the above means, it becomes possible to transmit to the communication partner an image in which the line of sight of the communication partner matches, and the lines of sight can be matched with each other.

[0039]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows a schematic block diagram of an embodiment of the present invention, and FIG. 2 shows an external view of a terminal.

As shown in FIG. 2, the terminal comprises a display device, two cameras, and a man-machine interface section (keyboard, mouse, switches, etc.). In FIG. 1, cameras 1 and 2 image the same subject. In a situation such as a video conference, the main subject is usually a person who is going to make a video conference. The captured image is
It is transmitted to the video signal processing units 3 and 4 having an image memory. The images processed by the video signal processing units 3 and 4 are applied to the corresponding point extracting unit 5 and the normal vector extracting unit 6. The corresponding point extraction unit 5 extracts the corresponding points, and the normal vector extraction unit 6
Extracts the normal vector.

The three-dimensional structure processing unit 7 uses the information extracted by the corresponding point extraction unit 5 and the normal vector extraction unit 6 to calculate the approximate three-dimensional position information of the subject. Using the structure information of the subject calculated in this way, the coordinate conversion unit 8
Aim the subject in any specified direction. The image of the subject pointed in the designated direction is transmitted from the transmission / reception transmission unit 9 to the communication partner.

The above is a very basic flow of information,
The details will be described below in detail.

The corresponding point extraction unit 5 will be described. The corresponding point extraction unit 5 calculates, at each intersection point on the parallax screen formed from two epipolar lines extracted from the two captured images,
A parallax line is calculated by performing a local calculation having excitatory coupling and inhibitory coupling based on the actual pixels of these two binary images. When extracting the corresponding points of these two binary images, virtual pixels indicating continuity between the actual pixels are provided between adjacent real pixels of each binary image, and excitability based on the virtual pixels is provided. A local operation with associative and inhibitory connections is performed in parallel with a local operation with excitatory and inhibitory connections based on real pixels. In addition, the excitatory connection based on the virtual pixel counteracts with the excitatory connection based on the real pixel,
Inhibitory connections based on virtual pixels compete with inhibitory connections based on real pixels.

In the cooperative algorithm proposed by David Marr, there are the following three rules. That is, Rule 1 (Compatibility) ... Black dots can only match black dots.

Rule 2 (Uniqueness) —Almost always one black dot in one image can match the only black dot in the other image.

Rule 3 (Continuity) ... The parallax of matching points changes smoothly over almost the entire area.

This embodiment further adds to these three rules:
The following rules 4 to 6 are added. That is, rule 4 ... The degree of continuity of parallax is set stronger at the center of the local processing and weaker at the periphery of the local processing.

Rule 5: In order to strengthen uniqueness, the inhibitory coupling is strengthened toward the periphery of the local processing.

Rule 6 ... Processing in which virtual pixels indicating continuity between real pixels are provided between adjacent real pixels in an image, and rules 1 to 5 are matched based on the provided virtual pixels. Give.

Note that the local processing in rule 5 is the processing performed on a local area centered on the real pixel when one real pixel of one image is focused, and the center of the local processing is
At this time, the position of one real pixel of interest is referred to, and the periphery of the local processing means a position away from the one real pixel of interest at this time. By repeating this local processing, the parallax line is calculated and the corresponding points are extracted.

The normal vector extraction unit 6 is composed of three modules as shown in FIG. The SS module extracts three-dimensional structural information based on the normal direction of the occluding contour portion of the object on the image. This module assumes a smooth constraint on the surface of a three-dimensional object and uses the relaxation method to find the state with the smallest error globally. As the initial condition, the normal direction information of the shield contour is used. The function representing the normal direction at the pixel point <x, y) is f (x,
y) and g (x, y). The quantity eg, which represents the smoothness of the question with the normal line in the vicinity, is defined as the following equation. That is,

[0053]

## EQU17 ## eg = ∫∫ ((f _x ² + f _y ² ) + (g _x ² + g _y ² )) dxdy where f _x is the partial differential of the function f in the x direction and f _y is the function f partial differential in the y direction, g _x is the partial differential in the x direction of the function g, and g _y is the function g
Is a partial differential in the y direction.

Further, the measured luminance value I and the theoretical value R (f, g)
The quantity ef representing the error between and is defined as the following equation. That is,

[0055]

[Equation 18] ef = ∫∫ (I (x, y) −R (f, g)) ² dxdy SS module solves f and g by the variational method to minimize eg + λef and performs discretization. , The relaxation method is executed to extract the normal direction information.

The LSAM module assumes a spherical surface as the surface shape of the object and calculates the normal direction based on the local luminance change information of the pixel. The second derivative d ² I of the luminance I in the direction ξ on the pixel is equal to the value of the second partial derivative I _uu in the u direction in the coordinate system (u, v) rotated by ξ. This value is obtained from the following coordinate conversion formula I _uu = I _xx cos ² ξ + I _yy sin ² ξ + 2I _xy cos ξ sin ξ.

The condition for minimizing I _uu is dI _uu / dξ = 0. Therefore, tan ξ = 2I _xy / (I _xx −I _yy ).

Assuming that the surface of the object is spherical and substituting the values of I _xx , I _yy , and I _xy at that time, tan2ξ =
It becomes 2xy / (x ² -y ² ). Since tan θ = y / x, tan 2θ = ² tan θ / (1-tan ² θ) = 2xy / (x ² −y ² ) = tan ² ξ.

Therefore, assuming that the surface shape is spherical,
The azimuth angle θ of the normal is equal to the minimum direction of the ^second derivative d ² I of luminance. The LSAM module calculates the normal direction by such calculation.

Details of the SS module and the LSAM module are described in Ikeuchi, K. et al. & B. K. P. Ho
rn "Numerical Shape from
Shading and Occluding Boun
"Daries", A1, vol. 17, # 1-3, P
P. 141-184, 1981 and Pentlan
d A. P. "Local Shading anal
ysis ”, IEEE Trans, PAM1-6, #
2, 1984.

The normal direction information to be newly used is defined as follows from the normal direction information calculated by the SS module and the LSAM module. That is, assuming that the normal vector calculated by the SS module is f _SS and the normal vector calculated by the LSAM module is f _LSAM , newly used normal direction information f is obtained by the following equation. That is, f = (f _SS + f _LSAM ) / 2, or f may be defined from the overall average of f _SS and f _{LSAM in} the vicinity of an arbitrary point (x, y). If four f _SS1 , f _SS2 , f _SS3 , f _SS4 ; f _LSAM1 , f from the neighboring region
_{When LSAM2} , f _LSAM3 , and f _LSAM4 are obtained, f = (1/8) (Σ (f _SSi + f _LSAMi )). However, i is 1 to 4.

The coordinate conversion unit 8 will be described. Coordinate conversion unit 8
Selects and executes either the automatic selection mode or the arbitrary selection mode. The automatic conversion mode uses the normal vector calculated by the normal vector extraction unit 6. Normally, the normal vector is oriented in various directions depending on the shape of the subject. For example, in a video conference or the like, where the person is the center of the subject, the person is almost centered as the subject to be imaged. From the average value of the area, look at the whole direction and rotate it so that it coincides with the optical axis of the camera.
Mathematically, the conversion is as follows.

[0063]

[Formula 19]

Here, (ω, φ, χ) is the rotation angle.

In the optional mode, the communication partner in the TV conference can change the orientation of the captured image on the transmission side. Although the conversion formula is the same as that of the equation (19), it is characterized in that the information about the rotation angle of the conversion is transmitted from the communication partner.

Next, the compound eye optical system will be described. FIG. 4 shows the basic arrangement of the compound-eye imaging system in this embodiment. The right imaging system 100 includes a right imaging optical system 102 and a right image sensor 103, and the left imaging optical system 200 includes a left imaging optical system 202 and a left image sensor 203. Here, the right imaging optical system 102 and the left imaging optical system 2
02 has specifications (or performance) equivalent to each other, and is composed of a zoom lens here. Also, the right image sensor 1
03 and the left-side image sensor 203 have specifications (or performance) equivalent to each other, and in this case, they are image pickup tubes such as Sachicon or solid-state image pickup elements such as CCD image pickup elements.

The right side imaging system 100 and the left side imaging system 200 are
The optical axes 101 and 201 substantially intersect at a point O on the object plane 1 and are arranged so as to be line-symmetric with respect to a normal line OO ′ of the object plane 1000. When the angles formed by the optical axes 101 and 201 and the normal line OO ′ of the object plane 1 (hereinafter referred to as tilt angles) are θ,
2θ is defined as the vergence angle. In this compound-eye imaging device, when the subject distance changes, for example, the rotation centers F ₁ and F ₂ are rotated.
The right-side imaging system 100 and the left-side imaging system 200 are respectively rotated around the center, and the convergence angle 2θ is changed according to the change in the subject distance, and then the imaging is performed.

Next, the compound-eye image pickup apparatus and its processing system in this embodiment will be specifically described. FIG. 5 is a schematic configuration diagram showing a first embodiment of the compound-eye image pickup apparatus of this embodiment, and FIG. 6 is a processing block diagram thereof. In the compound-eye imaging device shown in FIG. 5,
Based on the basic arrangement shown in FIG. 4, two sets of image pickup systems (right side image pickup system 100 and left side image pickup system 200) are used to synthesize two images obtained by picking up an image of a common subject to obtain a high-definition image. One image is obtained.

The structures of the right side image pickup system 100 and the left side image pickup system 200 will be described in detail with reference to FIG. Right imaging system 1
Reference numeral 00 denotes a right side image pickup optical system 102 and a right side image sensor 103 formed of an image pickup tube.
Is composed of a left-side image pickup optical system 202 and a left-side image sensor 203 formed of an image pickup tube.

The right-side imaging optical system 102 and the left-side imaging optical system 202 are respectively lens groups 102a to 102 including a zooming group 102b, 202b and a focusing group 102d, 202d.
d, 202a to 202d, and the zoom groups 102b and 202b
A zoom motor 106, which is a drive system for driving the
206, the focus motors 107 and 207 which are drive systems for driving the focusing groups 102d and 202d, and the imaging optical systems 102 and 2 in a plane including the optical axes 101 and 201.
02 and the image sensors 103 and 203 as a unit, a mechanism system (not shown) and a drive system (convergence angle motors 104 and 204), and convergence angle motors 104 and 20.
The angle encoders 105 and 205 for detecting the rotation angle of No. 4 are included. Note that external members such as potentiometers may be used as the angle encoders 105 and 205. For example, a drive signal of a motor capable of detecting a rotation angle from a drive signal such as a pulse motor can be used as a rotation angle. It may be a means for converting.

The focus motor 107 and the zoom motor 106 of the right imaging optical system 102 are the zooming group 102b.
The left imaging optical system 202 is controlled by the output signal of the focus encoder 109 for obtaining the position information of the focusing group 102d and the output signal of the zoom encoder 108 for obtaining the position information of the focusing unit 102d in the optical axis direction. The focus motor 207 and the zoom motor 206 are used to obtain the output signal of the focus encoder 209 for obtaining the position information of the variable power group 202b in the optical axis direction and the position information of the focusing group 202d in the optical axis direction. Zoom encoder 20
8 output signals. As a result, the focal length of the right imaging optical system 102 and the focal length of the left imaging optical system 202 are controlled so as to always match, and as a result, the imaging magnification of the right imaging optical system 102 and the left imaging optical system 202 are controlled.
The imaging magnifications of are always the same.

With the above configuration, first, the rotation angle information detecting means 105, 205 detect the convergence angle 2θ. Next, the focal lengths of the image pickup optical systems 102 and 202 are obtained from the output signals of the zoom encoders 108 and 208. Then, the subject distances to the respective image pickup optical systems 102 and 202 are obtained from the output signals of the focus encoders 109 and 209, and the object distances of the respective image pickup optical systems 102 and 202 are combined with the focal lengths of the respective image pickup optical systems 102 and 202 described above. A lens back is required. The zoom encoder 10
8, 208 and focus encoders 109, 209
With the output signal of, the focal lengths of the respective image pickup optical systems 102 and 202 and the lens back always match.

As shown in FIG. 6, the converted signal generator 12
R and 12L are image sensors 103 and 20, respectively.
An epipolar line, which will be described later, is reconstructed based on the image signal from the signal No. 3 and the angle signals from the angle encoders 105 and 205. The coordinate conversion units 11R and 11L perform coordinate conversion processing on the output images of the image sensors 103 and 203 according to the output signals of the conversion signal generation units 12R and 12L. The image memories 111 and 211 have video signals 112 and 212 whose coordinates have been converted by the coordinate conversion units 11R and 11L, respectively.
Temporarily store. The interpolation processing units 13R and 13L include the video signals 112 and 21 stored in the image memories 111 and 211, respectively.
Interpolation processing is performed on 2 and the sampling points are interpolated.

The corresponding point detecting section 5 includes an interpolation processing section 13R, 1R.
Video signal 113 interpolated in 3L,
The corresponding points in 213 are detected. The memory 15 stores the position of the corresponding point obtained by the corresponding point detecting unit 5 and the sum of the absolute values of the differences between the picture lines at the corresponding positions (hereinafter referred to as “residual”). The normal vector extraction unit 6 uses two images to obtain a normal vector. Image memory 311
The video extracted by the normal vector extraction unit 6 is written in.

Finally, the transmission section 9 will be described. Transmission unit 9
Transmits an image to a communication partner and receives a coordinate conversion control signal from the communication partner. Usually, the communication partner is equipped with a pointing device such as a joystick. The communication partner sets an arbitrary point of the re-normal image,
Arbitrary azimuth (ω, φ, χ) used in the conversion processor
The information corresponding to is transmitted.

As another embodiment, in the coordinate conversion section,
The coordinate conversion may be performed based on the azimuth angle of the coordinate conversion and the normal direction of the highest area of the subject, that is, the nearest area closest to the display. This is usually the case in video conferencing, for example.
The face of the subject is relative to the display, and at this time, the area around the nose is closest to the display, and in the area near it,
The normal direction of the area determined by the magnification of the camera and the approximate size of the nose is converted so that the normal direction is perpendicular to the display, and then transmitted.

[0077]

As can be easily understood from the above description, according to the present invention, the lines of sight coincide with each other, so that it is possible to realize a face-to-face conference or a call without feeling uncomfortable with the communication partner. Also, the configuration for that is simple, and the terminal device can be constructed at low cost.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

FIG. 2 is an external view of a terminal.

FIG. 3 is a block diagram of the internal configuration of a normal vector extraction unit 6.

FIG. 4 is a schematic configuration diagram of a compound-eye imaging device.

FIG. 5 is a schematic configuration diagram of an imaging system.

FIG. 6 is a schematic functional block diagram according to the flow of processing.

FIG. 7 is an explanatory diagram of a basic principle of a compound-eye imaging device.

8 is an explanatory diagram of a basic structure of a left imaging system and a right imaging system of FIG.

FIG. 9 is an explanatory diagram of triangulation for obtaining a distance image.

FIG. 10 is an explanatory diagram of object coordinate calculation when the optical axes of the right camera and the left camera are parallel to each other.

FIG. 11 is an explanatory diagram of object coordinate calculation when the optical axes of the right camera and the left camera intersect.

FIG. 12 is a front view of a display and a side view of a terminal of a conventional terminal device for making the lines of sight substantially coincide with each other in a video conference.

[Explanation of symbols]

1, 2: camera 3, 4: video signal processing unit 5: corresponding point extraction unit 6: normal vector extraction unit 7: three-dimensional structure processing unit 8: coordinate conversion unit 9: transmission / reception transmission unit 11R, 11L: coordinate conversion unit 12R, 12L: Converted signal generator 13R, 13L: Interpolation processor 15: Memory 100: Right imaging system 101: Optical axis 102: Right imaging optical system 103: Right image sensor 200: Left imaging system 201: Optical axis 202: Left Imaging optical system 203: Left side image sensor 102b, 202b: Variable magnification group 102d, 202d: Focusing group 102a-102d, 202a-202d: Lens group 104, 204: Convergence angle motor 105, 205: Angle encoder 106, 206: Zoom -Motors 107 and 207: Focus motors 108 and 208: Zoom encoders 109 and 209: Okasu encoder 111, 211: image memory 112, 212: coordinate-converted video signal 101: object 102: object plane 110 _L: left imaging system 110 _R: right-side imaging system 111 _L: left imaging optical system 112 _L: left image Sensor 111 _R : Right side imaging optical system 112 _R : Right side image sensor 120: Microprocessor 311: Image memory 1000: Subject plane L _L , L _R : Optical axis I _L : Left side image I _R : Right side image F ₁ , F ₂ : Rotation center

 ─────────────────────────────────────────────────── ─── Continued Front Page (72) Masayoshi Sekine, Inventor Masayoshi Shimomaruko, 3-30-2, Ota-ku, Tokyo Canon Inc. (72) Inventor Hideaki Mitsutake, 3-30-2, Shimomaruko, Ota-ku, Tokyo Canon Within the corporation

Claims (2)

[Claims] 1. An image communication apparatus for displaying an image of a communication partner on a display device of each other for conversation, each communication terminal comprising: a plurality of imaging devices; and images captured by the plurality of imaging devices. And a line-of-sight matching processing device for processing to match the line-of-sight of the image communication device. 2. The image communication device according to claim 1, wherein the line-of-sight coincidence processing device includes a calculation unit that calculates a distance structure and normal line information from images captured by the plurality of image-capturing optical devices and performs coordinate conversion.