JP2002042158A - Image synthesizing apparatus, image synthesizing method, and medium recording program - Google Patents

Abstract

(57) [Summary] To realize a real-time property by calculating a projection position (P) in pixel units and reducing the amount of calculation for image synthesis in order to synthesize an arbitrary viewpoint image of a three-dimensional object. A three-dimensional object (Dt) having brightness data (Dlum) and coordinate data (Dxz) corresponding to each pixel is captured by shooting a three-dimensional object from a plurality of surrounding points using a camera and a range finder.
dm), and based on the pixel unit Dtdm and the observer's viewpoint position information, project the pixel unit Dtdm onto the image plane to calculate the projection position and combine the luminance data of the P pixels to obtain an arbitrary value. In the synthesizing method for obtaining the viewpoint image (Pim), a part of the Dtdm of the pixel unit Dtdm is held as projection coordinate data (Dpxz) for projection, and the remaining Dxz is linear for the linear approximation operation. Conversion step S2 for converting to coordinate data (Dra) for approximation calculation
And a projection step S3 for projecting the transformed three-dimensional data 106 onto the image plane to obtain an arbitrary viewpoint image (Pim) to obtain P on the image plane based on Dpxz, and performing a linear approximation operation based on the obtained P and Dra. And a calculation step S4 for calculating the projection position on the image plane.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing method for synthesizing an image of a three-dimensional object viewed from an arbitrary viewpoint.

[0002]

2. Description of the Related Art Virtual reality (virtual reality) which is a technology of using a computer, CG (computer graphics) when used for moving images and games, and
In a virtual store or the like on the Internet, an image-based rendering technique for following an ever-changing viewpoint direction to obtain an image of an object viewed from the viewpoint direction is important. In this case, the image needs to change smoothly and naturally as the direction in which the object is viewed changes.

[0003] That is, what is important here is an image-based rendering technique for a three-dimensional object, and it is necessary to create a high-definition image as quickly as possible.

The image-based rendering technique is a technique for modeling a three-dimensional object based on a real image and synthesizing an arbitrary viewpoint image. As a conventional method for this, an image from an arbitrary viewpoint is obtained using three-dimensional data in pixel units. There is a method as shown in the flowchart of FIG.

As shown in FIG. 6, first, three-dimensional data is obtained (a three-dimensional data obtaining step (step S
51)). This three-dimensional data acquisition step (S51)
In, using a camera for imaging a subject and a range finder for acquiring a distance image, a three-dimensional object to be a subject is photographed from a plurality of points around the subject,
The obtained image is obtained as three-dimensional data Dtdm. That is, for the obtained image, three-dimensional data Dtdm which is data having luminance data Dlum and coordinate data Dxz corresponding to each pixel is obtained.

Then, using the three-dimensional data Dtdm, in the coordinate data projecting step (step S52), coordinate data is obtained for each pixel based on the three-dimensional data Dtdm for each pixel and the viewpoint position information of the observer. Dxz is projected on an image plane to calculate a projection position. Then, in the luminance data synthesizing step (step S53), the arbitrary viewpoint image Pim is synthesized by synthesizing the luminance data Dlum corresponding to the projected position with the frame memory.

The conventional acquisition of the three-dimensional arbitrary viewpoint image Pim is performed through the three-dimensional data acquisition step, the coordinate data projection step, and the luminance data synthesis step as described above. The details are as follows.

First, referring to FIG. 7, three-dimensional data obtaining step S51 in FIG. 6, and three-dimensional data Dtdm (luminance data Dlum, coordinate data Dxz) in pixel units obtained in the three-dimensional data obtaining step S51. Will be described.

In FIG. 7, reference numeral 601 denotes a subject,
A three-dimensional object from which three-dimensional data is to be obtained,
02 is a turntable that is rotated for photographing with the three-dimensional object 601 mounted thereon, 603 is a digital camera that obtains luminance data of the three-dimensional object 601, 604 is a range finder that obtains coordinate data of the three-dimensional object 601,
Reference numeral 605 denotes luminance data obtained by the digital camera 603, 606 denotes coordinate data obtained by the range finder 604, and 607 denotes luminance data 60 of a plurality of sheets.
5. Three-dimensional data consisting of coordinate data 606.

The range finder 604 uses a laser beam, for example, to control each position of the subject surface (the three-dimensional object 60).
By measuring the distance to each position on the surface of the subject, coordinate data of each position on the surface of the subject is obtained.

To obtain three-dimensional data of a three-dimensional object 601 as a subject, the three-dimensional object 601 is placed on a turntable 602 and its optical axis is directed to a fixed position outside the turntable 602 toward the three-dimensional object 602. Digital camera 6
03 and the range finder 604 are fixedly arranged. Then, the optical axes of the digital camera 603 and the range finder 604 are aligned, and calibration (position adjustment) is performed so that data for each pixel overlaps. And
The photographing is started, and the turntable 602 is rotated every photographing, for example, by 15 degrees. Thus, 1
By photographing all the stations (entire surroundings) of the three-dimensional object 601 in increments of 5 degrees, luminance data 605 and coordinate data 606 can be photographed in all stations 360 [degrees] / 15 [degrees] = 24 [frames].
Each data has pixels of the image size. In the luminance data 605, each pixel has a luminance (rdij, gri).
j, blij), and in the coordinate data 606, each pixel has a floating point coordinate (Xij, Xij, X) at the position of a point on the surface of the three-dimensional object with the origin at the center of the three-dimensional object 601.
zij). All of the luminance data 605 and coordinate data 606 composed of the plurality of images are collectively referred to as 3.
This is called dimension data 607.

Next, a method of synthesizing an arbitrary viewpoint image Pim in the coordinate data projecting step S52 and the luminance data synthesizing step S53 in FIG. 6 will be described with reference to FIG.

Generally, based on such three-dimensional data,
When synthesizing an arbitrary viewpoint image, image synthesis is performed by strictly reproducing the horizontal parallax and ignoring the vertical parallax that is considered to have a small visual effect.

Therefore, in the horizontal direction, synthesis is performed by a projection calculation in consideration of parallax, and in the vertical direction, a subject is approximated by a plane to expand and contract in the vertical direction.
This method is generally used because of problems such as the amount of calculation and the amount of data. FIG. 8 also describes a method of synthesizing an arbitrary viewpoint image (arbitrary viewpoint line) for a cross section of the three-dimensional object 601.

FIG. 8 illustrates the relationship between a projection image and data at a certain cross section of a three-dimensional object viewed from a certain viewpoint. Reference numeral 701 denotes three-dimensional data of a certain cross section of the three-dimensional object 601; One point at which projection is performed in the three-dimensional data 701 is shown. Reference numeral 703 denotes a new viewpoint position of the observer, 704 denotes an image plane corresponding to an image viewed from the viewpoint position 703, and 705 denotes a projection position at which the three-dimensional data 702 is projected on the image plane 704.

One point 70 in the three-dimensional data 701
2 has coordinate data (x, z) and luminance data (rd, gr, bl) (however,
“Rd” indicates luminance information of a red component, “gr” indicates luminance information of a green component, and “bl” indicates luminance information of a blue component).

Accordingly, in this case, the projection position 705 of the three-dimensional data 702 on the image plane 704 is expressed as g (x, z) as a function of x and z, and the viewpoint position 703
Between image and image plane 704 (focal length of camera 603)
Is represented by f and the distance between the viewpoint position 703 and the origin is represented by r.

G (x, z) = − fx / (z−r) The value of this function g (x, z) is
Projection position 70 of three-dimensional data 702 which is perspectively projected onto
5 In the case of a digital image, the image to be displayed is placed on a frame memory. Therefore, the perspective projection 3
The projection position 705 of the dimensional data 702 can be said to be a position corresponding to g (x, z) on the frame memory. That is, the luminance data of the pixel is placed at the position of the obtained value of g (x, z), so that g
If the luminance data of the pixel is written at the corresponding position (x, z), it is an image of the point 702 viewed from an arbitrary viewpoint on the image plane 704.

Thus, the coordinate data projecting step S5
2, a certain point 70 in the three-dimensional data 701
2 is viewed from a desired viewpoint, the projection position on the image plane 704 can be obtained by obtaining the projection position g (x, z) from the coordinate data (x, z) of the point 702. become.

When the position is determined, the point 702 is determined.
Is displayed at this projection position, an image of a certain point 702 viewed from an arbitrary viewpoint is expressed on the screen plane 704. In this case, the luminance data is combined with the luminance data “(rd, gr, bl)” of the point 702 of the three-dimensional data at the projection position 705 on the image plane 704 in the luminance data synthesizing step S53 (frame memory). To be written in).

Similarly, in addition to the projection of a point 702,
The same projection is performed on all of the three-dimensional data 701, and the projection position g (x, z) is calculated.

In the luminance data synthesizing step S53, while judging all three-dimensional data by the Z buffer, the luminance data (rd, gr, By writing bl), an arbitrary viewpoint image Pim is synthesized on the frame memory.

The Z-buffer is a process of overwriting a point on the near side with a frame memory according to a viewpoint position and coordinate data.

Thus, an arbitrary viewpoint image Pim can be synthesized on the frame memory.

This is the outline of the conventional method.

[0026]

However, in the above-mentioned conventional method, since the calculation is performed for each pixel, it takes a long time to perform the operation including the division when projecting all the pixels of the three-dimensional data. However, there is a problem in image synthesis in real-time processing.

In virtual reality, CG for use in moving images and games, and in virtual shops on the Internet, images that follow an ever-changing viewpoint and look at an object from that viewpoint naturally develop. Therefore, it is necessary to develop a new method capable of performing image synthesis more efficiently and at high speed.

Then, the object of the present invention is to
It is an object of the present invention to provide an image synthesizing method capable of synthesizing a three-dimensional object at a higher speed by reducing the amount of projection calculation of three-dimensional data, and securing real-time properties.

[0029]

In order to achieve the above object, the present invention is configured as follows. That is, the present invention captures a three-dimensional object from a plurality of surrounding points to obtain three-dimensional data having luminance data and coordinate data corresponding to each pixel, and obtains three-dimensional data in pixel units in the obtained image. Based on the viewpoint position information of the observer, three-dimensional data in pixel units is projected on an image plane to calculate a projection position,
In the image synthesizing apparatus for synthesizing an arbitrary viewpoint image by synthesizing luminance data corresponding to the projected position, a part of the coordinate data of the pixel-based three-dimensional data among the acquired three-dimensional data is used for the projection. Therefore, three-dimensional data conversion means for holding the coordinate data as it is as projection coordinate data and converting the remaining coordinate data into linear approximation calculation coordinate data for linear approximation calculation, and the three-dimensional data conversion means Coordinate data projection means for calculating a projection position on the image plane based on the projection coordinate data, in order to project the transformed three-dimensional data onto the image plane and synthesize an arbitrary viewpoint image;
A linear approximation calculating means for calculating a projection position calculated by the coordinate data projecting means on the image plane by a linear approximation calculation based on the projection position and the coordinate data for linear approximation calculation.

According to this configuration, a three-dimensional object to be image-combined is photographed from a plurality of points around the image by a video information acquiring unit and a distance information acquiring unit (for example, a camera and a range finder), and each pixel is corresponded. To obtain three-dimensional data with the brightness data and coordinate data
Part of the acquired coordinate data of the three-dimensional data in pixel units, for example, the central pixel in each line direction of the blocked image is held as it is as projection coordinate data for projection, and the remaining coordinate data is stored in the remaining coordinate data. On the other hand, the linear approximation operation coordinate data is converted for the linear approximation operation, and the converted three-dimensional data is projected by the coordinate data projection means in order to project the image data on an image plane to synthesize an arbitrary viewpoint image. The projection position on the image plane is calculated based on the coordinate data for use, and based on the projection position and the coordinate data for linear approximation calculation, the projection position on the image plane is calculated by the linear approximation operation by the linear approximation calculation means. Then, the luminance data of the pixel of the converted three-dimensional data is arranged at the calculated projection position to synthesize an arbitrary viewpoint image.

In the present invention, the position calculation for projection is performed accurately for some pixels of the blocked image, and the projection of most of the coordinate data is obtained by a linear approximation operation. In synthesizing the arbitrary viewpoint image, the calculation is reduced, and therefore, the arbitrary viewpoint image can be synthesized in real time.

[0032]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) In the present embodiment, for each block of an image divided into an 8 × 8 pixel configuration, calculation is performed for each line and only for one pixel at the center of the line. The amount of calculation is reduced for the other seven pixels of the line by approximating the value obtained at the center pixel by approximation, and the details will be described below.

FIG. 1 is a schematic block diagram of an image synthesizing apparatus according to the present invention. In the figure, reference numeral 101 denotes a CPU (processor), which plays a central role in calculation and control of the system of the present invention. Reference numeral 102 denotes a memory, and reference numeral 103 denotes a frame memory used for image synthesis processing of an arbitrary viewpoint.

Reference numeral 104 denotes a digital camera, which is an imaging device for capturing a digital image of a subject.
Reference numeral 5 denotes a distance information acquisition device that acquires distance information of a subject by a range finder and obtains coordinate data of each pixel.

Reference numeral 106 denotes an interface for taking in data of the digital camera 104 and the range finder 105 into the CPU 101. Reference numeral 107 denotes a turntable for mounting and rotating a subject. Reference numeral 108 denotes a driver for rotating the turntable 107 at predetermined angle intervals under the control of the CPU 101, and 109 denotes a large-capacity storage medium for storing acquired data and the like. . An input device 110 corresponds to a keyboard, a mouse, a trackball, or the like operated by an operator. Reference numeral 111 denotes an output device such as a display.

The system of the present invention is capable of synthesizing a three-dimensional image of an image of an object viewed from an arbitrary viewpoint in real time from image data of the object obtained by imaging at predetermined intervals. By performing the processing as shown in FIG. 2, the amount of calculation is reduced and real-time synthesis of an arbitrary viewpoint image is enabled.

FIG. 2 is a flowchart showing an image synthesizing method according to the first embodiment of the present invention.

In FIG. 2, in step S1, a three-dimensional object is photographed from a plurality of surrounding points by a camera 104 and a range finder 105 to acquire three-dimensional data Dtdm having luminance data Dlum and coordinate data Dxz corresponding to each pixel.
Dimensional data acquisition step, S2 is three-dimensional data D per pixel
A part of the coordinate data Dxz of tdm is held as projection coordinate data Dpxz as it is for projection, and the remaining coordinate data Dxz is converted to linear approximation operation coordinate data Dra for linear approximation operation. A three-dimensional data conversion step of creating converted three-dimensional data Dttdm together with the luminance data Dlum, S3 is a coordinate data projection step of calculating a projection position on an image plane based on projection coordinate data Dpxz, and S4 is a coordinate data projection step S3. A linear approximation operation step of calculating a projection position on an image plane by a linear approximation operation based on the calculated projection position and the linear approximation operation coordinate data Dra; S5
Is a luminance data synthesizing step of synthesizing the arbitrary viewpoint image Pim by synthesizing the luminance data Dlum corresponding to the projected position with the frame memory.

<Processing Flow> The basic flow in the system of the present invention is the same as that of the prior art described with reference to FIG.

That is, the CPU 101 firstly
Acquire three-dimensional data (implementation of three-dimensional data acquisition step S1). In the three-dimensional data obtaining step S1, a three-dimensional object serving as a subject TG is photographed from a plurality of points around the camera 104 using a camera 104 for capturing an image of a subject and a range finder 105 for obtaining a range image. Then, the obtained image is obtained as three-dimensional data Dtdm. That is, for the obtained image, three-dimensional data Dtdm which is data having luminance data Dlum and coordinate data Dxz corresponding to each pixel is obtained.

Specifically, the subject TG is placed on the turntable 107, the camera 104 and the range finder 105 are fixed at fixed positions, and the camera 104 and the range finder are fixed to the subject TG of the turntable 107. The directions of the 105 optical axes are matched.

Under the control of the CPU 101, the turntable 107 is rotated, for example, every 15 degrees, and the data of the subject TG is acquired from the camera 104 and the range finder 105 at each angle. The acquired data is stored and stored in the large-capacity external storage device 109.

That is, the range finder 105
For example, by using a laser beam to measure the distance to each position on the subject surface, coordinate data of each position on the subject surface is obtained. To obtain three-dimensional data of the subject TG, the subject TG must be turned. The digital camera 104 and the range finder 105 are fixedly arranged at a fixed position outside the turntable 107 with the optical axis thereof facing the subject TG. Then, the optical axes of the digital camera 104 and the range finder 105 are aligned and calibration (position adjustment) is performed so that data for each pixel overlaps, and shooting is started. Turntable 1 every time
07 is driven to rotate so that the large-capacity external storage device 10
9, the image data of the entire circumference of the subject TG (2
4) are obtained.

Each image data is composed of brightness data obtained by the camera 104 and coordinate data obtained by the range finder 105. Each data has pixels of the image size. In the luminance data, each pixel has luminance data (rdij, grij, blij), and in the coordinate data, each pixel is a subject T which is a three-dimensional object.
It has data of floating point coordinates (xij, zij) of each point (each position) on the three-dimensional object surface with the center of G as the origin. All the luminance data and coordinate data composed of the plurality of images are collectively referred to as three-dimensional data Dtdm.

Then, the coordinate data projecting step (step S
In 3), using this three-dimensional data Dtdm, the projection position is calculated by projecting coordinate data Dxz for each pixel on an image plane based on the three-dimensional data Dtdm in pixel units and the viewpoint position information of the observer.

Then, the luminance data synthesizing step (step S
In 5), a position on the frame memory 103 corresponding to the projected position (calculated projection position)
By writing the luminance data Dlum of the projection pixel, the arbitrary viewpoint image Pim is synthesized in the frame memory 103.

The basic flow is like this,
The present invention differs from the prior art in that coordinate data D
When the projection position of the luminance data is obtained by projecting xz, the projection position is calculated only in the coordinate data projection step S52 in the related art, but the present invention is newly provided with a linear approximation operation step S4. , The projection of most of the coordinate data Dxz,
In order to easily perform the linear approximation operation, a point where the coordinate data Dxz of the three-dimensional data Dtdm is converted into projection coordinate data Dpxz and coordinate data Dra for the linear approximation operation in order to facilitate the linear approximation operation. The point is that a three-dimensional data conversion step S2 for creating the dimension data Dttdm is provided.
Thus, the processing is speeded up, and an arbitrary viewpoint image of the subject can be synthesized in real time.

<Principle of Acceleration of Projection Operation> In the system of the present invention, the speed of the operation process is increased by performing the projection operation by a linear approximation operation. I will explain why.

The method of calculating the projection position g (x, z) on the image plane by projecting the coordinate data (x, z) described above is as follows.

G (x, z) = − fx / (z−r) Here, certain coordinate data D having a data value (x0, z0)
Based on the value of the projection position g (x0, z0) on the image plane of xz,
The projection position g (x1, z1) of the next point (x1, z1) on the image plane
z1) = g (x0 + h, z0 + k) is obtained by linear approximation.

[0052]

(Equation 1) Becomes

Therefore, the projection position g (x1, z1) of the data value (x1, z1) as the coordinate data Dxz on the image plane is obtained.
In order to obtain 1), instead of the data value (x1, z1) as the coordinate data Dxz, (x1 ′, z1 ′) =
((X1 / x0), (z1-z0) / fx0) and hold as three-dimensional data Dtdm, g (x
The projection position g (x1, z1) can be approximated only by a combination of multiplication and addition of (0, z0), x1 ′, z1 ′. That is, this makes it possible to eliminate the division operation from the operation processing.

Therefore, for example, as the eight points P0 to P7 located in the vicinity, P0 (x0, z0), P1 (x1, z1), P2 (x2, z2), P3 (x3, z3), P4 ( x4, z4), P5 (x5, z5), P6 (x6, z6), P7 (x7, z7), a point at the center position, for example, a point P3 (x3, z
Only the projection position g (x3, z3) of 3) is calculated by the calculation of g (x3, z3) = − fx3 / (z3-r), and the remaining seven points Pi (i = 0 to 2,
The projection positions of 4 to 7) are

(Equation 2) Approximately calculated by calculating

First, the case where the projection positions of eight points are calculated without performing linear approximation for comparison will be described. In this case, the following calculation is performed.

G (x0, z0) = − fx0 / (z0−r), g (x1, z1) = − fx1 / (z1-r), g (x2, z2) = − fx2 / (z2-r) , G (x3, z3) =-fx3 / (z3-r), g (x4, z4) =-fx4 / (z4-r), g (x5, z5) =-fx5 / (z5-r), g (X6, z6) = − fx6 / (z6-r), g (x7, z7) = − fx7 / (z7−r) That is, in this method, (subtraction × 1 + multiplication × 1 + division × 1) × 8 Is required. That is, the division is performed eight times in total, and the operations other than the division are performed sixteen times in total.

On the other hand, in the case of the system adopted in the system of the present invention, in which linear approximation is performed to calculate eight projection positions, the following operation is performed.

[0058]

(Equation 3)

That is, in this case, (subtraction × 1 + multiplication ×
1 + division × 1) + multiplication + (multiplication × 2 + addition × 1) × 7 That is, there is a total of one division, and a total of 24 operations other than division. In the case of this method, the operation itself is (24 + 1) − (8 + 16) = compared to the case of the comparative example method in which eight projection positions are calculated without performing linear approximation.
If one simply compares the number of operations with 1, the number is larger by one time. However, in the case of the present invention, the division is performed once and the other operations are performed 24 times.

In the arithmetic processing by the CPU 101, the division has the heaviest load, and the other operations have a much lighter load.

Now, assuming that the time required for other operations other than the division is the same T time, and assuming that the division process requires an operation time 1.1 times or more of the T, the former becomes (8) × T × 1.1) + (16 × T) = 26.4T, and the latter is (T × 1.1) + (T × 23) = 24.1T
In the linear approximation operation, the total amount of the operation is smaller by "2.3T". In general, the division is more complicated than other operations, and is about 1.5 to 2 times. It takes (or more) time.

Therefore, the operation by the linear approximation employed in the system of the present invention is extremely effective in increasing the speed as compared with the conventional system.

In the system according to the present invention, for each block, each line of the block corresponds to 8 lines of each line.
Of the constituent pixels of the point, one point at the center position is calculated with high accuracy, and the other seven pixels are subjected to linear approximation to achieve high-speed processing.

Based on the above description of the linear approximation operation effective for speeding up, the image synthesis processing by the system of the present invention will be described.

This processing is performed by the CP in the system of the present invention.
This is realized by U101 according to FIG.
First, three-dimensional data is obtained.

<Acquisition of Three-Dimensional Data> In the three-dimensional data acquisition step S1, as described above, an image over the entire circumference of the subject TG (three-dimensional object) which is the object of image composition is set at, for example, each angular position of 15 degrees. In each case, the brightness data D of the subject TG captured by the digital camera 104 and the range finder 105 at a fixed position and captured at each angular position
lum and coordinate data Dxz are obtained.

Here, as the luminance data Dlum, each pixel constituting the screen has a luminance (rdij, grij, blij).
j), and in the coordinate data Dxz, each pixel constituting the screen has 3
Floating point coordinates (xij,
zij) is obtained as data having the form of data.

Then, for each of the plurality of images, three-dimensional data Dtdm including luminance data Dlum and coordinate data Dxz of all pixels constituting the image are obtained, and the three-dimensional data Dtdm is large. It is stored in the capacity storage device 109.

The above processing is the processing of the three-dimensional data acquisition step S1. When the three-dimensional data Dtdm is obtained in this way, the CPU 101 enters a next process of a three-dimensional data conversion step S2.

<Three-dimensional data conversion> The three-dimensional data conversion step S2 is a process for facilitating the linear approximation operation. In the three-dimensional data conversion step S2, the obtained three-dimensional data Dtdm is converted into converted three-dimensional data Dttdm.

Here, the converted three-dimensional data Dttdm is data obtained by converting the coordinate data Dxz of the three-dimensional data Dtdm into coordinate data Dpxz for projection and coordinate data Dra for linear approximation calculation.

By performing the processing in the three-dimensional data conversion step S2, the coordinate data Dx of the three-dimensional data Dtdm is previously determined.
z is converted into converted three-dimensional data Dttdm consisting of projection coordinate data Dpxz and linear approximation calculation coordinate data Dra.

The flow of the method of converting the three-dimensional data Dtdm to the converted three-dimensional data Dttdm in the three-dimensional data conversion step S2 will be described with reference to FIGS.

First, as shown by reference numeral 201 in FIG. 3A, the CPU 101 divides the three-dimensional data Dtdm, which has been handled on a pixel-by-pixel basis, into blocks of an n × n pixel size. They are handled collectively in block units of nxn pixels.

This is carried out collectively at the end of the three-dimensional data acquisition step S 1, or, as shown in FIG. 7, every time an image is taken by the camera 603 and the range finder 604, that is, the image is Luminance data Dlum
When the coordinate data Dxz is obtained, the block is formed.

In the image 201 shown in FIG. 3A, an image of a 400 × 400 pixel configuration in which a hut is shown is
A state in which the image is divided into blocks of × 40 pixels and divided into 100 blocks is shown.

For each of the 100 blocks, the projection operation is strictly performed only for a certain pixel in the line by utilizing the correlation in the coordinate data in the block, and the remaining pixels are linearly adjusted as described above. Processing is performed so as to obtain the projection destination by an approximate calculation.

Naturally, in the case of a block in which the luminance data and coordinate data are filled with meaningless values in all the pixels in the block, a block image 202 shown in FIG. Then, the meaningless block is determined as an invalid block and deleted, and only the meaningful block is left.

The CPU 101 performs coordinate data conversion in the block on the image thus divided.

<Coordinate Data Conversion in Block> A method of converting coordinate data in a block will be described with reference to FIG. In FIG. 4, a case where the block size is 8 × 8 pixels will be described as an example.

In the cross section of the three-dimensional data described with reference to FIG. 8, eight points that exist in the vicinity are arranged in a certain row in this block. This three-dimensional data conversion step S2
Only the data in the third column of the × 8 pixel block stores coordinate data of (xij, zij) as projection coordinate data Dpxz in the same manner as before, and the columns (0 to 2) other than the third column are stored. Columns, 4 to 7 columns), as shown in FIG.
(Xij, zij) = (xij / xi3, (zij-z
i3) / fxi3) is calculated in advance and stored as coordinate data Dra for linear approximation calculation.

As a result, the three-dimensional data Dtdm is converted into converted three-dimensional data Dttdm in the three-dimensional data conversion step S2.

Finally, on the basis of the converted three-dimensional data Dttdm, how the coordinate data projection step S
3. Whether the arbitrary viewpoint image Pim is synthesized by the linear approximation operation step S4 and the luminance data synthesis step S5 will be described.

Since independent calculations are performed in the vertical direction, it is considered that the transformed three-dimensional data Dttdm of a certain row of a certain 8 × 8 block is projected on an image plane.

At this time, first, the projection position g (xi3, zi3) of the point (xi3, zi3) in the third column, which is the projection coordinate data Dpxz, on the image plane is determined in the coordinate data projection step S3. 8, the projection position on the image plane is calculated as g (xi3, zi3) = − fx3 / (zi3-r). Columns other than the third column (0-2 columns, 4
~ 7 columns) in the linear approximation operation step S4
Output g (xi3, zi) in the coordinate data projection step S3
Using 3) and the linear approximation operation,

(Equation 4)

Then, the projection position g (xij, zij) is calculated.

In the luminance data synthesizing step S5, the luminance data Dlim of the converted three-dimensional data Dttdm is converted to the calculated image plane irrespective of whether they are strictly projected or linearly approximated. Projection position g (xi
(j, zij) (overwrite the frame memory 103)
To synthesize an arbitrary viewpoint image Pim.

The same projection and linear approximation are performed for all the pixels in all the blocks included in the converted three-dimensional data Dttdm to calculate the projection position g (xij, zij) and make a determination using the Z buffer. While the projection position g
The data value (r) of the luminance data Dlum is stored in (xij, zij).
dij, grij, blij) are overwritten to synthesize an arbitrary viewpoint image Dim. (However, rd
ij indicates red component information, grij indicates green component information, and blij indicates blue component information.) It has been mentioned earlier that the synthesis of an arbitrary viewpoint image by this linear approximation is faster than the synthesis method in which all coordinates are directly projected. It is on the street.

As a result, the system of the present invention can synthesize an arbitrary viewpoint image in real time.

In the above, with respect to each block of the image divided into the 8 × 8 pixel configuration, the calculation is properly performed for only one central pixel among the eight pixels constituting one line. With respect to the other seven pixels of the line, the amount of calculation was reduced by adopting a method of obtaining a value by approximation from the value obtained at the center pixel. A second embodiment will be described as an example in which the calculation is further performed to further speed up the calculation processing.

(Second Embodiment) FIG. 5 shows a second embodiment of the present invention.
It is a figure explaining an embodiment. The blocks 401 and 402 having the 8 × 8 pixel configuration in FIG. 5 are the same as the blocks having the 8 × 8 configuration in FIG. 4. Only the data in the third column has information (xij, zij) as the projection coordinate data 108. Columns other than the third column (columns 0 to 2 and columns 4 to 7) are stored as (xij ′, zij ′) = (xij / zi3, (zij) as the linear approximation calculation coordinate data Dra.
-Zi3) / fx3) are stored.

Generally, coordinate data and the like are as shown in FIG.
As shown in the block 401 of the 8 × 8 pixel configuration (pixel configuration of 8 rows × 8 columns) shown in FIG. 1, the pixels of one row (horizontal one line) of the eight rows forming the block are arranged from upper left to right. And stored in memory. When the CPU 101 performs a 4-parallel floating-point operation on these pixels, as in block 401 shown in FIG. 5A, (x00, x01, x02, x03), (x04, x0)
5, x06, x07),..., (Xj0, xj1, xj
2, xj3), (xj4, xj5, xj6, xj7),
... and perform parallel operation collectively.

In the image synthesizing method according to the first embodiment, the third column (x03, x1
, X23, x33, x43,..., Xj3,...) And the linear approximation calculation is performed on the other columns, so that a block 401 shown in FIG. It is memory aligned.

However, such a memory alignment does not allow a parallel operation.

Therefore, in the second embodiment, the rows and columns of the block shown in FIG.
(X00, x10, x20, x30), (x40, x5) as in the 8 × 8 pixel configuration block 402 shown in FIG.
0, x60, x70),..., (X0j, x1j, x2
j, x3j), (x4j, x5j, x6j, x7j),
… Is performed to perform memory alignment in the vertical direction.

That is, in the second embodiment,
In the three-dimensional data conversion step S1 in FIG. 2, not only the coordinate data Dra for the linear approximation calculation is calculated in advance to create the converted three-dimensional data Dttdm as in the first embodiment, but also in FIG. By performing the above-described arrangement process in the row and column directions, the 8 × 8 block 40 shown in FIG.
2, the data is stored in this arrangement, and the parallel operation in the vertical direction is performed in the coordinate data projection step S3 and the linear approximation operation step S4. By doing so, it is possible to synthesize an arbitrary viewpoint image at a higher speed than in the first embodiment.

As described above, according to the present invention, a three-dimensional object is photographed from a plurality of surrounding points to obtain three-dimensional data having luminance data and coordinate data corresponding to each pixel. Based on the three-dimensional data and the observer's viewpoint position information, three-dimensional data in pixel units is projected onto an image plane to calculate a projection position, and luminance data corresponding to the projected position is synthesized to synthesize an arbitrary viewpoint image. In the image synthesizing apparatus, part of coordinate data of pixel-based three-dimensional data in the acquired three-dimensional data is held as it is as projection coordinate data for the projection, and the remaining coordinate data is not stored. Three-dimensional data conversion means for converting the coordinate data for linear approximation for linear approximation;
In order to project the converted three-dimensional data by the three-dimensional data conversion means onto an image plane and synthesize an arbitrary viewpoint image,
Coordinate data projection means for calculating a projection position on the image plane based on the projection coordinate data, and coordinate data projection means for calculating the projection position on the image plane by linear approximation calculation based on the projection position and the coordinate data for linear approximation calculation. And a linear approximation calculating means for calculating the projected position. With this configuration, a part of the coordinate data of the three-dimensional data in pixel units, for example, the central pixel in each line direction of the blocked image is held as it is as the projection coordinate data for projection, and the remaining pixel data is retained. The coordinate data is converted to the linear approximation operation coordinate data for the linear approximation operation, and the converted three-dimensional data is projected onto an image plane to synthesize an arbitrary viewpoint image. Means for calculating a projection position on the image plane based on the projection coordinate data, and, based on the projection position and the coordinate data for linear approximation calculation, using a linear approximation means to perform a linear approximation calculation on the projection position on the image plane. Is calculated, and the luminance data of the pixel of the converted three-dimensional data is arranged at the calculated projection position to synthesize an arbitrary viewpoint image.

In the present invention, the position calculation for projection is performed accurately for some pixels of the blocked image, and the projection for most other coordinate data is obtained by a linear approximation operation. In synthesizing the arbitrary viewpoint image, the calculation is reduced, and therefore, the arbitrary viewpoint image can be synthesized in real time.

The present invention is not limited to the examples shown in the above embodiments, and can be variously modified in the practical stage without departing from the gist thereof. In the above-described embodiment, in the above embodiment, for each block of the image divided into the 8 × 8 pixel configuration, only the center pixel of the eight pixels constituting one line is calculated for each line. The calculation amount is reduced by approximating the values of the other seven pixels of the line from the values obtained at the central pixel by approximation. Instead of one pixel each time, it is conceivable to use a plurality of pixels within a range in which a load is not imposed on the arithmetic processing.

Further, in the present invention, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some components are deleted from all the components shown in the embodiment, at least one of the problems described in the column of the problem to be solved by the invention can be solved, and the problem described in the column of the effect of the invention can be solved. In the case where at least one of the effects described above is obtained, a configuration from which this component is deleted can be extracted as an invention.

The method described in the embodiment of the present invention can be executed by a computer as a program, such as a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, CD-ROM).
-R, CD-RW, DVD, MO, etc.), and can be stored in a recording medium such as a semiconductor memory and distributed.
It can also be distributed by transmission over a network.

[0102]

As described above, according to the image synthesizing method of the present invention, in order to calculate the projection position in pixel units and synthesize an image of a three-dimensional object viewed from an arbitrary viewpoint, the conversion 3 By creating dimension data and performing a linear approximation operation, the amount of operation can be reduced, and higher real-time properties can be realized. Further, further speedup is possible by a special parallel operation.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the present invention, and is a block diagram showing a schematic configuration example of a system of the present invention.

FIG. 2 is a view for explaining the present invention, and is a flowchart for explaining an image synthesizing procedure according to the first embodiment of the present invention;

FIG. 3 is a diagram for explaining the present invention, and is a diagram for explaining how to block an image in a three-dimensional data conversion step according to the first embodiment of the present invention.

FIG. 4 is a diagram for explaining the present invention, and is a diagram for explaining a coordinate data conversion method in a three-dimensional data conversion step according to the first embodiment of the present invention.

FIG. 5 is a diagram for explaining the present invention, and is a diagram for explaining a three-dimensional data conversion method according to the second embodiment of the present invention.

FIG. 6 is a flowchart showing a flow of an image synthesizing method as a conventional technique.

FIG. 7 is a diagram for explaining a three-dimensional data acquisition step in a conventional image combining method.

FIG. 8 is a view for explaining a coordinate data projecting step and a luminance data combining step in an image combining method as a conventional technique.

[Explanation of symbols]

101: CPU (processor) 102: Memory 103: Frame memory 104: Digital camera (imaging device) 105: Range finder (distance information acquisition device) 106: Interface (I / F) 107: Turntable 108: Driver 109: Storage medium 110: Input device (keyboard, etc.) 111: Output device (display, etc.) TG: Subject (3D object) 201: Blocked luminance data / coordinate data 202: Luminance data / coordinate data from which meaningless portions have been deleted 401: 8 × 8 blocks aligned in the horizontal direction 402: 8 × 8 blocks aligned in the vertical direction

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (reference) H04N 5/225 H04N 5/225 Z 5L096 5/232 5/232 Z J 5/262 5/262 F term (reference) 5B050 BA04 BA09 BA13 CA07 DA07 EA07 EA13 EA19 EA24 EA27 EA28 FA02 FA09 5B057 BA02 CA12 CB13 CD14 CE08 DA07 DB03 5B080 BA07 DA07 DA08 GA02 GA22 5C022 AA13 AB24 AB45 AB61 AB62 AB68 AC02 AC69 AC74 CA00 CA01 5C023 AA11 AA11 DA11 FA66 FA69

Claims (4)

[Claims] 1. A three-dimensional object is photographed from a plurality of surrounding points to obtain three-dimensional data having brightness data and coordinate data corresponding to each pixel, and observe and obtain three-dimensional data in pixel units in the obtained image. Image synthesizing apparatus that projects three-dimensional data in pixel units on an image plane based on the viewpoint position information of a user to calculate a projection position, and combines luminance data corresponding to the projected position to synthesize an arbitrary viewpoint image. In the acquired three-dimensional data, part of coordinate data of pixel-based three-dimensional data is held as it is as projection coordinate data for the projection, and the remaining coordinate data is subjected to a linear approximation operation. Three-dimensional data conversion means for converting the coordinate data for linear approximation operation into the image data, and projecting the converted three-dimensional data by the three-dimensional data conversion means onto an image plane to generate an arbitrary viewpoint image. In order to synthesize,
Coordinate data projection means for calculating a projection position on the image plane based on the projection coordinate data, and coordinate data projection means on the image plane by a linear approximation operation based on the projection position and the linear approximation operation coordinate data. A linear approximation calculating means for calculating the projected position. 2. A three-dimensional data acquiring step of photographing a three-dimensional object from a plurality of surrounding points and acquiring three-dimensional data having luminance data and coordinate data corresponding to each pixel, A coordinate data projecting step of projecting pixel-based three-dimensional data on an image plane based on the observer's viewpoint position information to calculate a projected position, and combining luminance data corresponding to the projected position to form an arbitrary viewpoint image In an image synthesizing method including a luminance data synthesizing step of synthesizing, a part of coordinate data of three-dimensional data in pixel units is directly held as projection coordinate data for projection, and a linear approximation operation is performed on the remaining coordinate data. A three-dimensional data conversion step of converting the coordinate data for linear approximation calculation into a three-dimensional data, and projecting the converted three-dimensional data onto an image plane to synthesize an arbitrary viewpoint image. A coordinate data projection step of calculating a projection position on the image plane based on the target data; and a linear approximation calculation step of calculating a projection position on the image plane by a linear approximation operation based on the projection position and the coordinate data for linear approximation operation. An image synthesizing method, comprising: 3. The coordinate data projecting step and the linear approximation operation step collectively perform a parallel operation on a plurality of pixel data, and the three-dimensional data conversion step further converts the three-dimensional data in a format corresponding to the parallel operation in advance. 3. The image synthesizing method according to claim 2, wherein: 4. A pixel having luminance data and coordinate data corresponding to each pixel obtained by photographing a three-dimensional object from a plurality of surrounding points.
Using three-dimensional data, of the three-dimensional data, part of the coordinate data of the three-dimensional data in pixel units is held as it is as projection coordinate data, and the remaining coordinate data is linearly approximated for linear approximation calculation. A three-dimensional data conversion step of converting the coordinate data for calculation; and projecting the converted three-dimensional data in the three-dimensional data conversion step onto an image plane to synthesize an arbitrary viewpoint image. A coordinate data projection step of calculating a projection position on the image plane based on the viewpoint position information of the image data; and a linear approximation operation based on the projection position and the coordinate data for the linear approximation operation. A linear approximation calculating step of calculating the projected position, and an image combining step of combining luminance data corresponding to the projected position to form an arbitrary viewpoint image. Step a, medium recording the read and executable image synthesis program on the computer, characterized in that it comprises a.