JP2003090714A - Image processing apparatus and image processing program - Google Patents

Abstract

(57) [Summary] To create three-dimensional computer graphics of a subject from a stereo pair image of the subject. SOLUTION: A stereo pair image is photographed with an object to be expressed in three-dimensional computer graphics as a subject. Extract the corresponding points of the object from the captured stereo pair image, left image of these points,
The two-dimensional coordinate value on the right image is calculated. Then, from the combination of these points, the stereo pair image is photogrammetrically measured. At this time, the accuracy of calculation is increased while updating the combination of points using a genetic algorithm. When the three-dimensional coordinate data of the object is obtained by the calculation, a polygon is generated using the data, and a three-dimensional model of the object is described by VRML. Then, an image is cut out from one of the stereo pair images in accordance with the polygon, and this is mapped as a texture to the three-dimensional model.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing program, for example, to generate three-dimensional computer graphics of a subject from a stereo pair image.

[0002]

2. Description of the Related Art Due to the recent development of computer technology,
Three-dimensional computer graphics (hereinafter referred to as 3DCG) in which three-dimensional information is expressed by a computer is used in various fields. For example, the field of entertainment such as games and movies using 3DCG, and three-dimensional CAD (Computer Aided Design)
There is a field of construction using n).

The 3DCG is a technique for expressing a solid (hereinafter referred to as an object), which is an expression target in a three-dimensional space, by a two-dimensional still image or moving image. In the 3DCG, the appearance of the object when the point of view of the object changes or when the object moves is calculated from the three-dimensional information of the object and is expressed by a two-dimensional plane.

There are various methods of creating the 3DCG. For example, there is a method of creating a 3D CG by defining a three-dimensional model of an object and pasting a texture on it (texture mapping). The three-dimensional model includes a wireframe model in which an object is represented by a straight line, a surface model in which polygons such as triangles and quadrilaterals are combined and represented as a polyhedron, or a solid model including information on the internal configuration of the object. is there.

A surface model will be described as an example. The surface model is an approximate representation of an object as a polyhedron by combining constituent units each having a polygonal shape such as a triangle or a quadrangle called a polygon.
A three-dimensional model can be defined by generating the surface of an object from a combination of these polygons and giving the computer the three-dimensional coordinate values of each vertex of the polygon.

[0006] Normally, different types of polygons such as triangles and quadrilaterals are not mixed, and polygons are generated by a single polygon such as only triangles in the case of triangles. Also, the shape of each polygon is usually different to best represent the object. By the way, when the polygon is a triangle, the three vertices are always on the same surface, which has an advantage that the analysis is easy, but has a disadvantage that the number of polygons increases. In addition, when the polygons are formed in a quadrangle or a pentagon, there is an advantage that the number of polygons is reduced, but there is a disadvantage that a problem occurs when each vertex is not on the same plane.

Texture mapping is performed by pasting a two-dimensional picture such as a photograph or a pattern onto a modeled object. For example, after generating a three-dimensional model of an automobile, a photograph of the automobile is pasted on the surface of this three-dimensional model to obtain a realistic 3DCG. In this way, by inputting the three-dimensional information of the automobile and the texture information of the automobile into the computer, it is possible to display images of the automobile viewed from various angles and distances on the screen of the computer.

[0008]

However, the production of 3DCG requires a high degree of specialized knowledge, specialized skills, and expensive equipment. Moreover, there are many parts that rely on humans (and experts) in the generation of the three-dimensional model, the pasting of textures, and other processes. Recently, application software, etc., which partially assists in the creation of 3DCG, has also come into the market, but in order to use this, the user requires a fairly high level of computer knowledge and is complicated. It is difficult to generate a large 3DCG. In particular, the face of a person who has a great demand for 3DCG is composed of a complex free-form surface, and it is difficult to form 3DCG. Furthermore, a system that automatically acquires three-dimensional data and automatically performs three-dimensional modeling including texture mapping has not been realized.

Therefore, an object of the present invention is to provide an image processing apparatus capable of automatically photogrammetrically measuring the object from a stereo pair image of the object, and automatically generating 3DCG of the object from the survey result.
And to provide an image processing program.

[0010]

In order to achieve the above-mentioned object, the present invention is, in the invention described in claim 1, a first image acquisition for acquiring a first image of a subject taken from a predetermined direction. Means, second image acquisition means for acquiring a second image of the subject taken from a direction different from the predetermined direction, and extracting characteristic points of the subject from the first image, Moreover, the characteristic point of the subject is extracted from the second image, and the point extracted from the first image and the point extracted from the second image correspond to each other, and the extracted point is the corresponding point. Photogrammetric means for photogrammetrically measuring the subject and acquiring three-dimensional information of the subject using the two-dimensional coordinate values of the points on the first image and the two-dimensional coordinate values on the second image. An image processing apparatus is provided, which comprises: In the invention according to claim 2, a subject whose distribution tendency of characteristic feature points in the first image and the second image is known in advance is photographed from a predetermined position with respect to the subject. An image, the tendency of the distribution of the feature points recorded in the first image and the second image is known, and the corresponding point extraction means knows the feature points of the subject in advance. The image processing apparatus according to claim 1, wherein the characteristic points extracted from the first image and the second image are associated with each other by using a tendency of distribution. In the invention according to claim 3, the photogrammetric means is a combination generation means for generating a plurality of point combinations from the points acquired by the corresponding point acquisition means, and an orientation means for orienting each of the generated combinations. A convergence determination means for determining the convergence of the orientation result, and until the convergence determination means determines that the calculated value has converged, by a genetic operation in a genetic algorithm,
An image processing apparatus according to claim 1 or 2, further comprising: an updating unit that updates the combination. According to a fourth aspect of the present invention, there is further provided polygon generating means for generating a polygon for generating a stereo model of the subject using the survey result obtained by the photogrammetric means. An image processing apparatus according to claim 1, claim 2 or claim 3 is provided.
In the invention according to claim 5, the polygon generating means is
Voronoi region generation means for generating Voronoi from points on the subject whose three-dimensional coordinate values are known by photogrammetry, and Delaunay division means for generating polygons by performing Delaunay division on the generated Voronoi regions. An image processing apparatus according to claim 4, wherein: In the invention according to claim 6, a description means for describing the three-dimensional model of the subject represented by the generated polygon in a predetermined computer language, and the contents described by the description means in a predetermined file. An image processing apparatus according to claim 4 or 5, further comprising: an output unit that outputs the image. According to a seventh aspect of the present invention, a texture acquisition unit that acquires a texture to be mapped to the three-dimensional model of the subject from at least one of the first image and the second image, and a texture acquired by the texture acquisition unit. Update means for mapping the three-dimensional model of the subject and updating the file;
The image processing apparatus according to claim 6, further comprising: The invention according to claim 8 provides the image processing apparatus according to claim 6 or 7, wherein the predetermined computer language is VRML. In the invention according to claim 9, a first image acquisition function of acquiring a first image of a subject taken from a predetermined direction, and a second image capturing function of the subject taken from a direction different from the predetermined direction. A second image acquisition function for acquiring an image,
The characteristic points of the subject are extracted from the first image, the characteristic points of the subject are extracted from the second image, and the points extracted from the first image and the second point are extracted. Using the corresponding point extraction function that associates the points extracted from the image, the two-dimensional coordinate value of the extracted point on the first image and the two-dimensional coordinate value of the second image, the subject And a photogrammetry function for obtaining three-dimensional information of the subject by photogrammetry, and an image processing program or a computer-readable storage medium storing the image processing program. Also, a first image acquisition unit, a second image acquisition unit, a corresponding point extraction unit,
In a computer equipped with photogrammetric acquisition means,
The first image acquisition step of acquiring a first image of the subject taken from a predetermined direction by the first image acquisition means, and the second image acquisition means defining the subject as the predetermined direction A second image acquisition step of acquiring a second image photographed from a different direction, the characteristic point of the subject is extracted from the first image by the corresponding point extraction means, and the second image acquisition step is performed. A characteristic point of the subject is extracted from the image, and a corresponding point extraction step of associating the point extracted from the first image with the point extracted from the second image; Photogrammetric surveying the subject using the two-dimensional coordinate value of the point on the first image and the two-dimensional coordinate value on the second image to obtain three-dimensional information of the subject. Characterized by being composed of steps and To provide an image processing method.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION [Outline of First Embodiment] In the present embodiment, first, three objects are determined by photogrammetry.
Obtain dimensional data automatically. In this embodiment, a genetic algorithm described later is used to speed up the calculation of photogrammetry. Photogrammetry is a technique that is generally used in the field of civil engineering. It is a technique that takes a subject from two different directions and calculates the three-dimensional data of the subject from the coordinate values in both photos of the subject. is there.

After obtaining the three-dimensional data of the object by photogrammetry, this is Delaunay divided (described later) to generate polygons, and a three-dimensional model of the object is generated. Appropriate polygons for creating a three-dimensional model can be generated by Delaunay division. Then, the photograph taken during the photogrammetry is mapped as a texture on the three-dimensional model to obtain the 3DCG of the object.

[Details of the First Embodiment] The preferred embodiments of the present invention will be described below. In the present embodiment, the upper body including a human head is used as an example of the object. This is because there is a great demand from the market to make the face 3DCG, and the free phase that composes the face is very complicated. If this can be automatically made 3DCG, it is relatively easy to make another object 3DCG. This is because.

FIG. 1A is a diagram showing a configuration of a photographing box 10 for photographing the user's head 8 as seen from the top of the head, and FIG. 1B shows the photographing box 10. It is the figure seen in the arrow line A direction of Fig.1 (a). The shooting box 10 is a small room for shooting a stereo pair image of the user's head 8. Here, the stereo pair image is 1
A set of images of two subjects taken from different angles.

A chair 9 whose height is adjustable is fixed to the floor of the photographing box, and a mirror 7 is installed on the front wall of the user. The eye-gaze target line 6 is drawn on the mirror, and the user adjusts the height of the chair 9 so that his / her eyes reflected on the mirror 7 overlap the eye-gaze target line 6. By adjusting the height of the chair 9 in this way, the user's head 8 is positioned at a predetermined place.

A digital left camera 61 and a right camera 62 are installed on the wall in front of the user, and the installation positions of these cameras are known in advance. In addition, when the head 8 is photographed, the reference point 5 is set on the background wall.
a, 5b, and 5c are drawn. Reference points 5a, 5b, 5
The setting position of c is known in advance, and is used as a reference point when performing photogrammetry later. In addition, the digital camera is a CC with a large number of light receiving elements arranged instead of film.
It is a camera that electrically acquires image data of a subject by a D (Charge Coupled Device).
The camera data of the left camera 61 and the right camera 62 (lens focal length, CCD pixel size, vertical and horizontal pixel numbers, etc.) are aligned, and these values are known in advance. The camera data will be photogrammetrically measured for the user's face later.
Used when acquiring dimensional data.

As described above, since the positions of the left camera 61 and the right camera 62 are fixed at known positions in advance, for example, the center of the CCD of the left camera 61 is the origin, the photographing direction is the Z axis, and the vertical upward direction. When turning the right-hand screw in the Y-axis and in the direction from the Y-axis to the Z-axis, and letting the right-hand screw advance direction be the X-axis
Left camera 61, right camera 62, and reference points 5a, 5b,
The coordinate value of 5c is known in advance. The coordinate system set in this way is called a camera coordinate system. Further, the user sits on the chair 9 fixed at a predetermined position, and adjusts the height of the head 8 so that the line of sight of the head 8 matches the line of sight target line 6. The coordinate value of will be determined to some extent. As will be described later, when the position of the camera and the position of the subject are determined in this way, it becomes easy to acquire corresponding points between the left image and the right image, and it becomes easy to perform photogrammetry.

The stereo pair image of the user's face is obtained by simultaneously photographing the user's face using the left camera 61 and the right camera 62 in the photographing box 10 configured as described above. A 3DCG creating device 11, which is an image processing device composed of a computer, is installed near the photographing box 10, and a stereo pair image of a user's face is transmitted to the 3DCG creating device 11. The 3DCG creation device 11 generates a 3DCG of the user's face from the stereo pair image of the user.

The generated 3DCG is a 3DCG creating device 11
It can be displayed on the display device installed in. In addition, the data of 3DCG is transferred to, for example, a mobile phone to display the 3DCG of the user's face on the liquid crystal screen of the mobile phone, or transmitted to another terminal device via the Internet.
Alternatively, it can be stored in a magnetic disk or a semiconductor memory.

FIG. 2 is a diagram showing an example of the configuration of the 3DCG creating apparatus 11. The 3DCG creation apparatus 11 is configured by connecting an input device 34, an output device 38, a communication control device 42, a storage device 48, a storage medium drive device 46, an input / output interface 44, etc. to the control unit 26 via a bus line 43. ing. The bus line 43 is a transmission line that transmits signals between the CPU 28 and other devices.

The control unit 26 is a CPU (Central).
Processing Unit 28, ROM (Re
ad Only Memory, RAM (Ran)
Dom Access Memory) 30 and the like. The control unit 26 is controlled by the CPU 28 to generate the 3DCG of the user's face from the stereo pair image according to the 3DCG generation program 50, control the input / output with the left camera 61, the right camera 62 and the Internet, or the 3DCG. It controls the entire creation device 11, and so on.

The ROM 30 is a read-only memory that stores various programs, data and parameters for the CPU 28 to perform various calculations and controls. CPU2
8 can read programs, data, parameters, etc. from the ROM 30. Since the ROM 30 is read-only, the stored contents of the ROM 30 cannot be rewritten or erased.

The RAM 32 is a random access memory used by the CPU 28 as a working memory.
The CPU 28 can write and erase programs and data in the RAM 32. In the present embodiment, the RAM 32 can secure an area for the CPU 28 to perform photogrammetry, generation of a three-dimensional model of the face, texture (photograph of the face) mapping, and the like.

The input device 34 is composed of an input device such as a keyboard, a mouse and a joystick. The keyboard is a device for inputting information such as characters and numbers to the 3DCG creating device 11. The keyboard is composed of keys for inputting kana and English characters, ten keys for inputting numbers, various function keys, cursor keys and other keys.

The keyboard is, for example, the generated 3DCG
It is used when inputting the file name of the file or when performing maintenance on the 3DCG creating apparatus 11. Further, by operating a predetermined function key, the 3DCG of the face displayed on the display device can be rotated or moved in parallel. Further, when the generated 3DCG file is transmitted to another terminal on the Internet, it can be used when inputting the address of the transmission destination.

The mouse is a pointing device. When the user moves the mouse over the mouse pad,
3DC of the face displayed on the display device as the mouse moves
G rotates or translates. In addition, the 3DCG creation device 11 uses a GUI (Graphical User).
If it can be operated using an interface, it is possible to input predetermined information by clicking a button or icon displayed on the display device with a mouse.

The output device 38 is composed of, for example, a display device and a printing device. The display device is composed of, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, or the like, and is a device for presenting image information on a screen. The display device is capable of displaying the generated 3DCG or displaying a screen by GUI.

The printing device is a device for printing, for example, a stereo pair image or the generated 3DCG on a printing medium such as paper. The printing device is composed of various printer devices such as an inkjet printer, a laser printer, a thermal transfer printer, and a dot printer.

The communication control device 42 uses the communication line 3
The device is a device for connecting the DCG creating device 11 to a server device, and is composed of a modem, a terminal adapter and other devices. The communication control device 42 is, for example, the Internet or a LAN (Local Area Ne
3D generated on other terminal devices or server devices connected to these networks.
The CG file can be transmitted to other terminal devices.

The communication control device 42 is controlled by the CPU 28, and is, for example, TCP / IP (Transmission).
n Control Protocol / Intern
It is possible to communicate with other terminal devices and server devices according to a predetermined protocol such as et Protocol).
The CPU 28 can send the 3DCG file to a terminal device such as a user's personal computer or a mobile phone via the communication control device 42.

The storage device 48 is composed of a readable / writable storage medium and a drive device for reading / writing programs and data from / to the storage medium. A hard disk is mainly used as the storage medium, but other storage media such as a magneto-optical disk, a magnetic disk, and a semiconductor memory can be used instead.

The storage device 48 includes a 3DCG creation program 50, camera data 52, a camera control program 54,
An image database 56, a 3DCG database 58, etc. are stored. The 3DCG creation program 50 is C
It is a program for causing the PU 28 to exert a function for creating 3DCG from a stereo pair image. 3DC
The G creation program 50 is loaded into the CPU 28,
Generates various modules with the functions described later.

The camera data 52 is a database that stores parameters necessary for photogrammetry among the parameters unique to the left camera 61 and the right camera 62. Camera data 52
Is the focal length of the lens, CCD size, C
It is the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the CD light receiving surface. In the present embodiment, the left camera 61 and the right camera 62 use the same camera. Therefore, only one of the camera data may be stored. It is also possible to configure the left and right cameras with different cameras. In that case, the left camera 61 and the right camera 62 are
It is necessary to store the camera data of each.

The camera control program 54 uses the left camera 6
1. Controls the right camera 62, and at the same time controls the stereo pair images taken by these cameras to generate 3DCG 11
And a program to be stored in the storage device 48. The user turns on a switch (not shown) when taking a stereo pair image. this is,
A predetermined button may be pressed, or it may be detected that the user has inserted a coin into a predetermined insertion slot.

The camera control program 54 has a detection function for detecting that the switch is turned on, a countdown function for performing a countdown to the user until a stereo pair image is photographed by a speaker (not shown), and a left camera 6 at the same time when the countdown is completed.
1. A shooting function for shooting the user's face with the right camera 62,
The CPU 2 has a storage function of transferring the captured stereo pair image to the 3DCG creating device 11 and storing it in the storage device 48.
Demonstrate to 8.

The image database 56 is a database for storing the stereo pair image data transferred from the left camera 61 and the right camera 62. In the present embodiment, the stereo pair image data is JPEG (Joint
It shall be stored as a file in the Photograph Group format. This is a bitmap format or GIF (Graphics Interchange).
It is also possible to save as a file of another format such as e Format) format. The image data stored in the image database 56 is used when the CPU 28 performs photogrammetry of the user's face, generates a texture to be attached to the stereo model, and the like.

The 3DCG database 58 is stored in the CPU 28.
This is a database that stores the 3DCG data created by, for example, a VRML file that describes a stereo model, an image file (in JPEG format) that records an image (texture) to be attached to the stereo model, and the like are stored. In this embodiment, as an example, VRML is used as a language for describing a solid. VR
ML (Virtual Reality Modelin)
g Language) is a language use for describing 3DCG handled on the Internet, and conforms to the ISO standard. In addition, 3DCG in other formats
You may generate the file.

The storage device 48 stores other programs (not shown). An example of such a program is a VRML viewer. This program is a program for displaying 3DCG defined in the VRML file on the display device. By this program, the object (3DCG of the face in this embodiment)
Can be rotated or translated on the display. In addition, for example, a communication program that controls the communication control device 42 to maintain communication between the 3DCG creation device 11 and a terminal device or a server device connected to the network, and the 3DCG creation device 11 such as memory management and input / output management. OS that is the basic software for operating
(Operating System) and the like are stored.

The storage medium drive device 46 is a drive device for driving a removable storage medium to read / write data. Examples of removable storage media include magneto-optical disks, magnetic disks, magnetic tapes, semiconductor memories,
There is a paper tape punched with data, a CD-ROM, etc. Note that CD-ROMs and paper tapes can only be read. By driving the storage medium driving device 46, the data stored in the 3DCG database 58 can be transferred to a removable storage medium such as a magnetic disk or a semiconductor memory. The user can use these 3DCGs by mounting these storage media on a personal computer or a mobile terminal device at home.

The input / output interface 44 is, for example,
It is composed of a serial interface and other standard interfaces. The input / output interface 44 is an interface for connecting an external device to the 3DCG creating apparatus 11, and in the present embodiment, is connected to the left camera 61, the right camera 62, a speaker (installed in the photographing box 10), and the like. Has been done. CPU28
Is the left camera 6 via the input / output interface 44.
1. The right camera 62 can be controlled, and the countdown of shooting can be performed from the speaker.

In FIG. 3, the 3DCG creation program 50 is C
It is a figure showing an example of composition of each module realized when loaded in PU28. In the present embodiment, these modules are configured as software, but the present invention is not limited to this, and an integrated circuit having these functions can be created and implemented by hardware.

These modules are roughly divided into an image acquisition section 75, a corresponding point acquisition section 77, a photogrammetric section 72 and 3D.
It is composed of a CG file generator 73. The image acquisition unit 75 is a module that acquires image data (image data of a pair of stereo pair images) stored in the image database 56 and provides the image data to the corresponding point acquisition unit 77 and the 3DCG file generation unit 73. FIG. 4 is a diagram schematically showing a stereo pair image acquired by the image acquisition unit 75.
(A) is a left image, which is taken by the left camera 61. (B) is a right image, the right camera 6
It was taken by 2. The stereo pair images have reference points 5a, 5b, 5 used for photogrammetry.
c is shown together with the subject (head 8).

The corresponding point acquisition unit 77 (FIG. 3) identifies, for example, about 20 corresponding points in the left image and the right image of the stereo pair image using the characteristics of the human face, and identifies these points. This is a module that acquires two-dimensional coordinate values in an image and provides the two-dimensional coordinate values to the photogrammetry unit 72. Later, a three-dimensional coordinate value of these points is used to generate a stereo model of the face. Since the face is composed of complicated free phases, many points need to be extracted. In photogrammetry, it is necessary to know at least six two-dimensional coordinate values of the same point on the subject on the photograph for each of the left image and the right image. The coordinate value is obtained in units of millimeters, with the center of the image as the origin. This can be done by obtaining the coordinate value in pixels and then multiplying the coordinate value by the size of the pixel.

In this embodiment, a technique called face-it is used to automatically extract a set of corresponding points in the right image and the left image. This technique was originally developed to identify the face of a person who was captured by a security camera.
This technique uses edge detection to find the contour of a face and the positions of eyes, nose, and mouth.

The corresponding point acquiring unit 77 performs the processing in the order of face contour extraction → pupil position extraction → face organ rectangular area identification → face organ contour extraction → feature point extraction. Here, the facial organs are eyes,
The eyebrows, nose, mouth, and ears. Also, the corresponding point acquisition unit 7
7 extracts the face and also the reference points 5a, 5b, and 5c (to be used as the reference point in photogrammetry). Face contour extraction detects the edge of the face contour from above the face and background colors,
The face part is extracted from the stereo pair image. It should be noted that if the background of the photographing box 10 is plain, it will be easier to detect edges.

Since the position of the head 8 in the photographing box 10 is approximately determined, the pupil position in the stereo pair image is also approximately determined. In the vicinity, the position of the pupil is detected by using the brightness distribution from the eyebrows to the pupil. The position of the eyes is determined from the position of the pupil, and this is surrounded by a rectangle and extracted (rectangle 21 in FIG. 5). Once the eye position is determined,
Since the positions of the eyebrows and the nose and mouth are roughly determined (using the statistical position information of the facial organs obtained from many subjects), these are enclosed by a rectangle and extracted for each rectangle. Next, the outline of the eye is further extracted from the rectangle including the eye. Then, for example, the inner and outer corners of the eye and other characteristic points (for example, a mole) and the corresponding points in the left and right images are extracted, and the two-dimensional coordinate values (in units of pixels) on those images are extracted. Object, which is later converted to millimeters by internal orientation) for the left and right images.

In addition, the rectangle 22 including the nose, the rectangle 23 including the mouth, the rectangle 24 surrounding the eyebrows, and the rectangle 25 including the ears are similarly image-processed to extract the contours of the nose, mouth, and ears, respectively. The two-dimensional coordinate value of the characteristic point corresponding to the image is acquired. Further, the corresponding point acquisition 77 is performed on the contour of the face, the cheek,
If there are characteristic points (such as moles, scratches, and wrinkles) on the forehead, these are also extracted. In this way, a set of corresponding points in the left image and the right image is extracted. These points will later be used as observation points in photogrammetry.

The photogrammetry section 73 (FIG. 3) is an internal orientation section 7.
8 and a suboptimal solution search unit 79. As will be explained in detail later, photogrammetry generally involves internal orientation of a pair of stereo pair images of a subject taken from different directions.
The orientation is performed in the order of the mutual orientation and the absolute orientation, and the three-dimensional coordinate value of the subject (face) is calculated. In this way, photogrammetry is an internal orientation,
It consists of three phases: mutual orientation and absolute orientation.

Internal orientation means that the subject is a photosensitive surface (CCD).
Correct the distortion when projected on the screen,
It is the work of converting dimensional coordinates (called machine coordinates) into photographic coordinates used for mutual orientation. In the machine coordinates, the unit of length is a pixel unit, and the coordinate origin is set to one of the four corners of the image. In the photographic coordinate system, the unit of length is millimeter, and the coordinate origin is set at the center of the image. Affine transformation was used for coordinate transformation from machine coordinates to photographic coordinates.

The mutual orientation is an operation for obtaining a relative tilt or positional relationship between stereo pair images and reproducing the photographed state. By the relative orientation, a model coordinate system which is a relative three-dimensional coordinate system is obtained, and the subject is represented as a stereo model similar to the subject in the model coordinate system. At the stage of mutual orientation, the scale and tilt of the model coordinate system and the amount of translation are unknown.

Absolute orientation is a work for obtaining the scale and inclination of model coordinates and the amount of parallel movement by using the absolute coordinates (actual survey values) of the reference points that are copied in the stereo pair image. It is possible to obtain a solid with the same scale as the subject by absolute orientation. More specifically, conversion to the barycentric coordinate system for rotational conversion of the subject around the center of gravity, calculation of cumulative scale, calculation of cumulative parallel movement amount, calculation of cumulative rotation matrix, coordinate conversion of all observation points, elevation Performs adjustment calculation, elevation calculation of plane position, etc.

The internal orientation section 78 receives the coordinate values of the observation points (corresponding points) from the corresponding point acquiring section 77, and also stores in the storage device 4.
The camera data 52 is received from 8 (FIG. 2). And
The internal orientation section 78 internally orients the digital pair image using these coordinate values and camera data, and outputs the orientation result to the suboptimal solution searching section 79.

The sub-optimal solution search unit 79 has an observation point selection unit 8
0, mutual orientation section 81, absolute orientation section 82, evaluation / judgment section 8
It consists of three. The suboptimal solution selection unit 84 generates various combinations of a plurality of observation points on the stereo pair image acquired from the corresponding point acquisition unit 77, and searches for a combination of observation points whose orientation accuracy satisfies a predetermined degree. That is,
Search for a combination of observation points where the accuracy of orientation is high to some extent, and use this combination to orient a stereo pair image. As described above, the combination of observation points searched by the suboptimal solution search unit 79 is not necessarily the optimum combination among all the combinations, and is therefore called a suboptimal combination.

As for the accuracy of orientation, it is assumed that a desired accuracy is obtained when a residual is defined by a predetermined method and the residual satisfies a predetermined condition. For example, the residual can be defined as: First, a stereo pair image is located from a combination of observation points. Then, using the orientation result, the three-dimensional coordinate values of the reference points 5a, 5b, and 5c are calculated. Next, the measured values (known in advance) of the reference points 5a, 5b, and 5c are compared with the calculated values. The accuracy of orientation can be verified by this comparison value. For example, the reference points 5a, 5b,
It is possible to obtain the residual by adding the difference between the calculated value of 5c and the measured value to each of the X axis, the Y axis, and the Z axis. Then, when various combinations of observation points are generated using a genetic algorithm which will be described later and iteratively oriented, the residual becomes smaller and converges to a certain value. This combination of observation points when the residuals converge is used as the orientation result.

The outline of the sub-optimal solution searching unit 84 is as follows. After the observation point selection unit 80 generates a combination of observation points, the relative orientation unit 81 performs the mutual orientation, and the absolute orientation unit 82 performs the absolute orientation, and then the accuracy of the orientation (here, the convergence is determined) is evaluated and judged. The determination is made by the section 83. Observation point selection unit 80
At, a new combination of observation points is generated using a genetic algorithm, and similarly oriented, and the convergence is judged. The above work is repeated until the results converge. If the absolute orientation results converge, the result obtained by combining the observation points at this time is the orientation result. The photogrammetry unit 72 passes the result of the orientation to the 3DCG file generation unit 73. The details of the method for searching the suboptimal solution using the genetic algorithm will be described later.

The 3DCG file generation unit 73 is composed of an observation point coordinate calculation unit 86, a polygon generation unit 88, a VRML file generation unit 90, a texture generation unit 92, a texture pasting unit 94 and the like. Observation point coordinate calculator 9
2 receives the three-dimensional coordinates of the observation points used in the absolute orientation from the photogrammetry section 72 and also the observation point selection section 80.
The three-dimensional coordinate value of the observation point that is missed from the combination of the reference points is calculated from the absolute orientation result. In this way, the three-dimensional coordinate values of all the observation points taken on the face are obtained. A polygon is generated using these observation points whose three-dimensional coordinate values are known.

In order to generate polygons on the entire face later, new points are set on the stereo pair image so that points (including observation points) for that purpose are uniformly distributed on the entire face, and the points are set. It is also possible to add a function of calculating a three-dimensional coordinate value from the result of absolute orientation to the observation point coordinate calculation unit 92. All human faces have statistically similar free-form surfaces (for example, cheeks are rounded), and such a tendency can be found in advance by examining information on the free-form surfaces of many subjects. You can find out.

For example, a complementary equation for inferring the three-dimensional coordinate values of the point on the cheek is created from the three-dimensional coordinate values of the end points of the lips, the end points of the nose, the end points of the outer corners of the eyes, etc. It is possible to infer the three-dimensional coordinate values of points that are not present.
Or, for example, the end points of the lips and nose, the end points of the corners of the eyes, etc.
It is also possible to prepare a table or the like for estimating the three-dimensional coordinate value of a point on the cheek from the dimensional coordinate value. Thus, when the three-dimensional coordinate values of the observation points are determined by the orientation, the three-dimensional coordinate values of the points of the other parts of the face are estimated based on them, and there are places where the distribution of the observation points is small. In this case, points for generating polygons can be set at these places.

The polygon generator 88 acquires from the observation point coordinate calculator 86 three-dimensional coordinate values of the observation point (including the three-dimensional coordinate value of this point, if there is a point at which the three-dimensional coordinate value is estimated). A Voronoi region is generated, and the Voronoi region is further Delaunay divided to generate polygons. Note that the subsequent work is performed on the left image (since the three-dimensional data of the face has been obtained, the work can be performed on either one of the images thereafter).

FIG. 6A is a diagram for explaining Voronoi regions and Delaunay division. What is Voronoi region?
A set of points in a two-dimensional space or a three-dimensional space is divided into regions where the distance between the points is the shortest. Point 91
, a, 91b, 91c, ... Are observation points on the face whose three-dimensional coordinate values are known. The plane 92a (which is represented by a line segment in the figure for convenience) has points 91a and 9 on the plane.
It is a plane equidistant from 1b. Also, the plane 92b
Is a plane in which points on the plane are equidistant from the points 91a and 91c. In this way, the planes 92a, 92b, 92
c, ... Is generated. As a result, the Voronoi region 9
4a, 94b, 94c, ... Are generated.

The Delaunay division is a technique for forming a triangle based on the Voronoi region and dividing the region. Figure 6 (a)
Then, the points 91d, 91c, and 91b at the centers of the adjacent Voronoi regions 94a, 94b, and 94c are divided into line segments 93c and 93.
A Voronoi region 95 is generated by connecting a and 93b. The Voronoi region can be used as a polygon. In this way, a polygon is generated from a set of points on the face. When polygons are generated using the Voronoi region as described above, it is possible to suppress the generation of elongated polygons 96 as shown in FIG. 6B.

In this way, the polygon generator 88
Triangular polygons can be automatically generated from a set of points on the face. Then, from the generated polygons, it is possible to easily acquire the line / plane information required when a VRML file is generated later. Here, the line information is the three-dimensional coordinate information of the line segment forming the polygon, and the surface information is the three-dimensional coordinate information of the surface forming the polygon.
The polygon generator 88 passes the line / plane information to the VRML file generator 92.

The VRML file generator 90 (FIG. 3) is
A VRML file is generated using the line / face information received from the polygon generation unit 88. This function is called the translator function. A wireframe model can be created using line information, and a surface model or solid model can be created using surface information. Here, VRML is a language specification for describing three-dimensional computer graphics used on the Internet, and straight lines, planes, spheres, cylinders, etc. in space can be defined according to a predetermined grammar. By using VRML, a stereo model can be expressed in a virtual space constructed on the Internet. The solid described in VRML can be displayed on the display using a VRML viewer. It should be noted that VRML can not only stereoscopically represent an object but also audio and movement at the same time, and a user can move in a virtual space.

In the present embodiment, the faces related to the stereo pair image are represented by triangular polygons as shown in FIG. A free-form surface forming a face is represented as a polyhedron of triangles forming a polygon.

The texture generation unit 92 is a module that cuts out the image to be attached to each polygon generated by the polygon generation unit 88 from the left image. Texture generator 92
Receives the left image from the image acquisition unit 75, and receives information representing the shape and position of each polygon on the left image from the polygon generation unit 88. Then, the texture generation unit 92
The area on the left image corresponding to the shape and position of each polygon is extracted by extracting for each polygon. The cropped image is an image file in JPEG format, for example.
It is stored in the DCG data storage unit 58 (FIG. 2).

The texture pasting unit 94 pastes the texture cut out by the texture generation program 92 onto the stereo model generated by the VRML file generation unit 90 (mapping). VRML file generation program 90
The three-dimensional model of the face generated by is composed of triangular polygons. The texture pasting unit 94 receives information specifying the image data to be pasted to each polygon from the texture generation program 92, and pastes the image data corresponding to each polygon based on this information to update the VRML file. The pasting of the texture is performed by writing the file name of the corresponding texture at a predetermined position in the VRML file (text file). Then, the texture pasting unit 94 outputs the updated VRML file to the 3DCG database 58.

FIG. 8 is a flow chart for explaining the operation of the 3DCG creating apparatus 11. First, the CPU 28
In FIG. 2, the left camera 61 and the right camera 62 are remotely operated via the input / output interface 44 to photograph the user's face. Then, the CPU 28 receives the photographed left image data and right image data from the left camera 61 and the right camera 62 via the input / output interface 44, and stores these in the image database 56 (stereo pair image 1).
0). The above operation is performed by the CPU 28 according to the camera control program 54.

The following operation is performed by the CPU 28 according to the 3DCG creating program 50. When the stereo pair image is stored in the image data 56, the 3DCG creation program 50 is started. First, the CPU 28 reads the stereo pair image into the RAM 32. After that RAM3
Processing is performed using the image data read in 2. CP
U28 extracts face organs from the stereo pair image using the face-it technique, and further extracts feature points of the corresponding face organs in the left and right images. Then, the CPU 28 acquires the two-dimensional coordinate values on the image and stores them in a predetermined area of the RAM 32 (step 12).

Next, the CPU 28 acquires the camera data 52 and uses it to orient the stereo pair image internally (step 14). Next, the CPU 28 generates N (for example, 10) combinations of observation points and stores them in a predetermined area of the RAM 32 (step 14). In addition, this N
In the genetic algorithm (described later), the first combination
It is called the gene of generation. Next, the CPU 28 performs relative orientation for each combination of observation points (step 18),
Absolute orientation (step 20).

Next, the CPU 28 calculates the coordinate value of the reference point in the stereo pair image using the result of the absolute orientation, and obtains the residual difference from the actual coordinate value of the reference point. The residual is the sum of the difference between the calculated value of the reference point and the actual value over the X, Y, and Z coordinates. The residual is applied to all the combinations of the first generation, and the combination with the smallest residual is specified. Next, the CPU 28 determines whether the difference between the minimum value of the residuals of the previous time and the minimum value of the residuals of this time is larger or smaller than a fixed value. If the difference is small, it is determined that the difference has converged. It is determined that there is not (step 22). In addition, regarding the combination of the first generation, since there is no residual in the previous comparison, it is considered that the combination has not converged.

When the CPU 28 determines that the convergence has not occurred (step 22; N), the CPU 28 returns to step 16 and performs crossover and mutation (described later) according to the genetic algorithm to combine the second generation observation points. To generate. Then, the CPU 28 updates the combination of the first generation stored in the RAM 32 with the combination of the second generation. Similarly, mutual orientation (step 18), absolute orientation (step 20), and convergence determination (step 22) are performed.

When the CPU 28 determines that it has converged (step 22; Y), it calculates all three-dimensional coordinate values of the observation point (step 24). The three-dimensional coordinate values of the observation points belonging to the set used for the calculation of the absolute coordinates have already been obtained during the calculation of the absolute coordinates. The three-dimensional coordinate values of the observation points that do not belong to the set used for the calculation of absolute coordinates are absolute oriented using the results of absolute orientation. Also, new points are generated so that the points are evenly distributed on the face, and the three-dimensional coordinate values of these points are calculated using the results of absolute orientation.

Next, the CPU 28 performs Delaunay division on the points generated on the face, and further generates Voronoi regions by using the result of the Delaunay division to generate polygons (step 2).
6). The CPU 28 calculates line / face information from the generated polygon and stores it in a predetermined area of the RAM 32. next,
The CPU 28 reads the line / surface information from the RAM 32,
A VRML file is generated using this (step 2).
8). The CPU 28 stores the generated VRML in a predetermined area of the RAM 32.

Next, the CPU 28 reads out the line / face information and the stereo pair image stored in the RAM 32, and cuts out (texture) the image to be attached to the polygon from the image data using the line / face information, and for each polygon. JPE
It is stored in the 3DCG database 58 as a G file (step 30). Next, the CPU 28 pastes the texture corresponding to each polygon, updates the VRML file, and stores it in the 3DCG database 58 (step 32).

Next, a sub-optimization search method using the genetic algorithm performed by the photogrammetry section 72 will be described. In this embodiment, a method of shortening the calculation time required for processing is adopted by using a genetic algorithm. When performing photogrammetry, some (at least 6) of the observation points on the stereo pair image are selected, and the orientation calculation is performed using the selected observation points. Since the orientation accuracy depends on the method of selecting the observation points, it is necessary to find a combination of the observation points that can perform the orientation calculation with high accuracy from a large number of observation points. In conventional photogrammetry, a worker with a high degree of specialized knowledge and abundant experience searches for combinations of observation points. However, even such an expert finds it difficult to find the optimum combination of observation points.

By the way, when the subject is composed of a smooth curved surface such as a spherical surface or a free curved surface like a ball or a human face, many observation points are required to reproduce the curved surface. In such a case, the combination of observation points increases exponentially, and the time required for calculation increases significantly. Therefore, it is difficult to use all-solution search method in which all combinations are obtained and orientation calculation is performed for each of them, and the combination having the highest orientation accuracy is obtained.

Therefore, in this embodiment, a plurality of combinations of observation points are arbitrarily created, and genetic operations such as crossover and mutation are repeated on these combinations until the residuals converge to a certain range. The method of converging the residuals is adopted. If this method is used, it is not necessary to perform orientation calculation for all combinations of observation points even if there are many measurement observation points, and it is possible to reduce the calculation time, and the orientation result that meets the required accuracy is obtained. Can be obtained.

The genetic algorithm performs operations such as crossover, selection, and mutation in a solid population,
It originates in Darwinism evolution theory in which individuals with a high degree of compatibility with the environment survive after survival competition. According to this, there is a high probability that a person who is more suitable for the environment will survive in selection, and the survivors cross each other to leave a superior trait in the next generation and to acquire a new trait by mutation. If an inferior trait is acquired by mutation, that individual will be culled, and as a result, those who acquire a better trait by mutation will survive, their progeny will increase, and the fitness of the entire population will increase. improves.

Mutations can give rise to genes that cannot be obtained by crossover alone. Mathematically speaking, by making mutations, local solutions can be avoided and more optimal solutions can be obtained. The genetic algorithm is an algorithm of this process. In a genetic algorithm, an object is represented by a gene (also called a gene code) by some method, and the fitness of a gene is obtained using an evaluation function.

The evaluation function corresponds to the natural environment in the natural world. Individuals with genes that have a high degree of fitness to this natural environment have a higher probability of surviving and leaving offspring, and individuals with genes that have a low degree of fitness have a lower probability of leaving offspring, and as a result, are generally superior. The next generation with traits is formed. The application of genetic algorithms to engineering is being actively researched and applied to many combinatorial optimization problems such as optimization of jet engine design, optimization of cogeneration, and determination of road maintenance order. .

FIG. 9 is a diagram showing combinations of observation points. In this example, 20 observation points were taken. 20 genes
Consisting of individual loci 58a, 58b, ... 58t,
Two numerical values, 0 or 1, are assigned to each locus. A locus is a structural unit of a gene.
By assigning observation points from P1 to P20 to these loci, a numerical value of 0 or 1 is assigned to each observation point. When 1 corresponds to an observation point, that point is adopted as a set of points for orientation, and when 0 corresponds, it is not adopted. For example, P1, P2, P5, P7, P
If the loci of 9, P12, and P18 are assigned 1 and the other loci are assigned 0, P1, P2,
It means that P5, P7, P9, P12, and P18 are selected as a combination of the measurement observation point and the arbitrary observation point for orientation. In the present embodiment, the fitness of these genes is the reciprocal of the residual obtained from the orientation of the gene. That is, the smaller the residual, the greater the degree of conformity. The residual was determined by the same method as in Example 1. Further, the method of obtaining the goodness of fit is not limited to this, and any gene having a higher accuracy of orientation may have a higher goodness of fit.

As a genetic manipulation, crossover and mutation were performed. As for the crossover, two genes having high fitness were selected and two-point crossover was performed. The two-point crossover is, for example, a region between a point 51 between the second locus and the third locus and a point 52 between the ninth locus and the tenth locus in FIG. The exchange of data for a gene between two genes. The selection of the points 51 and 52 may be arbitrary or may be fixed. In addition to this, one-point crossover, three-point crossover, or the like may be performed as the crossover method. Mutations were made by inverting some 0's and 1's in any gene with a certain probability. As a result, a new gene that cannot be generated by crossover alone is formed and a local optimal solution is prevented.

In the present embodiment, the orientation is performed for the combination of the observation points for which the orientation accuracy is high. Therefore, the observation points that are out of this combination are removed and the orientation calculation is not performed. By the way, when the orientation is performed by the algorithm of the present embodiment and all the orientation elements (tilt of the left camera 61 and the right camera 62, the scale of model coordinates, rotation, parallel movement, etc.) are obtained, these orientation elements are determined. By using it, it is possible to calculate the three-dimensional coordinate value of the selected observation point in the real space. Therefore, even when the measurement observation points are selected in the orientation work and the coordinate values are not calculated, the coordinate values can be calculated using the orientation result.

FIG. 10 shows that the fitness of a gene rises and converges to a certain value each time the genetic operation is repeated. Point 54 is the maximum value of the fitness of each gene of the first generation. The fitness of each gene is calculated each time the genetic manipulation is performed. The maximum value of their fitness increases with each generation. Then, after a certain number of generations, the maximum value becomes almost constant. At this time, it is determined that the goodness of fit has converged. In the present embodiment, as indicated by point 56 in FIG. 10, when the maximum value of the goodness of fit of each gene within a certain calculation time ± t, for example, ± 3 minutes is within ± 1%, it is determined that it has converged. did. The convergence condition of the fitness is not limited to this. For example, when the maximum value of the fitness falls within a predetermined percentage over a predetermined generation, it is determined to be the convergence, and the maximum fitness of each gene is determined. The average value may be used instead of the value.

The following effects can be obtained in the first embodiment described above. The corresponding point acquisition unit 77 can automatically calculate the corresponding points of the left and right face photographs formed of complicated free-form curved surfaces that are difficult to acquire corresponding points. A genetic algorithm can be used to efficiently obtain a set of observation points for suboptimal photogrammetry from a large number of observation points. It is possible to program three stages of photogrammetry (internal orientation, mutual orientation, absolute orientation) and automatically calculate three-dimensional data of a subject (face in this embodiment).

The set of points for which the three-dimensional coordinate values have been calculated is divided into Delaunay divisions, and further, by generating Voronoi regions, appropriate polygons can be automatically generated to generate a surface model of the subject. Also, line / face information can be acquired from the generated surface model. A VRML file that describes a surface model of a subject can be automatically generated. Texture mapping can be performed on the surface model of the subject described in the VRML file using the image used in the photogrammetry.

The generated VRML file can be transferred to a server device or a mobile terminal device via the Internet, and can also be stored in a magnetic disk or a semiconductor memory.

In this embodiment, the coordinates in the stereo pair image are acquired in pixel coordinates (coordinate values are acquired in units of pixel size), but the present invention is not limited to this. The image may be divided by meshes, and the coordinate values may be measured with the size of each mesh as a unit. In addition, in the present embodiment, the corresponding points are acquired by using the facial features, but similarly, by preliminarily examining the features of other objects, 3DCG can be automatically performed even for objects other than faces. Can be created. Furthermore, for example, an object having a simple shape composed of a straight line and a plane, such as a square object such as a box, easily obtains corresponding points (for example, the square points of the box) by edge processing. be able to.

(Second Embodiment) In the second embodiment, the user's head 8 is photographed from four directions to create three-dimensional computer graphics of the head. Figure 1
FIG. 1 is a diagram showing a configuration of a photographing box 100 according to the second embodiment. The shooting box 100 includes a left camera 101 and a right camera 102 for shooting a stereo pair image (hereinafter referred to as a left pair image) on the left side surface of the head 8 in addition to the shooting box 10 of the first embodiment. , A left camera 104 for capturing a stereo pair image on the right side (hereinafter referred to as a right pair image), a right camera 105, and a left camera for capturing a stereo pair image on the occipital region (hereinafter referred to as an occipital pair image) 107 and the right camera 108 are equipped. All of these cameras have the same specifications (the same camera parameters).

The three-dimensional positional relationship of these cameras is known in advance. That is, for example, when the camera coordinate system of the left camera 61 is used, the three-dimensional coordinate values of other cameras are obtained in advance. In addition, on the four wall surfaces, reference points (not shown) used by the cameras installed on the opposite walls for surveying are drawn. The three-dimensional coordinate values of these reference points are also known in advance.

When the user turns on a switch (not shown), the countdown is performed from a speaker (not shown), and at the end of the countdown, all the cameras simultaneously photograph the head 8. In this way, in the photographing box 100, the stereo pair image of the front of the head 8, the stereo pair images of the left and right sides, and the stereo pair image of the occipital region are simultaneously photographed,
The three-dimensional data of the head 8 can be acquired over 360 degrees around.

FIGS. 12A and 12B are schematic diagrams showing a front stereo pair image (hereinafter referred to as a front pair image). (A) is a left image and (b) is a right image. Reference points 5a, 5b and 5c used in the photogrammetry are transferred to the stereo pair image together with the head 8. The head 8 (including ears and hair) and the reference points 5a, 5b, and 5c are extracted from the front pair image by image processing, and are used for photogrammetry and texture sampling.

3DCG when the head 8 is viewed from the front is created from the front pair image. At the time of photogrammetry, observation points 110a, 110b, and 111a, 111b are set at the apexes of the ears and the end points of the lips. Observation points 110a and 111
Let a be a reference point for superimposing the 3DCG created from the front pair image and the 3DCG created from the left pair image.

12 (c) and 12 (d) are schematic diagrams schematically showing the left pair image. (C) is the left image,
(D) is the right image. Reference points 5e, 5f, and 5g used for photogrammetry are transferred to the left pair image together with the head 8. The reference points 5e, 5f, and 5g are drawn on the wall surface in the background of the head 8. From the left pair image, the head 8 and the reference points 5e, 5f, 5g are extracted by image processing and used for photogrammetry and texture collection.

From the left pair image, 3DCG in which the head 8 is viewed from the left side is created. In addition, at the time of photogrammetry, observation points 110c and 111c are taken at the apexes of the ears and the end points of the lips, and they are used as reference points when the 3DCG created from the left pair image and the 3DCG created from the front pair image are superimposed. To be done.

FIGS. 12 (e) and 12 (f) are schematic views of the occipital region pair image. (E) is the left image,
(F) is the right image. Reference points 5h, 5i, and 5j used for photogrammetry are shown together with the head 8 in the occipital region pair image. The reference points 5h, 5i, and 5j are drawn on the wall surface that is the background of the head 8. From the occipital region pair image, the head 8 and the reference points 5h, 5i, 5j are extracted by image processing and used for photogrammetry and texture sampling.

From the occipital region pair image, 3DCG in which the head 8 is viewed from the rear is created. Further, at the time of photogrammetry, the observation point 110d is taken at the apex of the ear and used as a reference point when the 3DCG created from the occipital pair image and the 3DCG created from the left pair image are superimposed.

Although not shown, the right pair image is used to create 3DCG when the head is viewed from the right, similarly to the left pair image. At the time of photogrammetry,
3D created from right paired images by taking observation points at the end points of the lips
It is used when superimposing the CG and the 3DCG created from the front pair image.

Next, the 3DCG creating apparatus 5a according to the second embodiment will be described. The hardware configuration of the 3DCG creating apparatus 5a is the same as that of the computer shown in FIG. However, the input / output interface 44 further includes a left camera 101, a right camera 102, a left camera 104,
The right camera 105, the left camera 107, and the right camera 108 are connected. The camera control program causes the CPU 28 to exhibit a function of simultaneously photographing the head 8 with these cameras and storing the obtained stereo pair image in the image data pace 56. Further, the 3DCG creation program 50 according to the first embodiment is the 3DCG creation program according to the second embodiment.
It has been replaced by the creation program.

FIG. 13 shows a 3DC according to the second embodiment.
It is a figure showing an example of composition of G creation device 5a. In this configuration, the 3DCG creation program according to the present embodiment is C
It is loaded into the PU 28 and realized by software. 3DCG creation apparatus 5 according to the first embodiment
The parts having the same functions as in FIG.

The 3DCG creating apparatus 5a includes an image acquisition section 7
5, the corresponding point acquisition unit 77, the photogrammetric unit 72, the 3DCG file generation unit 73, the front CG storage unit 121, the left side CG storage unit 122, the right side CG storage unit 123, the occipital region CG storage unit 124, and the synthesis unit 125. Has been done.

The image acquisition section 75 is connected to the image database 56.
Read front pair image data, left side pair image data, right side pair image data, occipital pair image data,
It is provided to the corresponding point acquisition unit 77. The photogrammetry and the creation of the VRML file are performed in the order of front, left side, right side, and occipital region, and the image acquisition unit 75 reads the left side pair image after the processing of the front pair image is completed. As described above, the stereo pair images are sequentially provided to the corresponding point acquisition unit 77 according to the progress status of the processing.

The corresponding point acquiring unit 77 calculates a set of corresponding points on the stereo pair image acquired from the image acquiring unit 77,
The two-dimensional coordinate values on these images are provided to the photogrammetry unit 72. Similar to the first embodiment, corresponding points are extracted from the front pair image by using the features of the human face.

Also in the left side pair image and the right image pair image, the features of a human face viewed from the side are used, and like the front pair image, for example, the apex of the contour of the ear, the lower end, the end points of the lips, etc. Get corresponding points of. In addition, if the result of photogrammetry of the front pair image is used, the positional relationship between the camera for the front pair image and the camera for the side shooting is known in advance, so the positions of the pupils in the left pair image and the right pair image are calculated. You can also do it. By thus obtaining the position of the pupil, it is possible to detect and extract the positions of the ears, eyebrows, nose, and mouth with the position of the pupil as a reference.

Also in the occipital region pair image, the vertex of the ear, the lower end point, the apex of the head, etc. can be calculated by using the feature of the human head viewed from the rear. Further, the photogrammetric results of the front pair image, the left side pair image, and the right side pair image can be used for extracting the ear position. In addition, 3DCG of the back of the head
If the strictness of is not required, it is roughly known where the specific point on the left photograph of the occipital region is on the right photograph (the head 8 is in a predetermined position, and the human head is almost the same). Since it has a shape), the corresponding point of the occipital region can be calculated by estimation.

The structure and function of the photogrammetry section 72 are the same as those in the first embodiment. The photogrammetry unit 72 receives the two-dimensional coordinate values on the image of the corresponding points of the stereo pair image from the corresponding point acquisition unit, and executes the photogrammetry using the genetic algorithm. The photogrammetric unit 72 first orients the front pair image, and then the left side pair image, the right side pair image, and the occipital region pair image in this order.

The structure and function of the 3DCG file generating section 73 are the same as those in the first embodiment. The 3DCG file generation unit 73 generates a polygon using the result of photogrammetry, generates a VRML file, generates a texture from a stereo pair image, and attaches this to a stereo model described by VRML. The 3DCG file generation unit 73 sets the VR in the order of front, left side, right side, and occipital region.
The ML file is generated and these are output to a predetermined area of the RAM 32 (FIG. 2).

Front CG storage 121, left side CG storage 122, right side CG storage 123, occipital CG storage 1
An area 24 is provided in the RAM 32. Front C
The G storage unit 121 stores the VR generated from the front pair image.
The ML file is stored. Similarly, the left side CG storage unit 1
A VRML file generated from the left side pair image is indicated at 22 and a VRML file generated from the right side pair image is indicated at the right side CG storage unit 123 in the occipital region CG storage unit 12.
In 4, a VRML file generated from the occipital region pair image is stored.

The synthesizing unit 125 includes the front CG storage unit 121,
This module edits the VRML files stored in the left side CG storage section 122, the right side CG storage section 123, and the occipital CG storage section 124 to generate one VRML file. Each of these storage units has a front view of the head 8 viewed from the front (a surface model with texture attached), a left side view of the left side, a right side view of the right side, and a rear view. The occipital model is described by VRML, and these can be combined to create an all-round model over the entire circumference of the head 8.

When synthesizing the omnidirectional models, it is necessary to align the positional relationship in the three-dimensional space of each model related to the synthesis. For this reason, it was decided to move each model in parallel so that the corresponding point in each model coincides with the reference point. As an example of the reference points, the vertices of both ears and both ends of the lips in the front model, the vertices of the ears and the lip end points in the left side model and the right side model, and the binaural vertices in the occipital model are adopted.

For example, when synthesizing the front face model and the left side face model, the synthesizing unit 125 uses the photogrammetric unit 72 to calculate the three-dimensional coordinate values of the vertex of the right ear and the left end point of the lip in the front face model and the left side face model. The three-dimensional coordinate values of the left ear vertex and the point at the left end of the lip are received, and the translation amount of the left side stereo model is calculated from the difference between these coordinate values. The synthesizing unit 125 translates the left side face model based on the calculated value and synthesizes the left side face model with the front face model. It was decided to display the overlapping portions when they were combined. In the case of displaying on a display device, in this overlapping portion, a portion that is more exposed is displayed.

Similarly, the synthesizing unit 125 translates the right side face model into a front face model and then translates the occipital region model into a left side face model and a right side face model. The synthesizing unit 125 synthesizes the four models to generate an omnidirectional model of the head 8, and outputs this to the 3DCG database 58 as a VRML format file. In the present embodiment, the overlapping portions of the models are combined and synthesized, but the present invention is not limited to this, and the overlapping portions may be trimmed (cut and removed) and the two models may be attached to each other.

In addition to the method described above, there is a method of synthesizing models by arranging reference points. In this method, by unifying the reference points in the same coordinate system, it is possible to combine them only by performing absolute orientation. However, in some cases, it is necessary to perform processing such as cutting off the overlapping portion.

FIG. 14 is a flow chart for explaining the operation of the 3DCG creating apparatus 5a. First, CPU2
8 acquires a front pair image from the image database 56 (step 40). Next, the CPU 28 calculates the two-dimensional coordinates on the image of the corresponding points of the stereo pair image for each image (step 42).

Next, the CPU 28 performs photogrammetry with the corresponding points on the stereo pair image as observation points, and calculates the three-dimensional coordinate values of these observation points (step 44). next,
The CPU 28 creates a VRML file that defines the surface model of the subject using the three-dimensional information obtained by photogrammetry (step 46). Also, paste textures.

Next, the CPU 28 stores the generated VRML file in a predetermined area of the RAM 32 (step 48). Next, the CPU 28 determines whether the above processing has been performed for all stereo pair images (step 50). Note that stereo pair images include a front pair image, a left side pair image, a right side pair image, and an occipital region pair image, and processing is performed in this order.

When there is a stereo pair image which has not been processed yet (step 50; N), the process returns to step 40,
Acquire the next stereo pair image. Processing of all stereo pair images is completed, and VRMs generated from these are processed.
When the L file is stored in the predetermined area of the RAM 32 (step 50; Y), the CPU 28 determines whether these VR files are stored.
The ML file is read from the RAM 32, synthesized based on the reference points set on the model, and a VRML file that defines the entire circumference model of the entire head 8 is generated (step 52). Next, the CPU 28 stores the file generated in step 52 in the 3DCG database 58 (step 54).

In the present embodiment, the 3DCG database 5
The VRML file stored in 8 is similar to that of the first embodiment in that it can be distributed via the Internet or stored in a storage medium such as a magnetic disk or a semiconductor memory.

As described above, the following effects can be obtained in the second embodiment. It is possible to shoot a stereo pair image over the entire circumference of the head 8 with one shooting. Photogrammetry can be performed not only on the face of the head 8 but also on the left side, right side, and occipital region, and the front model, left side model, right side model, and occipital model can be described in a VRML file. You can The front model, the left side model, the right side model, and the occipital model are combined, and the entire circumference model over the head 8 can be output as a VRML file.

(Modification of Second Embodiment) FIG.
(A) is the figure which showed the photography box at the time of photographing a to-be-photographed object using convergent photography. In the second embodiment,
Two cameras were installed in parallel on each of the four walls of the photographing box, and the head 8 was photographed. Such a method in which the cameras are installed in parallel and the object is imaged from the same direction is called parallel imaging, and it is an imaging method that enables highly accurate surveying.
On the other hand, the case where a subject is photographed from different directions is called convergent photography. In the example of FIG. 15A, the photographing box 200
Digital cameras 201a, 201b, 20 at the four corners of the
1c and 201d are installed. By adopting the convergent shooting as described above, the number of cameras can be reduced. However, generally, in the case of convergent photography, the photogrammetric accuracy is lower than in the case of parallel photography.

FIG. 15B is a diagram showing a camera arrangement for enhancing the accuracy of the front model. When creating the all-round model of the head 8, the front model is the most important, and the accuracy of the left side model, the right side model, and the occipital model may not be so demanded. The photographing box 300 includes cameras 301a and 301b for photographing the right face of the head 8 in parallel.
Is installed in front of the head 8 on the right side, and cameras 301c and 301d for capturing the left face of the head 8 in parallel are installed in front of the head 8 on the left. Further, cameras 301e and 301f for parallel photographing of the occipital region of the head 8 are installed behind the head 8. In this way, by capturing the face of the head 8 in parallel with the four cameras, a more accurate front model can be obtained.

As described above, in the first embodiment, the second embodiment, and the modification of the second embodiment, the following application fields can be considered. (1) Stereoscopic Shooting Box In recent years, an entertainment device which automatically takes a digital photograph of a user at an amusement facility or the like and prints it by superimposing a picture of a flower on it, for example, has become popular. A user's three-dimensional photograph can be taken in the same manner as this. The generated 3DCG data of the user can be transferred to, for example, the mobile phone of the user and displayed on the liquid crystal screen. User's 3DCG displayed on the LCD screen
Can be rotated or translated by operating a predetermined button on the mobile phone. However,
It is assumed that the user's mobile phone is equipped with a viewer for displaying 3DCG.

(2) Make-up support at a beauty salon etc. A 3DCG of the head of the customer is created at the beauty salon, and various hairstyle data prepared in advance is synthesized with the 3DCG of the customer to clarify the hairstyle requested by the user. be able to. In addition, the user can search for a hairstyle that he / she prefers by combining various hairstyles into 3DCG. (3) Three-dimensionalization of the criminal image of the wanted criminal 3DCG of the wanted criminal is created and published on the Internet. The user can observe the wanted 3DCG on the Internet, and obtain, for example, the impression of the criminal when viewed from the front obliquely, the impression when viewed obliquely behind, the impression when viewed from the front. You can As a result, it is possible to memorize the impression of the criminal that cannot be obtained from the plane photograph.

(4) Three-dimensionalization of a product catalog When presenting a product catalog on the Internet,
These products are converted into 3DCG and provided to users. Especially in the case of products whose design point is an important selling point, such as vases and dresses, the user can observe the product from various viewpoints such as rotating the product, looking at it from the near side, and looking away from it. . Further, the product can be advertised by distributing a storage medium storing 3DCG of the product, not limited to the Internet.

(5) When ordering an easy-to-order suit such as clothes or a kimono, the 3DCG of the customer's head is generated, and the 3DCG of the suit or kimono is added to this.
By superimposing G, the customer can see himself wearing the garment before making it. (6) Surveying By using the photogrammetry unit 72 of the present embodiment, surveying of a structure having a free-form surface can be performed at high speed by a general-purpose digital camera. (7) Design Design In recent years, design designs are often created by computer using CAD or the like. Therefore, the data generated by CAD or the like is used as the 3DCG file generation unit 73 of the present embodiment.
With the VRML file, the 3DCG of the design can be observed. The design handled here can be converted to 3DCG as long as it can acquire three-dimensional data such as an architectural design, an indoor layout, or a car design.

[0126]

According to the present invention, an object can be easily photogrammetrically measured from a stereo pair image of the object, and a 3DCG of the object can be automatically generated.

[Brief description of drawings]

FIG. 1 is a diagram showing a photographing box according to a first embodiment.

FIG. 2 is a diagram showing an example of a configuration of a 3DCG creating apparatus.

FIG. 3 is a diagram showing an example of a configuration of modules of a 3DCG creation program.

FIG. 4 is a schematic diagram showing a stereo pair image.

FIG. 5 is a diagram showing a rectangle set when extracting a facial organ.

FIG. 6 is a diagram for explaining Voronoi regions and Delaunay division.

FIG. 7 is a schematic diagram showing a face of a stereo pair image represented by a triangular polygon.

FIG. 8 is a flowchart for explaining the operation of the 3DCG creation device 11.

FIG. 9 is a diagram for explaining a combination of observation points when a genetic algorithm is used.

FIG. 10 is a diagram showing that the fitness of a gene increases and converges to a certain value each time a genetic operation is repeated.

FIG. 11 is a diagram showing a configuration of a photographing box according to a second embodiment.

FIG. 12 is a diagram showing a stereo pair image according to the second embodiment.

FIG. 13 is a diagram showing an example of a configuration of a 3DCG creating apparatus according to a second embodiment.

FIG. 14 is a flowchart for explaining the operation of the 3DCG creation device.

FIG. 15 is a diagram showing a photographing box according to a modification of the second embodiment.

[Explanation of symbols]

6th line target line 7 mirror 10 shooting box 11 3DCG creation device 26 Control unit 28 CPU 30 ROM 32 RAM 34 Input means 38 Output means 42 Communication control device 43 bus line 44 I / O interface 46 storage medium drive 48 storage 50 3DCG Program 52 camera data 54 Camera Control Program 56 image database 58 3DCG database 61 left camera 62 right camera 72 Photographic Survey Department 73 3DCG file generator 75 Image acquisition unit 77 Corresponding point acquisition part 78 Internal orientation section 80 Observation point selection section 81 Mutual orientation section 82 Absolute orientation section 83 Evaluation / Judgment Department 84 Suboptimal solution search unit 86 Observation point coordinate calculator 88 Polygon generator 90 VRML file generator 92 Texture generator 94 Texture pasting section 100 shooting boxes 101 left camera 102 right camera 104 left camera 105 Right camera 107 left camera 108 Right camera 121 Front CG storage 122 Left side CG storage 123 Right side CG storage 124 Back head CG storage 125 Synthesis Department 200 shooting boxes 300 shooting boxes

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G06T 15/00 300 G06T 17/40 F 5L096 17/40 G01B 11/24 K (71) Applicant 501069555 Co., Ltd. Kansai Research Institute, Osaka Prefecture, Osaka City Yodogawa-ku 5-chome 1-28 Shin-Osaka Yachiyo Building Annex 3F (71) Applicant 394024477 Fukui Computer Co., Ltd. Fukui-City Fukui-Chuo 1250-1 (72) Inventor Shigenori Tanaka 4-7-1 Takemidai, Suita-shi, Osaka 2-606 (72) Inventor Hitoshi Furuta 2-1-246, Hatoyama, Uji-shi, Kyoto (72) Inventor Etsushi Kitagawa 5-chome, Miyahara, Yodogawa-ku, Osaka-shi, Osaka No. 28 Shin-Osaka Yachiyo Building Annex 3F Kansai Research Institute (72) Inventor Kotaro Nakayama 5 Miyahara, Yodogawa-ku, Osaka-shi, Osaka No. 1-28 Shin-Osaka Yachiyo Building Annex 3F Kansai Research Institute (72) Inventor Yoshitaka Minami 5-1-2 Miyahara, Yodogawa-ku, Osaka-shi Osaka Prefecture Kansai Sogo Inc. 3F In-house (72) Inventor Masanori Ikebe 5-1-2 Miyahara, Yodogawa-ku, Osaka-shi, Osaka 3rd floor, Annex, Shin-Osaka Yachiyo Building Kansai Co., Ltd. Inside Information Laboratory (72) Hiroya Yoshida, Yodogawa-ku, Osaka-shi, Osaka Miyahara 5-1-28 Shin-Osaka Yachiyo Building Annex 3F Kansai Research Institute F-Term (reference) 2F065 AA04 AA53 BB05 CC16 DD06 FF05 JJ03 JJ05 JJ26 PP12 QQ03 QQ24 QQ27 QQ28 QQ32 QQ36 QQ41 SS02 SS06 SS13 2F112 AC06 2 CA08 DA28 FA03 FA07 FA21 FA31 FA38 FA41 FA45 FA50 5B050 AA09 BA04 BA09 BA15 DA07 EA05 EA26 FA02 FA06 5B057 AA20 CA12 CB13 CD14 CH01 DA07 DC05 DC34 5B080 AA13 GA22 5L096 BA20 DA04 FA09 FA69 FA76 JA11

Claims (9)

[Claims] 1. A first image acquisition unit for acquiring a first image of a subject photographed from a predetermined direction, and a second image of the subject photographed from a direction different from the predetermined direction. Second image acquisition means, extracting characteristic points of the subject from the first image,
And a corresponding point extracting unit that extracts the characteristic points of the subject from the second image and associates the points extracted from the first image with the points extracted from the second image. Photogrammetric means for photogrammetrically measuring the subject and acquiring three-dimensional information of the subject using the two-dimensional coordinate values of the points on the first image and the two-dimensional coordinate values on the second image. An image processing apparatus comprising: 2. The first image and the second image are
An image obtained by photographing a subject whose distribution of characteristic feature points is known in advance from a predetermined position with respect to the subject, the image being recorded in the first image and the second image. The tendency of the distribution of the characteristic points is known, and the corresponding point extracting means uses the known tendency of the distribution of the characteristic points of the subject to obtain the first image and the second image.
The image processing apparatus according to claim 1, wherein the feature points extracted from the image are associated with each other. 3. The photogrammetric means includes combination generating means for generating a plurality of point combinations from the points acquired by the corresponding point acquiring means, orientation means for orienting each of the generated combinations, and the orientation Convergence determining means for determining the convergence of the result, and updating means for updating the combination by a genetic operation in a genetic algorithm until the convergence determining means determines that the calculated value has converged. The image processing apparatus according to claim 1 or 2, wherein 4. A polygon generating means for generating a polygon for generating a three-dimensional model of the subject using the surveying result obtained by the photogrammetric means, and further comprising: The image processing apparatus according to claim 2 or claim 3. 5. The polygon generating means includes Voronoi area generating means for generating Voronoi from points on a subject whose three-dimensional coordinate values are known by photogrammetry, and the generated Voronoi area is Delaunay divided to form polygons. The image processing apparatus according to claim 4, further comprising a Delaunay dividing unit that generates the image. 6. A description means for describing the three-dimensional model of the subject represented by the generated polygon in a predetermined computer language, and an output for outputting the contents described by the description means in a predetermined file. An image processing apparatus according to claim 4, further comprising: a means. 7. A texture acquisition unit for acquiring a texture to be mapped to the stereo model of the subject from at least one of the first image and the second image; and a texture acquired by the texture acquisition unit for the stereo of the subject. The image processing apparatus according to claim 6, further comprising: an updating unit that maps the model and updates the file. 8. The predetermined computer language is VRML
The image processing apparatus according to claim 6 or 7, wherein 9. A first image acquisition function for acquiring a first image obtained by photographing a subject from a predetermined direction, and a second image obtained by photographing the subject from a direction different from the predetermined direction. A second image acquisition function, extracting characteristic points of the subject from the first image, and extracting characteristic points of the subject from the second image,
A corresponding point extraction function that associates the points extracted from the first image with the points extracted from the second image; two-dimensional coordinate values of the extracted points on the first image; An image processing program, comprising: a photogrammetric function of photogrammetrically measuring the subject and acquiring three-dimensional information of the subject using the two-dimensional coordinate values on the image.