JPH11185059A - Three-dimensional shaped data processor and modeling system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To permit creation of a model on the shape of which visual characteristics of an object are reflected and exaggerated if necessary. SOLUTION: Characteristic areas a1 to a4 in first classification to satisfy a first setting condition are extracted from a two-dimensional image G2 formed by picking up the object, characteristic areas b1, b2 in second classification to satisfy a second setting condition are extracted from a distance image G3 obtained by three-dimensional measurement for the object and data correction to deform a part corresponding to the extracted characteristics areas in the first classification and the second classification is added to a shape model of the object obtained by the three-dimensional measurement.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to three-dimensional shape data processing for creating a model of an existing object.

[0002]

2. Description of the Related Art A portable non-contact type three-dimensional measuring device (three-dimensional camera) has been commercialized and used for data input to a CG system or a CAD system, body measurement, visual recognition of a robot, and the like. As a non-contact measurement method, a slit light projection method (light cutting method) is generally used, but a pattern light projection method, a stereo vision method, an interference fringe method, and the like are also known.

[0003] Also, three-dimensional CG software that can be used in personal computers and small three-dimensional cutting machines for hobbies are commercially available. If these are used, models and creations can be easily manufactured even in ordinary households.

[0004] On the other hand, a kind of vending machine for creating a face photograph sticker of a customer on the spot is gaining in popularity. The customer inserts coins for the fee and takes a desired pose in front of the camera while watching the monitor screen. Then, when a predetermined operation is performed, a sheet on which a fixed number of seals are arranged is created and discharged to an outlet. Most models have multiple options for the shape of the facial photograph, the imprint pattern, and the like.

[0005]

According to the above-described three-dimensional measuring apparatus, the shapes of various objects including a human body can be converted into data with the same ease as taking a photograph. Since it is a non-contact type, even when measuring a human body, the person to be measured does not feel troublesome. Therefore, it is conceivable to use this three-dimensional measuring device for creating a facial model that is not a facial photograph. That is, when combined with a three-dimensional processing machine, it is possible to measure a person's face and create a model with an appropriate magnification on the spot.

[0006] However, in the non-contact three-dimensional measurement, no facial element indistinguishable in shape does not appear on the model. For example, the so-called iris portion of the eyeball is not distinguished from the iris portion even though it is an important element characterizing the face. Eyebrows drawn with makeup tools also assimilate with the forehead.

[0007] On the other hand, there is a case where the person himself has a sense of incompatibility with a model that faithfully reproduces the undulation of the face. This is because the color affects the perception of undulation, such as the warm-colored lips appearing more raised than they actually are. Even though the shape is faithful, the uncolored model looks flatter than a familiar face. Also, a model that intentionally exaggerates facial elements is preferred, and a model whose nose is slightly higher than the actual one is delighted.

In other words, when creating a model of a human face, it is necessary to correct the face elements such as the eyes, irises, lips, and nose in the shape model obtained by the three-dimensional measurement. It is an object of the present invention to allow the visual features of an object to be reflected in the shape and, if necessary, to create an exaggerated model.

[0009]

According to the present invention, a portion is specified using both two-dimensional photographing information of an object and a range image obtained by three-dimensional measurement, and a shape model that is locally emphasized is generated. .

According to a first aspect of the present invention, a first-class characteristic region satisfying a first set condition is extracted from a two-dimensional image obtained by photographing an object, and extracted from a distance image obtained by three-dimensional measurement of the object. A feature area of a second classification that satisfies a second setting condition is extracted, and a part corresponding to the extracted feature area of the first and second classifications is extracted from the shape model of the object obtained by the three-dimensional measurement. Is modified to modify the data.

In the three-dimensional shape data processing apparatus according to the second aspect of the present invention, at the time of extracting the feature area of the first classification, a shaded part of the object is divided by referring to image information around the shaded part. Things.

According to a third aspect of the present invention, there is provided a modeling system comprising: a photoelectric conversion device for photographing an object; a three-dimensional measuring device for measuring a shape of the object; and a two-dimensional image of the object obtained by the photoelectric conversion device. And a three-dimensional shape data processing device for correcting data on the shape model of the object obtained by the three-dimensional measurement device based on the distance image obtained by the three-dimensional measurement device. .

[0013]

FIG. 1 is an external view of a three-dimensional model forming apparatus 1 according to the present invention. The three-dimensional model creation device 1 has a function of measuring the shape of an object and processing the material on the spot based on the measurement data, and is used as a vending machine for small articles in the shape of a customer's face. . The created article is a three-dimensional object in which a model of the face protrudes from a plate of a predetermined shape (for example, a square). It is also possible to add a specific undulating pattern to the plate surface (background portion). If an appropriate metal fitting is attached to such an article, it becomes an accessory such as a pendant, a broach, or a key holder. A metal fitting may be attached to the material in advance.

On the front surface of the upper half of the substantially life-size housing 10,
A light emitting window 12 and a light receiving window 14 for optical three-dimensional measurement are provided together with a display 16 for a user to check a pose. The light receiving window 14 is also used for two-dimensional color photography. The lower half of the housing 10 projects forward from the upper half, and the upper surface thereof is an operation panel 18. The product outlet 20 is provided on the front surface of the lower half.

The user stands toward the three-dimensional model forming device 1 and inserts coins for the fee. Thereafter, when the user performs a start operation, the three-dimensional model creation device 1 measures the shape of the object existing within a certain range in front and displays a three-dimensional shape model (for example, a surface model) indicating the measurement result. Then, when the user performs a confirmation operation for instructing determination of the composition, the three-dimensional model creation device 1 starts three-dimensional processing according to the measurement result. The product is completed in a matter of minutes. The user takes out the product from the outlet 20.

FIG. 2 is a plan view of the operation panel 18. The operation panel 18 includes a start button 181, a confirmation button 182, a cancel button 183, a joystick 1
84 and a coin slot 185 are provided. The start button 181 is a start operation unit, and the confirmation button 182 is a confirmation operation unit. Joystick 18
Reference numeral 4 is used for changing the composition of the model. Rotation processing of the three-dimensional shape model is performed in response to a pan operation for tilting left and right, a tilt operation for tilting up and down, and a roll operation for rotating a knob, and the processing results are sequentially displayed. The cancel button 183 is an operation unit for instructing re-measurement when the user does not like the displayed three-dimensional shape model. However, the number of valid times is set in the cancel button 183, and it is not possible to instruct re-measurement without any restriction.

FIG. 3 is a functional block diagram of the three-dimensional model creation device 1. The three-dimensional model creation device 1 includes a modeling system 1A that generates a three-dimensional shape model having a model size, and a processing system 1B that makes the three-dimensional shape model visible.

The modeling system 1A includes a photographing system 30 for converting (converting) the appearance information of a user who is an original object into digital data. The imaging system 30 includes a three-dimensional measurement device 34 for converting shape information into data by a slit light projection method, a two-dimensional imaging device 36 for converting color information into data, and a controller 38. Note that other methods may be used for the three-dimensional measurement instead of the slit light projection method. The shape data DS, which is information measured by the three-dimensional measuring device 34, and the color image data DC, which is image information obtained by the two-dimensional image capturing device 36,
Is input to Since the relative relationship between the camera coordinates of the three-dimensional measurement and the two-dimensional imaging is known, it is easy to align the three-dimensional shape model based on the shape data DS with the two-dimensional image. For example, if the apparatus is configured as a three-dimensional input device disclosed in Japanese Patent Application Laid-Open No. 9-145319, two-dimensional imaging and three-dimensional measurement can be performed from the same viewpoint, so that this alignment can be performed more easily. Data processing device 4
0 is provided with an image processing circuit (not shown), and performs various data processing including data correction specific to the present invention. The controller 42 of the data processing device 40 also performs overall control of the three-dimensional model creation device 1 and gives appropriate instructions to the controller 38 of the imaging system 30 and the controller 76 of the processing system 1B. The display 16 and the operation input system 80 are connected to the controller 42. The operation input system 80 includes the above-described operation panel 18 and a fee receiving mechanism.

On the other hand, the processing system 1B includes a processing device 72 for cutting a material such as a resin block, a material supply device 74 for supplying a material to a processing position and transporting the processed product to the outlet 20; a controller 76; An outlet sensor 78 is provided. The detection signal of the outlet sensor 78 is input to the controller 42.

It is also possible to adopt a circuit configuration in which the controller 42 is in charge of the control of the photographing system 30 and the processing system 1B, and the controller 38 and the controller 76 are omitted.

FIG. 4 is a perspective view showing an example of the mechanism configuration of the processing system 1B. The material supply device 74 has a stock section 210 for storing a total of eight types of materials. The storage spaces are provided on both sides of the linear transfer path 212, and four elevators 220 are arranged along the transfer path 212 in the storage spaces on each side. A plurality of materials of the same type are stacked on each elevator 220, and the vertical movement of the elevator 220 is controlled so that the uppermost material is located at a predetermined height. When one type of material suitable for the model to be created is designated, the designated material is transferred from the storage space to the transfer path 212 by the pushing rod 218 as the work 216.
Sent out to Then, the work 21 on the transfer path 212
6 is sent to the table 200 of the processing device 72 by the transfer rod 214 with the chuck.

In the table 200, the work 216 is fixed by two stoppers 202 and a clamp jig 204. Then, it is cut by a blade 208 attached to a rotating shaft 206 that can move up and down, left and right, and back and forth.

When the three-dimensional processing is completed, the work 216 is nipped by the chuck at the tip of the transfer rod 214, carried to the discharge-side end of the transfer path 212, and sent to the discharge port 222. Instead of using the transfer rod 214, the work 216 may be moved from the table 200 to the discharge port 222 in the form of a slide.

The mechanism configuration of the processing system 1B is not limited to the example. For example, the number of elevators can be reduced by arranging the same type of material horizontally on each of the multi-stage shelves, arranging an elevator at one end in the arrangement direction, and extruding the materials from the shelves to the elevators. The work may be carried from the storage position to the processing position to the discharge position by the arm robot. Instead of cutting, it is also possible to create a model by a method such as an additive manufacturing method (including a stereolithography method), a laser processing (thermal processing), and a molding processing (pressing and the like). In addition, regarding the material shape, the user may be able to select a desired external shape, or a material having the shortest processing time is automatically selected from a plurality of types of materials in which a standard face model has been created in advance. You may do so.

In the three-dimensional model creating apparatus 1 having the above configuration, a three-dimensional model obtained by three-dimensional measurement is used to create a face model having undulations representing irises and eyebrows and having emphasized nose and lips. Is automatically corrected by the data processing device 40 by using the color photographing information of the face.

FIG. 5 is a schematic diagram of extraction of a face element. The data processing device 40 includes a two-dimensional image G2 represented by the color image data DC and a distance data G3 represented by the shape data DS.
, A region of a specific face element is extracted. In this embodiment, the eyebrows, eyes, irises (iris and pupil),
And the lips are extracted. This is because these facial elements have minute undulations and are difficult to extract from the distance data G3.
On the other hand, the nose and neck are extracted from the distance data G3. This is because the nose is similar in color to its surroundings (such as the cheek) and is difficult to extract from the two-dimensional image G2. The same goes for the neck.

The extraction from the two-dimensional image G2 is performed in the following manner. By performing clustering in a specific color space (for example, L ^* a ^* b ^* color space), the two-dimensional image G2 is divided into regions of the same hue. Labeling is performed on the result to extract a continuous area having the same hue. Matching is performed using a template created in advance based on statistics on the position and color of the face element, and a feature region a corresponding to eyebrows, eyes, irises, and lips
1, a2, a3, and a4 are selected. For example, if the lower side of the lower lip is shaded and darker than it should be, the boundary between the red lip and the shade and the boundary between the shade and the flesh-colored chin are extracted, and the middle between these boundaries is extracted. A virtual line passing through the part is determined, and this virtual line is defined as a part of the contour of the lower lip. In other words, region division is performed on the shaded portion with reference to the surrounding color information.

The extraction from the distance data G3 is performed in the following manner. By performing clustering on the distance distribution from the three-dimensional measurement reference point, the distance data G3 is divided into regions within the same distance range. The result is subjected to labeling to select continuous characteristic regions (corresponding to the nose and neck) b1 and b2 within the same distance range.

FIG. 6 is a schematic diagram of partial data correction. First, among the three-dimensional shape models specified by the shape data DS, the first-class feature regions a1 to a4 extracted from the two-dimensional image G2 and the second-class feature regions b1 extracted from the distance image G3 in the manner described above. , B2 are set as correction targets. Next, a deformation of a preset degree is performed for each correction target portion Up.

The deformation is performed in the following manner. A mesh M perpendicular to the camera line of sight is projected on the correction target portion Up, and the position where the grid point of the mesh M is projected on the correction target portion Up is set as a sampling point p. A correction vector v having a set length that is directed to the camera side from the sampling point p and is perpendicular to the model plane is calculated. The length of the correction vector v is set for each face element. For example, eyebrows are longer than eyes. Mesh M and correction target part Up
(This is referred to as a contour point) q is determined.
The position (tip point) p2 of the tip of the correction vector v and the contour point q
Find the surface passing through The obtained curved surface and the original three-dimensional shape model are synthesized. That is, the correction target portion Up of the three-dimensional shape model is replaced with the obtained curved surface Up2. As a result, the correction target portion Up rises, and, for example, the positions of the eyes and irises are represented by shapes. The same applies to eyebrows drawn with eyebrows. Undulating nose and lips are emphasized. The neck is assimilated with the background. Since the position of the contour point q does not change, the continuity of the surface is maintained between the eyebrows and irises and their surroundings.

Hereinafter, the operation of the three-dimensional model forming apparatus 1 will be described with reference to a flowchart. FIG. 7 is a main flowchart showing a schematic operation. After the power is turned on, the two-dimensional imaging and the display of the imaging result are repeated in a standby period waiting for the operation by the user (# 10, # 12, # 1).
4). In addition, a guidance message is displayed periodically. When a fee is input and the start button 181 is pressed, two-dimensional imaging is performed again and three-dimensional measurement is performed (# 1).
6, # 18). Predetermined data processing is performed (# 20), and the obtained three-dimensional shape model is displayed (# 22). At this time, the appearance is enhanced by applying a known graphic method such as adding a shadow. Then, it waits for an instruction operation. However,
The waiting time is finite, and after the time limit, it is considered that the confirmation operation has been performed.

When the joystick 184 is operated,
As described above, the three-dimensional shape model is rotated and displayed according to the operation (# 24, # 38). Cancel button 18
When 3 is pressed, the operation returns to the operation in the standby period (# 40, # 1).
0). However, in this case, there is no need for the customer to re-enter the fee, and if the start button 181 is pressed, re-measurement is performed.

When the confirmation button 182 is pressed (# 2
6) Data for processing control is generated with reference to the processing condition database based on the three-dimensional shape model (# 28).
The material is processed (# 30). When the processing is completed, the product is discharged (# 32), and it is confirmed that the product has been taken out by the takeout port sensor 78, and the process returns to the standby operation (# 34, # 1).
0).

FIG. 8 is a flowchart showing the contents of the data processing of FIG. Here, as described above, the following processing including data correction for partially raising the projection and compression in the depth direction for shortening the processing time and intentionally flattening the design is performed.

A smoothing process is performed to remove abnormal data due to noise and to avoid excessive reproduction of fine irregularities (# 200). Perform re-sampling processing (# 21
0). This is a process of converting input data into data aligned with grid points at equal intervals projected in parallel from a certain direction in order to directly face the input data in the processing direction when the face is oblique. For example, if a person's face cannot be measured due to shadows below the ears, the data can be converted to represent a normal face-up face after three-dimensional measurement with the face facing up. If there is no measurement point at the position where the grid point is projected, linear interpolation is performed using the measurement values around the measurement point.
At this time, the projected direction is vertically above when processing, and each grid point has height data. Also,
Even when the input data is perspective projection, the input data can be converted to parallel projection data by this processing.

Completion of missing part without data (# 22)
0). Various methods such as linear interpolation and weighted completion can be applied as the complementing method. For example, all missing parts of the data are replaced with fixed values (simple interpolation). As the fixed value, a set value, a minimum height, and an average value of the outer peripheral position of the face can be considered. If the missing part is completely surrounded by the valid data part, linear interpolation is performed from the surrounding data. In addition, for parts where it is expected that accurate data cannot be obtained by optical three-dimensional measurement due to the properties of the target, such as black eyebrows and hair on a human face, existing three-dimensional shape data can be replaced. Good. In this case, a standard model of the face (front half of the head) is prepared, and the data of the standard model whose position and size have been adjusted are used for the data missing part. The adjustment of the standard model is performed in the following procedure. The eyes and mouth are extracted from the two-dimensional image in the same manner as in the case of the data correction described above, and the positions of the three reference points are calculated. Then, the standard model is linearly converted so that each reference point of the standard model matches the actually measured shape model.
It should be noted that such a combination is applicable not only to the missing part of the face but also to any part.

After obtaining a three-dimensional shape model that is faithful to the real shape in each of the above-described processes, as described above, data correction specific to the present invention, which makes the eyebrows and irises visible and emphasizes the lips and nose, is performed (# 230). ). It is also possible to provide a mode switching function so that the user or the device manager can select whether or not to perform data correction and which part (face element) to correct.

A height compression process is performed to reduce the size of the three-dimensional model in the depth direction (# 240). That is,
Reduce the height difference in the depth direction to shorten the processing time.
A flat model is suitable for pendants and medals. Either the uniform compression method or the non-uniform compression method can be applied to the compression, and the compression can be used for each part.

The background portion of the three-dimensional shape model is detected (# 250). This is a pre-process for correcting the background portion. Leave the back of the customer as a blue background,
If the result of the color determination of the two-dimensional image is used, the background portion can be easily and reliably detected.

A background conversion for replacing the background portion with other data is performed (# 260). For example, since the background portion is extremely deep, the data is converted into data having a small depth in order to reduce the processing time. The data to be replaced may be plane data or three-dimensional plane data representing a pattern such as a flower or a tree or a geometric pattern.

The size of the full-size three-dimensional shape model is adjusted to match the product size (# 270). Further, resolution conversion is performed to adapt the data amount to the accuracy of the processing device 72 (# 280). In this processing, a mesh having a predetermined grid width is projected and resampled at grid points, but the projection direction is fixed to the vertical direction during processing. As a point of the resolution conversion (data number conversion), first, the constituent points of the shape model for machining are defined by the point pitch and the vector change amount, and the point pitch range corresponding to the vector change amount is stored in advance. Read from the specified characteristic data table and set. That is, the pitch is increased by thinning out the data, or the pitch is reduced by interpolating the data. When the measurement resolution is sufficiently large, only the thinning may be performed. If the resolution conversion function is provided, the resolution of the three-dimensional measuring device 34 is not limited, so that a usage form in which the measuring means is replaced depending on the application is allowed.

Lastly, positioning is performed by moving the origin of the coordinates in parallel so that the reference position of the three-dimensional shape model matches the reference position of machining (# 290). In the case of using a material in which predetermined irregularities have been formed in advance as described above in the processing, in the processing data generation processing (# 28 in FIG. 7) corresponding to the confirmation operation, 3
The cutting amount is calculated by comparing the three-dimensional shape model with the built-in irregularities.

FIG. 9 is a flowchart showing the contents of the partial data correction processing of FIG. As described with reference to FIG.
From the two-dimensional image G2, the face elements of the first classification, which are difficult to be distinguished by the shapes such as eyebrows and eyes, are extracted (# 2310).
A face element of the second classification, which is difficult to discriminate by color such as nose and neck, is extracted from # 3 (# 2320). Either the first classification or the second classification may be extracted first, and the extraction processing of both classifications may be performed in parallel.

The part U to be corrected for the three-dimensional shape model
p is set (# 2330), and the correction vector v is calculated by projecting the mesh M (# 2340, # 2350). Further, an intersection (contour point) q between the contour of the correction target portion Up and the mesh M is obtained (# 2360). Either the correction target portion Up or the contour point q may be obtained first, and the operations for obtaining them may be performed in parallel.

Then, a surface including the tip point p2 and the contour point q of the correction vector v is obtained as a correction partial model (# 23).
70), the obtained surface and the original three-dimensional shape model are combined (# 2380).

In the above-described embodiment, the three-dimensional model creating apparatus 1 is supposed to be used as a vending machine. However, the data processing according to the present invention does not matter whether the model creation is charged or free. The size of the model is not limited to the reduced size,
It may be full size or enlarged size. The original object may be a creature other than a person.

[0047]

According to the first to third aspects of the present invention,
The visual features of the object can be reflected in the shape and enable the creation of exaggerated models as needed.

[Brief description of the drawings]

FIG. 1 is an external view of a three-dimensional model forming apparatus according to the present invention.

FIG. 2 is a plan view of an operation panel.

FIG. 3 is a functional block diagram of the three-dimensional model creation device.

FIG. 4 is a perspective view showing an example of a mechanism configuration of the processing system.

FIG. 5 is a schematic diagram of extraction of a face element.

FIG. 6 is a schematic diagram of a partial data correction.

FIG. 7 is a main flowchart showing a schematic operation.

FIG. 8 is a flowchart showing the contents of the data processing of FIG. 7;

FIG. 9 is a flowchart showing the contents of a partial data correction process of FIG. 8;

[Explanation of symbols]

 40 data processing device (three-dimensional shape data processing device) 1A modeling system 36 two-dimensional photographing device (photoelectric conversion device) 34 three-dimensional measuring device DS shape data (shape model) G2 two-dimensional image G3 distance data Up Correction target part (feature Part corresponding to the area)

Claims (3)

[Claims] 1. A first classification feature region that satisfies a first setting condition is extracted from a two-dimensional image obtained by photographing an object, and a second classification feature region that satisfies a second setting condition is obtained from a distance image obtained by three-dimensional measurement of the object. Extracting a characteristic region of the classification, and modifying the shape model of the object obtained by the three-dimensional measurement to modify a portion corresponding to the extracted characteristic region of the first classification and the second classification. A three-dimensional shape data processing apparatus for creating a model characterized by the following. 2. The method according to claim 1, further comprising the steps of:
2. The three-dimensional shape data processing apparatus for model creation according to claim 1, wherein a region of a shadow in the object is divided by referring to image information around the shadow portion. 3. A photoelectric conversion device for photographing an object, a three-dimensional measurement device for measuring a shape of the object, and the three-dimensional measurement device based on a two-dimensional image of the object obtained by the photoelectric conversion device. 3. A modeling system for creating a model, comprising: the three-dimensional shape data processing device according to claim 1 or 2, wherein data correction is performed on the shape model of the object obtained by the method.