JP2000322588A - Image processing apparatus and method - Google Patents

Abstract

(57) [Summary] [Problem] Even if a face moves on a screen, the position and size of a figure are determined in accordance with the position of the part, and synthesis is performed by following the figure. And an image processing apparatus capable of obtaining a different face composite image. SOLUTION: An image input unit 100 for inputting an image, a face region detection unit 200 for detecting a face region from an image, a face component detection unit 300 for detecting a position of a face component, and a A graphic calculation unit 500 for determining a predetermined graphic or image, and an image synthesizing unit 6 for embedding a graphic or image according to the position and size of a face area or a face part in an input image
00 and an image output unit 400 that outputs a synthesized image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing apparatus and method for synthesizing a graphic or the like with an image.

[0002]

2. Description of the Related Art There are various systems using face image processing.

For example, there are advanced uses such as presence detection, personal authentication, gaze / face direction detection, and the like. In addition, the followings are mainly intended to deform an image while focusing on a face area.

[0004] 1. Glasses changing system 2. Video editing system 3. Hairstyle change system 4. 4. Couple child simulation system 5. Frame decoration of photo sticker making machine (so-called Purikura (trademark)) There is a face deformation system proposed in SIGGRAPH97, and the like.

[0005] 1. In this case, a feature point is manually designated, and a process for a still image is commercialized. There is a known example such as a spectacle wearing simulation apparatus (Japanese Patent Laid-Open No. 6-139318). Patents have also been filed for automation of feature points.

[0006] 2. Is a process for moving images, and a system for manually designating corresponding points and the like is used in television stations and the like, and is used for broadcasting. As for morphing, various morphing software for still images are commercially available.

[0007] 3. Is an image processing apparatus (Japanese Patent Laid-Open No. 8-329)
No. 278), the face contour is detected, the hair part is determined as an area, and then another hairstyle is superimposed.

[0008] 4. Is a device that morphs two persons or a person and another image and synthesizes, for example, an image of a child of the two persons, and has been commercialized. The detection of the face parts is simplified by matching the face to the position displayed on the half mirror.

[0009] 5. With regard to, the position of the face is not specified, and the main purpose is to decorate the frame of the image. In addition, it is necessary to adjust the position of the face to a specific area and move to the position of the frame.

[0010] 6. Is described in the literature [Darrell, T. Gordo
n, F., Woodfill W., Baker, H.: '' A MagicMorphin Mirro
r '', SIGGRAPH'97 Visual Proceedings, ACM Press, 19
97], a device that finds the position of a person's face by combining color, distance information and pattern information, and deforms the image of the face position to display an effect as if the shape of the face was bent. .

[0011] Regarding their uses and features, 1 and 3 are systems for studying fashion such as hairstyles and glasses. Reference numerals 2, 4, and 6 denote systems for locally deforming an image to improve a visual effect. No. 5 does not require image recognition and does not automatically detect.

[0012]

The present invention does not fall under these devices, and each was functionally inadequate to achieve the following objects of the present invention.

That is, the present invention mainly targets a moving image and follows a face region in which the position and the size are variously changed.
Using the function to automatically detect the face area and face parts, vivid colors are applied to various parts of the face area according to the position and size of the face area and the position and size of the face parts. The purpose of this is to instantly disguise or decorate a person in an image by performing graphic display.

[0014] Further, since the face is automatically detected, the synthesized image having the same effect can be obtained at the moved screen position even if the user moves, without bothering the user. And

Accordingly, the present invention provides an image processing apparatus and a method thereof which have a great effect as an amusement and which can automatically obtain disguise images of various persons having different impressions instantaneously.

[0016]

According to a first aspect of the present invention, there is provided an image input means for inputting an image, a face area detecting means for detecting a face area of a subject in the image input by the image input means, A face part position detecting means for detecting a position of a face part such as an eye or a nose from the face area detected by the face area detecting means; and a face area detected by the face area detecting means or detected by the face part position detecting means. In accordance with the position and size of the face part, the contents and position of a figure or image to be combined with an image corresponding to the image input by the image input means are determined, or the corresponding image is converted. A graphic calculation means for determining the contents and position, and combining the graphic or image determined by the graphic calculation means with the image input by the image input means, or Image combining means for converting an image input by a stage with contents determined by the graphic calculation means, and image output means for outputting an image combined or converted by the image combining means. Image processing device.

According to a second aspect of the present invention, the face area detecting means comprises:
Or, if the image input by the image input unit is a moving image, the face part position detecting means detects only the face area of the previous frame of the moving image, or only the vicinity of the position of the face part as a search area, 2. The image processing apparatus according to claim 1, wherein a face area or a face part of a current frame of the moving image is detected.

According to a third aspect of the present invention, the graphic calculation means includes:
A face region or a position of a face part, a relative positional relationship between the face parts, and a face region or a set of a plurality of figures or images that can be set based on the size of the face part. Information holding means for holding registered figure pattern information; information on the face area detected by the face area detecting means or the face part position detecting means; 2. The image processing apparatus according to claim 1, further comprising: superposition position calculating means for determining a position and a size of a graphic or an image to be synthesized using the registered graphic pattern information held in the means. It is.

According to a fourth aspect of the present invention, in the image processing apparatus according to the third aspect, the registered figure pattern information held in the information holding means can be added, deleted, edited, or exchanged. It is.

According to a fifth aspect of the present invention, the graphic calculation means includes:
Selecting means for randomly selecting or changing, based on random numbers, registered figure pattern information to be used by the superposition position calculating means from among registered figure pattern information held by the information holding means; An image processing apparatus according to claim 1.

According to a sixth aspect of the present invention, there is provided the image processing apparatus according to the first aspect, further comprising an audio output unit for outputting an audio in synchronization with the graphic or image synthesized by the image synthesizing unit. .

According to a seventh aspect of the present invention, the graphic calculation means includes:
Voice input means for inputting a user's voice, and registered graphic pattern information used by the superposition position calculating means, based on the voice input by the voice input means, from among the registered graphic pattern information held in the information holding means. 2. The image processing apparatus according to claim 1, further comprising: a selection unit that selects or changes the image data.

According to an eighth aspect of the present invention, the graphic calculation means includes:
Based on the face area or the position of the face part, the relative positional relationship of the face part, and the face area or the size of the face part, the area to be converted from the image input by the image input means is determined. 2. The image processing apparatus according to claim 1, wherein the image processing unit determines the image and converts the area of the image determined by the graphic calculation unit with predetermined contents.

According to a ninth aspect of the present invention, the image input means includes:
2. The image processing apparatus according to claim 1, wherein a digital moving image or a digital still image is input.

The invention according to claim 10 is the image processing apparatus according to claim 1, wherein said image output means outputs a digital moving image or a digital still image.

The invention according to claim 11 is an image input step of inputting an image, a face area detecting step of detecting a face area of a subject in the image input in the image input step, and a detecting step by the face area detecting step. A face part position detection step of detecting the position of a face part such as an eye or a nose from the detected face area; and a face area detected by the face area detection step, or a face part position detected by the face part position detection step. According to the size, determine the contents and position of a figure or image to be combined with the image corresponding to the image input in the image input step, or determine the contents and position of converting the corresponding image A figure calculation step and a figure or image determined in the figure calculation step are combined with the image input in the image input step. Or an image synthesizing step of converting the image input by the image input step with the content determined by the graphic calculation step, and an image outputting step of outputting the image synthesized or converted by the image synthesizing step. And an image processing method.

According to a twelfth aspect of the present invention, there is provided an image input function for inputting an image, a face area detection function for detecting a face area of a subject in the image input by the image input function, and a face area detection function. A face part position detection function of detecting the position of a face part such as an eye or a nose from the detected face area, a face area detected by the face area detection function, or a position of the face part detected by the face part position detection function. According to the size, determine the content and position of the graphic or image to be combined with the image corresponding to the image input by the image input function, or determine the content and position of converting the corresponding image A graphic calculation function and an image input by the image input function are combined with a graphic or an image determined by the graphic calculation function, or an image is input by the image input function. A program for realizing an image synthesizing function for converting the converted image with the content determined by the graphic calculation function, and an image outputting function for outputting the image synthesized or converted by the image synthesizing function. Is a recording medium of an image processing method.

According to a thirteenth aspect of the present invention, in the case where a predetermined figure or image is combined with an image, the recording medium of the registered figure pattern information related to the predetermined figure or image is provided. This is a recording medium on which registered graphic pattern information characterized by including vector information on graphic composing points composing a graphic or an image based on a certain characteristic point of a target person in the medium.

According to a fourteenth aspect of the present invention, the configuration of the registered graphic pattern information is such that the registered graphic pattern information can be added, deleted, edited, or exchanged. This is a recording medium on which registered graphic pattern information is recorded.

According to the first, eleventh and twelfth aspects, it is possible to automatically detect the face position and instantaneously obtain an image obtained by combining the image with the decoration.

According to the second aspect of the present invention, the processing for detecting the position of the face part can be speeded up, and it becomes possible to instantaneously combine the moving images.

According to the third aspect of the present invention, figures can be superimposed in accordance with the position and size of the detected face area, and even if the user moves on the screen, the position of the combined pattern,
It is possible to follow the size.

According to the invention of claim 4, a plurality of various patterns can be registered at the same time, and it is easy to edit and improve the patterns, so that it is possible to set a pattern according to the user's preference. To

According to the fifth aspect of the invention, it is possible to automatically change the combined pattern at regular intervals.

According to the sixth and eighth aspects of the present invention, a face position is automatically detected, and an image obtained by combining a decoration with the image can be obtained instantaneously, and it is possible to add an acoustic effect.

According to the seventh aspect of the present invention, by inputting a voice, not only can it be easily recognized that the synthesized pattern has changed, but it is also possible to produce the change more effectively.

According to the ninth and tenth aspects of the present invention,
Processing can be performed on moving images and still images, conversion of stored image data is possible, and the effect in each use situation, such as attaching moving images to e-mail or the like, increases.

[0038]

Embodiments of the present invention will be described below.

[Embodiment 1] An image processing apparatus according to Embodiment 1 of the present invention will be described.

In the image processing apparatus, as shown in FIG.
The size and the position of the face area and the face part are detected with respect to the input image, and a predetermined figure is relatively determined according to the detected face part position as shown in FIGS. 2B and 2C. Then, the display and processing are superimposed together with the input image as shown in FIGS. 2B and 2C. Thereby, the displayed content can be made to appear as if the person to be recognized is disguised instantaneously.

In order to generate an image aiming at such an effect, the image processing apparatus is provided with an image input unit 10 as shown in FIG.
0, a face area detection unit 200, a face part detection unit 300, an image output unit 400, a graphic calculation unit 500, and a graphic synthesis unit 600 are required, and each component will be described below.

The image processing apparatus can be realized by, for example, a personal computer having a camera, and a program that performs the function of each component may be installed and stored in the personal computer. In addition, this program is FD, C
The program may be recorded on a recording medium such as a D-ROM, an MO, and a DVD, and may be transferred to another personal computer. Furthermore, you may use it for a photo sticker manufacturing apparatus etc.

(Image Input Unit 100) Image Input Unit 100
Is for obtaining a still image or a moving image of a person. As shown in FIG. 3, the basic configuration includes a camera 101 and a frame grabber 102, and is used for photographing a human face. The input image is color,
Not limited to monochrome.

As another configuration of the image input unit 100, a TV tuner, a VCR, a DV
D or the like.

Further, as another configuration of the image input unit 100, digital video data recorded in a file or video data obtained via a network may be input.

Of course, any type of moving image and still image can be used.
The input image is stored in the memory 103 on the frame grabber 102 and is accessed from each processing unit.

(Face Area Detector 200) Face Area Detector 20
0 detects a face area of a person from an image obtained from the image input unit 100 from the image.

As a method for detecting a face area, a template generated from a previously collected face image is used to apply the method to each part of the image, and a part having a high correlation with the template is defined as a face area.

As shown in FIG. 4, the face area detecting section 200
Comprises an image conversion unit 201, a template calculation unit 202, and an area determination unit 203.

When the input image is a color image, the image conversion unit 201 converts the image into a monochrome image, further reduces the input image to several steps, and generates images of several sizes. The reduction ratio and the like are determined in advance from the information such as the lens system of the camera and the angle of view, the size of the image, etc., and the size of the assumed face area is determined in advance to determine the ratio of several steps.

The template calculation unit 202 raster-scans the reduced image using the prepared template, and obtains the degree of similarity between each partial image and the template. As a method of calculating similarity, a subspace method [Erkki Oya, Hidemitsu Ogawa, Makoto Satoh translation, “Pattern recognition and subspace method”, Industrial Books, 198
6] can be used to determine the similarity. In this case, a template of a pseudo face image similar to a face but not a face is also prepared, and by calculating the similarity at the same time, an erroneous area can be prevented from being detected.

The area determining section 203 outputs information on the area determined to be most likely to be a face, based on the similarity between each partial image in each reduced image. Specifically, it is the reduction ratio of the reduced image and the detection position of the detected location. Considering the size of the template and the reduction ratio, the size of the rectangle representing the face area and the position of the rectangle are output according to the size of the original input image.

(Face part detection unit 300) Face part detection unit 30
0 detects face parts such as eyes, nose, and mouth in the area determined to be a face area.

As shown in FIG. 5, the face part detecting unit 300
Is composed of an image conversion unit 301, a candidate point extraction unit 302, a candidate point verification unit 303, and a candidate point determination unit 304.

The image conversion unit 301 sets, from the input image, the region determined by the face region detection unit 200 as a region of interest, and determines a processing pixel for performing processing on a portion having a low gray value of the region. .

In the candidate point extraction unit 302, a document [Kazuhiro Fukui, Osamu Yamaguchi: "Face feature point extraction by combination of shape extraction and pattern matching", IEICE Transactions (D), vol.
-D-II, No.8, pp2170--2177 (1997)], and detects candidate points in a circular area.
Candidates are represented as multiple points.

The candidate point verification section 303 determines the similarity of each face part with the template by using a pattern recognition method.

The candidate point determining unit 304 narrows down the candidate points using a combination of the positional relationships of the candidate points, and outputs the positions of the candidate points.

This will be described with reference to FIG.

Reference numeral 351 in FIG. 6 represents an input image at time T. As a result of detection of a face area by the face area detection unit 200, a rectangular area 352 is shown in FIG.

The face part detecting section 300 detects a candidate point of a circular area inside the rectangular area (35 in FIG. 6).
3), 4 that satisfies the positional relationship, pattern similarity, etc.
One of the candidates is selected to obtain an eye-nose feature point (354 in FIG. 6).

In this embodiment, when a pupil position and a nostril position are detected for a certain input image, only the vicinity of the detected position is set as a component search range in the next frame.
Performs a tracking search to detect parts at high speed.

In FIG. 6, reference numeral 355 denotes an input image at time T, and it is assumed that an eye / nose feature point has been detected by the above-described detection method. Then, at time T + 1, face detection and eye / nose detection are not performed again from the entire screen. As shown at 356 in FIG. 302: detection of candidate points of the circular region;
The candidate point verification unit 303 determines the similarity between each face part and the template, narrows the candidate points, and outputs the positions of the candidate points.

At time T + 1, the image is at 35 in FIG.
Even if the direction changes in a different direction as shown in Fig. 7, the movement of the person is small for a short time, so the feature points can be extracted even if the search range is narrowed as described above. Becomes

When a large operation is performed and the image is out of the component search range as indicated by 358 in FIG. 6 at the time T + a, the face area is detected by the face area detection unit 200. Search for facial parts again.

By this tracking processing, the time required to determine the position of the face part is reduced, and smooth synthesis and conversion of images can be performed.

In this embodiment, the center coordinates of the position of the pupil of the eye,
Outputs the center coordinates of the nostril position.

Of course, the position of the mouth and other face parts may be detected and output.

(Image Output Unit 400) Image Output Unit 400
Sends the input image to which the combined content is added to the output device.

As shown in FIG. 7, when outputting an image to a screen, the result of synthesis in the image memory of the image synthesis unit 600 is transferred to the video RAM 410. The information is displayed on the display 411 via D / A conversion.

In the case of outputting to a file, image data is transferred to a file pointer or a buffer.

Files are stored on the magnetic disk 430 and the memory 4
40 and so on.

The image may be stored and output through an encoder 420 for compressing the image, converting the format of the image, and the like.

The data may be sent to another device via the network 440.

In the case of performing communication, it is also possible to transmit an image obtained as a result of the synthesis. A method of transmitting and combining on the receiving side may be used.

When the output medium is an image or a digital video, the information of each pixel is changed in the memory of each image according to the drawing contents, and is stored in a recording medium (file).

(Figure Calculation Unit 500) Figure Calculation Unit 500
Holds, selects, and calculates a position of a predetermined figure or image to be superimposed on the input image. The predetermined graphic or image information for superposition as shown in FIGS. 2B and 2C is called “registered graphic pattern information”.

1. Contents of Registered Figure Pattern Information First, the registered figure pattern information will be described.

The contents of the registered figure pattern information indicate the figure attributes such as a specified figure such as a line, a polygon, a circle, an ellipse, and an arc to be displayed, and a specified image such as a pixel set called a bitmap. There are objects, attributes such as the color and texture of the figure, positional information indicating a relative positional relationship with the face area, and the like. These are called objects.

As an example, each object is represented by a structure as shown in FIG.

Here, each argument is used to represent a figure.
It is for expressing parameters and relationship information for expressing a relative positional relationship. An object having a plurality of such objects and performing a graphic arrangement having a certain meaning is referred to as registered graphic pattern information.

In the example shown in FIG. 2B, the registered figure pattern information represents "cat", and is composed of nine objects as shown in FIG. 511, 512, 513 polygonal objects, 514, 515, 516, 517, 51
8, 519 are line objects. Cats can be "disguised" by arranging figures in relative positional relation to the face.

Each object has a different structure for expressing it depending on the graphic attribute of the object. For example, in the case of a polygon object, a variable representing the number of corners of the polygon and position information of each point is defined. In the case of a line object, position information of two points is defined. Also, variables indicating RGB values for indicating color attributes, variables indicating texture, and the like are defined.

FIG. 12 is an explanatory diagram of a specification method for specifying a relative position of an object.

When a face as shown in FIG. 12A is input to the camera, the points shown in FIG. 12B are detected by the functions of the face area detecting section 200 and the face part detecting section 300.

FIG. 12 (c) is a diagram in which some reference points are labeled from the detected feature points, and the coordinates of each feature point are obtained by indicating their names. Can be.

FIG. 12D shows some pieces of vector information for relatively indicating a position using each feature point, and position information can be designated by each label.

Actually, with respect to the input image, FIG.
A method for drawing a polygon (triangle) as shown in FIG.

First, the type of the figure (currently a polygon) is specified, and its position information is specified.

As shown in FIG. 12E, p1, p2,
Each point of p3 is represented by a relative positional relationship from the eye position b1.

For this purpose, the vectors a1, a2 and a3 are specified as follows.

First, b1 (eye) is selected as a reference, and a vector a1 expressing a relative position is represented.

For this purpose, for example, using a vector called FACE_WIDTH, FACE_HEIGHT based on the face area, a1 = t1 * FACE_WIDTH + t2 * FACE_HEIGHT Therefore, p1 = b1 + a1 = b1 + FACE_WIDTH * t1 + FACE_HEIGHT *
A description such as t2 may be provided (t1 and t2 are real values).

Then, in order to write a white triangle as shown in FIG. 12 (f), the following description is possible.

FG_FILL_POLYGON 3 RIGHT_EYE ADD FACE_WIDTH t1 ADD FACE_HEIGHT t2 RIGHT_EYE ADD FACE_WIDTH -t3 ADD FACE_HEIGHT t4 RIGHT_EYE ADD FACE_WIDTH Specify the object at 255. Specify the feature points of the facial parts.

As the feature points of the face parts, there are a plurality of candidates such as the pupil position, the nostril position, the center of the face area, and the midpoint between the parts. In the case of 511, the midpoint of the two nostrils is used as the reference feature point. And sets a relative position from the reference feature point.

The relative position is set so that even when the size of the face changes, the figures are superposed and the figures are superimposed, and the relative position is designated using the relative reference amount.

As the relative reference amount, the size of the detected face area, the distance between two eyes, and the like can be selected.

In step 511, in order to display a triangle, the size of the face area is selected as a relative reference amount, and three points are designated at x and y coordinate positions that are constant multiples of the size. In the case of the color attribute 511, the color is painted and the RGB value is designated.

Similarly, in the case of 512 and 513, each pupil position is selected as a reference feature point, and each point is designated using a relative reference amount above the pupil position.

Since 514 to 519 are line objects, two points are designated. Also in this case, the designation is performed using the reference feature point and the relative reference amount. In the case of a line, designation of a solid line, a broken line, etc., designation of a color attribute, and the like can also be performed.

In addition, for the circle object, variables indicating the center of the circle, variables indicating the size of the circle, and the like are defined.
The structure is similarly defined for other graphic objects.

As for the bitmap object, a position for display, an enlargement ratio for display, and the like are specified.

Note that these objects have different drawing results depending on the drawing order. Therefore, it is assumed that the drawing order is specified and the objects are applied to the images in the order of description.

Next, the behavior of the graphic calculation unit 500 will be described.

As shown in FIG. 9, the figure calculating section 500 includes a figure database 501 and a superposition position calculating section 501.

A plurality of registered graphic pattern information is stored in the graphic database 501, and these registered graphic patterns are managed as files and can be freely newly added,
Deletion is performed. The database manages information such as the number and size of pattern information.

As shown in FIG. 13, the information structure of each pattern has a plurality of registered graphic patterns in the database, and each registered graphic pattern is configured as a set of objects.

This database is composed of a magnetic disk or a memory unit. FD, CD-RO storing a database of registered graphic patterns
A recording medium such as M, MO, DVD or the like may be separately prepared, and this may be stored in an image processing apparatus which is a personal computer by installation or the like. In this case, only the recording medium of the database of the registered graphic patterns may be manufactured and sold.

The pattern editing unit 504 enables input / output of registered graphic patterns. For example, it may have a medium for exchanging a portable magnetic disk, a portable memory unit, or the like, or an interface for exchanging data via a network, or may have an editing editor function.

The registered graphic pattern information can be described in a certain script language.

For example, the registered graphic pattern information representing “cat” is expressed as a script described by text information as shown in FIG. As a result, by editing the text file, the graphic information to be synthesized can be corrected, and this file can be distributed through a network.

The superposition position calculation unit 502 reads necessary reference feature point information and relative reference amount information from the face area detection unit 200 and the face part detection unit 300, and determines the respective positions specified by the arguments. calculate.

This will be described with reference to the flowchart of FIG.

First, one of the registered graphic data information is selected from the graphic database 501 and read into the memory (step 50201). The selection is made by the selection unit 503.

The selecting section 503 sends the ID (for example, identification number) of the registered figure pattern selected by the user to the superposition position calculating section 502, and reads the pattern of the ID.

Here, the selection unit 503 may be provided with a random number generation mechanism and a timer to introduce effects such as performing a pattern change sequence at regular time intervals and switching patterns randomly.

For the registered graphic pattern information loaded into the memory, the following processing is performed for a plurality of objects.

First, one object is read in the order described (step 50202).

An argument defined in the object is read (step 50203).

The arguments include those representing positional information and those representing parameters (such as the size of the radius of a circle). Processing is performed according to the information of the designated arguments for each.

The description will be made in the case of position information. First, variables for representing position information are initialized. Assuming that the point (ax, ay) representing the position information is:
Substitute 0 for both ax and ay (step 50204).

Next, the designated length is read using the position information of the designated face part or the like or the vector shown in FIG. 12 obtained from the face part (step 5020).
5).

Apply a coefficient to the position information (multiply by a constant)
Processing is performed (step 50206).

The variables ax and a representing the position information
It is added to y (step 50207).

Further, if there is a description in the argument, the processing is repeated from step 50205. Otherwise, (ax, a
y) is the position information (step 50208).

If there are more arguments, the process is repeated (step 50209). When the values have been determined for all the arguments of the object, the process ends (step 5021).
0).

According to this procedure, the position of the composite image corresponding to the information of the position of the face detected is determined.

(Image Synthesizing Unit 600) Image Synthesizing Unit 600
Stores the designated registered figure pattern information in the figure calculating section 50.
Combined with each attribute at the position calculated by 0,
Or perform image processing.

FIG. 15 shows an example of the configuration of the image synthesizing section 600.

An image memory 610 holds an input image.
The information is passed to the respective combining units 612.

Each of the synthesizing units 612 is selectively used according to the type of shape described in the registered graphic pattern information and the processing method. As for the instruction, the object information control unit 611 receives the shape, position information, parameters, and image information passed from the graphic calculation unit 500, and repeats the processing by the number of designated objects.

The image stored in the image memory 610 is converted by the synthesizing unit 612 and stored in the image memory 613. If the processing is repeated, the image memory 61
3 to the image memory 610, and the process is repeated.

The object information control section 611 transfers the image in the image memory 613 to the outside when all the syntheses are completed.

So far, the description has been given of the case where a figure is synthesized with an image. Next, superposition of a bitmap image will be described.

The bit maps are superimposed as shown in FIG. The flowchart is shown in FIG.

In FIG. 16, reference numeral 650 denotes an input image to be processed, in which the detected eyes and nose are circled and displayed for easy understanding.

A bit map image (monkey) represented by 652 in FIG. 16 is superimposed on a position calculated according to the detected position to obtain a composite image 651 in FIG. I do.

In step 661 of FIG. 17, a bitmap image to be combined is read, and at the same time, an exclusion color at the time of embedding (representing a region not to be combined and indicating the color of a white region 652 in FIG. 16) is read.

In step 662, the position where the bitmap image is to be synthesized is calculated from the above-described position designation as indicated by 653 in FIG.

At step 663, the input image is read and the synthesis of the bitmap image is started. Since the combining position may be enlarged or reduced, the pixel position in the target area is determined (step 664), and the processing differs depending on the combining position (step 665). FIG.
At the position 655 in FIG. 16 corresponding to the position 654 in FIG. 6, in the case of the excluded color (656 in FIG. 16), the pixel value of the input image is output as it is (step 666), and there is a pattern instead of the excluded color (step 666). Step 657 in FIG. 16 outputs the pixel value of the bitmap image (step 667).

It is determined whether the processing has been performed for all the target areas (step 668).
A composite image as shown at 651 is output.

Of course, a plurality of bitmaps may be synthesized for one input image.

The composition of the synthesizing unit 612 may be changed to one image memory, and the composition is not limited.

The output medium is a display or the like.
In the case of displaying in real time, a configuration in which image conversion is directly applied to the VRAM may be employed.

When the output medium is an image or a digital video, the information of each pixel may be changed in the memory of each image according to the drawing content and stored in the recording medium.

[Embodiment 2] An image processing apparatus according to Embodiment 2 will be described.

In the image processing apparatus according to the second embodiment, an image conversion section 700 is added to the image synthesis section 600 described in the first embodiment.

FIG. 18 shows a configuration example in which the function of the image conversion unit 700 is added to the image synthesis unit 600.

The same functions as those described in the image synthesizing section 600 and having a function of converting an image, for example, a mosaic processing section 701 for performing mosaicization on a certain area of an image, or enlarging a partial area of an image, A partial enlargement / reduction processing unit 702 to be reduced and another image conversion unit 703 are added.

In the embodiment using the image synthesizing unit 600, only synthesis by superimposing figures and images is performed. However, by applying image conversion to the image itself, it is possible to obtain images having various impressions. Become.

FIG. 19 shows an example of image conversion.

FIG. 19A shows an input image to be processed, in which the detected eyes and nose are circled for easy understanding.

Using the information of the detected face area and the position of the eyes and nose for this image,
FIG. 19B shows the result of the mosaic processing.

FIG. 19C shows the result of performing the mosaic processing only on the vicinity of the face area, and the processing for performing image conversion on such an image itself is provided.

Of course, since the face area is automatically tracked, even if a person moves, as shown in FIGS. It is possible to obtain an image in which the face area has been subjected to mosaic processing.

The function of enlarging or reducing a part of the input image can also provide the effect of enlarging the eyes and the nose as shown in FIGS. 19 (e) and 19 (f).

[0158] Therefore, since deformation control can also be performed using part information such as eyes and nose, the conventional technique of 6. This is different from the system described above.

The processing procedure is as follows.

The image memory 610 holds the input image,
It passes to each image conversion part.

Each of the image conversion units 701, 702, 703
It is selectively used depending on the type of image conversion described in the registered graphic pattern information and the processing method.

In response to the instruction, the object information control unit 611 receives the shape, position information, and parameters passed from the graphic calculation unit 500, converts the image held in the image memory 610 by the conversion unit 701, and
Hold at 3. When the process is repeated, the image is copied from the image memory 613 to the image memory 610, and the process is performed again. The object information control unit 611 transfers the image in the image memory 613 to the outside when all the synthesis and the image conversion are completed.

Each image converter operates according to the flowchart shown in FIG. Here, the operation of the partial enlargement / reduction processing unit 702 that performs the partial enlargement processing will be described as an example.

First, in step 711, information of image conversion is read. The partial enlargement processing unit 702 reads the enlargement ratio of the eye area and the like.

In step 712, position information of image conversion for performing image conversion is calculated.

The partial enlargement / reduction processing unit 702 obtains the size of the eye area of the original image and the size of the eye area after enlargement. The position of the eye region before the enlargement is obtained as shown by 721 in FIG. 20, and the position of the eye region after the enlargement of 722 in FIG. 20 is obtained based on the enlargement ratio obtained in step 711.

In step 713, an image to be subjected to image conversion is read from the memory, and in step 714, image conversion is performed.

In the partial enlargement / reduction processing 702, the above enlargement is performed on the target area, and an image like 723 is obtained. Then, in step 715, the image is output.

Of course, such image deformation may be performed a plurality of times, or multiple operations may be performed. Further effects can be expected by using it in combination with the synthesis of figures.

In addition to such image conversion, other processing such as color tone correction or extraction of only the edge of the target area may be performed.

[Modifications] Several modifications of the above embodiment will be described below.

(Modification 1) As shown in FIG. 21, an audio output unit 700 may be added to output audio, music, and the like in synchronism with image synthesis.

In this case, the graphic calculation section 500 may output a sound in synchronization with the frame in which the synthesis is started. In addition, background music may be output when synthesis is being performed. The audio information may be stored along with the registered graphic pattern information.

(Modification 2) As shown in FIG. 22, a speech recognition section 800 may be added to select synthesized contents by speech.

For example, when a voice "monkey" is input to the system, the voice recognition unit 800 recognizes the word "monkey" and selects the disguise content corresponding to the word by an instruction to the selection unit 503. The disguise as shown in FIG. 16 can be realized.

Of course, an extension for adding a keyword to the registered graphic pattern information may be provided for this purpose.

Thus, by uttering the keywords attached to the combined information one after another, various image deformations can be enjoyed one after another.

(Modification 3) In the first and second embodiments, the face area,
Although only the position and the size of the face part were used, additional variations can be added, in addition to the secondary features obtained from the face.

Here, the secondary features refer to blinking, face direction, pupil position, line-of-sight direction, how to open the mouth, personal identification information, and the like.

For example, by deforming a figure according to the direction of a face, a deformation operation such as affine transformation is applied to each figure object, and a figure corresponding to the direction of the face can be output.

Further, the number of objects to be displayed and combined may be changed depending on the state of the face.

For example, the blinking of human eyes is used as a switch, and in a frame with closed eyes, another object is displayed or the attributes (color, size, position, shape) of the object are changed. It is also possible to make it. For such an extension, it is necessary to change the description method of the object and the structure of the registered graphic pattern information.

Although the facial feature is only the position of the eyes and nose, the expression ability can be further enhanced by adding a part for detecting the mouth and the like.

(Modification 4) At present, no change is made to the input image itself, but it is possible to enhance the expressive power by adding a deformation such as morphing to the image and combining it with a graphic display. it can.

For example, by deforming the face area in accordance with the skeleton of a certain animal and then performing graphic display, the reality can be further improved.

(Modification 5) It is also possible to change the attribute of each object according to the information of the input image.

For example, the color attribute is changed in accordance with the color tone of the input image. When a dark image is input, it is effective to reduce the brightness and saturation and superimpose the images.

(Modification 6) In the embodiment, the face detection was explained by one person. However, the method is changed to a method of detecting a plurality of persons and simultaneously detecting face parts information of a plurality of persons, and disguising a plurality of persons simultaneously. It is also possible to make it.

[0189]

According to the present invention, a face image including various deformation patterns can be obtained instantaneously, and it is possible to improve image effects such as amusement, presentation, image communication, and video mail.

Further, since a moving image different from the usual one can be easily generated, it can be expected that the effect as a communication tool is brought about.

[Brief description of the drawings]

FIG. 1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is an example of a synthesis application.

FIG. 3 is a block diagram of an image input unit.

FIG. 4 is a block diagram of a face area detection unit.

FIG. 5 is a block diagram of a face part detection unit.

FIG. 6 is an explanatory diagram of face part detection.

FIG. 7 is a block diagram of an image output unit.

FIG. 8 is a script example of registration pattern information.

FIG. 9 is a block diagram of a graphic calculation unit.

FIG. 10 is an example of a registered graphic pattern.

FIG. 11 is an example of an object structure.

FIG. 12 shows a method for specifying a relative position of an object.

FIG. 13 shows the structure of a registered graphic pattern.

FIG. 14 is a flowchart of a method for calculating a relative position of an object.

FIG. 15 is a block diagram of an image synthesis unit.

FIG. 16 is an explanatory diagram of a bitmap synthesis unit.

FIG. 17 is a flowchart of bitmap synthesis.

FIG. 18 is a block diagram of an image conversion unit.

FIG. 19 is an example of image conversion application.

FIG. 20 is an explanatory diagram and a flowchart of image conversion.

FIG. 21 is a block diagram illustrating an image processing apparatus according to a second embodiment.

FIG. 22 is a block diagram showing a modified example of the image processing apparatus using a voice recognition unit.

[Description of sign]

 DESCRIPTION OF SYMBOLS 100 Image input part 200 Face area detection part 300 Face part detection part 400 Image output part 500 Graphic calculation part 600 Graphic synthesis part

Claims (14)

[Claims] 1. An image input means for inputting an image, a face area detection means for detecting a face area of a subject in an image input by the image input means, and an eye from a face area detected by the face area detection means. A face part position detecting means for detecting the position of a face part such as a nose; and a face area detected by the face area detecting means, or
Depending on the position and size of the face part detected by the face part position detection means, determine the contents and position of a figure or image to be combined with an image corresponding to the image input by the image input means, or A graphic calculating means for determining the content and position of converting the corresponding image; and combining the graphic or image determined by the graphic calculating means with the image input by the image input means,
Alternatively, there is provided: an image synthesizing unit that converts an image input by the image input unit with contents determined by the graphic calculation unit; and an image output unit that outputs an image synthesized or converted by the image synthesizing unit. An image processing apparatus characterized in that: 2. The method according to claim 1, wherein the face area detecting means or the face part position detecting means, when the image input by the image input means is a moving image, the face area or the position of the face part in the previous frame of the moving image. The image processing apparatus according to claim 1, wherein only the vicinity of the image is detected as a search area, and a face area or a face part of a current frame of the moving image is detected. 3. The method according to claim 1, wherein the figure calculating means includes: a plurality of face regions or face parts; a plurality of positions that can be set based on a relative positional relationship between the face parts and a size of the face region or the face parts. An information holding unit that holds registered figure pattern information composed of a set of figures or images, and a position of a face region or a face part detected by the face area detection unit or the face part position detection unit And superimposition position calculation means for determining the position and size of a graphic or image to be synthesized using information such as the size and size and the registered figure pattern information held in the information holding means. The image processing apparatus according to claim 1, wherein: 4. The image processing apparatus according to claim 3, wherein said registered graphic pattern information held in said information holding means can be added, deleted, edited, or exchanged. 5. The graphic calculating means randomly selects or changes registered graphic pattern information to be used by the superposition position calculating means from the registered graphic pattern information stored in the information storing means, by using a random number. 2. The image processing apparatus according to claim 1, further comprising a selection unit that performs the selection. 6. An image processing apparatus according to claim 1, further comprising an audio output unit for outputting an audio in synchronization with a figure or an image synthesized by said image synthesizing unit. 7. The graphic calculation means includes: a voice input means for inputting a voice of a user; and a voice input by the voice input means, from among the registered graphic pattern information held in the information holding means. 2. The image processing apparatus according to claim 1, further comprising: selecting means for selecting or changing registered graphic pattern information used by the superposition position calculating means. 8. The image input means based on a face area or a position of a face part, a relative positional relationship between the face parts, and a size of the face area or the face part. 2. The image according to claim 1, wherein a region to be converted is determined from the input image, and the image combining unit converts the region of the image determined by the graphic calculation unit with predetermined contents. Processing equipment. 9. The image processing apparatus according to claim 1, wherein said image input means inputs a digital moving image or a digital still image. 10. An image processing apparatus according to claim 1, wherein said image output means outputs a digital moving image or a digital still image. 11. An image inputting step of inputting an image, a face area detecting step of detecting a face area of a target person in the image input by the image inputting step, and an eye from the face area detected by the face area detecting step. A face part position detecting step of detecting the position of a face part such as a nose; and a face area detected by the face area detecting step, or a position or a size of the face part detected by the face part position detecting step. A figure calculation step of determining the contents and position of a figure or image to be combined with an image corresponding to the image input in the image input step, or determining the contents and position of converting the corresponding image; With respect to the image input in the image input step, the figure or image determined in the figure calculation step is synthesized, or An image synthesizing step of converting an image input by the image input step with contents determined by the graphic calculation step; and an image outputting step of outputting an image synthesized or converted by the image synthesizing step. An image processing method comprising: 12. An image input function for inputting an image, a face area detection function for detecting a face area of a target person in the image input by the image input function, and an eye from the face area detected by the face area detection function. A face part position detection function of detecting the position of a face part such as a nose, and a face area detected by the face area detection function, or
According to the position and size of the face part detected by the face part position detection function, determine the content and position of a figure or image to be combined with an image corresponding to the image input by the image input function, or A graphic calculation function for determining the content and position of converting the corresponding image, and combining the image or image determined by the graphic calculation function with the image input by the image input function,
Alternatively, an image synthesizing function of converting an image input by the image input function with contents determined by the graphic calculation function, and an image output function of outputting an image synthesized or converted by the image synthesizing function are realized. A recording medium for an image processing method characterized by recording a program to be executed. 13. When a predetermined figure or image is combined with an image, the recording medium is a recording medium of registered figure pattern information related to the predetermined figure or image, wherein the registered figure pattern information is A recording medium on which registered graphic pattern information characterized by including vector information on graphic composing points constituting a graphic or an image based on a certain characteristic point. 14. The configuration of the registered graphic pattern information, wherein the registered graphic pattern information can be added, deleted, edited, or exchanged.
3. A recording medium on which the registered figure pattern information described in 3 is recorded.