HK1062854A - Method and system for enhancing portrait images - Google Patents

Description

Method and system for enhancing portrait images

Technical Field

The present invention relates generally to the field of digital image processing, and more particularly to improved image production from portrait-type images of targeted persons.

Background

For centuries, only the wealthy or privileged classes of the upstream society have the ability to hire skilled craftsmen to laboriously produce elaborate portraits with paint, sculpture and drawing. In many cases, portraits offer the goal of easy creation of a more than satisfactory actual portraits. In a subtle or overt way, the painter's job will interact with the intent and purpose of the target object. The second category of artistic breaks includes improvements to the entity. The target object is thus rendered in such a way as to minimize its physical imperfections and present the most attractive allowable external characteristics.

In modern society, portrait is no longer an exclusive area of wealthy and brute force. The advent of photography into all levels of society has reflected that the creation of portrait images is a ubiquitous part of many life's major events. In western cultures, weddings, graduations, birthdays, arrival of babies, etc., all of which are typically captured in relatively standardized portrait images. While the state of the art allows individual hobbyists to capture, and even enhance images such as these, there still exists a category of professional photographers that provide for the generation of high quality portrait images. Not surprisingly, the purpose of the portraiture expert is still the same as in the last hundred years to present the target object in the most pleasing possible way. In essence, the target object wants to be what they are considered to be, rather than what they actually are.

In response to the desire of people to be viewed as they wish, rather than as they actually are, professional photographers resort to retouching the portrait image to give people their preferred appearance. Embellishments include altering the photographic image in some way, i.e., an image that was not captured or described in the original photographic image. One purpose of retouching portrait images is to make a person look beautiful by removing temporary imperfections, such as blemishes or dark circles under the eyes, or permanent imperfections, such as moles or wrinkles, while still maintaining the personality of the individual. Removing facial blemishes, moles and scars, softening lines and wrinkles, reducing bags under the eyes, whitening teeth and white eyes are examples of modifications that may be performed to improve or enhance the appearance of an individual in a portrait image.

Prior to the advent of the digital information age, either the portrait image was modified by using coloring to mask or alter imperfections in the image, or image modifications were performed on the negative or printed image. Because digital image capture devices are commonly used, it is a preferred retouching method via digital image technology implemented on the captured digital image. Digital methods allow enhancements to be implemented that are either very difficult to do or previously impossible to implement on analog images. Image editing software, such as Adobe Photoshop, can be used to beautify the figure by removing the blemishes, straightening the nose, balancing the eyes, and applying digital makeup.

Improvements in computer technology and image processing algorithms allow for a new class of automatic and semi-automatic image enhancement. With respect to the target object of the portrait image, related technical modifications include face detection and recognition, feature detection and implementation of mask (mask) for the face, face repositioning and red-eye detection and correction.

In published PCT patent application WO 00/76398A 1, "skin image analysis System and method", Hillebrand et al disclose a system that can detect skin defects and calculate a skin cleanliness index (skin safety index). This system is directed towards targeting cosmetics and skin care. The system may also simulate ameliorating defective skin areas that would be achieved when using a recommended treatment or product that eliminates or hides the skin defect. The skin defects are detected using color information and standard morphological techniques are used to simulate improvement in the skin area of the defect.

In published european patent application EP1030276 a1, "method of correcting face images, makeup simulation method, makeup kit and base decal", Utsugi describes a method for preparing a desired face after makeup via image processing based on a desired face or a model face. The technique includes making a bright portion, and a shape of eyebrows, an eye line, and a lip line that are close to a desired face or that model the face within a range that is possible by cosmetic modification.

A disadvantage of such systems, especially for a set of portrait uses, e.g. for weddings, graduations, schools and moving pictures, birthdays, arrival of babies, etc., requires close interaction with the user in order to enter selections and evaluation results. For example, in Utsugi, the cosmetic user's ceremony requirements are developed from the model face, which is selected, for example, from a popular talent, actress, or actor, and different adjustment preparations are made to reach the model face. In addition, a skilled operator is required to work with the user to produce a satisfactory result.

Thus, retouching a portrait is a technique of its own, even if digital images are presented, and has heretofore kept more artistic than a subject. Furthermore, the process of retouching portrait images is a highly manual and time consuming process performed by skilled operators. It would therefore be advantageous to develop a system that uses automated and semi-automated portrait image enhancement methods to enable easy retouching of the portrait. The present invention addresses the shortcomings of the prior art described above by providing a method and system for automatically enhancing the appearance of a target object in an image.

Disclosure of Invention

It is an object of the present invention to overcome one or more of the problems set out above. Briefly summarized, according to one aspect of the present invention, a retouching method for enhancing an appearance of a face located in a digital image, comprising the steps of: (a) acquiring a digital image comprising one or more faces; (b) detecting locations of facial feature points in one or more faces, the facial feature points including points identifying salient features including one or more of skin, eyes, eyebrows, nose, mouth, and hair; (c) using the locations of the facial feature points to segment the face into different regions, the different regions including one or more of skin, eyes, eyebrows, nose, mouth, neck, and hair regions; (d) determining one or more face-related features representing different locations; (e) based on the determined face-related features representing different parts of the face, selecting (1) two or more enhancement filters, each specifically dedicated to a particular part, and (2) default parameters for the enhancement filters; and (f) performing the enhancement filter on the particular region to produce an enhanced digital image from the digital image.

The present invention has the advantage of efficiently using automated and semi-automated portrait image enhancement methods to enable the retouching correction to be made and monitored to retouch the portrait without the need for skilled operator intervention. Thus, a highly manual and time consuming process performed by skilled operators is avoided and the retouching method can be used for batch processing.

These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings.

Drawings

FIGS. 1A and 1B are, respectively, a system for practicing the present invention and a graphical user interface for use with the system.

FIG. 2A is a flow chart for implementing an embodiment of the present invention with a selectable degree of user intervention.

FIG. 2B is a flow chart for automatically implementing embodiments of the present invention in a batch mode without user intervention.

Fig. 3 is a flowchart of the face detection and facial point location steps shown in fig. 2A and 2B.

Fig. 4 is a diagram illustrating an example of the positions of feature points on a detected face.

Fig. 5A is a flow chart of a method for determining a neck region.

5B-5D are pictorial examples that visually illustrate the process of generating the final neck probability map.

Fig. 6 is a schematic diagram of an ensemble of enhancement filters for use in the system illustrated in the flow diagrams of fig. 2A and 2B.

Fig. 7A is a flowchart for the skin texture enhancement filter shown in fig. 6.

Fig. 7B is a schematic diagram of a pixel adjacency including the trough edge filter kernal used by the skin texture enhancement filter.

FIG. 8 is a flow chart illustrating a preferred embodiment of selecting and modifying skin features in accordance with the present invention.

FIG. 9 is a diagrammatic example of a pinwheel filter illustrating a line segment drawn around a defect pixel.

Figures 10A and 10B together represent a flow chart illustrating a process for estimating correction values for defect pixels using the pinwheel filter of figure 9.

FIG. 11 is a flow chart illustrating a preferred embodiment for calculating new pixel values in the process illustrated in FIG. 10.

Fig. 12 is a flowchart illustrating a process for generating a line segment via a feature pixel in the pinwheel filter shown in fig. 9.

Fig. 13 is a flow diagram of a presently preferred embodiment of the skin tone enhancement filter shown in fig. 6.

Fig. 14 is a flow diagram of a presently preferred embodiment for calculating the shading/highlighting intensity for use in the skin tone enhancement filter shown in fig. 13.

Fig. 15A, 15B and 15C illustrate several functions showing the results of different blending coefficients used in the skin enhancement filter shown in fig. 13.

Fig. 16 is a flow chart of a presently preferred embodiment of the tooth and eye enhancement filter shown in fig. 6.

Fig. 17 is a diagram illustrating a mixing function used in the skin texture enhancing filter.

Fig. 18 is a diagram illustrating an example of control points used for adjusting the shape of the eye.

Fig. 19 is a flow chart of a presently preferred embodiment for the shape enhancement filter shown in fig. 6.

Detailed Description

As image processing systems utilizing facial feature detection and feature enhancement are well known, the present specification will address features that form, inter alia, part of, or more directly cooperate with, the method and system in accordance with the present invention. Features not explicitly shown or described herein may be selected from those recognized in the art. In the following description, the preferred embodiments of the present invention will typically be implemented as software programs, although those skilled in the art will readily appreciate that the equivalent of such software may also be constructed in hardware. The software, which is not explicitly shown, suggested or described herein, is conventional in the art and is within the routine skill in such art, and is useful in implementing the present invention, as described in accordance with the present invention. As a matter of terminology, in the description of the present invention, reference is made to enhancement filters as enhancing visual appearance of a face in a digital image. For example, an eye enhancement filter is a method of enhancing the white and/or iris color of the eye.

If the present invention is implemented as a computer program, the program may be stored in a conventional computer-readable storage medium, which may include, for example: magnetic storage media such as a magnetic disk (such as a floppy disk or a hard disk) or magnetic tape; optical storage media such as optical disks, optical tape, or machine-readable bar codes; solid state electronic storage devices, such as Random Access Memory (RAM) or Read Only Memory (ROM); or any other physical device or medium employed to store a computer program.

Fig. 1A illustrates a system 10 that is effective in practicing the present invention. The system 10 includes a personal computer PC12 including a Central Processing Unit (CPU) that can perform a set of predetermined steps for implementing the method of the present invention. A digital storage medium 20 is also provided in connection with the PC12 for storing digital images. The digital storage medium 20 may include different types of devices such as RAM, ROM, hard and floppy disk drives, and the like. The digital storage medium 20 may also be used to store the generated enhanced image locally. In addition, digital image capture devices such as a scanner 28 and a digital camera 30 may be provided to the computer 12, with the scanner 28 and digital camera 30 being additional sources of digital images. However, it should be understood that the digital image may be obtained from any source. A user interacts with the computer 12 via an input device 40, such as a mouse and/or keyboard, and a display monitor 50 is connected to the computer 12. The system 10 may also include a device, such as a printer 34, for outputting the image locally. Typically, the components described above would be stored on the PC12 in the sense of being directly connected to the PC 12.

Alternatively, the components described above need not all be stored on the host computer 12, but may be stored on a server 62, the server 62 being connectable to a client PC12 via a communication network 70. The server may also include a Central Processing Unit (CPU) that may perform a set of predetermined steps to implement the method of the present invention. The server may also be connected to a storage medium 65 and one or more printers 60. This may allow images to be remotely acquired, stored and printed via the communications network 70 using the storage media 65 and printer 60 connected to the server 62. The software for implementing the present invention is typically stored on the storage medium 20. Alternatively, this software may be downloaded from the server via the communication network 70. Software for implementing the present invention may be executed either on the client using a CPU contained in the PC12 or on the server side using a CPU contained in the server 62. The communication network 70 may comprise a private communication network, such as a Local Area Network (LAN), or a public network, such as the internet, which may be accessed by an individual using an Internet Service Provider (ISP). In such a network, the remote network service provider may also be accessed by the user using a retail kiosk or any other suitable communication device, as is conventional.

FIG. 1B shows an example of a Graphical User Interface (GUI) for executing the software of the present invention that runs locally on the system 10 or remotely on the server 62 and produces a GUI screen 78 as shown in FIG. 1B. The user loads the software and downloads an image to be enhanced, and the graphical user interface panel 78 is displayed on the display 50 as the image is downloaded. The image 80 on the left of the GUI screen 78 is the original downloaded image with the face to be enhanced by the method of the present invention. On the right, an image 82 with an enhanced face is presented. In one embodiment, the user taps the eyes of the face 80 to be enhanced when the image is downloaded. As will be described in detail later, in response, the system automatically finds feature points of the face and segments the face into different features (e.g., eyes, eyebrows, etc.) and the neck. The system establishes default parameters and applies all enhancement filters to the original image in a predetermined order.

The resulting image 82 is displayed on the right side of the GUI screen 78. The sliders 90, 92, 94, 96 and 98 allow the user to interactively change the parameters of the different enhancement filters. The initial position of the slider corresponds to the default value, which is automatically set by the system. The main appearance enhancer slider 90 combines all of the component enhancement sliders. The assembly sliders include a texture enhancer slider 92, a skin enhancer slider 94, an eye enhancer slider 96 and a teeth enhancer slider 98. The texture enhancement slider 92 controls the parameters of the texture enhancement filter. The skin enhancer slider 94 controls the parameters of the skin tone enhancement filter. The eye enhancer slider 96 and the tooth enhancer slider 98 control the parameters of the eye and tooth whitening filters, respectively. All enhancement filters are described in detail in the following sections. The minimum and maximum values for all sliders are set to "no enhancement" (e.g., at the left end of each slider) and "maximum enhancement" (e.g., at the right end of each slider), respectively.

The user can control the degree and mood of facial enhancement by using a universal extrinsic feature enhancer slider 90 or separate component sliders 92-98. Whenever the user changes the position of the main appearance enhancer slider 90, the system maps the slider position to the appropriate parameter values for the enhancement filters and applies all enhancement filters to the original image in a predetermined order. The enhanced image 82 is then displayed on the right side of the GUI screen 78. Whenever the user changes one of the component enhancer sliders 92-98, the system applies all enhancement filters to the original image in a predetermined order based on the position of each component enhancer slider. The enhanced image 82 is then displayed on the right side of the GUI screen 78. Part of the GUI structure in the preferred embodiment is the option to modify the facial feature points and neck contour. When the user selects an option from the menu "edit" from the top bar of the GUI screen 78, the facial feature points and neck contour points are superimposed on the original image 80, and the user can change the position of the displayed points using the user input device 40, such as a pointing device. The toolbar 84 includes specialized tools that the user may use to interact with and modify the displayed image. For example, a tool for zooming in and out, a tool for editing feature points, a blending tool for locally blending the original image with the enhanced image, a tool for intermittently modifying the results of the enhancement filter, and so forth.

Fig. 2A is a simplified flow diagram illustrating an embodiment of a method for enhancing a portrait image according to the present invention. After the process is initiated at step 200, a digital image is acquired by the system and then displayed on the display monitor 50 at an acquisition and display step 205. In the present invention, a digital image refers not only to an image obtained from a picture, but also to a digital image obtained without limitation from any source, for example, from a digital camera, scanning of a hard copy image, or electronically originating from another source. In a locating step 210, the faces of the individual in the image are detected and the locations of the facial feature points on each face are identified. The process of locating faces and their associated feature points may be performed manually or semi-automatically or automatically by a user using image processing techniques. The locations of the facial feature points are used to identify and segment the face into different regions (such as skin, eyes, nose, mouth, hair, etc.) and neck. The feature points of the face are superimposed on the image and displayed on the display monitor 50 in a display step 215. Optionally, feature points outlining the neck are also displayed. If the feature points of the face are determined automatically or semi-automatically, the user has an opportunity to determine whether the feature points need to be adjusted at decision step 220. If the user determines that the feature points need to be adjusted, the user may adjust the feature points of the face using input device 40 at a modification step 225. Optionally, the user may also adjust the feature points outlining the neck.

In a default setting step 230, the system uses the locations of the facial feature points to identify and segment the face into different regions, such as skin, eyes, nose, mouth, hair of the face, etc., and determines appropriate default parameters for the enhancement filter. In a preferred embodiment, the neck is automatically segmented based on the location of the facial feature points. The method of segmenting the neck will be described in detail later. Thus, in step 230, the system also determines the appropriate default parameters for the neck for the enhancement filter. Optionally, at this stage, the gender and age of the face may be determined manually or automatically using gender and age classification algorithms. An example of an automatic gender classification is described in proc.of4th IEEE Int' l conf.on Face & getture Recognition, 3 months 2000, b.moghaddam and m.h.yang, "GenderClassification with Support Vector Machines," which is incorporated herein by reference. Typical algorithms for age-based work perform analysis of digital images of the face of a target object for specific detailed facial features, such as facial feature rates and wrinkle analysis. One automatic age classifier that may be used in practicing the present invention is disclosed in U.S. Pat. No.5,781,650 to Lobo and Kwon, which is incorporated herein by reference. Gender and age classification may be used to select parameters for which the enhancement filter should be performed along with gender-specific parameters. For example, as a default, a large amount of skin texture and skin enhancement may be applied to a female's face as compared to a male's face.

The system then executes the enhancement filter using the appropriately selected parameters at execution step 235. Alternatively, as shown by the dashed lines, the user may determine that the enhancement filter should be performed along with the appropriate parameters for the enhancement filter at a user selection step 240. After the system completes executing the enhancement filter at step 235, the enhanced image is displayed in display 245 on display monitor 50. At this time, the user can determine whether the enhanced image is satisfactory. If the image is not satisfactory, the user may choose to either change the enhancement filter and/or its parameters at user selection step 240 or adjust the locations of the facial feature points at modification step 225. In one embodiment, the user may choose to sequentially perform different enhancement filters to see the effect of a given enhancement filter having the image thereon. In this embodiment, the user will cycle through steps 240, 235, 245, 250, changing the enhancement filters and parameters until a satisfactory result is obtained. If the enhanced image is deemed satisfactory at decision 250, the user may choose to either print locally at printer 34, at local print 255, or archive locally on storage medium 20, at local archive 260. Alternatively, if the system is connected to server 62 via communication link 70, the user may also choose to either print remotely on printer 60, at remote printing step 265, or archive remotely on storage media 65, at remote archiving step 270. After the appropriate action has been selected and completed, the system queries for the presence of a new image (query step 275) and, depending on the response, loops back to the acquire and display step 205 for the next image or ends the process (end step 280).

Fig. 2B illustrates an alternative embodiment of enhancing a portrait image according to the present invention that is more conducive to highly automated and efficient production at commercial photography facilities. This embodiment describes an automatic batch process for computing many images via the portrait enhancement system without user intervention. In an acquisition step 282, an image is acquired from a list of images (computer files) that have been selected to be enhanced. In a face detection and localization step 284, faces in the image are automatically detected and the locations of the facial feature points on each face are automatically identified. In an enhancement selection step 286, the system characterizes the face and determines an aggregation of appropriate enhancement filters, as well as default parameters for the enhancement filters. Optionally, at this stage, the gender and age of the face may be automatically determined using a gender and age classification algorithm (as described above).

Alternatively, the age and gender of the face in an image may be provided to the system via metadata associated with the image, e.g., age and gender may be provided from or otherwise queried by a user providing a photofinishing envelope. The default filters and parameters may also be provided as metadata-dependent images or as a separately set image before performing the enhancement process. The system then executes the ensemble of enhancement filters using the appropriately selected parameters, at an enhancement execution step 288. If there is more than one face in the image, this fact is annotated in the face query step 290, and the system repeats via steps 286, 288, and 290 until all faces in the image are enhanced. In a storing step 292, the enhanced image is stored. The process of enhancing the image is continued until all input images have been enhanced (image query step 294). After all the images have been processed (end step 296), the enhanced images may be applied to the application level 255-270 shown in FIG. 2A. Optionally, after all images have been processed (end step 296), the enhanced image can then be incorporated into the user interaction system, as described in FIG. 2A, verified and if necessary modified before the application level (step 255 in FIG. 2A and 270).

Fig. 3 is a flow chart for the face detection and point location step 210 of fig. 2A and 2B, showing the location process of salient facial feature points on a face present in an image according to the present invention. A face detection module 310 is adapted to the digital image to mark the locations and approximate sizes of all of the human faces present in the image, and a facial feature detector 315 locates the salient facial feature points on the detected faces. The face detection step may be performed manually or automatically by the user using one of many face detection methods already described in academic literature. One preferred automatic face Detection method for this application consists of the method described in HenrySchneiderman, a Statistical model for 3D Object Detection Applied faces and cars, ph.d. thesis Robotics Institute, geographic MellonUniversity, month 5 2000, which is incorporated herein by reference. Alternatively, if a manual method is used to detect faces, the preferred method is for the user to tap both eyes of one or more faces present in the image. The spatial separation of the eyes can be used to estimate the size of the face.

One or more faces are next detected, and the image coordinates and estimated size of each face are in turn provided to the facial feature detector 315, the facial feature detector 315 having the task of locating the salient facial feature points on the detected faces. In the preferred embodiment, an active shape model is used as the facial feature detector. The active shape models are described in "Automatic interpretation and coding of facial using flexible models", IEEE trans. on PAMI, Vol.19, No.7, pp 743756, 1997, by A.Lanitis, C.J.Taylor, and Cootes, which are incorporated herein by reference. The detected feature points are used to identify, outline and segment different parts of the face, such as skin parts of the face, eyes, nose, mouth, eyebrows, hair of the face, etc., and the neck. The detected sites are identified by corresponding binary signatures. The binary token for that location is then spatially decorated to produce an alpha token. The alpha-mask is used in steps 235 and 288 to blend the results of the enhancement filter (e.g., a dermato-texture enhancement filter) with the original image. Decorating the binary token in the blending operation and applying the resulting alpha token ensures a smooth transition between the regions that have been enhanced and those that have not. To generate the alpha token, the binary token is decorated by blurring the binary token with a blurring function, where the blurring boundary is selected based on the size of the face. As shown in fig. 2A and 2B, the binary signature is used to determine where to de-gapped apply the enhancement filter.

Referring to fig. 4, an example of the visible location of feature points 420 on a detected face 410 is shown. Typically, these facial feature points are located using image processing techniques, either manually or automatically.

In many images, it is critical to apply the skin enhancement filter not only to the facial region but also to the neck. In a presently preferred embodiment, the neck is determined by synthesizing a modified generic neck shape model with a skin color classifier. A flow chart of a method for determining neck is shown in fig. 5A. In a generic mapping step 550, a generic probability map for the neck is generated. The generic neck probability map is based on a priori knowledge of the neck shape. In the preferred embodiment, a generic probability map is generated by averaging a sampled population of normalized images representing an expected population of images to be enhanced (e.g., portrait type images). The sampled images are normalized by scaling each image to the same predetermined location of the eye. In each sample image, one neck is outlined manually. The final neck probability for each pixel is the average sum of the scoring functions equal to 1 if the given pixel is part of the neck, and equal to 0 for all sampled images otherwise. A generic neck probability map may be generated using progressive approximations. If the gender of the person is known in the image, in the preferred embodiment, different probability maps are used for men and women. Generally, skin sites are more visible in a portrait of a woman than a man. In a scaling step 554, a generic neck probability map is scaled to fit the feature points of the individual's chin line and facial contour.

In the skin color classification step 556, a neck probability map is generated based on the color segmentation. A pixel-based supervisory color classifier is employed to label all pixels within a specified interval of skin color. The pixel-based color classifier is modeled as a gaussian, which is a conditional probability function of the pixel color C due to skin,(equation 1)

Here the mean vector u and the covariance matrix M are estimated from the defined skin site. The vector C corresponds to the red (R), green (G) and blue (B) signals of the pixel. The above method is also applicable when C is represented in other color configurations, such as CIELAB, YUV, HSV, etc. A subset of the facial skin sites is used to determine a conditional skin probability distribution. In the presently preferred embodiment, skin sites over the eyes and skin sites of detected facial hairs are excluded from the use of the estimated mean vector p and covariance matrix M in equation eq.1. The neck probability is defined for all pixels outside the facial region and below the chin line according to equation eq.1, otherwise it equals 0.

The final probability map is generated by synthesizing the scaled generic neck probability map and the skin tone based probability map, at a final map generation step 558. In the preferred embodiment, the two probability maps are mathematically and logically multiplied. The resulting neck probability map is used as an alpha channel to determine how to blend the enhancement filter results with the original image. The binary neck mask is generated by thresholding the final neck probability map. If the probability is greater than 0 for a given pixel, the corresponding token value is equal to 1, indicating the neck, otherwise the token value is equal to 0, indicating a non-neck. The binary neck region characterization code is used to determine where to apply the enhancement filter.

5B-5D demonstrate a process for visually generating the final neck probability map. Referring to FIG. 5B, a scaled generic neck probability map is shown. A generic probability map is scaled to match the individual feature points outlining the face. A generic neck probability map is based on a priori knowledge of neck position relative to the facial contour as described in the previous section. The generic neck probability shown in FIG. 5B is an example of what is possible based on progressive rules rather than the statistical analysis described in the previous section. Which may serve the purpose of demonstrating the quality characteristics of the process. The high probability that the central portion 570 within the generic token corresponds to the neck has a high value (e.g., greater than 0.5 and less than or equal to 1). The low probability boundary portion 572 corresponding to the neck has a low value (e.g., greater than 0 and less than 0.5). Outside the region 572, the probability of the neck decreases gradually to a value of 0. Generally, the probability decreases in a continuous manner from the center of the region 570 to the edge of the region 572 in the horizontal direction and from top to bottom in the vertical direction. The central portion of the mask just below the chin line has the greatest probability associated with it.

An outline 574 of a neck probability map determined by the skin tone classifier is shown in fig. 5C. As described in the previous section, this skin tone based on probability is calculated according to equation eq.1. The probability values outside the contour region 574 are equal to 0. The probability value within this region 574 is greater than 0, as defined by equation eq.1. The two neck probability maps: the scaled generic neck probability map and the neck probability map based on skin color classification are synthesized by multiplication of the two. The outline of the resulting final neck probability map is shown in fig. 5D. The central region 576 corresponds to the high probability region 570 clipped by the skin tone probability region 574. The region 578 corresponds to the low probability region 572 clipped by the skin tone probability region 574.

Once the feature points and neck of the face have been located, one ensemble (i.e., two or more) of extrinsic feature enhancement filters may be applied to the face and neck in the image. Referring to fig. 6, several different enhancement filters are shown. In a preferred embodiment, the following enhancement filters are performed: a skin texture enhancement filter 610, a skin tone enhancement filter 620, a teeth enhancement filter 630, an eyes enhancement filter 640, and a facial feature shape enhancement filter 650. All of these filters will be described in detail in the following sections of this specification. The enhancement filters may be applied in any order. In one embodiment, the user may select any of the enhancement filters in any order he/she wants to apply them to the face and neck in the image. However, in the preferred embodiment, the system sets a default order in which the enhancement filters are applied along with the appropriate parameters.

If the user chooses to run the default settings, then all enhancement filters are applied in the default order along with the default parameters and the enhanced image is displayed on the monitor. The default order of enhancement filters is as follows: skin texture enhancement filters, skin tone enhancement filters, tooth and eye whitening filters (in two orders), and facial feature shape enhancement filters. The enhancement filters are described in the following sections in the same order as the default order in which the filters are applied to the original image.

Skin texture enhancement filter

The task of the skin texture enhancement filter is to smooth the local skin texture of the skin, remove blemishes, dark spots, etc., and either remove or reduce the amplitude and depth of wrinkles. Referring to fig. 7A, a flow chart for a skin texture enhancement filter according to the present invention is shown. In a skin feature generation step 710, the skin feature within the skin region delineated by the facial feature points is detected. The detected skin features are then altered at a feature modification step 720. The skin feature layout includes features that we want to alter, such as spots, dark spots, wrinkles, etc.

In the prior art, color analysis is used to detect specific skin defects. Also in the prior art, the detection of skin defects is performed by analyzing the frequency curve of the color signal associated with the pixels included within the limited 2-dimensional window slid over the skin site. If the associated frequency profile is found to be bimodal, the site is considered to include a skin defect. Since these techniques use color information as part of their primary detection means, they require that the imaging system be color calibrated and/or that the image to be resolved be properly colorimetrically adjusted. When controlled light and/or color filters are used to capture the image, the efficacy of using color information is enhanced. Furthermore, if the detected defect is to be further classified as a stain, brown spot, or the like, accurate color reproduction is necessary based on the color of the defect.

In the picture, the skin imperfections, such as wrinkles, spots, etc., manifest themselves as (in fact exactly) highlights and shadows of the positions where they should not be. The skin imperfections thus correspond to local maxima and minima in brightness. The highlights correspond to peaks and the shadows correspond to valleys in the luminance image. In a process that departs from the prior art, and in accordance with a preferred embodiment of the present invention, skin imperfections are detected by applying peak and valley detectors to a luminance image formed from the component red, green and blue signals. The valley/peak detector is a second order difference controller. The luminance is defined as a weighted linear combination of the red R, green G and blue B signals, as described below,

l ═ K1R + K2G + K3B (equation 2)

Where the value of K represents the weighting. A currently preferred weighting for generating the luminance image is k 1-k 2-k 3-0.333. Examples of trough/peak controllers can be found in the IEEE scientific research report set, Vol.73, No.4, 4 months 1985, D.E.Pearson and J.A.Robinson, "visual communication at version Low Data Rates".

The presently preferred peak/valley controller is a series of four directional controllers of the type employed by pilson et al. A vertical operator V is defined by equation eq.3, which is applied to the center pixel m of the 5x5 neighboring pixels shown in fig. 7B, where the pixel locations are designated a through y, as follows,

v ═ f + k + p + j + o + t-2(h + m + r) (equation 3)

This operator is applied at each pixel location in the delineated skin site. Similarly, the form of one horizontal operator H is shown in equation eq.4, and a pair of left and right diagonal operators DL and DR are shown in equations eq.5 and eq.6. Is applied individually at each pixel location in the delineated skin site, as described below,

h ═ b + c + d + v + w + x-2(l + m + n) (equation 4)

DR ═ c + g + k + o + s + w-2(i + m + q) (equation 5)

DL ═ c + i + o + k + q + w-2(g + m + s) (equation 6)

These operators not only detect the valleys/peaks, but they also have a re-response to the edge bottom/shoulder. They are therefore called valid/peak detectors. The Valedge feature corresponds to a positive value output by the operator, whereas the peakedge feature corresponds to a negative value. A valley/peak detector sensitive to only valleys/peaks is obtained by applying a logical condition to the operator. For vertical valleys, the uniformity of the logical valley detector is given by:

if (f + k + p) > (h + m + r) and (j + o + t) > (h + m + r)

Then V ═ f + k + p + i + o + t) -2(h + m + r) (equation 7)

Otherwise V is 0

For vertical peaks, the uniformity of the logical peak detector is given by:

if (f + k + p) < (h + m + r) and (j + o + t) < (h + m + r)

Then V ═ f + k + p + j + o + t) -2(h + m + r) (equation 8)

Otherwise V is 0

The logic detectors for horizontal and diagonal troughs/peaks have a similar form. Both the valley/peak and the valley/peak operators are valid for generating a skin feature map in accordance with the present invention. From this point on, the term valley/peak operator used for us relates to both the valley/peak and the valley/peak operators.

The effect of noise in the image is suppressed by applying a noise reduction filter before applying the directional trough/peak filter to the image. Suitable noise filters are low pass filters, median filters and other linear and non-linear filters commonly employed in digital images to reduce noise.

The directional trough/peak image produced by the operator is thresholded to retain only the (strongly) relevant skin features. For valley feature maps, pixels having values less than a specified threshold Tv are zeroed out. For a peak feature map, pixels having values greater than a specified threshold Tp are zeroed out. The threshold may be either a fixed general threshold or an adaptive threshold. The presently preferred thresholding method is to use an adaptive threshold whose value is given by equation eq.9.

T＝βL_avg(equation 9)

Where B is a constant and Lavg is the local average luminance around the pixel. Different values of B may be used for the vertical, horizontal and diagonal components. The local average luminance Lavg may be the value of the pixel itself or the average luminance of a neighborhood of pixels, e.g. a 3 x 3 neighborhood of pixels.

The presently preferred step is to generate a composite skin feature layout by synthesizing the individual orientation feature maps.

F ═ max { H, V, DR, DL } (equation 10)

In addition, each pixel in F can be labeled by the filter that originates from the orientation. The orientation label data and/or the independently oriented skin feature map may be useful in removing features such as wrinkles where particular directional features are preferentially removed.

The task of the skin texture enhancement filter is to smooth the skin's local areaThe texture of the skin. The spatial size of the skin texture enhancement filter is varied with the size of the face. Thus, the spatial size of the valley/peak filter is adaptively determined based on the size of the face. Specifically, the spacing between the eyes is used to determine the size of the valley/peak filter. A common representation for the vertical valley/peak operator is(equation 11) where w and h are chosen as a function of face size. A generic valley/peak detector for horizontal and diagonal valleys/peaks has a similar form.

One presently preferred non-directional valley/peak detector is by subtracting the blurred luminance image I from the luminance image I_bGiven by

F(x，y)＝I(x，y)-I_b(x, y) (equation 12) the blurred luminance image may be generated by applying a blurring filter, such as a gaussian or a box filter, to the luminance image. The output of the valley/peak detector is thresholded as described above to produce a peak and valley signature. The limits of the blurring filter are chosen as a function of the face size (which is an example of default parameters) and the size of the skin feature that is desired to be detected.

The feature map is further refined by grouping pixels that are connected to each other via connected component labels. Connected component labeling scans an image, grouping its pixels into components based on pixel connectivity, i.e., all pixels in the connected components are connected to each other in some way. Once all of the groupings have been determined, each pixel is labeled according to its assigned component. Each connected component corresponds to a skin feature. Each skin feature is characterized and classified by its feature-based features, such as size, shape, and location, in the skin site. Where the size corresponds to the number of pixels (which is an example of a default parameter). Shape information is useful in describing wrinkle features from other skin features, such as scars. Furthermore, the feature depicted in the skin feature layout may be further classified using color information about pixels that have been determined to be a skin feature. Once the skin feature maps are generated, they are used to select the skin features to be altered at step 720.

Referring to fig. 8, a flow chart illustrating a preferred embodiment of selecting and modifying skin features according to the present invention is shown. In a feature selection step 810, features to be altered based on their features are selected. As described above, the feature characteristics correspond to size, shape, and color, and are selected based on the location of the features in the skin site. The selected skin feature is thus (adaptively) enlarged, at an enlargement step 820, and then modified, at a filtering step 830. The result of the dilation for a skin feature is to enlarge the size of the skin feature. The amount of augmentation may be either a fixed amount for all selected skin features or an adaptive amount based on the feature representing the skin feature. In a presently preferred embodiment, the skin feature is adaptively augmented based on its size and the size of the face (which is an example of a default parameter). Large skin features are more exaggerated than smaller skin features. The process of expansion may be performed using standard morphological operators.

Although all desired skin features may be selected and modified in one cycle via steps 810, 820 and 830, the best skin enhancement is obtained by cyclically modifying the skin features sequentially via steps 810, 820 and 830 while changing the type of skin feature to be modified. In the presently preferred embodiment, the trough skin feature is first selected and modified at step 810- > 830, and then the remaining peak skin features are selected and modified in two iterations through step 810- > 830. To preserve the skin texture, only skin features whose dimensions are between the specified minimum and maximum dimensions are modified. Alternatively, to preserve the skin texture, only features of the skin whose dimensions are greater than a specified minimum dimension are modified. Furthermore, the minimum and maximum sizes of the skin feature to be modified are directly scaled by the size of the face.

At step 830, a pinwheel filter is applied to each pixel of the enlarged feature. The pixels of the enlarged feature are referred to as feature pixels. All other remaining pixels are referred to as non-feature pixels. In an embodiment of the invention, the characteristic pixels are defined by binary characterization codes, where a value of 0 corresponds to a characteristic pixel and a value of 1 corresponds to a non-characteristic pixel. The pinwheel filter interpolates new values of existing feature pixels by using neighboring non-feature pixels centered on the feature pixel and adjusted by a line segment. The pinwheel filter is described in detail in commonly assigned U.S. patent No.6,104,839, "method and apparatus for correcting pixel values in digital images," which is issued in the name of David r. A brief description of the pinwheel filter is included herein to clarify and specify certain modifications to the original algorithm described in the patent taken as Cok et al. Referring to fig. 9, a set of four line segments 930 is exemplarily graphed (vertical V, horizontal H, two diagonal line segments D1 and D2) for the selected feature pixel 940. The four line segments are spaced at 45 degree increments. The enlarged feature 920 is shown in gray (shaded). The line 910 represents a facial edge. Each line segment in the set consists of both characteristic and non-characteristic pixels on both sides of the pixel 940. The non-feature pixel feature and feature pixels in the line segment are local pixels of the feature pixel 940 selected in a given direction defined by the line segment. The generation method of the line segment group is described in detail later.

Referring to fig. 10A and 10B, a method for calculating a new value suitable for each feature pixel by applying the pinwheel filter in the image is shown. In step 1004, the number of line segments NL, the maximum number of pixels on one side of the line segment MAX NP1 and the minimum number of pixels on one side of the line segment MIN NP1 are set. These parameters will be explained in detail with reference to fig. 12. Based on the size of the face (which is an example of a default parameter), the maximum number of pixels on one side of the line segment MAX NP1 and the minimum number of pixels on one side of the line segment MIN NP1 are set. The larger the size of the face, the larger the values of MAX NP1 and MIN NP 1. These two parameters are unique to the present invention, depending on the size of the face, as compared to the method described by Cok et al. In step 1006, the first characteristic pixel PSEL is selected. At step 1008, a set of NL line segments is generated by the pixel PSEL. In the preferred embodiment, the number of line segments NL in a group is equal to 4, corresponding to vertical, horizontal and two diagonal line segments.

The method of generating line segments is shown in FIG. 12. In step 1110 of FIG. 12, a first line segment is selected, e.g., the vertical line segment (V). At step 1120, the direction of one side of the line segment is selected. This is the direction in which a pixel is being added to the line segment starting from the pixel PSEL. For each line segment, two directions corresponding to both sides of the line segment concentrated at the pixel PSEL are defined. In step 1130, the feature pixels are added to the line segment until the first non-feature pixel arrives along a given direction. In step 1140, the first non-feature pixel is added to the line segment and the neighboring non-feature pixel is added to the line segment in a given direction until one of the following conditions is met:

maximum number of non-characteristic points on one side MAX NP1,

-a facial edge or a facial feature edge is reached,

new feature pixel arrival.

Based on the face size, at step 1004 (fig. 1A), the maximum number of non-feature points on one side of the line segment MAX NP1 is set. The line segment need not pass the facial edge or the edge of the region that is excluded from the skin texture enhancement treatment (e.g., eye region, mouth region, etc.). The constraints described above for generating valid line segments are unique to the present invention as compared to the method described by Cok et al. When the addition of pixels to one side of the line segment is complete, then at step 1160, the direction for the second side of the line segment is set, and the process is repeated for the second side of the line segment at steps 1130 and 1140. When both sides are completed (step 1150), then the next line segment from the group is selected (step 1180), and the process is repeated at step 1120-1160. When all NL segments in the group are generated (step 1170), the process is complete (step 1190).

Referring back to FIG. 10A, as described above, after generating the NL segment set at step 1008, segments having less than non-characteristic points of MIN NP1 on at least one side are removed from the set (step 1010). If no valid line segments remain in the group (step 1014), then the PSEL pixel value does not change for that pixel (step 1052) and a new feature pixel PSEL is selected (steps 1046 and 1048), and the process is repeated (beginning with step 1008). If the number of remaining valid line segments in the group is greater than 0 (step 1014), then for each remaining valid line segment, a linear fit is calculated for the non-characteristic pixels in the valid line segment for each channel (channel) (step 1016). At step 1018, a mean square fit error for each valid line segment of the non-feature pixels for each channel is calculated based on the linear fit calculated at the previous step 1016. At step 1020, as the mean-squared fit error for all channels for a given active line segment, the total mean-squared fit error for each active line segment is calculated according to the following equation.Where N is 1.., N (equation 13)

Where K is the number of channels in the image. For black and white images, K equals 1. For color images, K is typically equal to 3 (e.g., 3 channels correspond to R, G, B channels).

At step 1022, based on the linear fit calculated at step 1016, the value PSEL_n，kThe pixel PSEL for each active line segment of each channel is calculated. At step 1024, the new pixel value PSEL_kIs calculated for each channel k. The final pixel values of the enhanced image are blended by the new pixel values PSEL for each channel k_kAnd the original pixel values of the alpha mask generated in accordance with the previous section. The blending operation ensures a smooth transition between the enhanced and non-enhanced regions of the skin. This blending operation and alpha token at step 1024 is unique to the present invention as compared to the method described by Cok et al. IN an alternative embodiment, the final pixel value PSEL _ OUT of the enhanced image is determined by adaptively blending the new pixel value PSEL with the original pixel value PSEL _ IN for each channel k.

PSEL_OUT_K＝aPSEL_K+(1-a)PSEL_IN_K(equation 14)

The blending coefficient alpha for blending new and original pixel values is dependent on the feature representing the skin feature from which the pixel originates. For example, as shown in FIG. 17, the blending coefficient α may vary with the size of the skin feature from which the pixel originates. Referring to fig. 17, the original pixel values are unmodified for small skin feature sizes, which are smaller than min. For skin features with a size between min and mid, this new pixel value is used as the last pixel value. And for large skin features whose size is greater than mid, the final pixel value is determined by blending the new and original pixel values. In some areas (e.g., around the eye), it is sometimes preferable not to completely remove skin features, but to modify their rough external features. Thus, in another embodiment, the blending coefficient a also depends on the position of the skin feature pixel. Different methods of calculating the new pixel value pselic are described later. If the selected feature pixel PSEL is not the last feature pixel (step 1046), then the next feature pixel is selected (step 1048) and the process is repeated starting at step 1008. Otherwise, the entire process is complete (step 1050).

The number of line segments NL is equal to 4 in the preferred embodiment, and the step of calculating a new value for the selected characteristic pixel PSEL (step 1024 in fig. 10) is shown in detail in fig. 11. The method presented in fig. 11 here is unique to the present invention, as compared to the method described by Cok et al. It is assumed that there is at least one valid line segment in the group (N > 0). Referring to FIG. 11, if the number of active segments N in the group is equal to 1 (step 1026), then the new value PSEL k for each channel k is equal to the value PSELi, k calculated for its segment (step 1028). Otherwise, at step 1030, the active line segment groups are sorted in descending order based on the root mean square error value MSEn for each line segment n. As a result of the classification at step 1030, the first active line segment (N ═ 1) in the group has the highest root mean square error, and the last active line segment (N ═ N) in the group has the lowest root mean square error. If the number of active line segments N in the group is equal to 2 (step 1032), then the new value PSEL k for each channel k is equal to the mean of PSELi, k and PSEL2, k (step 1034). Otherwise, if the number of active segments N in the group is equal to 3 (step 1036), the new value PSEL k for each lane k is equal to the value of PSEL2, k calculated for that active segment at the median value of the root mean square error (N-2) (step 1038). Otherwise, the number of active line segments N is equal to 4, and the new value PSEL k for each channel k is equal to the mean of PSELz, k and PSEL2, k (step 1040). The final pixel value of the enhanced image is computed by blending the new pixel values PSELk for each channel k with the original pixel values of the alpha-mask generated according to the previous section (step 1024).

In use for calculating new pixelsIn a second embodiment of the values, the new values PSEL k for each channel k (step 1024 in fig. 10) are calculated in the following manner. First, a weighting value Wn, k for each line segment n of each channel k is calculated according to equation eq.15.Where n 1., N K ═ 1.., K (equation 15)

Second, as a weighted sum of the PSELn, k values determined for each active line segment n and for each channel k, a new value PSEL k for each channel is calculated according to equation eq.16.(equation 16)

The final pixel value of the enhanced image is calculated by blending the new pixel values PSELi < for each channel k with the original pixel values of the alpha-mask generated according to the previous section (step 1024). This blending operation and alpha token are unique to the present invention, as compared to the method described by Cok et al.

Most skin features we want to alter correspond to valley features, i.e. dark areas surrounded by bright areas. In most cases, the skin features will be altered without regard to the color information associated with the pixels that have been determined to be skin features, although there may be circumstances under which an individual may not want to define characteristics of the face, such as beauty marks to be removed from the individual's picture. In this case, color information relating to the skin feature pixels may be used to determine the type of skin feature that should be removed. An alternative approach is to create a tool into the graphical user interface that will allow the operator to cancel an undesirable modification (e.g., removal of the desired aesthetic mark). In a preferred embodiment, the user selects a cancel/redo tool from the graphical tool list 84 and moves the cancel/redo tool via the pointing device 40, such as a mouse, to the location of the skin feature that the user wants to repair in the original image 80. Tapping a skin feature in the original image 80, which is displayed on the left side, causes the feature to be added back to the enhanced image 82, which is displayed on the right side. Clicking again on the repaired skin feature in the original image now causes the skin feature to be removed from the enhanced image. Thus, the cancellation/redo tool toggles back and forth if it is present in the enhanced image by either removing a skin feature from the enhanced image or restoring it to the enhanced image if it is present in the enhanced image. In an alternative embodiment, the texture enhancer slider 92 may be configured to not enhance and the tool 84 may be used to allow the user to select which skin feature to remove.

In another embodiment, the graphical user interface captures and displays a digital image that includes one or more faces. A skin feature map is generated by identifying and mapping the skin features using any of the techniques previously described, and thus, the skin feature map represents the skin features on one or more faces in the digital image. The pointing device 40 is thus used to point to a particular skin feature. In response to pointing and clicking operations of the pointing device, the skin feature layout is referenced such that the particular feature and appropriate enhancement for the feature provided by an appropriate enhancement filter is enabled for the skin feature pointed at.

Skin color enhancement filter

The task of the skin tone enhancement filter 620 (fig. 6) is to improve the overall appearance of skin tones. The skin tone filter flattens out uneven skin tones and generates more evenly colored skin tones. In addition, the skin tone filter is used to modify the color of skin tones to produce a more desirable skin tone representation. This corresponds to modifying both the lightness and chroma of the skin to match the individual and foster preference for skin tone appearance. Furthermore, skin tone problems tend to exist on a larger spatial scale than skin texture. It is important not to make the skin tone too flat because faces without highlights or shadows are flat and tend to be tiring.

Referring to fig. 13, a flow diagram of a presently preferred embodiment for a skin tone enhancement filter in accordance with the present invention is shown. The skin tone enhancement filter inherently compresses highlights and shadows by adaptively blending (at blending step 1240) the input image i (x) with a blurred version i (x) of the input image obtained from blending step 1210, as described below.

O(x)＝α(x)I_b(x) + (1-. alpha. (x)) I (x) (equation 17)

Where α (X) is a blending coefficient obtained from the coefficient generation step 1230, and X ═ X, y is the position of one pixel. The mixing operation is applicable to all color signal channels. The blending coefficient is a function of the x-shadow/highlight SH { x) intensity image obtained from the shadow/peak generation step 1220, as follows.

α (x) ═ f [ sh (x) ] (equation 18)

Referring to FIG. 14, there is shown a presently preferred embodiment for calculating the shadow/highlight intensity image in accordance with the present invention. The shadow/luminance intensity image from step 1220 is generated by subtracting the average luminance image 1320 from the luminance image 1310. The luminance image is calculated by using equation eq.2. The average luminance image may be either a local average luminance image or a global average luminance. The local average luminance image may be generated by applying a blurring filter, such as a gaussian or box filter, to the luminance image, but the global average luminance is calculated by determining the average luminance of the skin within the skin region delineated by the feature points. When using the locally averaged image to compute a shadow/highlight image, the shadow/highlight intensity is similar to a valley/peak image. In a preferred embodiment, the blur limit used to calculate the shadow/highlight map should be greater than the blur limit or magnitude of the trough/peak detector used in the skin texture enhancement filter. In either case, the fuzzy limit depends on the size of the face (which is an example of a default parameter).

In its simplest embodiment, the blending coefficient is a constant f [ sh (x) ] ═ α (x) ═ α, which is an example of a default parameter, and does not depend on the shading/highlight intensity image as shown in fig. 15A. The blending functions shown in fig. 15B and 15C are useful for toning along highlights and hot spots caused by bright and/or oily skin while leaving the shadow regions unchanged.

The skin tone filter may also be used to modify the coloration of the skin. In a presently preferred embodiment, the color C of a site, such as a skin site, may be modified C by moving the device and altering the color distribution of the site_modAs follows

C_mod(x，y)＝k(C(x，y)- C)+C_preferred(x, y) (equation 19)

Where the vector C (x, y) corresponds to the red (R), green (G) and blue (B) signals of the pixel, C_preferedIs the preferred color vector, C is the current average color vector, and K is a constant that can vary between 0 and 1. This preferenceColor C of_prefered(x, y) may vary spatially depending on the location of the skin to be altered.

A preferred embodiment for illuminating or obscuring a skin site is to modify the contrast of the skin site as follows

C_mod(x，y)＝C^γ(x, y) (Eq. 20)

Where the gamma value is less than a corresponding lit skin tone and the gamma value is greater than a corresponding darkened skin tone.

One embodiment for illuminating the skin site replacement is given by:(equation 21)

And for darkening skin sites is given by:(equation 22)

Where γ varies between 0 and 1.

The above method is also applicable when C is represented in other color configurations, such as CIELAB, YUV, HSV, etc. Furthermore, these equations may be applied to all or a subset of the C components. When C is represented in a color configuration (e.g., CIELAB), where the component is associated with a luminance (L)^*) And chroma (a)^*b^*) This is especially true in connection with this.

Tooth and eye enhancement filter

The task of the tooth and eye enhancement filters 630 and 640 (fig. 6) is to improve the brightness and whiteness of the teeth and eyes. Referring to fig. 16, a flow chart of a presently preferred embodiment for a tooth and eye whitening filter in accordance with the present invention is shown. The salient facial feature points 420 (fig. 4) are used to generate a mask that identifies pixels corresponding to the eyes and tooth sites, at an eye/tooth mask generation step 1510. The input image is used in a valley/peak layout generation step 1520, which is generated using equation 12, where the bounds of the blur are determined by the eye separation (which is an example of a default parameter). The valley/peak layout and color information are used to calculate scaling factors (which are examples of default parameters) for the luminance and chrominance values of pixels within the eye and tooth characterization code, at a scaling factor generation step 1530. The scaling factors are then applied to the luminance and chrominance values of the pixels within the eye and tooth portion that generated the new luminance and chrominance values, at a pixel modification step 1540.

In a presently preferred embodiment, the RGB values for pixels within the codebit are converted to CIELAB (L)^*a^*b^*) Configured, and the luminance and chrominance values are modified,

L^*＝L^*(1+ kP) (Eq. 23)

a^*＝a^*/1+ kP) (Eq. 24)

b^*＝b^*/(1+ kP) (equation 25)

Where K is the aforementioned default parameter and P is the probability that the pixel belongs to either the white of the eye or the tooth. One presently preferred expression for the probability P is as follows,(equation 26)

Where the blur limit is chosen as a function of face size and β is a threshold, F is calculated using equation 12.

Shape enhanced filter

The task of the shape enhancement filter 650 (shown in fig. 6) is to change the shape of facial features such as eyes, nose, and mouth. The shape of the facial feature is enhanced by changing its shape using standard bending techniques. An example of a bending technique useful for practicing the present invention is given by t. Feature-based Image metamorphism, computerpraphics (Feature-based Image morphological change, computer graphics), 26 (2): 35-42, New York, NY, July, 1992, Proceedings of SIGBRAPH' 92, which is incorporated herein by reference.

Referring to fig. 19, a flow chart of a presently preferred embodiment for a shape enhancement filter 650 according to the present invention is shown. At a source control point determination step 1910, the source control points 1810 (see fig. 18) are used to curve the image determined by the feature points 420 (see fig. 4), the feature points 420 depicting the shape of the facial features (e.g., eyes) to be enhanced. The target control point 1820 is determined, at a target control point determination step 1920. The location of the target control point 1820 defines the new location of the source control point 1810 in the shape-changed image. The target control points 1820(D1, D2, D3, and D4) are the new locations of the source control points 1810(S1, S2, S3, and S4), respectively. In a warping step 1930, the source and target control points are used to warp the image. The location of the target control point 1820 is defined by the shape of the facial feature that is desired to be changed. For example, if it is desired to increase the size of the eye, the target control point 1820 is positioned as shown in FIG. 18.

In a presently preferred embodiment, the parameters of the shape enhancement filter are used to define whether the facial feature shape is increased or decreased by specifying the location of the target control point 1820. The shape enhancement filter may be incorporated into the GUI screen 78 (see fig. 1B) by an additional slider control. Furthermore, there may be a unique shape enhancing slider for each facial feature. The system maps the position of the slider into the appropriate parameter values for the shape enhancement filter. Moving the slider in one direction causes the facial feature size to decrease, while moving the slider in the opposite direction causes the facial feature size to increase. Thus, the default position of the respective slider may be a neutral position (until the slider is moved in a sense) that does not impose any shape enhancement.

Throughout the above description, certain parameters have been specified as candidates for system default parameters, e.g., to determine the initial settings of the enhancement filter and the enhancement slider used in the graphical user interface. These parameters have been chosen without limitations as examples of suitable default parameters and should not be considered as limitations or limitations of parameter settings. It should be clear to one of ordinary skill in the art that many other parameters, including those cited in this specification, may be selected and/or specified as default parameters.

Claims (12)

1. A method for enhancing an appearance of a face located in a digital image, the method comprising the steps of:

(a) acquiring a digital image comprising one or more faces;

(b) detecting locations of facial feature points in one or more faces, the facial feature points including points identifying salient features including one or more of skin, eyes, eyebrows, nose, mouth, and hair;

(c) using the locations of the facial feature points to segment the face into different regions, the different regions including one or more of skin, eyes, eyebrows, nose, mouth, neck, and hair regions;

(d) determining one or more face-related features representing different locations;

(e) based on the face-related features representing different locations, selecting (1) two or more enhancement filters, each specifically dedicated to a particular location, and (2) default parameters for the enhancement filters; and

(f) the enhancement filter is performed on the particular region to produce an enhanced digital image from the digital image.

2. The method of claim 1, further comprising the step of: (g) an image utilization process is applied to the enhanced digital image.

3. The method of claim 1, wherein the enhancement filters include at least two of skin texture, skin tone, eye, tooth, and shape filters.

4. The method of claim 1, wherein a gender classification algorithm is applied to determine the gender of the face prior to selecting the enhancement filter in step (e).

5. The method of claim 1, wherein an age classification algorithm is applied to determine the age of the face prior to selecting the enhancement filter in step (e).

6. The method of claim 1, wherein the distance between the eyes of the face is used to determine default parameters for the one or more enhancement filters selected in step (e).

7. The method of claim 2, wherein the utilizing applied in step (g) comprises printing the image on a local device.

8. The method of claim 2, wherein the utilizing applied in step (g) comprises archiving the image on a local device.

9. The method of claim 2, wherein the utilizing applied in step (g) comprises printing the image on a remote device.

10. The method of claim 2, wherein the utilizing applied in step (g) comprises archiving the image on a remote device.

11. The method of claim 1, wherein the facial features determined in step (d) include at least one feature selected from the group consisting of facial size, distance between facial features in a particular region, gender of a target object in the image, and age of the target object.

12. A computer storage medium having stored therein instructions for causing a computer to perform the method according to claim 1.