JP7604804B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

In recent years, the number of companies that have introduced remote work has increased, and opportunities for online meetings or video calls have increased. During online meetings or video calls, it is desirable to enable video in order to check the facial expression of the person on the other end of the line. When enabling video, the person on the other end of the line must take the time to dress appropriately for the call, such as by putting on makeup if the person is female, or shaving if the person is male. In response to this, a technology has been proposed that corrects video for video calls to a state suitable for the call. For example, Patent Document 1 discloses a technology that, in data communication involving image data, determines the degree of fatigue of a user based on the number of blinks or the degree of bloodshot eyes, and synthesizes the image with a previously captured image if the degree of fatigue is above a certain level. Patent Document 2 discloses a technology that selects or processes video to be used for video communication from "video that is acceptable to show to the other party" that the user has previously determined, and outputs the video.

JP 2001-016564 A JP 2012-142925 A

When video calling images are used solely to determine the degree of fatigue of the caller, unnecessary corrections may be made to the image even if the caller is well-groomed. In addition, when images for communication are constantly selected or processed from images that are acceptable for the other party to see, the images for communication may be corrected regardless of whether the person is well-groomed or not. Furthermore, it may be time-consuming for the user to decide in advance what images are acceptable for the other party to see, and usability may not be optimal.

In one aspect, the present invention aims to provide a technology for correcting a captured image of a person on a video call when that person is not well-groomed.

To achieve the above objective, the present invention adopts the following configuration.

The first aspect of the present disclosure provides an image processing device that includes an extraction unit that extracts features of a user's captured image and a user's reference image, a determination unit that determines whether to enable or disable a correction process for the user's captured image based on a result of comparing the features of the user's captured image with the features of the user's reference image, a correction unit that generates a corrected image of the user's captured image when it is determined that the correction process is enabled, and an output unit that outputs the corrected image when it is determined that the correction process is enabled, and outputs an uncorrected user's captured image when it is determined that the correction process is disabled.

The "reference image" is, for example, an image of a user in a well-groomed state. The image processing device can determine whether or not to correct the captured image based on the result of comparing the captured image of the user with the reference image. The image processing device automatically corrects the captured image according to the determination of whether or not to correct it, so that the user can participate in a video call without worrying about the state of his/her grooming or performing a special operation to instruct image correction.

The feature amount may be a feature amount of a part of the user's face where changes are easily detected. By extracting feature amounts of parts of the user's face where changes are easily detected from the captured image and the reference image, the image processing device can accurately determine whether the grooming of the captured image is well-groomed, i.e., whether to disable correction processing, when comparing images of the same user.

The feature amount may include at least one of Haar-like features, color histograms, and color moments. The feature amount may also be calculated by an algorithm using a learning model that has been trained on images of well-groomed and ungroomed individuals. The image processing device can use various features or a combination of these features to determine whether or not to enable correction processing.

The determination unit may calculate the degree of coincidence between the features of the user's captured image and the features of the user's reference image, and determine to enable the correction process if the degree of coincidence is less than a predetermined threshold, and to disable the correction process if the degree of coincidence is equal to or greater than the predetermined threshold. The image processing device can determine whether to correct the captured image based on the degree of coincidence between the user's captured image and the reference image, and can automatically correct the captured image. This allows the user to participate in a video call without worrying about their appearance or performing any special operations to instruct image correction.

The correction unit may change the amount of correction applied to the captured image of the user depending on the degree of similarity. By changing the amount of correction depending on the degree of similarity between the captured image and the registered image, the user can suppress unnecessary correction when their appearance is relatively neat.

The correction unit may generate the corrected image based on the captured image of the user and the user's reference image. For example, the correction unit may generate the corrected image using a Generative Adversarial Network (GAN) that has been trained on the captured image of the user and the user's reference image. The correction unit may also generate the corrected image by cutting out a part or the entire face of the user's reference image and replacing the corresponding part of the captured image of the user with the cut-out image. By generating the corrected image based on the captured image of the user and the reference image, the image processing device can generate a corrected image that is closer to the reference image.

The correction unit may generate a corrected image by performing a filter process to remove noise or adjust saturation on the user's captured image based on facial feature information. Since the image processing device can generate a corrected image based on various facial feature information without using a reference image, the user can be saved the trouble of preparing a reference image.

The determination unit may determine whether to enable or disable the correction process for each predetermined number of frames of the user's captured image. The image processing device can correct the captured image even if makeup such as lipstick comes off during a video call, allowing the user to continue the call without worrying about their appearance being ruined.

The image processing device may further include an imaging unit that captures an image of the user. The image processing device can be configured simply by being integrated with the imaging unit.

A second aspect of the present invention provides a human body detection method including: an extraction step of extracting features of a user's captured image and a user's reference image; a determination step of determining whether to enable or disable a correction process for the user's captured image based on a result of comparing the features of the user's captured image with the features of the user's reference image; a correction step of generating a corrected image of the user's captured image when it is determined that the correction process is enabled; and an output step of outputting the corrected image when it is determined that the correction process is enabled, and outputting an uncorrected captured image of the user when it is determined that the correction process is disabled.

According to the present invention, when a person on a video call is not well groomed, the captured image of the person can be corrected.

FIG. 1 is a diagram illustrating an application example of an image processing apparatus according to an embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of the image processing apparatus. FIG. 3 is a flowchart illustrating an example of the image correction process. FIG. 4 is a diagram showing a first example of how facial feature amounts are extracted. FIG. 5 is a diagram showing a second example of extracting facial feature amounts. FIG. 6 is a diagram illustrating an example of a correction process for a captured image.

Below, an embodiment of one aspect of the present invention will be described with reference to the drawings.

<Application Examples>
1 is a diagram for explaining an application example of an image processing device according to an embodiment. The image processing device acquires a camera image (captured image) input from a camera and a registered image (reference image) registered in advance in a DB (database), and extracts features from each image. The registered image is, for example, an image of a user in a well-groomed state, and serves as a reference image for determining whether or not to correct the captured image.

The image processing device compares the features of the camera image and the registered image and evaluates the degree of match. If the degree of match is equal to or greater than a predetermined threshold, the image processing device determines that the user is well-groomed and disables correction processing on the camera image. If the degree of match is less than the predetermined threshold, the image processing device determines that the user is not well-groomed and enables correction processing. In this way, the image processing device determines whether to enable or disable correction processing on the camera image based on the result of comparing the features of the camera image and the registered image.

When the correction process is enabled, the image processing device corrects the user's camera image to generate a display image (corrected image), which is then transmitted to and displayed on another computer used by the call partner. The corrected image may also be displayed on the display of the image processing device. The display image may be generated based on a registered image of the user in a well-groomed state. This allows the user to participate in a video call without worrying about the state of their appearance, since the corrected image is displayed without the user having to perform any special operations when their appearance is not good.

<Embodiment>
(Device configuration)
An example of the functional configuration of the image processing device 1 will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating an example of the functional configuration of the image processing device 1. The image processing device 1 includes an imaging unit 10, a registered image database 11, a feature extraction unit 12, a correction determination unit 13, a correction processing unit 14, and an output unit 15.

The imaging unit 10 captures an image of the user who is making a call. The registered image database 11 stores a registered image (reference image) that serves as a reference for determining whether or not to correct the captured image of the user. The registered image database 11 may store a plurality of registered images for each user. The registered image may be, for example, an image captured when the user makes a call for the first time using the image processing device 1. The registered image may also be an image selected by the user while checking an image displayed on the display device. Note that the registered image database 11 is not limited to the registered image of the user, and may also hold information on the feature amount of a well-groomed state.

The feature extraction unit 12 (extraction unit) extracts features from the user's captured image and the registered image. The features are, for example, Haar-like features, color histograms, and color moments. The feature extraction unit 12 may also use a combination of these features as features for determining whether to enable or disable the correction process.

The correction determination unit 13 (determination unit) determines whether or not to enable correction processing for the captured image based on the features extracted by the feature extraction unit 12. Specifically, the correction determination unit 13 compares the features of the captured image of the user with the features of the registered image of the user, and calculates the degree of match.

For example, the correction determination unit 13 can calculate the Haar-like feature amount around the boundary between the eyebrows and the skin on the forehead for each of the captured image and the registered image, and calculate the degree of match as "(feature amount of registered image - difference between each feature amount)/feature amount of registered image." The correction determination unit 13 may calculate Haar-like feature amounts for multiple sites other than the boundary between the eyebrows and the skin on the forehead, and use the average value of these as the degree of match.

The correction determination unit 13 can determine that correction processing for the user's captured image is disabled if the degree of match is equal to or greater than a predetermined threshold (e.g., 80%), and can determine that correction processing is enabled if the degree of match is less than the predetermined threshold.

The correction processing unit 14 (correction unit) corrects the captured image of the user when the correction determination unit 13 determines that the correction process is enabled. The correction processing unit 14 can generate the corrected image, for example, by using a Generative Adversarial Network (GAN) that has learned images of well-groomed and poorly-groomed users. The correction processing unit 14 can also generate the corrected image by cutting out a part of the registered image or the entire face, and replacing the corresponding part of the captured image of the user with the cut-out image and synthesizing the two. Furthermore, the correction processing unit 14 can generate the corrected image by performing a filter process to remove noise or a saturation adjustment on the captured image of the user.

The output unit 15 outputs an image of the user who is the caller. The image output by the output unit 15 is transmitted to another computer. The image output by the output unit 15 may also be displayed on the display device of the image processing device 1. If the correction determination unit 13 determines that the correction process for the user's captured image is enabled, the output unit 15 outputs the user's captured image after correction (corrected image), and if the correction process is disabled, the output unit 15 outputs the user's captured image without correction. If the correction determination unit 13 determines that the correction process is enabled, the output unit 15 transmits the user's captured image after correction to an electronic device such as a tablet terminal used by the call partner for the call to display it.

The image processing device 1 of this embodiment may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. The image processing device 1 has a CPU (processor), a RAM (memory), non-volatile storage (HDD, SSD, etc.), an input device (touch panel, etc.), and a communication device (wired or wireless LAN module, etc.). The image processing device 1 also has a lens and an imaging element (image sensor such as CCD or CMOS).
The image capturing device includes a display device (such as a liquid crystal monitor), and other hardware resources.

The processor deploys a program stored in the storage in the RAM and executes it to realize the functions of each functional unit described in FIG. 2. Note that the method of realizing the image processing device 1 is not limited to this. For example, the image processing device 1 may be realized by distributed computing using multiple computer devices, and some of the functional units may be realized by a cloud server. In addition, some of the functional units of the image processing device 1 may be realized by a dedicated hardware device such as an FPGA or ASIC.

(Image correction processing)
The overall flow of the image correction process will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the image correction process. The image correction process is started, for example, when a user starts an application used for calling on the image processing device 1. Note that the image correction process shown in Fig. 3 is a process executed for each frame of a camera image (captured image).

In S101, the feature extraction unit 12 acquires a camera image of the user captured by the imaging unit 10. The feature extraction unit 12 proceeds to the process of S102 for each frame of the camera image. In the following explanation of each process, the camera image will be explained as an image of one frame of data received from the imaging unit 10.

In S102, the feature extraction unit 12 determines whether the camera image is an image at the timing for evaluating the feature amount. The timing for evaluating the feature amount may be, for example, the timing when the feature extraction unit 12 first recognizes a human face in each frame. Note that the timing for evaluating the feature amount is not limited to the timing when a human face is first recognized, and may be a predetermined interval such as every predetermined number of frames (e.g., 30 frames) or every predetermined time (e.g., 5 minutes). If the camera image is an image at the timing for evaluating the feature amount (S102: Yes), the process proceeds to S103. If the camera image is not an image at the timing for evaluating the feature amount (S102: No), the process proceeds to S108.

In S103, the feature extraction unit 12 extracts the feature amounts of the camera image. It also acquires the user's registered image from the registered image database 11 and extracts the feature amounts of the registered image. Here, the extraction of the feature amounts will be explained with reference to FIG. 4 and FIG. 5.

First Example of Feature Extraction Figure 4 is a diagram showing a first example of extracting facial features. The example in Figure 4 is an example in which feature points unique to a user that are unlikely to change over time are selected, and Haar-like features are extracted around the feature points. The circles shown in the camera image in Figure 4(A) and the registered image in Figure 4(B) indicate feature points unique to a user that are unlikely to change over time.

Fig. 4A shows the results of Haar-like feature calculations in the area around the user's left eye together with a camera image. Fig. 4A shows an example of Haar-like feature calculations using a rectangular pattern for identifying edges (a filter dividing the area vertically or horizontally into two parts, one white and the other black). The calculated Haar-like feature is represented by a rectangle with a shade corresponding to the calculated value. Haar-like feature at the outer corner 401a of the user's left eye
The feature amount is indicated in gray 401b.

FIG. 4B shows the registered image together with the results of calculating the Haar-like feature amount in the area around the left eye of the user. The Haar-like feature amount is calculated in the same manner as in FIG. 4A. The calculated Haar-like feature amount is represented by a rectangle of a shade corresponding to the calculated value. The Haar-like feature amount at the outer corner 402a of the user's left eye is shown by a shade 402b that is darker than 401b in FIG. 4A. In this way, in the registered image, the Haar-like feature amount is larger than in the camera image due to the difference in shade caused by makeup such as eyeliner, and so 402b is darker than 401b.

In the first example shown in FIG. 4, the degree of match between the camera image and the registered image can be calculated, for example, based on the degree of match of the Haar-like features at each of the feature points indicated by circles. The degree of match between the camera image and the registered image may be the sum or average of the degrees of match at each feature point. The degree of match at each feature point can be calculated, for example, as (X-|X-Y|)/X (where X is the Haar-like feature at the feature point of the registered image, and Y is the Haar-like feature at the corresponding feature point of the camera image).

The method of matching features using feature points that are unique to a user and do not change easily over time can also be achieved using a facial recognition algorithm. In the determination processes of S104 and S105, the correction determination unit 13 calculates a score indicating whether or not the person is the actual person using a facial recognition algorithm, and this score can be used as the degree of match between the camera image and the registered image.

Second Example of Feature Extraction Figure 5 is a diagram showing a second example of extracting facial features. The example in Figure 5 is an example in which feature points that are easy to capture changes between the same user are selected, and Haar-like features are extracted around the feature points. The circles shown in the camera image in Figure 5 (A) and the registered image in Figure 5 (B) indicate feature points that are easy to capture changes between the same user.

Fig. 5(A) shows the results of calculating Haar-like features in the area around the left cheek of the user together with a camera image. Fig. 5(A) shows an example of calculating Haar-like features using a rectangular pattern (a filter dividing the area vertically or horizontally into two parts, one side being white and the other being black) that identifies edges. The calculated Haar-like features are represented by a rectangle with a shade corresponding to the calculated value. Haar-like features at the center 501a of the user's left cheek
The feature amount is indicated by white 501b.

FIG. 5B shows the registered image together with the results of calculating the Haar-like feature amount in the area around the left cheek of the user. The Haar-like feature amount is calculated in the same manner as in FIG. 5A. The calculated Haar-like feature amount is represented by a rectangle of a shade corresponding to the calculated value. The Haar-like feature amount at the center 502a of the user's left cheek is shown in gray 502b. Thus, in the registered image, the Haar-like feature amount is larger than in the camera image due to the difference in shade caused by makeup such as blush, and so 502b is darker than 401b.

In the second example shown in FIG. 5, the degree of match between the camera image and the registered image can be calculated in the same manner as in the first example of FIG. 4, for example, based on the degree of match of the Haar-like features at each of the feature points indicated by circles. Note that the features extracted from areas whose color changes before and after grooming, such as a woman's cheeks or a man's beard, are not limited to Haar-like features, but may be features such as color histograms and color moments, or a combination of these features.

In the second example, since feature amounts are compared at feature points that are likely to capture changes between the same user, the correction determination unit 13 can accurately determine whether or not the person is well-groomed. Feature points that are likely to capture changes between the same user can be selected from areas such as the eyebrows, corners of the eyes, cheeks, and mouth for women, and areas such as around the mouth where beards grow for men. By changing the areas of the feature points to be compared depending on gender, it becomes possible to more accurately determine whether or not the person is well-groomed.

Third Example of Feature Extraction As a third example of extracting facial features, an example of using a learning model generated by deep learning such as CNN (Convolution Neural Network) will be described. The learning model used in the third example is, for example, a model generated by having CNN learn images of well-groomed people and images of poorly-groomed people. The images to be learned by CNN may be images other than the user himself/herself, and may include images of the user himself/herself. The feature extraction unit 12 uses the generated learning model to extract the registered image (well-groomed image) and the score of the camera image as features by the CNN algorithm. The correction determination unit 13 can determine whether to enable or disable the correction process for the camera image based on the degree of agreement between the score of the registered image and the score of the camera image in the determination processes of S104 and S105.

Returning to FIG. 3, in S104, the correction determination unit 13 compares the features of the camera image extracted in S103 with the features of the registered image. Specifically, the correction determination unit 13 calculates the degree of match between the camera image and the registered image based on the features of the camera image and the features of the registered image. The degree of match between the camera image and the registered image is calculated by a method according to the type of features extracted from each image, as described in the first to third examples above.

In S105, the correction determination unit 13 determines whether the degree of match between the camera image and the registered image calculated in S104 is equal to or greater than a predetermined threshold. If the degree of match is equal to or greater than the predetermined threshold (S105: Yes), the process proceeds to S107. If the degree of match is less than the predetermined threshold (S105: No), the process proceeds to S106.

In S106, since the degree of match between the camera image and the registered image is less than a predetermined threshold, the correction determination unit 13 sets the correction flag to on (ON) and enables the correction process of the camera image. In S107, since the degree of match between the camera image and the registered image is equal to or greater than a predetermined threshold, the correction determination unit 13 sets the correction flag to off (OFF) and disables the correction process of the camera image.

The correction flag is a flag for the correction processing unit 14 to determine whether or not to execute correction processing on the captured image of the user. When the correction flag is set to on in S106, the correction processing is enabled until the correction flag is set to off for a subsequent frame in S107. If the feature amount is evaluated only at the beginning of a call, when the correction flag is initially set to on, the correction processing unit 14 continues the correction processing of the camera image until the call ends.

In contrast, if the correction flag is set to OFF in S107, the correction process is disabled until the correction flag is set to ON for a subsequent frame in S106. If the feature is evaluated only at the beginning of a call, when the correction flag is initially set to OFF, the camera image is displayed on the display device without correction.

In addition, if the timing for evaluating the feature amount is every predetermined number of frames, when the correction flag is set to ON in S106, the correction process of the camera image will be valid until the next time the feature amount is evaluated. Conversely, when the correction flag is set to OFF in S107, the correction process of the camera image will be invalid until the next time the feature amount is evaluated, and the camera image will be displayed on the display device without correction.

In S108, the correction processing unit 14 determines whether the correction flag is on (ON). If the correction flag is on (S108: Yes), the process proceeds to S109. If the correction flag is off (S108: No), the correction processing unit 14 outputs the camera image to the output unit 15 without correction, and the process proceeds to S110.

In S109, the correction processing unit 14 corrects the camera image of the user to generate a corrected image. Here, three methods by which the correction processing unit 14 generates a corrected image will be described. The first and second methods are methods in which the corrected image is generated based on the camera image and the registered image. The third method is a method in which the corrected image is generated by correcting the camera image based on facial feature information prepared in advance.

The first method is to generate a corrected image by using a GAN that has been trained with images of the user in which they are well-groomed and images of the user in which they are not well-groomed. For example, the GAN can learn camera images when the correction flag is off as data of an image in which the user is well-groomed. The GAN can also learn camera images when the correction flag is on as data of an image in which the user is not well-groomed. The correction processing unit 14 can generate a corrected image in which the user is well-groomed by using the trained GAN.

The second method is to cut out a part of the user's registered image or the entire face, and replace the corresponding part of the user's captured image with the image cut out from the registered image to generate a corrected image. The second method will be explained in detail using Figure 1. In the example of Figure 1, the correction processing unit 14 cuts out images of the eyebrows, eyes, and mouth of the registered image. The correction processing unit 14 can replace the eyebrows, eyes, and mouth in the camera image with the images of the eyebrows, eyes, and mouth cut out from the registered image to generate a corrected image.

When replacing a part of the camera image with an image cut out from the registered image and combining these images, the amount of correction (here, the rate of combining) may be changed according to the degree of match between the camera image and the registered image. For example, the correction processing unit 14 may decrease the amount of correction as the degree of match increases and increase the amount of correction as the degree of match decreases.

The user may also be able to set whether to cut out and replace the entire face from the registered image, or to cut out and replace only a portion of it. If a portion of the registered image is cut out, the user may also be able to set which part of the face is to be cut out and replaced.

The third method is to correct the camera image based on facial feature information stored in advance in the registered image database 11 or the like, without using a registered image, to generate a corrected image. The facial feature information is, for example, information on the eyebrows, mouth, cheeks, skin color or brightness when makeup is applied. For example, multiple patterns of facial feature information may be prepared according to the video call scene, such as for work or personal use. This saves the user the trouble of preparing a registered image according to the video call scene.

The third method will be described in detail with reference to FIG. 6. In the example of FIG. 6, the correction processing unit 14 removes blemishes on the cheek in the camera image by filtering to remove noise. Furthermore, the correction processing unit 14 can generate a corrected image (display image) by adjusting the saturation of the eyebrows, mouth, cheeks, and skin based on facial feature information. Note that when performing filtering or saturation adjustment, the amount of correction may be changed depending on the degree of match between the camera image and the registered image.

Returning to FIG. 3, in S110, the output unit 15 outputs the image output from the correction processing unit 14. That is, when the correction flag is set to on, the output unit 15 outputs the corrected image generated by the correction processing unit 14. Also, when the correction flag is set to off, the output unit 15 outputs the uncorrected camera image of the user. The image output by the output unit 15 is transmitted to and displayed on another computer. Also, the image output by the output unit 15 is displayed on a display device.

The image processing device 1 repeats the above process for each frame until the user ends the call. When the user ends the call, the camera image is no longer acquired in S101, and the image correction process shown in FIG. 3 ends.

(Action and Effect)
In the above embodiment, the image processing device 1 acquires a camera image (captured image) and a registered image of a well-groomed person, extracts features from each image, and evaluates the degree of match. If the degree of match is equal to or greater than a predetermined threshold, the image processing device 1 determines that the person is well-groomed and disables correction processing. If the degree of match is less than a predetermined threshold, the image processing device 1 determines that the person is not well-groomed and enables correction processing. This allows the caller (user) to participate in a video call without worrying about the state of their appearance or performing a special operation to instruct image correction.

＜Other＞
The above-described embodiment merely describes an exemplary configuration of the present invention. The present invention is not limited to the above-described specific embodiment, and various modifications are possible within the scope of the technical concept thereof.

For example, in the above embodiment, the correction determination unit 13 compares the feature amounts of the camera image and the registered image and evaluates the degree of match, but this is not limited to this. The correction determination unit 13 may evaluate the difference between the feature amounts of the camera image and the registered image, and enable correction processing when the difference is equal to or greater than a predetermined threshold, and disable correction processing when the difference is less than the predetermined threshold.

Also, for example, in the above embodiment, the correction determination unit 13 disables the correction process when the degree of match between the camera image and the image of a well-groomed state is equal to or greater than a predetermined threshold, but this is not limited to this. An image of an unkempt state may be stored in the registered image database 11 as a reference image of the user. In this case, the correction determination unit 13 may disable the correction process when the degree of match between the camera image and the image of an unkempt state is less than a predetermined threshold, and enable the correction process when the degree of match is equal to or greater than the predetermined threshold.

<Appendix 1>
(1) an extraction unit (12) that extracts features of a captured image of a user and a reference image of the user;
a determination unit (13) that determines whether to enable or disable a correction process for the captured image of the user based on a result of comparing a feature amount of the captured image of the user with a feature amount of a reference image of the user;
a correction unit (14) that generates a corrected image of the captured image of the user when it is determined that the correction process is enabled;
an output unit (15) that outputs the corrected image when it is determined that the correction process is enabled, and outputs an uncorrected captured image of the user when it is determined that the correction process is disabled;
An image processing device (1).

(2) an extraction step of extracting features of a captured image of a user and a reference image of the user (S103);
a determination step of determining whether to enable or disable a correction process for the captured image of the user based on a result of comparing the feature amount of the captured image of the user with the feature amount of a reference image of the user (S104 to S107);
a correction step of generating a corrected image of the captured image of the user when it is determined that the correction process is enabled (S108, S109);
an output step of outputting the corrected image when it is determined that the correction process is to be enabled, and outputting an uncorrected captured image of the user when it is determined that the correction process is to be disabled (S110);
13. An image processing method comprising:

1: Image processing device, 10: Imaging unit, 11: Registered image database, 12: Feature extraction unit, 13: Correction determination unit, 14: Correction processing unit, 15: Output unit

Claims (14)

an extraction unit that extracts feature amounts of a captured image of a user's face and a reference image of the user's face ;
a determination unit that determines whether to enable or disable a correction process for the captured image of the user's face based on a result of comparing a feature amount of the captured image of the user's face with a feature amount of a reference image of the user's face ;
a correction unit that generates a corrected image of the captured image of the face of the user when it is determined that the correction process is enabled;
an output unit that outputs the corrected image when it is determined that the correction process is enabled, and outputs an uncorrected captured image of the user's face when it is determined that the correction process is disabled;
Equipped with
The image processing device according to claim 1, wherein the reference image of the user's face is an image of the user in an ungroomed state. The image processing apparatus according to claim 1 , wherein the feature amount is a feature amount of at least one of eyebrows and cheeks of the user. 3. The image processing apparatus according to claim 1, wherein the feature amount includes at least one of a Haar-like feature amount, a color histogram, and a color moment. 4. The image processing device according to claim 1, wherein the feature amount is calculated by an algorithm using a learning model that has been trained on images of the user in a well-groomed state and images of the user in an ungroomed state. The image processing device according to any one of claims 1 to 4, characterized in that the determination unit calculates a degree of similarity between features of a captured image of the user's face and features of a reference image of the user's face , and determines to disable the correction process if the degree of similarity is less than a predetermined threshold, and to enable the correction process if the degree of similarity is equal to or greater than a predetermined threshold. The image processing apparatus according to claim 5 , wherein the correction unit changes an amount of correction applied to the captured image of the user's face depending on the degree of coincidence. The image processing device according to claim 1 , wherein the correction unit generates the corrected image based on a captured image of the user's face and a reference image of the user's face . The image processing device according to claim 7 , wherein the correction unit generates the corrected image by a Generative Adversarial Network (GAN) that has been made to learn the captured image of the user's face and a reference image of the user's face . The image processing device according to claim 7, characterized in that the correction unit generates the corrected image by cutting out eyebrows, eyes, mouth or the entire face of the reference image of the user's face and replacing the corresponding part of the captured image of the user's face with an image cut out of the corresponding part. 7. The image processing device according to claim 1, wherein the correction unit generates the corrected image by performing a filter process for removing noise or a saturation adjustment on the captured image of the user's face based on facial feature information. The image processing device according to claim 1 , wherein the determination unit determines whether to enable or disable the correction process for each predetermined number of frames of a captured image of the user's face. The image processing device according to claim 1 , further comprising an imaging unit that captures an image of the face of the user. An extraction step of extracting feature amounts of a captured image of a user's face and a reference image of the user's face ;
a determination step of determining whether to enable or disable a correction process for the captured image of the user's face based on a result of comparing a feature amount of the captured image of the user's face with a feature amount of a reference image of the user's face ;
a correction step of generating a corrected image of the captured image of the face of the user when it is determined that the correction process is enabled;
an output step of outputting the corrected image when it is determined that the correction process is to be enabled, and outputting an uncorrected captured image of the user's face when it is determined that the correction process is to be disabled;
Including,
An image processing method, characterized in that the reference image of the user's face is an image of the user in an ungroomed state. A program for causing a computer to execute each step of the method according to claim 13.

Images