KR20090011581A - Apparatus and Method for Recognizing Eyeball in Mobile Device - Google Patents

Abstract

The present invention relates to an apparatus and method for photographing a portable terminal having a camera capable of recognizing an eyeball of a subject and photographing the subject by the movement of the eyeball. The present invention detects the eyeball of a subject and generates a photographing signal. And a process of checking the wound state of the eyeball before applying the photographing signal to the camera unit, and the process of suspending photographing if the wound state of the eyeball is greater than or equal to a set threshold. If the value is less than the value includes the step of automatically photographing the subject.

Description

APPARATUS AND METHOD FOR EYEBALL RECOGNITION PHOTOGRAPHING OF THE CAMERA IN A PORTABLE TERMINAL}

The present invention relates to an apparatus and method for photographing a camera provided in a portable terminal, and more particularly, to an apparatus and method for photographing a subject through movement of the eye after recognizing the eye of the subject to be photographed. It is about.

When taking a picture using a portable terminal equipped with a camera, a hand shake often occurs when the shutter key is pressed. Because the camera provided in the portable terminal is provided in the portable terminal in order to perform additional functions, most of the cases do not have an optimal appearance for taking a picture. Therefore, when taking a picture using the mobile terminal, the probability of obtaining a clear picture is much lower than that of a general camera.

The user using the camera wants to get a clear picture of the subject, regardless of his shooting technique. A clear picture may look different depending on the subjectivity or taste of an individual, but generally means a picture without a subject blur, a picture with a clear distinction between subjects, and the like, a picture with a shake and a picture with an indefinite distinction of a subject. Means unclear pictures.

For example, people often close their eyes when photographing. This situation often occurs because of the aesthetic tension caused by tensions in photography or the counts mentioned during shooting. Most of these photographs are judged to be indefinite, and thus, many shots are attempted again.

Accordingly, an object of the present invention is to provide a photographing apparatus and a photographing method of a mobile terminal for automatically photographing a subject after sensing an eye of a subject. .

Another object of the present invention is to check the eye-wound state of the subject before applying the shooting signal to the camera unit, and if the eye-wound state is more than the set threshold value, the shooting is suspended, and if the eye-wound state is less than the set threshold value, An apparatus and method for photographing a mobile terminal for automatically photographing a subject are provided.

In accordance with another aspect of the present invention, there is provided a method of recognizing an eyeball, the method of photographing a mobile terminal having a camera function, the method comprising: detecting an eye of a subject and generating a photographing signal; Checking the eye's winding state before applying the light to the camera unit; and if the eye's winding state is higher than or equal to a set threshold value, holding the photographing process; The process of shooting.

An eye recognition photographing apparatus according to an embodiment of the present invention for achieving the above object, an eye recognition module for detecting the eye of the subject to generate data representing the movement of the eye, the data and eye recognition imaging An automatic photographing module for applying a photographing signal to the camera unit when the setting data matches, and a camera unit for automatically photographing a subject when the photographing signal is input.

As described above, according to the eye recognition photographing apparatus and the photographing method proposed by the present invention, the user can photograph the subject by only a simple movement of the eye without a separate process of inputting a photographing key.

In addition, since there is no shaking of the portable terminal that may occur when the photographing key is input, the subject can be stably photographed.

In addition, if the subject's eye closed state is greater than or equal to the set threshold when taking a picture, the shooting can be suspended, thereby reducing the inconvenience of taking, deleting, and re-taking a picture in which the subject's eye is wound more than half. There is this.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, only the parts necessary for understanding the operation according to the embodiment of the present invention will be described, it should be noted that the description of other parts will be omitted so as not to distract from the gist of the present invention.

The terms or words used in the specification and claims described below should not be construed as being limited to the ordinary or dictionary meanings, and the inventors are appropriate to the concept of terms in order to explain their invention in the best way. It should be interpreted as meanings and concepts in accordance with the technical spirit of the present invention based on the principle that it can be defined. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

Prior to the detailed description of the present invention, for convenience of description, a portable terminal having a camera function according to an embodiment of the present invention will be described below using a mobile communication terminal as an example, but the present invention is not limited thereto. That is, the portable terminal according to the embodiment of the present invention is a terminal for providing convenience to a user. Preferably, the mobile terminal includes a camera function, a digital broadcasting terminal, an MP3 (MPEG Audio Layer-3). ) Player, Portable Multimedia Player (PMP), Digital Camera, Navigation Terminal, Personal Digital Assistant (PDA), Smart Phone, IMT-2000 (International Mobile Telecommunication 2000) Applicable to all information and communication devices such as terminals, wideband code division multiple access (WCDMA) terminals, universal mobile telecommunication service (UMTS) terminals, notebooks and personal computers, and applications thereof Will be self explanatory.

1 is a diagram illustrating an internal configuration of a portable terminal according to an exemplary embodiment of the present invention, and FIG. 2 is a screen example illustrating an eye recognition photographing setting mode of a portable terminal according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the configuration of a mobile terminal according to an exemplary embodiment of the present invention includes a wireless communication unit 110, an audio processor 120, an input unit 130, a storage unit 140, a camera unit 150, The signal processor 160, the image processor 170, the display unit 180, and the controller 190 are included.

The wireless communication unit 110 is responsible for a series of communication for wireless transmission and reception of signals such as user data, for example, messages and voice to other external portable terminals. The wireless communication unit 110 includes an RF (Radio Frequency) transmitter for upconverting and amplifying a frequency of a signal to be transmitted, and an RF receiver for low noise amplifying and downconverting a received signal. In addition, the modulated signal received from the controller 190 is converted into an intermediate frequency and then converted into a high frequency (RF) signal and transmitted to the base station through the antenna ANT, and receives a high frequency signal from the base station through the antenna ANT This is sequentially converted to the intermediate frequency and the baseband and provided to the controller 190.

The audio processor 120 plays a function of reproducing an audio signal output from the controller 190 or transmitting an audio signal such as a voice input from a microphone (MIC) to the controller 190. That is, the analog voice signal received from the microphone (MIC) is converted into a digital signal and transmitted to the controller 190, and the digital signal of the opposite voice from the controller 190 is converted into an audible analog signal speaker (SPK) It serves to send.

The input unit 130 includes a plurality of input keys and function keys for receiving numeric or character information and setting various functions. The function keys may include direction keys, side keys, shortcut keys, and the like, which are set to perform a specific function. In addition, the input unit 130 transmits a key signal input in relation to user setting and function control of the portable terminal to the controller 190.

In particular, the input unit 130 preferably includes a shortcut key for camera shooting. That is, the portable terminal may include a key corresponding to the shutter role of the general camera on one side of the portable terminal to facilitate camera shooting.

The storage 140 stores an application program required for operating a function according to an embodiment of the present invention, and stores downloaded content and user data generated by a user. In addition, one or more buffers may be temporarily stored to store user data (eg, photographed pictures) generated while executing an application program according to a camera function.

The storage unit 140 may include a program area and a data area. The program area stores an operating system (OS) for booting the portable terminal, an application program required for a text message, and the like. In addition, when activating each function according to a user's request, each function is provided through corresponding application programs under the control of the controller 190. The program area stores overall programs for the eyeball recognition module 191 that detects the eyeball of the subject and generates a photographing signal in the camera mode. The data area is an area in which user data generated by using a mobile terminal is stored, and images captured by the camera unit 150 are stored.

The camera unit 150 captures an image of a subject photographed through a lens and includes a camera sensor for converting an optical signal into an electrical signal. The camera sensor may be a CCD (Charge Coupled Device) sensor.

In particular, the camera unit 150 according to an embodiment of the present invention performs the function of transmitting the electrical signal to the eye recognition module 191 of the controller 190 to represent the image of the subject.

The signal processor 160 includes a signal processor for converting an electrical signal output from the camera unit 150 into a digital signal. The signal processing apparatus is typically implemented with a DSP (Digital Signal Processor), but is not limited thereto.

The image processor 170 generates image data for displaying the digital signal output from the signal processor 160, and transmits the image data to the display unit 180. In more detail, the image processor 170 processes the digital signal output from the signal processor 160 in units of frames, and then edits and outputs the digital signal according to the display characteristics and size of the display unit 180. do. The image processor 170 includes an image codec and performs a function of compressing image data displayed on the display unit 180 in a set manner or restoring the compressed image data to original image data. The image codec may be a JPEG codec, an MPEG codec, a wavelet codec, or the like.

The display unit 180 visually provides a menu of the portable terminal, user data input by the user, function setting information, and various information to the user. The display unit 180 may be formed of a liquid crystal display (LCD). In this case, the display unit 180 may include a control device for controlling the liquid crystal display, a video memory capable of storing image data, a liquid crystal display device, and the like. If the LCD is formed in the form of a touch screen, the display unit 180 may perform some or all of the functions of the input unit 130.

In particular, the display unit 180 may provide an image of a subject currently projected by a camera provided in the portable terminal as a preview screen. The preview screen function is a function of previewing an image of a subject incident through a lens before the camera actually photographs the image. In addition, the display unit 180 may display before storing the image photographed by the camera, and may display a confirmation message for the user's storage and deletion.

The controller 190 controls the overall operation of the portable terminal and the signal flow between the internal blocks of the portable terminal. That is, the controller 190 includes a wireless communication unit 110, an audio processing unit 120, an input unit 130, a storage unit 140, a camera unit 150, a signal processing unit 160, an image processing unit 170, and a display unit. It controls the signal flow between each configuration, such as (180).

In addition, the controller 190 performs each function of the mobile terminal according to an input signal input from the input unit 130, and displays information such as a current state and a user menu according to the execution of the function through the display unit 180. .

In particular, the controller 190 according to an embodiment of the present invention, the eye recognition module 191 for detecting the eye of the subject photographed through the camera unit 150 and the automatic shooting for generating a signal for automatically photographing the subject Module 193 may be included.

In the present invention, the eye recognition module 191 and the automatic photographing module 193 will be described separately. However, according to circumstances, the eye recognition module 191 and the automatic photographing module 193 may be integrated into the eye recognition photographing module and may be provided in the controller 190.

The eye recognition module 191 detects an eyeball of a subject and generates data representing motion of the eyeball.

As a method of recognizing the eyeball of the subject, a method of using the information of the light of the eye which emits distinct light from the face, a method of detecting the eye by detecting the iris of the eye, a method of using the color information of the eye contrasted with the face color, etc. Various methods can be used.

The eyeball recognition module 191 performs a function of detecting the eyeball of the subject. The eye recognition module 191 converts an electrical signal of the subject applied from the camera unit 150 into an image by converting the image into an image, and detects the eye from the image. Thereafter, the detected eye movement is recognized, data representing the eye movement is generated, and the data is transmitted to the automatic photographing module 193.

In addition, the eye recognition module 191 detects the eyeball of the subject, and generates a continuous captured picture of the eyeball. Thereafter, the eyeball recognition module 191 transfers the continuous captured picture of the eyeball to the automatic photographing module 193, and the automatic photographing module 193 may grasp the closed state of the eyeball through the captured picture. have.

The automatic photographing module 193 receives data from the eyeball recognition module 191 and generates a signal for automatically photographing the subject.

The automatic photographing module 193 receiving the data analyzes the data to determine what movement of the eyeball the data represents. As a result of the determination, if the data matches the eye recognition photographing setting data, the automatic photographing module 193 outputs a photographing guide message.

Next, the automatic photographing module 193 may analyze the continuous captured photograph of the eyeball and determine the eyeball winding state before transmitting the photographing signal to the camera unit 150. At this time, if the closed state of the eyeball is equal to or greater than the set threshold value, the automatic photographing module 193 outputs a photographing suspension message to stop photographing. Meanwhile, when the closed state of the eyeball is less than or equal to a set threshold value, the automatic photographing module 193 transfers the photographing signal to the camera unit 150 to photograph the subject.

The threshold value means a numerical value of the eye half wound. The automatic photographing module 193 may set a threshold by analyzing white and black color numerical values from successive captured photographs of the eye. That is, the automatic photographing module 193 identifies the captured picture of the eyeball including the most white and the black color among the captured pictures, and analyzes the white and black color values in the identified eyeball picture. Thereafter, the automatic photographing module 193 sets half of the numerical value as a threshold.

The eye recognition shooting setting data may be set in the eye recognition shooting setting mode before capturing a subject. 2 is a screen example illustrating an eye recognition shooting setting mode of a mobile terminal according to an exemplary embodiment of the present invention.

Referring to FIG. 2, a menu for setting the eye recognition photographing setting includes menu items such as “1. right eye blink for 2 seconds”, “2. left eye blink for 2 seconds”, and “3. binocular blink for 2 seconds”. May be included. Before photographing a subject, the controller 190 detects a selection signal of one of the menus from a user and stores the selection signal in the storage 140. When photographing the subject in the eye recognition photographing mode, when the data corresponding to the eye recognition photographing setting data that the selection signal means is input, the subject may be photographed.

The selection signal maintains the same value until the user changes the selection signal, and maintains a default value unless otherwise selected.

In addition, the menu for setting the eye recognition photographing is not limited to the menu illustrated in FIG. 2, but may be variously expanded according to an algorithm included in the controller 190.

In the above, the configuration of the mobile terminal according to an embodiment of the present invention has been described. Hereinafter, a method of photographing a portable terminal according to an exemplary embodiment of the present invention will be described.

3A is a flowchart illustrating a photographing process of a mobile terminal according to an embodiment of the present invention.

Referring to FIG. 3A, the controller 190 is performing the standby mode in step S401. If there is a key input for selecting a camera mode according to a user's selection during the standby mode, the controller 190 detects it in step S403 and displays a shooting menu.

In the shooting menu, when the user selects the self-taking mode in step S405, the controller 190 detects this and displays the self-taking menu.

Here, the self-photographing mode may be represented as shown in FIG. 3B. 3B is a screen example illustrating a self-taking setting mode of a mobile terminal according to an exemplary embodiment of the present invention. As shown in FIG. 3B, the self-shooting mode setting screen displays the normal mode and the eye recognition mode, and allows the user to select through the corresponding radio button.

The normal mode is a method in which the controller 190 photographs a subject after receiving a photographing signal through the input unit 130. This method is a method of photographing the subject manually. When the shutter is mounted on the portable terminal, a physical force must be applied to photograph the subject.

The eye recognition mode is a method in which the controller 190 detects the eye of the subject and automatically photographs the subject without an external shutter key input process.

In the eye recognition mode according to an embodiment of the present disclosure, the controller 190 of the portable terminal recognizes the eyeball of the subject to be photographed, and generates a photographing signal according to the movement of the eyeball to automatically photograph the subject.

The setting of the eye recognition mode may be set before entering the shooting mode, or may be set by calling the self-shooting setting mode screen of FIG. 3B while performing the preview mode in the self-camera mode.

Next, when the user selects the eye recognition shooting mode in step S407, the control unit 190 detects this and performs eye recognition shooting. However, if the user does not select the eye recognition photographing mode, the controller 190 captures the subject in the normal mode to receive the photographing signal through the input unit 130 in step S409.

In order to perform the eye recognition photographing mode, the controller 190 executes the preview mode in step S411. In this case, the controller 190 displays an image of the subject captured by the camera unit 150 on the screen of the portable terminal. At this time, the image processor 180 captures an image of the subject and transmits the image to the eye recognition module 191. The eye recognition module 191 receiving the image detects the eyeball of the subject from the image in step S413 and generates data representing the detected eye movement. Thereafter, the eye recognition module 191 transmits the data to the automatic photographing module 193, and the automatic photographing module 193 applies the photographing signal to the camera unit 150 in step S415. The camera unit 150 receiving the photographing signal automatically photographs the subject.

In operation S417, the controller 190 outputs a message to the display unit 180 asking whether to store the photographed picture. If the user selects a key for storing the photographed picture, the controller 190 stores the photographed picture after receiving a file name of the photographed picture from the user in step S419.

On the other hand, if the controller 190 receives a key that does not store the photographed picture in step S417, the controller 190 deletes the photographed picture in step S421 and proceeds to step S423.

In step S423, the controller 190 checks whether shooting is terminated. If the shooting continues, the controller 190 proceeds to step S409 to execute the preview mode. Otherwise, the controller 190 terminates the camera mode.

4 is a flowchart illustrating an eyeball recognition process illustrated in FIG. 3A.

In operation S501, the eyeball recognition module 191 recognizes a face from an image of the subject applied by the image processor 180. There are many ways to recognize a face, but commonly used PCA (Pricipal Component Analysis), Fisher Discriminarnt Analysis (FDA), and Independent Component Analysis (ICA) are commonly used.

The PCA analysis is a method of recognizing a face by finding a vector that can best express a face in the entire image space. This method is widely known as a practical way to reduce the dimensions of different spaces.

The FDA assay is an assay made to express the local features of the face well from the background. This analysis method is a method of efficiently recognizing a face when the storage unit 140 of the portable terminal stores a large number of various images having different lighting and facial expressions for each subject.

The ICA analysis method is an analysis method for recognizing a face by extracting a new feature that can highlight a face feature from a screen when it is difficult to distinguish a background and a face.

The eyeball recognition module 191 detects the eyeball from the face recognized in the step S501 in step S503. There are various methods for detecting the eye, such as using information on the light of the eye that emits distinct light from the face, how to detect the eye by detecting the iris of the eye, and using the color information of the eye as opposed to the face color. Can be used. In the present invention, a method of using eye color information contrasted with the face color will be used. Particularly, in the present invention, a method of recognizing the eye from the captured picture is used by using an algorithm for analyzing the captured picture of the face. For example, the eye recognition module 191 may detect the eye using an algorithm for extracting an image having a shape and color similar to a human eye from the captured photograph.

Next, the eyeball recognition module 191 recognizes the movement of the eyeball in step S505. That is, the eyeball recognition module 191 may recognize how many times the eyeballs flicker, which eyeballs flicker, and the like, by using an algorithm for analyzing a series of captured photographs of the eyeballs.

Thereafter, the eyeball recognition module 191 generates data representing motion of the eyeball in step S509. For example, when the controller 190 recognizes that the right eye of the subject blinks once, the binary signal 01, and recognizes that the left eye of the subject blinks once, the binary signal 10 and both eyes of the subject blink once. If it is recognized that the digital signal data such as binary signal 11 can be generated.

As described above, the eyeball recognition module 191 generates the data by detecting the eyeball of the subject, and then applies the data to the automatic photographing module 193.

5 is a flowchart illustrating an automatic photographing process illustrated in FIG. 3A.

Referring to FIG. 5, the automatic photographing module 193 detects data received from the eyeball recognition module 191 in step S601, and determines whether the data matches the eye recognition photographing setting data in step S603. As a result of the determination, when the data and the eye recognition photographing setting data match, the automatic photographing module 193 outputs a photographing guide message in step S605.

The photographing guide message may include at least one of a lamp, a vibration, a voice message, and a video message. The voice message may include a guide voice such as "take a picture" output by the automatic shooting module 193 through the audio processor 120. In addition, the video message may include a guide image, such as "take a picture" that is displayed on the display unit 180 by the automatic photographing module 193.

If the data does not match the eye recognition photographing setting data, the automatic photographing module 193 outputs an error message in step S607 and then terminates the automatic photographing module 193.

The error message may include at least one of a lamp, a vibration, a voice message, and a video message. The voice message may include a guide voice, such as, "Inconsistent with eye recognition recording. Please input a shooting signal again" output by the auto imaging module 193 through the audio processor 120. In addition, the video message may include a guide image, such as "inconsistent with eye recognition photographing. Please input a photographing signal" indicated by the automatic photographing module 193 on the display unit 180.

In operation S609, the automatic photographing module 193 outputs the photographing guide message and checks the eyeball winding state of the subject before applying a photographing signal to the camera unit 150. To this end, the automatic photographing module 193 analyzes a continuous captured picture of the eyeball inputted through the eyeball recognition module 191. As a result of the analysis, if the eye of the subject in the image is greater than the set threshold, the automatic photographing module 193 outputs a shooting suspension message in step S613, and returns to step S609 to check the eye state of the subject again, and then, in the next step Proceed.

The shooting suspension message may include at least one of a lamp, a vibration, a voice message, and a video message. For example, the voice message may include a guide voice, such as “open your eyes,” which is output by the auto imaging module 193 through the audio processor 120. In addition, the video message may include a guide image, such as "open your eyes," which is displayed on the display unit 180 by the automatic photographing module 193.

Meanwhile, when the eyeball winding state of the subject is less than or equal to a predetermined threshold value, the automatic photographing module 193 applies a photographing signal to the camera unit 150 in step S615 and automatically photographs the subject in step S617.

As described above, in the detailed description of the present invention has been described with respect to specific embodiments, various modifications are possible without departing from the scope of the invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims below, but also by the equivalents of the claims.

1 is a view showing the internal configuration of a mobile terminal according to an embodiment of the present invention.

2 is a screen example illustrating an eye recognition shooting setting mode of a mobile terminal according to an exemplary embodiment of the present invention.

3A is a flowchart illustrating a photographing process of a mobile terminal according to an embodiment of the present invention.

3B is a screen example illustrating a self-taking setting mode of a mobile terminal according to an embodiment of the present invention.

4 is a flowchart illustrating an eyeball recognition process illustrated in FIG. 3A.

FIG. 5 is a flowchart for explaining an automatic photographing process shown in FIG. 3A. FIG.

Claims (17)

In the photographing method of a mobile terminal having a camera function, Detecting the eyeball of the subject and generating data representing the eyeball movement; and And photographing the subject when the data and the eye recognition photographing setting data coincide with each other. The method of claim 1, The shooting process And outputting an error message if the data and the eye recognition photograph do not coincide with each other. The method of claim 2, The error message is And at least one of a lamp, a vibration, a voice message, and a video message. The method of claim 1, The shooting process And a photographing guide message is output before the photographing signal is applied to the camera unit, and the eyeball winding state of the subject is checked. The method of claim 4, wherein The shooting guide message is And at least one of a lamp, a vibration, a voice message, and a video message. The method of claim 4, wherein The shooting process And if the eye closed state of the subject is greater than or equal to a set threshold, photographing is suspended, and if the eye closed state is less than or equal to a set threshold, photographing the subject. The method of claim 6, The eye closed state of the subject The photographing method of the mobile terminal, characterized in that for analyzing and confirming the continuous captured picture of the eyeball. The method of claim 6, The threshold is And a white and black color numerical value is analyzed from the consecutive captured photographs of the eyeball. In the photographing method of a mobile terminal having a camera function, Detecting the eyeball of the subject and generating a shooting signal; Checking the eye state of the eyeball before applying the photographing signal to the camera unit; And automatically capturing or suspending the subject according to the eye state of the eyeball. In the photographing apparatus of a mobile terminal having a camera function, A camera unit for automatically photographing a subject when a shooting signal is inputted; and And a controller configured to determine eyeball movement from an image of a subject transmitted from the camera unit and to generate the photographing signal according to the eyeball movement. The method of claim 10, The control unit An eye recognition module for detecting an eye of a subject and generating data representing motion of the eye; And an automatic photographing module configured to apply a photographing signal to the camera unit when the data and the eye recognition photographing setting data coincide with each other. The method of claim 11, The automatic shooting module And outputting an error message if the data and the eye recognition photograph do not match. The method of claim 12, The error message is And at least one of a lamp, a vibration, a voice message, and a video message. The method of claim 11, The automatic shooting module And a photographing guide message is output before the photographing signal is applied to the camera unit. The method of claim 14, The shooting guide message is And at least one of a lamp, a vibration, a voice message, and a video message. The method of claim 14, The automatic shooting module And checking the eye state of the eyeball after outputting the photographing guide message, and shooting is suspended if the eyeball state of the eyeball is equal to or greater than a set threshold value. The method of claim 16, The automatic shooting module And analyzing the successive captured photographs of the eyeballs to determine the closed state of the eyeballs.

Images