JP2007310815A - Mobile terminal device - Google Patents

Abstract

Provided is a portable terminal device that can be operated by easier means using the movement of an eyeball.
The number of eyeball movements is detected by a camera built in the mobile terminal device, and an operation or an input to the mobile terminal device is performed based on the detected number of times. More preferably, the number and direction of eyeball movements are detected, and the operation for the mobile terminal device is determined based on the number and direction, and the processing for the determined operation is executed, so that an intended number or character can be easily obtained. Can be selected.
[Selection] Figure 2

Description

  The present invention relates to a mobile terminal device such as a mobile phone, and more particularly to a mobile terminal device that can operate a mobile terminal by detecting the movement of an eyeball using a built-in camera.

  The operation of a mobile terminal device such as a mobile phone that is currently popular is performed by an operation such as pressing a key button or the like arranged on the mobile phone with a finger. However, when the fingertip is inconvenient due to a failure or the like, the key operation is difficult.

  On the other hand, since most recent mobile phones have a built-in camera, a method of detecting the movement of the eyeball using the camera and operating the mobile phone using the movement of the eyeball is provided.

For example, Patent Document 1 below detects the movement of the pupil position to detect the movement direction of the user's line of sight, moves the pointing device in the detected movement direction of the user's line of sight, and performs the movement toward the pointing device. A mobile phone with a line-of-sight input function for detecting a decision making is disclosed. Patent Document 2 below discloses a mobile terminal device that captures an image of a user's eyeball, detects a user's line-of-sight direction from the captured image, and identifies a character to be input from the line-of-sight direction. Further, Patent Document 3 below also discloses an electronic device having a line-of-sight detection means.
JP 2004-21870 A JP 2004-194207 A JP 2002-132517 A

  However, it is relatively difficult to move the eyeball to adjust the line-of-sight direction to the intended position, the input accuracy is low, and the operability is poor.

  In particular, when displaying a list of characters, numbers, symbols, etc., and selecting characters from the line-of-sight direction, it is difficult to align the line of sight with the intended display position of the character because the information to be displayed is large and the characters are also small. Business.

  Therefore, an object of the present invention is to provide a portable terminal device that can be operated with easier means using the movement of the eyeball.

  In order to achieve the above object, a first configuration of a mobile terminal device of the present invention includes a camera that images a user's eyeball, a detection unit that detects the motion of the captured eyeball, and the number of times the detected eyeball has moved And a control unit that executes processing for a predetermined operation corresponding to the number of times the eyeball moves.

  According to a second configuration of the mobile terminal device of the present invention, in the first configuration, the detection unit detects a direction of movement of the eyeball, and the counter counts the number of times of movement of the eyeball for each direction of movement of the eyeball. The control unit executes a process corresponding to a predetermined operation corresponding to the direction and the number of movements of the eyeball.

  According to a third configuration of the mobile terminal of the present invention, in the second configuration, the control unit executes a process for an operation of selecting and inputting a number or a character in accordance with the direction and the number of movements of the eyeball. It is characterized by doing.

  According to the present invention, by enabling an operation according to the number of times of movement of the eyeball, operability and input accuracy are improved, and even a user with a fingertip can easily operate the mobile terminal device. Will be able to.

  Depending on the number and direction of eyeball movements, it is possible to easily select an intended number or character, and it is easy to input a number or character without using a key button.

  Embodiments of the present invention will be described below with reference to the drawings. However, such an embodiment does not limit the technical scope of the present invention.

  In the embodiment of the present invention, the number of movements (and direction) of the eyeball is detected by a camera built in a portable terminal device, for example, a cellular phone, and an operation or input to the portable terminal device is performed based on the number of times (and direction). I do.

  FIG. 1 is a diagram illustrating a block configuration example of a mobile terminal device according to an embodiment of the present invention. FIG. 1 shows a configuration example of a mobile phone as an example. The control unit 10 controls all functions such as a call function, an e-mail function, an Internet connection function, and a camera function of the mobile phone 1. It consists of a memory that stores programs to be executed. The eyeball motion detection process and the operation process associated with the detection in the embodiment of the present invention are executed as software processing by the control unit 10.

  The key button 11 is an input means for inputting numbers and characters, the speaker 12 is an audio output means, the microphone 13 is an audio input means, and the display 14 is a display means. The present invention provides a technique for inputting numbers and characters according to the number of eyeball movements without using the key buttons 11.

  The radio signal processing unit 15 processes a signal transmitted to and received from the base station via the antenna 16 and handles a signal related to a call, electronic mail, or Internet connection. The power source 18 supplies necessary power to the components of the mobile phone 1.

  The camera 17 is configured to include an image sensor such as a CCD element or a CMOS element, and the captured image data can be stored in a memory in the control unit 10. The control unit 10 detects the movement of the eyeball based on the captured image data, and executes an operation corresponding to the detected number of movements of the eyeball.

  FIG. 2 is a diagram for explaining eyeball motion detection. Four patterns are detected in which the black eyes of the eyeball are moved upward, downward, rightward, and leftward from the center position of the eyeball. In the detection method, an image of an eyeball in which the black eye is at the center position of the eyeball is captured in advance by the built-in camera 17 of the mobile terminal device 1, and the image is stored in the memory in the control unit 10. Is compared with the black eye position of the eyeball image captured by the camera 17, and when the black eye position is more than a predetermined value, the movement of the eyeball is detected and its direction is specified. .

  The position of the black eye is determined by, for example, determining the positions of the upper and lower left and right eyes of the black eye from the difference between the luminance data of the black eye data and the brightness data of other parts such as the white eye, Is detected. The position of the center of the black eye may be obtained from each end position of the black eye, and the positions of the center of the black eye may be compared.

  FIG. 3 is a diagram illustrating an operation process based on the number of eyeball movements according to the first embodiment of the present invention. In the first embodiment, an operation for determining the operation mode of the mobile phone is performed based on the direction and number of movements of the eyeball.

  When the camera 17 captures the user's eyeball, when the vertical eyeball motion detector B1 detects the vertical eyeball motion, the upward eyeball motion detector B3 detects the upward eyeball motion in the vertical direction, and the downward direction. The eyeball motion detection unit B4 detects the downward eyeball motion in the vertical direction. When the lateral eye movement detection unit B2 detects the lateral eye movement, the right eye movement detection unit B5 detects the right eye movement in the horizontal direction, and the left eye movement detection unit B4 The left eye movement in the direction is detected.

  The number of upward eyeball movements detected by the upward eyeball motion detection unit B3 is counted and stored by the counter C1. Similarly, the counters C2, C3, and C4 respectively count the number of downward eye movements detected by the downward eye movement detection unit B4, the number of right eye movements detected by the right eye movement detection unit B5, The number of left eye movements detected by the left eye movement detection unit B6 is counted and stored. Each eyeball motion detection unit B <b> 1 to B <b> 6, counters C <b> 1 to C <b> 4 and an operation mode determination unit D <b> 1 described later are functions of the control unit 10.

  The operation mode determination unit D1 reads out the number of times stored in the counters C1 to C4 and displays the number of times in each direction on the display 14 as illustrated. A predetermined operation mode corresponding to the number of times in each direction may be displayed. Then, the operation mode determination unit D1 determines the number of times in each direction when a fixed time has elapsed without detecting eye movements, determines a predetermined operation corresponding to each number of times, and performs processing for the operation. Execute. By executing the determined operation, the counter count is reset.

  For example, the operation is an operation for determining the operation mode of the mobile phone, and the phone mode for making a call by the mobile phone, the e-mail mode for sending and receiving e-mails, and the camera mode for taking a picture with the built-in camera When there are a plurality of operation modes such as the Internet mode for connecting to the Internet, when a movement of the eyeball in the upward direction, the downward direction, the right direction, and the left direction is detected once, the telephone mode and the e-mail respectively. Set to mode, camera mode, and Internet mode. By a combination of a plurality of times in each direction and a number of times in a plurality of directions, not only the operation mode but also a finer operation may be defined.

  FIG. 4 is a diagram illustrating an operation process based on the number of eyeball movements according to the second embodiment of the present invention. In the second embodiment, an operation for inputting a telephone number and making a call is performed in the telephone mode based on the direction and number of movements of the eyeball. The configurations of the eyeball motion detection units B1 to B6 and the counters C1 to C4 are the same as those in the first embodiment. Each eyeball motion detection unit B <b> 1 to B <b> 6, counters C <b> 1 to C <b> 4 and a transmission processing unit D <b> 2 described later are functions of the control unit 10.

  The outgoing call processing unit D2 reads the number of times stored in the counters C1 to C4, selects a number corresponding to the number of times in each direction, and displays the selected number on the display 14 as shown in the figure. The call processing to the telephone number composed of the entered numbers is executed.

  The display 14 displays operations assigned to the eye movements in the vertical and horizontal directions and information corresponding to the operations. For example, the movement of the eyeball in the left-right direction is a number selection operation corresponding to a telephone number described later, and “number” is displayed at the display position in the left-right direction of the display 14. The upward movement of the eyeball is an operation of confirming the selected number, and “confirm” is displayed at the upward display position of the display 14. Further, the downward movement of the eyeball is an operation of making a call to a telephone number with a fixed number, and “call” is displayed at the downward display position of the display 14.

  FIG. 5 is a diagram for explaining a method of selecting a number by the movement of the eyeball. As shown in FIG. 5A, numbers 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 are arranged in order from the left, and the initial position is assumed to be 0. In the number selection operation, the eyeball is moved a predetermined number of times in the left-right direction to move to a desired number position and select a number.

  As illustrated in FIG. 5B, for example, when it is desired to make a call to the telephone number “09057346288”, each number is selected as follows.

  Since the initial position is 0, first, in this state, 0 is fixed by moving the eyeball upward. Subsequently, since 9 is selected by moving the eyeball once in the left direction (1), 9 is determined by moving the eyeball upward in this state. Next, by moving the eyeball once in the right direction, 0 is selected again (2). In this state, by moving the eyeball in the upward direction, 0 is determined.

  Next, since 5 is selected by continuously performing the eyeball movement in the left direction five times (3), 5 is determined by moving the eyeball upward in this state. Then, 7 is selected by continuously moving the eyeball in the right direction twice from there (4), and 7 is confirmed by moving the eyeball in the upward direction in this state. Subsequently, in order to move from the position 7 to the position 3, the eyeball moves in the left direction four times in succession. As a result, 3 is selected (5), and 3 is fixed by moving the eyeball upward in this state.

  Next, since 4 is selected by moving the eyeball once in the right direction (6), the eyeball is moved upward in this state to confirm 4. Further, when the eyeball is moved twice in the right direction, 6 is selected (7). In this state, the eyeball is moved upward to confirm 6. Subsequently, by moving the eyeball four times in the left direction, the eyeball moves from the position 6 to the position 2 and 2 is selected (8). In this state, the eyeball is moved upward and 2 To confirm. Then, the eyeball is moved 6 times to the right, 8 is selected (9), 8 is confirmed by moving the eyeball upward, and without moving the eyeball in the left-right direction, 8 is continuously determined by moving the eyeball upward.

  After confirming the telephone number, the eyeball is moved once downward. As a result, a call operation is performed, and the mobile phone performs a call operation for a phone number composed of fixed numbers.

  In this way, it is possible to select a number according to the direction and number of movements of the eyeball and perform a call operation, so it is easy to select a number, the input accuracy improves, and even if the key button cannot be pressed, A desired telephone number can be easily called only by the movement of the eyeball.

  FIG. 6 is a diagram for explaining an operation process based on the number of eyeball movements according to the third embodiment of the present invention. In the third embodiment, an operation for inputting characters is performed in the e-mail mode based on the direction and number of movements of the eyeball. The configurations of the eyeball motion detection units B1 to B6 and the counters C1 to C4 are the same as those in the first embodiment. Each eyeball motion detection unit B1 to B6, counters C1 to C4, and a character selection unit D3 described later are functions of the control unit 10.

  The character selection unit D3 reads the number of times stored in the counters C1 to C4, selects a character corresponding to the number of times in each direction, and displays the selected character on the display 14 as illustrated.

  The display 14 displays operations assigned to the eye movements in the vertical and horizontal directions and information corresponding to the operations. For example, as described later, the movement of the eyeball in the left direction and the downward direction is an operation of selecting a character by specifying a row and a column. “Row” and “Column” are displayed. The upward movement of the eyeball is an operation for confirming the selected number, and “confirm” is displayed at the upward display position of the display 14. Further, the movement of the eyeball in the right direction is an operation of deleting the confirmed number, and “Delete” is displayed at the right display position of the display 14.

  FIG. 7 is a diagram for explaining a method of inputting characters by the movement of the eyeball. As shown in FIG. 7A, Japanese 50 sounds are arranged in a matrix, and characters can be selected by specifying row numbers and column numbers. It is assumed that the initial position is the character “A” in which both the row number and the column number are 0. In the character selection operation, the eyeball is moved a predetermined number of times in the left direction and downward direction to move to a desired character position and select a character.

  As illustrated in FIG. 7B, for example, when a character “Fujitsu Utarou” is to be input, each character is selected as follows.

  In order to move the initial position from “A” to “F”, the eyeball is rotated five times to the left and then moved downward twice to reach the “F” position. By moving the eyeball upward, “F” is confirmed.

  In the second embodiment, when moving to the position of the desired number, confirming the number, and moving to the next number position, the position of the next number is determined based on the position of the confirmed number. In the third embodiment, the reference position returns to the “A” position each time a character is determined. Therefore, when the next character is selected by performing the “F” determination operation, the character is moved again from the initial position “A” to the next character.

  In order to move from the initial position “a” to the next character “shi”, the eyeball is rotated twice in the left direction and further moved once in the downward direction to reach the position “shi”. In this state, by moving the eyeball upward, “Shi” is confirmed.

  As shown in the matrix of FIG. 5 (a), the next cloud point “” ”is placed under“ wa ”, so the eyeball is moved nine times in the left direction and one turn in the downward direction. By moving the eyeball in the upward direction, the muddy point “” is determined, and thereby the character “ji” is input.

  Similarly, the character “T” is input by moving the eyeball three times in the left direction and one turn in the downward direction, and moving the eyeball in the upward direction in this state. The character “U” is input by moving the eyeball downward two times and moving the eyeball upward in this state. The character “ta” is input by moving the eyeball three times in the left direction and moving the eyeball upward in this state. The character “RO” is input by moving the eyeball eight times in the left direction and four times in the downward direction, and moving the eyeball in the upward direction in this state. The character “U” is input by moving the eyeball downward two times and moving the eyeball upward in this state.

  When an input character is mistaken, one character determined most recently is deleted by moving the eyeball rightward once. By moving the eyeball a plurality of times continuously in the right direction, the number of eyeball movements is deleted in the order of the most recently determined character.

  After the character input is thus completed, the mail transmission operation may be performed by continuously moving the eyeball twice in the upward direction.

  Thus, after a character is input, an e-mail transmission operation may be performed by moving the eyeball twice continuously in the upward direction. As a result, the mobile phone performs an e-mail transmission operation composed of the input characters.

  In this way, characters can be selected and input according to the direction and number of movements of the eyeball, so that the selection of characters is easy, the input accuracy is improved, and even when the key button cannot be pressed, the movement of the eyeball Character input can be made easily with just this.

  In the embodiment of the present invention, a mobile phone will be described as an example. However, the present invention can be applied to a mobile terminal device such as a PDA or a notebook personal computer in addition to the mobile phone. Furthermore, the present invention is not limited to a portable terminal device, and can also be applied to an electronic device (including a portable terminal device) such as a stationary desktop personal computer incorporating a camera. Furthermore, the camera is not limited to a built-in camera, and may be configured to be connected externally. In this case, the electronic apparatus includes a receiving unit that receives image data of the eyeball captured by the camera. Furthermore, the camera may be provided with the configuration of the present invention that executes processing according to the number of eyeball movements.

(Appendix 1)
A camera that images the user's eyeball;
A detection unit for detecting the movement of the imaged eyeball;
A counter that counts the number of detected eye movements;
And a control unit that executes a process for a predetermined operation corresponding to the number of movements of the eyeball.

(Appendix 2)
In Appendix 1,
The detection unit detects the direction of movement of the eyeball,
The counter counts the number of eye movements for each direction of eye movement,
The said control part performs the process corresponding to operation predetermined according to the direction and frequency | count of the movement of an eyeball, The portable terminal device characterized by the above-mentioned.

(Appendix 3)
In Appendix 2,
The said control part performs the process with respect to operation which selects and inputs a number or a character according to the direction and frequency | count of a motion of an eyeball.

(Appendix 4)
A camera that images the user's eyeball;
A detection unit for detecting the movement of the imaged eyeball;
A counter that counts the number of detected eye movements;
An electronic device comprising: a control unit that executes processing for a predetermined operation corresponding to the number of movements of the eyeball.

(Appendix 5)
A receiving unit that receives image data of a user's eye image captured by the camera;
A detection unit for detecting the movement of the imaged eyeball;
A counter that counts the number of detected eye movements;
An electronic device comprising: a control unit that executes processing for a predetermined operation corresponding to the number of times of movement of the eyeball.

(Appendix 6)
An imaging unit that images the user's eyeball;
A detection unit for detecting the movement of the imaged eyeball;
A counter that counts the number of detected eye movements;
And a control unit that executes a process for a predetermined operation corresponding to the number of movements of the eyeball.

It is a figure which shows the block structural example of the portable terminal device in embodiment of this invention. It is a figure explaining the motion detection of an eyeball. It is a figure explaining the operation processing by the frequency | count of the movement of the eyeball in the 1st example of an embodiment of this invention. It is a figure explaining the operation processing by the frequency | count of the movement of the eyeball in the 2nd embodiment of this invention. It is a figure explaining the method of selecting a number by the movement of an eyeball. It is a figure explaining the operation processing by the frequency | count of the movement of the eyeball in the 3rd embodiment of this invention. It is a figure explaining the method of inputting a character by movement of an eyeball.

Explanation of symbols

  1: mobile phone, 10: control unit, 11: key, 12: speaker, 13: microphone, 14: display, 15: wireless signal processing unit, 16: antenna, 17: camera, 18: power supply

Claims (3)

A camera that images the user's eyeball;
A detection unit for detecting the movement of the imaged eyeball;
A counter that counts the number of detected eye movements;
And a control unit that executes a process for a predetermined operation corresponding to the number of movements of the eyeball. In claim 1,
The detection unit detects the direction of movement of the eyeball,
The counter counts the number of eye movements for each direction of eye movement,
The said control part performs the process corresponding to operation predetermined according to the direction and frequency | count of the movement of an eyeball, The portable terminal device characterized by the above-mentioned. In claim 2,
The said control part performs the process with respect to operation which selects and inputs a number or a character according to the direction and frequency | count of a motion of an eyeball.

Images