HK1119887B - Mobile information terminal apparatus - Google Patents

Description

Portable information terminal device

Technical Field

The present invention relates to a portable information terminal device, and more particularly, to an improvement of a portable information terminal device that performs authentication processing based on a captured image.

Background

A portable information terminal device such as a mobile phone includes a camera for capturing an object and generating image data. Among such portable information terminal devices, recently, the following ones have appeared: it is promoted to perform an authentication process (i.e., face authentication) by capturing a face image with a camera during operation and comparing feature information such as the shape or size of eyes, nose, mouth, and outline extracted from the captured face image with user feature information stored in advance as registered feature information in a memory of a portable information terminal device. As a result of the authentication processing, only when the degree of coincidence between the feature information extracted from the captured image and the feature information of the user exceeds the recognition threshold value, it is determined that the user who is performing the operation at that time is a genuine user, and the subsequent processing is allowed to be performed, whereby it is possible to improve security and protect privacy (for example, see patent document 1).

In general, an image captured at the time of authentication processing is not stored in a memory, and only feature information extracted from the image is stored in the memory, and the feature information is compared with registered feature information. Here, if an image captured at the time of authentication processing can be stored as history information when a person who is not a genuine user, for example, a person who operates a stolen personal digital assistant or the like illegally uses the personal digital assistant, the illegal user can be easily identified from the stored image after the personal digital assistant is retrieved. In such a technical field, various techniques have been proposed such as storing an image captured at the time of authentication processing as history information when authentication fails (for example, see patent documents 1 to 3).

Patent document 1: japanese laid-open patent publication No. 2000-278658

Patent document 2: japanese laid-open patent publication No. 2005-115485

Patent document 3: japanese laid-open patent publication No. 2005-084991

However, in the above-described conventional technique, when the authentication process fails, whatever image is captured at the time of the authentication process is stored as history information. Therefore, for example, at the time of authentication processing, even if the operator takes a surrounding landscape instead of the operator's own face or the like, the image is stored as a history in the memory. In other words, information that is completely unnecessary for determining an illegal user is stored in the memory, resulting in an increase in the amount of memory usage above that required.

Disclosure of Invention

The present invention has been made in view of the above circumstances, and an object thereof is to provide a portable information terminal device capable of storing only information useful for identifying an unauthorized user.

The invention provides a portable information terminal device capable of reducing the amount of memory used.

The invention provides a portable information terminal device capable of more accurately identifying an illegal user.

A first portable information terminal device according to the present invention includes: a photographing unit for photographing an object; a feature information storage unit that stores feature information extracted from the face image as registered feature information in advance; an authentication processing unit that performs authentication processing using the registered feature information based on the captured image captured by the capturing unit; and an error history management unit that stores history information of authentication failure as error history information in the error history storage unit in the authentication process, the authentication process unit including: a feature extraction unit that extracts feature information from the captured image; a matching degree calculating unit for comparing the feature information extracted by the feature extracting unit with the registered feature information stored in the feature information storing unit and calculating a degree of matching after digitization; and a matching degree determination unit that compares the matching degree with an authentication threshold value, wherein the error history management unit stores the error history information in the error history storage unit when the feature information is extracted by the feature extraction unit and the matching degree is determined to be smaller than the authentication threshold value by the matching degree determination unit.

According to the above configuration, although the feature information is extracted from the captured image, the error history information can be stored only when it is determined that the degree of coincidence of the feature information and the registered feature information is smaller than the authentication threshold value. In other words, although the captured image is a face image and the feature information of the face image such as the shape or size of the eyes, nose, mouth, and outline is extracted, the error history information can be stored only when the degree of coincidence with the user feature information stored in advance as the registered feature information is smaller than the authentication threshold and it is determined that it is not a genuine user.

In this way, a person who is not a genuine user, for example, a person who steals the portable information terminal device, and the like, wants to use the portable information terminal device illegally, and only when feature information is extracted from the face image of the person, history information in which authentication has failed in the authentication process is stored as error history information, and after the portable information terminal device is retrieved again, the person is specified based on the error history information. Therefore, it is possible to prevent the error history information from being stored when the user is not identified as an unauthorized user from the captured image, and therefore, it is possible to store only information useful for identifying an unauthorized user and reduce the amount of memory used.

A second portable information terminal device according to the present invention includes a timer unit for counting time, wherein the error history information includes: time information measured by the time measuring means when the authentication process is performed.

According to the above configuration, the time information when the authentication process is performed can be stored as the error history information. Therefore, after the portable information terminal device is retrieved again, the unauthorized user can be identified from the time information at the time of the authentication process, and thus the unauthorized user can be identified more accurately.

In the third portable information terminal device according to the present invention, the error history information includes a captured image captured by the imaging device for the authentication process.

According to the above configuration, a captured image used for authentication processing, that is, a face image from which feature information is extracted can be stored as error history information. Therefore, after the portable information terminal device is retrieved again, the unauthorized user can be identified from the face image, and thus the unauthorized user can be identified more accurately.

A fourth portable information terminal device of the present invention has a sequential authentication unit for performing imaging again when authentication fails in the authentication process by controlling the imaging unit and the authentication processing unit, and performing the authentication process based on the imaged image; the error history information includes: all or any one of the captured images captured by the capturing means for the authentication process.

According to the above configuration, when authentication fails in any one of the authentication processes repeatedly and continuously performed, all or any one of the captured images (face images) used for the authentication processes is stored as error history information. When all the face images are stored, an illegal user can be determined more accurately from these face images. On the other hand, when any one face image is stored, the amount of memory used can be reduced as compared with the case where all face images are stored.

A fifth portable information terminal device according to the present invention is a portable information terminal device including position information detecting means for detecting position information of the portable information terminal device, wherein the error history information includes: and position information detected by the position information detecting device when the authentication process is performed.

According to the above configuration, the position information at the time of the authentication process can be stored as the error history information. Therefore, after the portable information terminal device is retrieved, the unauthorized user can be identified from the position information at the time of the authentication process, and thus the unauthorized user can be identified more accurately.

In the sixth portable information terminal device according to the present invention, it is possible to sequentially store error history information of a predetermined number of times from the latest error history information in the error history information storage means, and the error history management means stores the error history information in the error history storage means when authentication fails after a predetermined time has elapsed since the last error history information was stored.

According to the above configuration, when the unauthorized user notices that history information in which authentication failed in the authentication process is stored as error history information and performs the authentication process in which his/her face cannot be recognized a plurality of times in a short time in order to delete the error history information from the error history storage means, it is possible to prevent error history information at the time of the authentication process from being stored. Therefore, it is possible to prevent an unauthorized user from intentionally deleting error history information useful for identifying the unauthorized user from the error history storage unit, and thus the unauthorized user can be identified more accurately.

A seventh portable information terminal device of the present invention comprises an error display unit operable to display, on a waiting screen, that the error history information is stored in the error history storage unit.

According to the above configuration, when the portable information terminal device is used illegally without the user knowing it and the error history information at that time is stored in the error history storage means, the user can be notified by displaying the prompt on the waiting screen.

According to the portable information terminal device of the present invention, the captured image is a face image, the feature information is extracted from the captured image, but only when it is determined that the degree of coincidence of the feature information with the registered feature information is less than the authentication threshold value, the error history information is stored. In this way, when an unauthorized user cannot be identified from the captured image, it is possible to prevent the storage of the error history information, so that it is possible to store only information useful for identifying an unauthorized user, and it is possible to reduce the amount of memory used.

According to the portable information terminal device of the present invention, since the time information when the authentication process is performed, the captured image used for the authentication process, and the position information when the authentication process is performed are stored as the error history information, the unauthorized user can be specified from these pieces of information after the portable information terminal device is retrieved again, and thus the unauthorized user can be specified more accurately. When the authentication process is repeatedly and continuously performed, an unauthorized user can be identified more accurately if all the captured images used for the authentication process are stored, and the amount of memory used can be reduced if any one of the captured images is stored.

According to the portable information terminal device of the present invention, the error history information is stored only when the authentication has failed after a predetermined time has elapsed since the last time the error history information was stored, whereby it is possible to prevent an unauthorized user from intentionally deleting the error history information useful for identifying the unauthorized user from the error history storage means, and thus it is possible to identify the unauthorized user more accurately.

According to the portable information terminal device of the present invention, when the portable information terminal device is used illegally without the user knowing it and the error history information is stored in the error history storage device, the user can be notified by displaying the prompt on the waiting screen.

Drawings

Fig. 1 is an external view of an example of a portable information terminal device according to an embodiment of the present invention, and shows an example of a portable information terminal device that is a mobile phone.

Fig. 2 is a block diagram showing an internal configuration of the portable telephone set shown in fig. 1.

Fig. 3 is a block diagram showing a configuration example of the imaging unit drive control section of fig. 2.

Fig. 4 is a block diagram showing a configuration example of the authentication processing section of fig. 2.

Fig. 5 is a schematic diagram showing an example of display of the main display unit when face authentication is set.

Fig. 6 is a schematic diagram of a display example of the image registration screen for face authentication.

Fig. 7 is a schematic diagram of an example of the display of the security level setting screen.

Fig. 8 is a schematic diagram of a display example of the error history screen.

Fig. 9 is a schematic diagram of an example of the display of the main display unit when performing face authentication.

Fig. 10 is a schematic diagram of display examples when error history information is selected from the error history screen, (a) is a selection status of the error history information, (b) is a first display example for the selected error history information, and (c) is a second display example for the selected error history information.

Fig. 11 is a schematic diagram of an example of a waiting screen displayed in the main display section.

Detailed Description

Fig. 1 is an external view of an exemplary portable information terminal device according to an embodiment of the present invention, and shows a mobile phone 1 as an example of the portable information terminal device. The mobile phone 1 is a so-called folding mobile phone, and the display housing 100 and the operation housing 200 are connected by a hinge 300, and the display housing 100 can be folded so as to face one surface of the operation housing 200.

In the display case 100, a main display unit 101, a receiver 103, and a sub camera 105 are disposed on an inner case surface when folded, and a sub display unit 102 and a main camera 104 are disposed on an outer case surface. The operation case 200 has a plurality of operation keys 201 and a microphone 203 for transmitting speech on the inner case surface when folded. Such a folding mobile phone 1 can be carried in a folded and compressed state, and when the case is unfolded, the operation key 201 can be pressed to operate while the display of the main display unit 101 is viewed. In short, the main information display is realized by the main display unit 101, and the main input operation is realized by using the operation keys 201.

Such a mobile phone 1 has the following functions (so-called face authentication function): when the user operates the operation key 201 while viewing the display of the main display section 101, the face image of the user is captured by the sub-camera 105, and authentication processing is performed based on feature information extracted from the face image, with the result that subsequent processing is permitted only when it is determined that the person who performs the operation at this time is a genuine user. By performing authentication processing when the function having high concealment is executed, it is possible to improve security and protect privacy.

Fig. 2 is a block diagram showing an internal configuration of the mobile phone shown in fig. 1. The control unit 400 is a processor that controls the main operation of the mobile phone 1, and includes an imaging unit drive control unit 401, an authentication processing unit 402, a continuous authentication unit 403, an error history management unit 404, a time counting unit 405, and a position information detection unit 406. An operation signal is input from the operation key 201 to the control unit 400, and the memory 410, the image input/output unit 430, the audio input/output unit 440, and the communication unit 450 are connected to the control unit 400 so as to be capable of input and output.

The image input/output unit 430 includes a main display unit 101, a sub display unit 102, a main camera 104, and a sub camera 105. The main camera 104 and the sub camera 105 are respectively unitized as an image pickup unit including a photoelectric conversion element such as a CCD (Charge-Coupled Device) and a cmos (complementary Metal oxide semiconductor) image sensor, and a control circuit. Here, a photoelectric conversion element that requires initialization processing such as offset and gain adjustment is used after power is supplied to the imaging unit.

The audio input/output unit 440 includes the receiver 103 and the microphone 203. The communication unit 450 transmits and receives radio waves to and from a base station, not shown, to realize transmission and reception of audio data, image data, and the like.

The image pickup unit drive control section 401 controls the drive of the main camera 104 and the sub camera 105 by transmitting a signal to the image input/output section 430. In this embodiment, the image pickup unit drive controller 401 controls the drive of the sub-camera 105 to pick up a face image of an operator who operates the operation keys 201 while viewing the main display unit 101 with the sub-camera 105, and performs authentication processing based on feature information extracted from the face image. However, the authentication process may be performed after the face image is captured by the main camera 104.

The authentication processing unit 402 compares the feature information extracted from the captured image with the feature information (registered feature information) of the user stored in the memory 400 in advance, and determines whether the operator who performs the operation at this time is a genuine user. The continuous authentication unit 403 continuously performs authentication processing a plurality of times by controlling the imaging unit drive control unit 401 and the authentication processing unit 402. When the authentication fails, the error history management section 404 stores history information at the time of this authentication in the memory 410 as error history information.

The timer unit 405 is a device for measuring time. The position information detecting unit 406 is a device that detects position information of the mobile phone 1, and here, the position information detecting unit 406 is configured by a gps (global Positioning system).

The memory 410 includes a display screen data storage section 411, an image data storage section 412, a feature information storage section 413, and an error history information storage section 414. The display screen data storage unit 411 stores screen data displayed on the main display unit 101 or the sub display unit 102. The image data storage unit 412 stores image data captured by the main camera 104 or the sub camera 105, image data received from the outside through the communication unit 450, and the like. The feature information storage 413 stores feature information of the face image used for the authentication process.

In the error history information storage section 414, error history information is stored a predetermined number of times in order from the latest error history information. That is, when the error history information stored in the error history information storage unit 414 fails to be authenticated in a state where the predetermined number of times is not satisfied, the error history information at the time of the authentication is added to the error history information storage unit 414 and stored. On the other hand, when the authentication fails in a state where the error history information is stored in the error history information storage unit 414 a predetermined number of times, the oldest error history information is deleted and new error history information at the time of authentication is stored.

For example, the following may be stored as the error history information: time information output from the timer unit 405 when performing the authentication process, a captured image captured by the sub-camera 105 used for the authentication process, position information output from the position information detection unit 406 when performing the authentication process, and the like.

However, the feature information of the face image or the image data of the face image used for the authentication process may be stored in a recording medium (not shown in the figure) detachable from the mobile phone 1, such as a sim (subscriber Identity module) card.

Fig. 3 is a block diagram showing a configuration example of the imaging unit drive control section 401 of fig. 2. The imaging unit drive control unit 401 includes an imaging unit activation unit 407, an imaging control unit 408, and an imaging unit stop unit 409. The image pickup unit activation unit 407 transmits a control signal to the main camera 104 and the sub-camera 105, and then activates these cameras, and the image pickup unit activation unit 407 includes: a power supply connection unit 407A for connecting power to the main camera 104 or the sub-camera 105 in accordance with a user operation, and an initialization processing unit 407B for performing initialization processing.

The image pickup control unit 408 controls the image pickup operation of the main camera 104 or the sub-camera 105 based on the input signal from the initialization processing unit 407B and the image pickup operation by the user, and generates still image data of the subject for the main camera 104 or the sub-camera 105 at that time. The image pickup unit stop unit 409 stops the driving of the image pickup units by stopping the supply of power to the main camera 104 and the sub-camera 105.

Fig. 4 is a block diagram showing a configuration example of the authentication processing section 402 of fig. 2. The authentication processing unit 402 includes a feature extraction unit 402A, a matching degree calculation unit 402B, and a matching degree determination unit 402C. When the authentication process is started, feature information such as the shape or size of each part of the face, such as the eyes, nose, mouth, and contour, is extracted from the face image captured by the sub-camera 105 by the feature extraction section 402A.

When the captured image is not a face image but a landscape image or the like, the feature information cannot be extracted from the captured image. In this case, the authentication fails, but the error history information at this time is not stored in the error history information storage section 414 of the memory 410. On the other hand, when the feature information is successfully extracted from the captured image, the matching degree calculating unit 402B compares the feature information with the registered feature information stored in the feature information storage unit 413 of the memory 410, and calculates a matching degree after digitization.

The matching degree determination unit 402C compares the matching degree calculated by the matching degree calculation unit 402B with a predetermined threshold (authentication threshold), and outputs the comparison result. At this time, when the matching degree exceeds the authentication threshold, it is determined that the user is a genuine user, and the authentication is successful. And when the consistency is smaller than the authentication threshold, the user is not judged as a real user, and the authentication fails. Here, the matching degree is a ratio (%) of matching feature information among all the compared feature information, and the authentication threshold is usually set to 90% or more of a predetermined value.

The comparison result output by the matching degree determination unit 402C is input to the error history management unit 404. When the authentication fails, the error history management section 404 stores the error history information at the time of the authentication in the error history information storage section 414. On the other hand, when the authentication is successful or when the authentication fails within a certain time period from the time when the previous error history information is stored in the error history information storage unit 414, the error history management unit 404 does not store the error history information at the time of the authentication in the error history information storage unit 414. That is, the error history management section 404 stores the error history information in the error history information storage section 414 only when the authentication fails after a certain time has elapsed since the previous error history information was stored.

In the present embodiment, if the feature information is extracted from the captured image, the error history information can be stored only when it is determined that the degree of coincidence between the feature information and the registered feature information is smaller than the authentication threshold value. That is, although the captured image is a face image, and the feature information of the face image such as the shape or size of the eyes, nose, mouth, and outline is extracted, the error history information can be stored only when the degree of matching with the user feature information stored in advance as the registered feature information is smaller than the authentication threshold value and it is determined that the user is not a genuine user.

Thus, if a person who is not a genuine user, for example, who has stolen the mobile phone 1, wants to illegally use the mobile phone 1, only when the feature information is extracted from the face image of the person, the history information of the authentication failure in the authentication process is stored as the error history information, and after the mobile phone 1 is retrieved again, the person can be determined based on the error history information. Accordingly, it is possible to prevent the error history information from being stored when an unauthorized user cannot be identified from the captured image, so it is possible to store information useful for identifying an unauthorized user, and it is possible to reduce the amount of use of the memory 410.

Further, by adopting the above-described configuration, the error history information is stored only when the authentication has failed after a lapse of a certain time since the previous error history information was stored, and when the authentication process in which the unauthorized user has failed as the error history information during the authentication process is performed a plurality of times in a short time in order to delete the error history information from the error history information storage unit 414, it is possible to prevent the error history information during the authentication process from being stored. Therefore, it is possible to prevent the unauthorized user from intentionally deleting the error history information for specifying the unauthorized user from the error history information storage unit 414, and thus it is possible to specify the unauthorized user more reliably.

Fig. 5 is a schematic diagram showing an example of the display of the main display unit when setting is performed regarding face authentication. When the user operates the operation key 201 to read out the security setting screen (see fig. 5 a), and selects the "face recognition setting" item among the items displayed on the security setting screen, a screen urging the input of the terminal password number assigned in advance to the mobile phone 1 is displayed on the main display unit 101 (see fig. 5 b). In this display screen, when the clear key included in the operation keys 201 is operated, the display of the main display screen 101 returns to the security setting screen as shown in fig. 5.

The terminal password number is set in advance at the time of purchase of the mobile phone 1, and is constituted by an arbitrary character string of, for example, 4 to 8 digits, and can be changed by the user by operating the operation key 201. In the terminal password number input screen of fig. 5(b), when the character string input by the operation of the operation key 201 does not match the terminal password number set in advance, the presentation is displayed on the main display unit 101 (see fig. 5 (c)). When the character string input on the terminal password number input screen is 3 bits or less, a prompt to input a 4-8-bit character string is displayed on the main display unit 101 (see fig. 5 (d)).

After the main display unit 101 has been displayed as shown in fig. 5(c) or 5(d), if a predetermined time elapses and a timeout occurs or any one of the operation keys 201 is operated, the display of the main display unit 101 returns to the security setting screen shown in fig. 5 (a).

When the character string input in the terminal password number input screen of fig. 5(b) coincides with the terminal password number set in advance, the display of the main display portion 101 is switched to a face authentication setting screen for setting ON/OFF (ON/OFF) of a mode for performing face authentication (see fig. 5 (e)). When the clear key included in the operation key 201 is operated on this display screen, the display of the main display unit 101 returns to the terminal password number input screen shown in fig. 5 (b). Then, in the face authentication use setting screen, when the face authentication is finished by the operation of the operation key 201, the display of the main display unit 101 returns to the security setting screen shown in fig. 5 (a).

In the face authentication use setting screen of fig. 5(e), once face authentication is started by operation of the operation key 201, the display of the main display section 101 is switched to a face authentication setting screen for making settings regarding face authentication (see fig. 5 (f)).

In the present embodiment, feature information may be extracted from a maximum of 7 image data and stored in the memory 410 as registered feature information used when performing face authentication. The face authentication is performed based on the registered feature information extracted from at least 2 pieces of image data, whereby the probability of success of authentication (the acceptance rate of the person) when performing face authentication on a genuine user can be increased, and authentication failure due to a small difference in the expression of the user or the angle of light irradiation or the like can be prevented. However, the registration feature information used for the face authentication of the user is not limited to being extracted from 7 pieces of image data, and the face authentication may be performed based on the registration feature information extracted from one piece of image data, for example.

The face image of the user is captured by the main camera 104 or the sub-camera 105 in advance, and these image data are stored in the image data storage section 412 of the memory 410. The face authentication is performed based on the registered feature information extracted from these image data and stored in the feature information storage part 413 of the memory 410, and therefore, these image data themselves are necessary at the time of face authentication, so if stored in the memory 410, the user can confirm the face image by reading these image data, thereby improving convenience.

The face authentication setting screen in fig. 5(f) includes: an item corresponding to an image of the user (image used for face authentication) which is an extraction source of the registered feature information, an item corresponding to setting of an authentication threshold value (security level setting) at the time of face authentication, and an item corresponding to an error history. On this display screen, if an item corresponding to the image for face authentication is selected, a screen (image registration screen for face authentication) for performing rewrite registration or confirmation of the image for face authentication is displayed. If an item corresponding to the security level setting is selected, a screen for setting an authentication threshold value at the time of face authentication (security level setting screen) is displayed. If an item corresponding to the error history is selected, a screen for displaying the error history (error history screen) is displayed.

When the operation key 201 is operated after the various settings and the like are made, and the face authentication setting is completed, the display of the main display unit 101 is switched to the displays shown in fig. 5(g) and 5 (h). That is, if at least 1 face authentication image is registered, a prompt to perform face authentication is displayed (see fig. 5(g)) on the main display unit 101 based on feature information extracted from the face authentication image, and if the confirmation key 111 included in the screen is selected, the display of the main display unit 101 returns to the security setting screen shown in fig. 5 (a). On the other hand, if the face authentication image is not registered at a time, a prompt to register the face authentication image is displayed on the main display unit 101 (see fig. 5(h)), and when the confirmation key 112 included in the screen is selected, the main display unit 101 returns to the face authentication setting screen shown in fig. 5 (f).

Fig. 6 is a schematic diagram of a display example of the image registration screen for face authentication. When any one of the items corresponding to the face authentication image is selected in the face authentication setting screen in fig. 5(f), the display of the main display unit 101 is switched to the face authentication image registration screen corresponding to the face authentication image shown in fig. 6 (a). The image registration screen for face authentication includes an "overwrite registration" item and a "registration data confirmation" item.

When the "rewrite registration" item among the items displayed on the image registration screen for face recognition in fig. 6(a) is selected by an operation, the sub-camera 105 is started, and when the sub-camera 105 is in a state in which it can capture an image, an image of the subject with respect to the sub-camera 105 is displayed on the main display unit 101 in real time (see fig. 6 (b)). At this time, the main display unit 101 serves as a viewfinder of the sub-camera 105, and the face of the operator who operates the main display unit 101 while viewing the main display unit 101 appears on the viewfinder screen formed by the main display unit 101.

In the finder screen of fig. 6(b), when the user performs a registration operation by operating the enter key included in the operation keys 201 after adjusting the position of his/her face or the like, the feature extraction unit 402A of the control unit 400 performs a process of extracting feature information from the image captured by the sub-camera 105 at that time.

In this case, if the position of the face of the user is appropriately adjusted, feature information such as the shape and size of the eyes, nose, mouth, and outline can be extracted, and if the position is not appropriately adjusted, the feature information cannot be extracted if the face of the user is not displayed at all on the main display unit 101. When the feature information is successfully extracted, the feature information is stored in the feature information storage 413 of the memory 410, and the face image at that time is stored as an image for face authentication in the image data storage 412 of the memory 410.

When the feature information is successfully extracted, a prompt that the feature information based on the face image has been stored (registered) in the memory 410 is displayed in the main display section 101 (see fig. 6 (c)). When the user selects the confirmation key 113 included in such a display screen, the display of the main display unit 101 returns to the face authentication setting screen shown in fig. 5 (f). On the other hand, when the extraction of the feature information fails, the prompt and the reason for the failure of the extraction of the feature information are displayed on the main display unit 101 (see fig. 6(d)), and when the user selects the confirmation key 114 included in the display screen, the display on the main display unit 101 returns to the finder screen shown in fig. 5(b), and the face image can be captured again by the sub-camera 105.

Upon operation selection of the "registration data confirmation" item displayed in the image registration screen for face authentication of fig. 6(a), the corresponding image for face authentication stored in the image data storage section 412 of the memory 410 is read out and displayed in the main display section 101 (see fig. 6 (e)). The user can confirm the registered image for face authentication through the confirmation screen of the image for face authentication shown in fig. 6 (e). In this display screen, when the clear key included in the operation key 201 is operated, the display of the main display unit 101 returns to the image registration screen for face authentication shown in fig. 6 (a).

In the image confirmation screen for face authentication of fig. 6(e), when the user operates the operation key 201 to delete the corresponding image for face authentication, the display of the main display unit 101 is switched to a deletion confirmation screen for confirming whether or not the image for face authentication can be deleted (see fig. 6 (f)). In this display screen, a "yes" key 115 indicating that the image for face authentication should be deleted and a "no" key 116 presenting that the image for face authentication is not to be deleted are included. In the deletion confirmation screen of fig. 6(f), when the no key 116 is selected by the operation of the operation key 201, the display of the main display unit 101 returns to the image confirmation screen for face authentication shown in fig. 6 (e).

In the deletion confirmation screen of fig. 6(f), when the "yes" key 115 is selected by the operation of the operation key 201, the processing of deleting the corresponding image for face authentication and registered feature information from the image data storage portion 412 and the feature information storage portion 413 of the memory 410 is started, and a prompt indicating that deletion is being performed is displayed on the main display portion 101 (see fig. g). Then, when the process of deleting the image for face authentication and the registered feature information is completed, the prompt is displayed on the main display unit 101 (see fig. 6(h)), and if a time-out elapses after a certain time elapses or in response to an operation of any one of the keys included in the operation keys 201, the display on the main display unit 101 returns to the face authentication setting screen shown in fig. 5 (f).

Fig. 7 is a schematic diagram of an example of the display of the security level setting screen. In the face authentication setting screen of fig. 5(f), when an item corresponding to the security level setting is selected, the display of the main display section 101 is switched to the security level setting screen for authentication threshold setting at the time of face authentication shown in fig. 7. As for the security level, it is previously set: the highest authentication threshold is "level 3", the authentication threshold lower than "level 3" is "level 2", and the authentication threshold lower than "level 2" is "level 1"; in the security level setting screen, any one of the security levels can be selected.

Fig. 8 is a schematic diagram of a display example of the error history screen. In the face authentication setting screen of fig. 5(f), when the item corresponding to the error history is selected, the display of the main display unit 101 is switched to the error history screen for displaying the error history shown in fig. 8. In the error history screen, time information indicating the date and time when the authentication failed is displayed as error history information only for a predetermined number of times (for example, 9 times). When re-authentication fails in a state where the error history information of the predetermined number of times is stored in the error history information storage unit 414 of the memory 410, the oldest error history information is deleted, the newest error history information is stored in the memory 410, and the error history information is updated so that new error history information is displayed on the error history screen.

Fig. 9 is a schematic diagram of an example of display of the main display unit 101 when face authentication is performed. In this example, when switching on/off of the security mode included in the management function of the mobile phone 1, face authentication is performed. When the security mode is in the activated state, a state of displaying normal data and data registered in advance as the security data; when the security mode is in the off state, the state is such that only normal data is displayed without displaying the security data.

In the management function screen for setting the management function of the mobile phone 1 shown in fig. 9(a), when the "privacy mode" item among the items displayed on the management function screen is selected and operated, the sub-camera 105 is started. After the sub-camera 105 is in a photographable state, an image of the subject for the sub-camera 105 is displayed in a real-time manner in the main display section 101 (see fig. 9(b)), and automatic authentication is started. At this time, since the sub-camera 105 is in a photographable state after the initialization processing is finished, it is possible to photograph a face image for authentication and further extract feature information from the photographed image even if the user does not perform a photographing operation. Then, the authentication process is automatically performed by comparing the extracted feature information with the user feature information stored in advance in the feature information storage unit 413 of the memory 410.

In this authentication process, when the degree of matching between the feature information of the extracted face image and the feature information of the user stored in the memory 410 exceeds an authentication threshold value set in advance in the security level setting screen, it is determined that the operator is a true user. At this time, the face image of the user displayed on the main display unit 101 is emphasized and displayed by so-called action focus (action focus), and the success of authentication is notified by visual effect.

Then, a message indicating that the authentication is successful is displayed on the main display unit 101 (see fig. 9(c)), and after a predetermined time has elapsed, the screen is switched to the screen for switching on/off the security mode (see fig. 9 (d)). On the other hand, when the degree of coincidence of the feature information is smaller than the authentication threshold and authentication fails, the feature information is automatically extracted again from the image captured by the sub-camera 105 at that time, and is compared with the feature information of the user stored in the memory 410.

As described above, in the present embodiment, when authentication fails, authentication processing is repeatedly and continuously performed until a time-out elapses after a predetermined time (for example, 3 seconds) elapses, and the authentication processing is terminated when authentication succeeds. For example, when the timeout time is set to 3 seconds, the authentication process is performed 7 times at most. In other words, the time required for 1 authentication process is 0.4 seconds, and assuming that the authentication is successful in 1 authentication process, the next process can be entered with almost no waiting time after the initialization process of the sub-camera 105 is completed. While the authentication process is being repeated, a finder screen is displayed in the main display section 101, and an image of the subject for the sub-camera 105 is displayed in real time.

When the authentication process is repeated, every time the authentication process is performed a plurality of times, a display corresponding to the degree of matching of the feature information is displayed on the main display unit 101 as a result of comparison of the feature information in the authentication process. Specifically, the color of the frame 117 of the finder screen displayed on the main display unit 101 is displayed in a color corresponding to the degree of coincidence of the feature information. Therefore, the comparison result of the degree of matching of the feature information at the time of the authentication process is displayed, and therefore, the user can confirm whether or not the face image of the user photographed at the time of the authentication process is an easily recognizable image based on the display.

However, the configuration is not limited to this, and the main display unit 101 may display the matching degree of the feature information in a hierarchical manner using numerical values, graphs, or the like. Further, the comparison result is not limited to the configuration in which the comparison result is displayed for all authentication processes, and the comparison result may be displayed for at least one authentication process.

When the time-out occurs after a predetermined time has elapsed after the start of the automatic authentication, a prompt indicating that the authentication has failed is displayed on the main display unit 101 (see fig. 9 (e)). When the user selects the reconfirm key 118 included in the display screen, the display of the main display unit 101 returns to the finder screen shown in fig. 9(b), and the automatic authentication is restarted. On the other hand, in fig. 9(e), when the user selects the terminal password number input key 119 displayed on the main display portion 101, the display of the main display portion 101 is switched to a terminal password number input screen urging input of the terminal password number (see fig. 9 (f)).

When the character string input in the terminal password number input screen of fig. 9(f) coincides with the terminal password number set in advance, the display of the main display portion 101 is switched to a screen for switching on/off of the secret mode (see fig. 9 (d)).

On the other hand, in the terminal password number input screen of fig. 9(f), when the character string input by the operation of the operation key 201 does not match the terminal password number set in advance, the presentation is displayed on the main display portion 101 (see fig. 9 (g)). When the character string input on the terminal password number input screen is 3 bits or less, a prompt to input a 4-8-bit character string is displayed on the main display unit 101 (see fig. 9 (h)). Next, after the display of fig. 9(g) or 9(h) is performed, when a predetermined time elapses and a timeout occurs or any one of the operation keys 201 is operated, the display of the main display unit 101 returns to the management function screen shown in fig. 9 (a).

In the present embodiment, when authentication fails in the authentication process, the authentication process is automatically performed again, and when the true user performs an operation, if it cannot be determined that the user is the true user by the authentication process 1, the user performs the authentication process again without performing a photographing operation thereafter. In other words, since the authentication process is repeated automatically until the user is determined to be a genuine user, the time required for the authentication process can be shortened while reducing the labor required for the authentication process.

When the face image is captured in the authentication process, it is preferable that the shutter sound is not emitted. With this configuration, when the authentication process is performed in an environment where people are around the vehicle, such as an automobile, it is possible to prevent the shutter sound from being generated and being misinterpreted as a candid photograph or the like. In the present embodiment, since the photographing operation is performed at the time of the authentication process, it is possible to prevent misunderstanding as a candid photograph or the like due to the photographing operation being performed.

However, when the user is not determined to be a genuine user within a predetermined time, the authentication process may be repeated until the authentication is successful, without being limited to the timeout. Further, the configuration is not limited to the configuration in which the authentication processing is continuously performed, and the configuration in which the authentication processing is performed only once after the sub-camera 105 is started and is in the image-recordable state may be employed.

Fig. 10 is a schematic diagram of display examples when error history information in the error history screen is selected, (a) is a selection status of the error history information, (b) is a first display example for the selected error history information, and (c) is a second display example for the selected error history information.

In the error history screen shown in fig. 8, the user can select error history information (see fig. 10 a) by operating the operation key 201, pointing the pointer 120 to any one of the error history information (time information) displayed in the list on the main display unit 101, and then performing a determination operation.

In the first display example shown in fig. 10(b), in accordance with the selection operation of the error history information, the image captured by the sub-camera 105 for the authentication process when the error history information is stored is read out from the error history information storage section 414, and the captured image is displayed in the main display section 101. When the authentication is performed continuously as in the present embodiment, since a plurality of captured images are used for the authentication, all or any one of the captured images is displayed on the main display unit 101. When all of the plurality of captured images are displayed in the main display section 101, the captured images are thumbnail-displayed in the main display section 101, or when a selection operation is performed on the captured images, the selected captured images may also be displayed in an enlarged manner throughout the main display section 101.

However, when a configuration is adopted in which only one authentication process is performed, which is different from the embodiment of the present invention, the image captured by the sub-camera 105 in the 1-time authentication process may be displayed on the main display unit 101.

In the second display example shown in fig. 10(c), in the authentication process in which the error history information is stored, the position information detected by the position information detecting unit 406 is read from the error history information storing unit 414 in accordance with the selection operation of the error history information, and the position information is displayed on the main display unit 101. In this embodiment, the place name as the position information is displayed on the main display unit 101, but the present invention is not limited to this, and for example, a map may be displayed on the main display unit 101, and the detected position information may be specified on the map.

The present invention is not limited to the configuration in which the time information at the time of authentication processing as the error history information is list-displayed on the main display unit 101, and may be configured such that the error history information displayed in the list is selected, the time information at the time of authentication processing corresponding to the error history is read from the error history information storage unit 414, and the time information is displayed on the main display unit 101. Further, time information when the authentication processing is performed for each error history, a captured image used for the authentication processing, and position information when the authentication processing is performed may be displayed together, and a configuration may be adopted in which any one of these error history information is displayed for the user to select.

In the present embodiment, by storing time information at the time of authentication processing, a captured image used for authentication processing, or position information at the time of authentication processing as error history information, an unauthorized user can be identified from the information after the mobile phone 1 is retrieved again, and therefore, the unauthorized user can be identified more accurately. When performing the continuous authentication, assuming that all the captured images used for these authentication processes are stored, an illegal user can be more correctly determined from the captured images (face images), and if any one captured image is stored, the amount of use of the memory 410 can be reduced compared to the case where all the captured images are stored.

Fig. 11 is a schematic diagram of an example of the standby screen displayed on the main display unit 101. Here, the standby screen is a screen displayed on the main display unit 101 when the mobile phone 1 is powered on and the application is not started.

In the example shown in fig. 11, a notification area 121 for notifying the user by displaying a predetermined icon is provided on a part of the standby screen displayed on the main display unit 101. In this example, in the notification area 121, a mail reception icon 122 for notifying that a new electronic mail is intended to be received, and an error display icon (error display unit) 123 for notifying that new error history information is intended to be stored are displayed. Therefore, when the user does not know that the mobile phone 1 is used illegally and the error history information at that time is stored in the error history information storage unit 414, the user can be notified by displaying the prompt on the standby screen.

The face authentication is not limited to the on/off switching of the security mode, and may be performed in various setting operations for other functions with high security or privacy, such as card setting, reception restriction, lock setting, password number change, notification sender number/no notification setting, resetting, software update, detailed display of the own station number, and other data change.

The authentication process is not limited to face authentication based on face image data, and may be performed based on a fingerprint image of a user if the image of the user is displayed. In this case, the authentication process can be performed by comparing the extracted feature information of the fingerprint images.

The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the claims.

This application claims priority from Japanese patent application No. 2005-212806 (Japanese application No. 2005-212806) at 7/22/2005 according to the rules of the Paris convention, which is incorporated herein by reference in its entirety.

Description of the symbols

1. Portable telephone set

101. Main display part

102. Sub-display part

103. Receiver for receiving telephone

104. Main camera

105. Sub-camera

121. Notification area

123. Error display icon

201. Operating key

203. Microphone for transmitting telephone

400. Control unit

401. Image pickup unit drive control unit

402. Authentication processing unit

402A, a feature extraction unit

402B, a coincidence degree calculating unit

402C, coincidence degree determination unit

403. Continuous authentication unit

404. Error history management unit

405. Timing part

406. Position information detecting unit

407. Image pickup unit activation unit

407A, Power supply connection part

407B, initialization processing unit

408. Image pickup control unit

409. Image pickup unit stop unit

410. Memory device

411. Display screen data storage unit

412. Image data storage unit

413. Characteristic information storage unit

414. Error history information storage unit

450. Communication unit

Claims (6)

1. A portable information terminal device, comprising:

a photographing unit for photographing an object;

a feature information storage unit that stores feature information extracted from the face image as registered feature information in advance;

an authentication processing unit configured to perform authentication processing using the registered feature information based on the captured image captured by the capturing unit; and

an error history management unit that stores history information of authentication failure as error history information in the error history storage unit in the authentication process,

the authentication processing unit includes: a feature extraction unit that extracts feature information from the captured image; a matching degree calculating unit for comparing the feature information extracted by the feature extracting unit with the registered feature information stored in the feature information storing unit and calculating a degree of matching after digitization; and a coincidence degree judging unit for comparing the coincidence degree with an authentication threshold value,

when the feature information is extracted in the feature extraction unit and the degree of coincidence is judged to be smaller than the authentication threshold value in the degree of coincidence judgment unit, the error history management unit stores the error history information in the error history storage unit,

error history information that can be stored in the error history storage unit a predetermined number of times in order from the latest error history information;

the error history management means stores the error history information in the error history storage means when the authentication fails after a predetermined time has elapsed since the last error history information was stored.

2. The portable information terminal device according to claim 1,

has a timing unit for timing a time,

the error history information includes: and time information measured by the time measuring means when the authentication process is performed.

3. The portable information terminal device according to claim 1,

the error history information includes: a captured image for the authentication process captured by the capturing unit.

4. The portable information terminal device according to claim 1,

a continuous authentication unit for controlling the shooting unit and the authentication processing unit to shoot again when the authentication fails in the authentication processing and performing the authentication processing according to the shot image,

the error history information includes: all or any one of the captured images captured by the imaging device and used for the authentication process.

5. The portable information terminal device according to claim 1,

a position information detecting unit for detecting position information of the portable information terminal device;

the error history information includes: and position information detected by the position information detecting means when the authentication process is performed.

6. A portable information terminal device according to any one of claims 1 to 5,

having an error display unit for displaying in the waiting screen: error history information is stored in the error history storage unit.