HK1119801B - Mobile information terminal device - Google Patents

Description

Portable information terminal device

Technical Field

The present invention relates to a portable information terminal device, and more particularly, to an improvement of a portable information terminal device that performs authentication processing based on a captured image.

Background

A portable information terminal device such as a mobile phone generally includes a camera for capturing an image of a subject to generate image data. A camera is equipped with, for example, a CCD (Charge-Coupled Device) or a CMOS (Complementary metal oxide Semiconductor) image sensor as a photoelectric conversion element.

Recently, in such a portable information terminal device, it is promoted that a camera captures a face image during operation, and feature information such as the shape or size of eyes, nose, mouth, and outline extracted from the captured face image is compared with feature information of a user stored in advance in a memory of the portable information terminal device to perform authentication processing (so-called face authentication). As a result of the authentication processing, only in a case where the degree of coincidence between the feature information extracted from the captured face image and the feature information of the user exceeds the recognition threshold value, it is determined that the person who performs the operation at this time is a genuine user, and the processing after execution is permitted, whereby it is possible to achieve enhancement of security and protection of privacy (for example, refer to patent document 1).

In the authentication process, power is first supplied to the camera to start the camera, and the photoelectric conversion element provided in the camera is initialized to perform imaging. When shooting is possible by a camera, an image of a subject with respect to the camera is displayed in real time on a display unit of a portable information terminal device. The person who performs the operation performs adjustment so as to take a face image of himself with an image camera while viewing a display screen (viewfinder screen) of the display section, and performs a determination operation to take a face image. Thereby, it is possible to extract feature information from the captured face image, and perform authentication processing by comparing the feature information with the user feature information stored in advance.

Patent document 1: japanese laid-open patent publication No. 2000-278658

However, in the current authentication processing method, when an operation for determining execution of a process requiring an authentication process is performed, the camera is first started, and after photographing by the camera is possible, if a face image is not photographed according to the photographing operation, the authentication process is not performed. Since it usually takes about 0.5 seconds to shoot after the camera is started, and the user has to wait for the time to pass before shooting operation is performed, there is a problem that it is troublesome.

Further, when it is not determined that the user is a genuine user, the photographing operation must be performed again, which increases the trouble and also causes a problem that the authentication process takes time.

Disclosure of Invention

The present invention has been made in view of the above circumstances, and an object thereof is to provide a portable information terminal device capable of reducing labor and time required for authentication processing.

Another object of the present invention is to provide a portable information terminal device capable of shortening the time required for authentication processing.

The 1 st mobile information terminal device of the present invention includes: a photographing unit for photographing an object; a shooting unit starting part which supplies power to the shooting unit based on the operation of a user so as to carry out the initialization processing of the shooting unit; an imaging control unit that controls an imaging operation performed by the imaging unit; a feature information storage section that stores feature information extracted from the face image in advance as registered feature information; and an authentication processing unit configured to perform an authentication process based on a captured image of the imaging unit using the registration feature information, wherein the imaging control unit starts imaging in response to an end of the initialization process of the imaging unit performed by the imaging unit starting unit.

According to the above configuration, the imaging unit can be started based on the user operation, the imaging can be performed after the initialization processing is finished, and the authentication processing can be automatically performed based on the completion of the initialization processing. That is, the user automatically extracts feature information from the image captured by the capturing unit without performing a capturing operation after the capturing unit is started, and performs authentication processing by comparing the feature information with registered feature information stored in advance. Therefore, the trouble that the photographing operation is performed after waiting for the time from the start of the photographing unit until photographing becomes possible can be avoided, and the labor and time required for the authentication process can be reduced.

Further, as in the prior art, the time delay from when the user confirms that the shooting means is enabled after starting the shooting means to when the user performs the shooting operation can be reduced, and the authentication process can be performed in a shorter time after the shooting means is started, so that the time required for the authentication process can be shortened.

The 2 nd portable information terminal device of the present invention includes: a photographing unit for photographing an object; a shooting unit starting part which supplies power to the shooting unit based on the operation of a user so as to carry out the initialization processing of the shooting unit; an imaging control unit that controls an imaging operation performed by the imaging unit; a feature information storage section that stores feature information extracted from the face image in advance as registered feature information; an authentication processing unit that performs authentication processing based on a captured image of the capturing unit using the registration feature information; and a continuous authentication unit configured to control the imaging control unit and the authentication processing unit to perform imaging again when authentication in the authentication processing fails, and to perform authentication processing based on the captured image.

With this configuration, when authentication in the authentication process fails, the authentication process can be automatically performed again. Therefore, when the true user performs the operation, if it is not determined to be the true user by the 1 st authentication process, the authentication process can be performed again without performing the photographing operation by the user thereafter. That is, since the authentication process is automatically repeated until it is determined that the user is a genuine user, the labor and time required for the authentication process can be reduced, and the time required for the authentication process can be shortened.

In the 3 rd portable information terminal device according to the present invention, the authentication processing means includes: a feature extraction unit that extracts feature information from the captured image; a matching degree calculating unit that compares the feature information extracted by the feature extracting unit with the registered feature information stored in the feature information storing unit and calculates a degree of matching after digitization; the authentication device further includes a comparison result display unit for displaying a comparison result between the degree of agreement and the authentication threshold value during the authentication process.

With this configuration, since the result of comparison between the degree of matching and the authentication threshold value at the time of the authentication process is displayed, the user can confirm whether or not the face image of the user captured at the time of the authentication process is an image that can be easily recognized based on the display.

In the 4 th portable information terminal device according to the present invention, the authentication processing means performs the authentication processing based on a series of operations, and the photographing unit activating means activates the photographing unit based on an operation before the last operation among the consecutive operations.

With such a structure, the photographing unit can be quickly started based on the operation before the last operation. In this case, the photographing unit is brought into the photographable state before the last operation is performed, and the waiting time until the photographing unit is brought into the photographable state can be omitted. Further, if the configuration is adopted in which the authentication process is automatically performed once the imaging means enters the state in which imaging is possible, the authentication process can be completed before the last operation is performed, so that the time required for the authentication process can be further shortened.

A mobile information terminal device according to claim 5 of the present invention includes: and an imaging unit stopping unit configured to stop power supply to the imaging unit when a next operation of the series of operations is not performed for a predetermined time after the imaging unit is started.

With such a configuration, after the photographing unit is started, it is possible to prevent a long-time leaving without starting the authentication process. Since the imaging unit needs to consume a relatively large amount of drive power, the power consumption can be reduced by stopping the drive of the imaging unit without starting the authentication process.

A portable information terminal device according to claim 6 of the present invention includes: and an imaging unit stopping unit that stops power supply to the imaging unit when an operation different from an operation next to the series of operations is performed after the imaging unit is started.

By adopting the above configuration, after the photographing unit is started, it is possible to prevent a long-time leaving without starting the authentication process. Since the imaging unit needs to consume a relatively large amount of driving power, power consumption can be reduced by stopping driving of the imaging unit without starting the authentication process.

According to the portable information terminal device of the present invention, the imaging means can be started up in accordance with the user's operation, and imaging can be performed after the initialization processing is completed, whereby the authentication processing can be automatically performed. Therefore, the trouble of waiting for the time from the start of the imaging unit until the imaging is possible can be avoided, and the labor and time required for the authentication process can be reduced. In addition, since the time delay from the time when the user confirms that the photographing means becomes photographable after starting up to the time when the user performs the photographing operation can be reduced, and the authentication process can be performed in a shorter time after the photographing means starts up, the time required for the authentication process can be shortened.

Further, according to the portable information terminal device of the present invention, when authentication fails in the authentication process, the authentication process is automatically performed again, and the authentication process is repeatedly and automatically performed until it is determined that the user is a genuine user, so that labor and time required for the authentication process are reduced, and time required for the authentication process is shortened. If the result of comparison of the degree of agreement at the time of authentication processing with the authentication threshold value is displayed, the user can confirm whether or not the own face image captured at the time of the authentication processing is an easily recognizable image based on this display.

In addition, according to the portable information terminal device of the present invention, based on the operation performed before the last operation, the photographing unit can be quickly started. The photographing unit is brought into a photographable state when the last operation is performed, and a waiting time until the photographing unit is brought into the photographable state can be omitted. In addition, if a configuration is adopted in which the authentication process is automatically performed once the imaging unit enters a state in which imaging is possible, since the authentication process can be completed before the time when the last operation is performed, the time required for the authentication process is shortened.

In addition, according to the portable information terminal device of the present invention, by stopping the driving of the imaging unit without starting the authentication process, it is possible to prevent the imaging unit from being left for a long time without starting the authentication process after starting the imaging unit, and therefore, it is possible to reduce power consumption.

Drawings

Fig. 1 is an external view showing an example of a portable information terminal device according to embodiment 1 of the present invention, and shows a mobile phone as an example of the portable information terminal device.

Fig. 2 is a block diagram showing an example of the internal configuration of the mobile phone shown in fig. 1.

Fig. 3 is a block diagram showing a configuration example of the imaging unit drive control section of fig. 2.

Fig. 4 is a block diagram showing an example of the configuration of the authentication processing section of fig. 2.

Fig. 5 is a diagram showing an example of display of the main display section when setting relating to face authentication is performed.

Fig. 6 is a diagram showing a display example of the image registration screen for face authentication.

Fig. 7 is a diagram showing a display example of a security level setting screen.

Fig. 8 is a diagram showing a display example of an error history screen.

Fig. 9 is a diagram showing an example of display of the main display section when face authentication is performed.

Fig. 10 is a diagram showing an example of display of a main display section when face authentication is performed according to embodiment 2 of the present invention.

Fig. 11 is a diagram showing an example of display of a main display section when face authentication is performed according to embodiment 3 of the present invention.

Fig. 12 is a timing chart for comparing the time required until the authentication process ends, (a) in the case of the conventional mobile phone, (b) in the case of the mobile phone of embodiment 1, (c) in the case of the mobile phone of embodiment 2, and (d) in the case of the mobile phone of embodiment 3

Description of reference numerals:

1 Portable telephone

101 main display part

102 sub-display unit

103 receiver

104 main camera

105 sub-camera

201 operating key

203 microphone for transmitting

400 control part

401 imaging unit drive control unit

402 authentication processing unit

402A feature extraction unit

402B coincidence degree calculating unit

402C coincidence degree determination unit

403 continuous authentication unit

404 timing part

405 shooting unit starting part

405A Power supply connection part

405B initialization processing unit

406 shooting control part

407 shooting unit stop unit

410 memory

411 display screen data storage unit

412 image data storage unit

413 characteristic information storage unit

414 error history information storage unit

450 communication part

Detailed Description

Example 1

Fig. 1 is an external view showing an example of a portable information terminal device according to embodiment 1 of the present invention, and shows a mobile phone 1 as an example of a portable information terminal device. The mobile phone 1 is a so-called folding mobile phone, and the display housing 100 and the operation housing 200 are connected by the hinge 300, and one surface of the display housing 100 and one surface of the operation housing 200 can be folded in a facing manner.

In the display housing 100, a main display unit 101, a receiver 103, and a sub camera 105 are provided on the inner housing surface when folded, and a sub display unit 102 and a main camera 104 are provided on the outer housing surface. In the operation housing 200, a plurality of operation keys 201 and a microphone 203 for transmission are provided on the housing surface that is the inner side when folded. Such a folding mobile phone 1 can be carried in a folded state, and if the housing is unfolded, the operation keys 201 can be pressed while viewing the display of the main display unit 101. That is, main information display is performed using the main display unit 101, and main operation input is performed using the operation keys 201.

The mobile phone 1 has a function (i.e., a face authentication function) of: when the user operates the operation key 201 while viewing the display of the main display unit 101, the sub-camera 105 captures a face image of the user and performs authentication processing based on feature information extracted from the face image, and as a result, the subsequent processing is permitted only when it is determined that the person who operates the sub-camera at that time is a real user. By performing authentication processing when a function with high confidentiality is executed, it is possible to improve security and protect privacy.

Fig. 2 is a block diagram showing an example of the internal configuration of the mobile phone 1 shown in fig. 1. The control unit 400 is a processor for controlling the main operation of the mobile phone 1, and includes: an imaging unit drive control unit 401, an authentication processing unit 402, a continuous authentication unit 403, and a timer unit 404. The memory 410, the image input/output portion 430, the sound input/output portion 440, and the communication portion 450 are connected to the control portion 400 so as to be capable of input/output, while operation information from the operation keys 201 is input to the control portion 400.

The image input/output unit 430 includes a main display unit 101, a sub display unit 102, a main camera 104, and a sub camera 105. The main camera 104 and the sub camera 105 are each unitized as an imaging unit, and these imaging units include a photoelectric conversion element such as a CCD (Charge-coupled Device) or cmos (complementary Metal Oxide semiconductor) image sensor, and a control circuit. Among them, a photoelectric conversion element is used, which needs to perform initialization processing such as offset or gain adjustment after power is supplied to an imaging unit.

The audio input/output unit 440 includes the receiver 103 and the microphone 203.

The communication section 450 performs transmission and reception of audio data and image data by performing transmission and reception of electric waves with a base station not shown in the figure.

The photographing unit drive control section 401 controls the main camera 104 and the sub camera 105 by sending signals to the image input output section 430. In this embodiment, the imaging unit drive control section 401 controls the drive of the sub-camera 105 to capture a face image of an operator who operates the operation key 201 while viewing the main display section 101 with the sub-camera 105, and performs authentication processing based on feature information extracted from the face image. However, the main camera 104 may capture a face image and perform authentication processing.

The authentication processing unit 402 compares the feature information extracted from the captured image with the feature information (registered feature information) of the user stored in advance in the memory 410, and determines whether or not the operator who performs the operation at this time is a genuine user. The continuous authentication unit 403 continuously performs authentication processing a plurality of times by controlling the imaging unit drive control unit 401 and the authentication processing unit 402. The timer unit 404 measures time and outputs time information.

The memory 410 includes: a display screen data storage unit 411, an image data storage unit 412, a feature information storage unit 413, and an error history information storage unit 414. The display screen data storage unit 411 stores screen data to be displayed on the main display unit 101 or the sub display unit 102. The image data storage unit 412 stores image data captured by the main camera 104 or the sub-camera 105, image data received from the outside via the communication unit 450, and the like. The feature information storage 413 stores feature information of a face image used in the authentication process. When the authentication fails, information at the time of the authentication is stored as error history information in the error history information storage unit 414.

However, the feature information of the face image or the image data of the face image used in the authentication process may be stored in a storage medium (not shown) that is detachable from the mobile phone 1, such as a SIM (Subscriber Identity Module) card.

Fig. 3 is a diagram showing a configuration example of the photographing unit driving control section 401 of fig. 2. The imaging unit drive control unit 401 includes an imaging unit activation unit 405, an imaging control unit 406, and an imaging unit stop unit 407. The imaging unit activation unit 405 is configured to transmit a control signal to the main camera 104 and the sub-camera 105 to activate the cameras, and includes a power supply unit 405A configured to supply power to the imaging unit 106 (the main camera 104 and the sub-camera 105) based on a user operation, and an initialization processing unit 405B configured to perform initialization processing.

The imaging control unit 406 controls the imaging operation of the main camera 104 or the sub-camera 105 based on the input signal from the initialization processing unit 405B and the imaging operation by the user, thereby generating still image data of the subject with respect to the main camera 104 or the sub-camera 105 at this time. The shooting unit stop section 407 stops driving of these shooting units by stopping supply of power to the main camera 104 or the sub camera 105.

Fig. 4 is a diagram showing a configuration example of the authentication processing section 402 of fig. 2. The authentication processing unit 402 includes a feature extraction unit 402A, a matching degree calculation unit 402B, and a matching degree determination unit 402C. At the start of the authentication process, feature information on the shape or size of each part of the face such as the eyes, nose, mouth, and outline is extracted from the face image captured by the sub-camera 105 by the feature extraction section 402A. The matching degree calculation unit 402B compares the feature information extracted by the feature extraction unit 402A with the registered feature information stored in the feature information storage unit 413 of the memory 410, and calculates the matching degree of the number batch.

The matching degree determination unit 402C compares the matching degree calculated by the matching degree calculation unit 402B with a predetermined threshold (authentication threshold), and outputs the comparison result. At this time, if the matching degree exceeds the authentication threshold, it is determined that the user is a genuine user and the authentication is successful, and if the matching degree is less than the authentication threshold, it is not determined that the user is a genuine user and the authentication is failed. Here, the matching degree is a percentage (%) of the total feature information compared, in which matching feature information is matched, and the authentication threshold is usually set to 90% or more of a predetermined value.

Fig. 5 is a diagram showing an example of display of the main display section 101 at the time of setting relating to face authentication. When the user operates the operation key 201 to read out the security setting screen (see fig. 5 a), and selects the "face authentication setting" option from the options displayed on the security setting screen, the main display unit 101 displays a terminal password number assigned in advance to the mobile phone 1 (see fig. 5 b). When the clear key included in the operation key 201 is operated on this display screen, the display of the main display unit 101 returns to the security setting screen shown in fig. 5 (a).

The terminal password number is constituted by an arbitrary character string of 4 to 8 digits, for example, and is set in advance when the mobile phone 1 is purchased, but may be modified by the user operating the operation key 201. In the terminal password number input screen of fig. 5(b), when the character string input by operating the operation key 201 does not match the terminal password number set in advance, the message is displayed on the main display unit 101 (see fig. 5 (c)). When the character string input on the terminal password number input screen is 3 bits or less, a message indicating that a 4-to 8-bit character string should be input is displayed on the main display unit 101 (see fig. 5 (d)).

When the main display unit 101 times out after a predetermined time elapses after the display shown in fig. 5(c) or 5(d) is performed, or when any of the keys included in the operation keys 210 is operated, the display of the main display unit 101 returns to the security setting screen shown in fig. 5 (a).

When the character string input on the terminal password number input screen in fig. 5(b) matches the terminal password number set in advance, the display of the main display unit 101 is switched to the face authentication use setting screen (see fig. 5(e)) for setting on/off of the mode for performing face authentication. When the clear key included in the operation key 201 is operated on this display screen, the display of the main display unit 101 returns to the terminal password number input screen shown in fig. 5 (b). In the face authentication setting screen, when the face authentication is turned off by the operation of the operation key 201, the display of the main display unit 101 returns to the security setting screen shown in fig. 5 (a).

In the face authentication use setting screen of fig. 5(e), when face authentication is turned on by operation of the operation key 201, the display of the main display portion 101 is switched to a face authentication setting screen (refer to fig. 5(f)) for performing setting relating to face authentication.

In the present embodiment, as the registered feature information used at the time of face authentication, feature information may be extracted from a maximum of 7 image data and stored in the memory 410. By performing face authentication based on the registration feature information extracted from at least 2 pieces of image data, it is possible to improve the authentication success rate (principal acceptance rate) in the case where the true user performs face authentication, and prevent authentication failure due to subtle differences in the expression of the user, the irradiation angle of light, or the like. However, the registration feature information for face authentication is not limited to being extracted from 7 pieces of image data, and face authentication may be performed based on the registration feature information extracted from 1 piece of image data, for example.

The face image of the user is preferably captured by the user in advance using the main camera 104 or the sub-camera 105, and the image data is stored in the image data storage section 412 of the memory 410. Since the face authentication is performed based on the registered feature information extracted from the image data and stored in the feature information storage unit 413 of the memory 410, the image data itself is not required for the face authentication, but if the registered feature information is stored in the memory 410, the user can read the image data to confirm the face image, thereby improving the convenience.

The face authentication setting screen in fig. 5(f) includes: options corresponding to images (images for face authentication) representing users who are extraction sources of the registered feature information; an option (security level setting) corresponding to the setting of an authentication threshold value at the time of face authentication; and options corresponding to the error history. On this display screen, when an option corresponding to the image for face authentication is selected, a screen for performing rewriting registration or authentication of the image for face authentication (image registration screen for face authentication) is displayed. When an option corresponding to the security level setting is selected, a screen for setting an authentication threshold value in face authentication (security level setting screen) is displayed. In addition, when an option corresponding to the error history is selected, a screen for displaying the anchor error history (anchor error history screen) is displayed.

After these various settings and the like are made, when the operation key 201 is operated and the face authentication setting is completed, the display of the main display unit 101 is switched to the display shown in fig. 5(g) or 5 (h). That is, if at least one image for face authentication is registered, a prompt for face authentication based on feature information extracted from the image for face authentication is displayed in the main display portion 101 (refer to fig. 5(g)), and when the confirmation key 111 included in the screen is selected, the display of the main display portion 101 returns to the security setting screen shown in fig. 5 (a). On the other hand, when any one of the face authentication images is not registered, a prompt to register the face authentication image is displayed on the main display unit 101 (see fig. 5(h)), and when the confirmation key 112 included in the screen is selected, the display of the main display unit 101 returns to the face authentication setting screen shown in fig. 5 (f).

Fig. 6 is a diagram showing a display example of the image registration screen for face authentication. When any one of the options corresponding to the face authentication image is selected in the face authentication setting screen in fig. 5(f), the main display unit 101 switches the display to the face authentication image registration screen corresponding to the face authentication image as shown in fig. 6 (a). The image registration screen for face authentication includes an option of "rewrite registration" and an option of "confirmation of registered data".

When the "rewrite registration" option is selected from the options displayed on the image registration screen for face authentication in fig. 6(a), the sub-camera 105 is activated, and after the sub-camera 105 becomes in a photographable state, an image of the subject with respect to the sub-camera 105 is displayed in real time on the main display unit 101 (see fig. 6 (b)). At this time, the main display unit 101 functions as a viewfinder of the sub camera 105, and the face of the operator who operates while viewing the main display unit 101 is displayed on the viewfinder constituted by the main display unit 101.

In the viewfinder screen of fig. 6(b), when the user performs registration operation by operating the enter key included in the operation keys 201 after performing adjustment of the position of his/her face, for example, the feature extraction unit 402A of the control unit 400 performs processing for extracting feature information from the image captured by the sub-camera 105 at that time.

At this time, if the position of the face of the user is appropriately adjusted, feature information such as the shape or size of the eyes, nose, mouth, and outline can be extracted, but when the position is not appropriately adjusted and when the face of the user cannot be completely displayed on the main display unit 101, the feature information cannot be extracted. When the feature information extraction is successful, the feature information is stored in the feature information storage 413 of the memory 410, and the face image at that time is stored in the image data storage 412 of the memory 410 as an image for face authentication.

In the case where the feature information extraction is successful, a prompt to store (register) the feature information based on the face image in the memory 410 is displayed in the main display section 101 (refer to fig. 6 (c)). When the user selects the confirmation key 113 included in the display screen, the display of the main display unit 101 returns to the face authentication setting screen shown in fig. 5 (f). On the other hand, in the case where the feature information extraction has failed, the message and the reason for the failure of the feature information extraction are displayed in the main display portion 101 (refer to fig. 6(d)), and when the user selects the confirmation key 114 included in the screen, the display in the main display portion 101 returns to the finder screen shown in fig. 5(b), and the face image can be captured again with the sub-camera 105.

When the "registered data confirmation" option is selected from the options displayed on the image registration screen for face authentication in fig. 6(a), the corresponding image for face authentication stored in the image data storage 412 of the memory 410 is read out and displayed on the main display 101 (see fig. 6 (e)). The user can confirm the registered image for face authentication through the image confirmation screen for face authentication shown in fig. 6 (e). When the clear key included in the operation key 201 is operated on this display screen, the display of the main display unit 101 returns to the image registration screen for face authentication shown in fig. 6 (a).

In the image confirmation screen for face authentication of fig. 6(e), when the corresponding image for face authentication is deleted by the operation of the operation key 201 by the user, the display of the main display unit 101 is switched to a deletion confirmation screen for confirming whether or not the image for face authentication is to be deleted (see fig. 6 (f)). The display screen includes a yes key 115 indicating that the face authentication image should be deleted and a no key 116 indicating that the face authentication image should not be deleted. In the deletion confirmation screen of fig. 6(f), when the no key 116 is selected by the operation of the operation key 201, the display of the main display unit 101 returns to the image confirmation screen for face authentication shown in fig. 6 (e).

When the yes key 115 is selected by the operation of the operation key 201 on the deletion confirmation screen in fig. 6(f), the processing of deleting the corresponding image for face authentication and the registered feature information from the image data storage 412 and the feature information storage 413 of the memory 410 is started, and the deletion is displayed on the main display unit 101 (see fig. 6 (g)). When the process of deleting the image for face authentication and the registered feature information is finished, this message is displayed on the main display unit 101 (see fig. 6(h)), and when a predetermined time elapses, the display on the main display unit 101 is returned to the face authentication setting screen shown in fig. 5(f) in response to an operation of any key included in the operation keys 201.

Fig. 7 is a diagram showing an example of display of the security level setting screen. When an option corresponding to the security level setting is selected on the face authentication setting screen in fig. 5(f), the display of the main display unit 101 is switched to a security level setting screen for setting an authentication threshold value in face authentication as shown in fig. 7. As the security level, the authentication threshold may be set in advance to the highest "level 3", the authentication threshold "level 2" lower than the "level 3", and the authentication threshold "level 1" lower than the "level 2", and any one of the security levels may be selected in the security level setting screen.

Fig. 8 is a diagram showing a display example of an error history screen. When the option corresponding to the error history is selected on the face authentication setting screen in fig. 5(f), the display of the main display unit 101 is switched to an anchor error history screen for displaying the anchor error history as shown in fig. 8. The anchor error history screen displays time information indicating the date and time when the authentication failed as the anchor error history information in a list for a predetermined number of times (for example, 9 times). When a new authentication failure occurs while the anchor error history information is stored in the anchor error history information storage unit 414 of the memory 410 a predetermined number of times, the oldest error history information is deleted, and the new error history information is stored in the memory 410 and updated so that the new error history information is displayed on the error history screen.

Fig. 9 is a diagram showing an example of display of the main display unit 101 when face authentication is performed. In this example, face authentication is performed when switching on/off of the security mode included in the management function of the mobile phone 1. In the security mode on state, not only general data but also data registered in advance as the security data is displayed, and in the security mode off state, the security data is not displayed and only general data is displayed.

When the "privacy mode" option is selected and operated from the options displayed on the management function screen for setting the management function of the mobile phone 1 as shown in fig. 9(a), the sub-camera 105 is started. After the sub-camera 105 enters a state in which the image can be captured, an image of the subject for the sub-camera 105 is displayed in real time on the main display unit 101 (see fig. 9(b)), and automatic authentication is started. At this time, at the point in time when the sub-camera 105 becomes a photographable state after the initialization processing is finished, the face image for authentication is photographed even if the user does not perform a photographing operation, and the feature information is extracted from the photographed image. And automatically performs authentication processing by comparing the extracted feature information with the user feature information stored in advance in the feature information storage part 413 of the memory 410.

In this authentication process, when the degree of matching between the feature information of the extracted face image and the feature information of the user stored in the memory 410 exceeds the authentication threshold value preset in the security level setting screen, it is determined that the operator is a true user. In this case, the face image of the user displayed on the main display unit 101 is highlighted by so-called action focusing, and the success of authentication is notified with a visual effect.

Thereafter, a message indicating that the authentication has succeeded is displayed on the main display unit 101 (see fig. 9 c), and after a predetermined time has elapsed, the screen is switched to a screen for switching the security mode on/off (see fig. 9 d). On the other hand, when the degree of coincidence of the feature information is smaller than the authentication threshold value and the authentication fails, the feature information is automatically extracted again from the image captured by the sub-camera 105 at that point, and the feature information is compared with the feature information of the user stored in the memory 410.

As in the present embodiment, when authentication fails, the authentication process is continuously repeated until a timeout elapses after a predetermined time (for example, 3 seconds) elapses, and the authentication process is terminated when authentication succeeds. For example, when the timeout period is set to 3 seconds, the authentication process is performed 7 times at most. Further, the time required for the 1-time authentication process is about 0.4 seconds, and if the authentication is successful in the 1 st authentication process, the subsequent process can be performed with almost no waiting time after the initialization process of the sub-camera 105 is completed. While the authentication process is being repeated, the main display unit 101 displays a viewfinder screen, and displays an image of the subject facing the sub camera 105 in real time.

When the authentication process is repeated, each time the authentication process is performed a plurality of times, the main display unit 101 displays a matching degree corresponding to the feature information as a result of comparing the feature information in the authentication processes. More specifically, the color of the frame 117 of the finder screen displayed on the main display unit 101 is displayed in a color corresponding to the degree of matching of the feature information. In this way, since the comparison result of the feature information matching degree at the time of the authentication process can be displayed, the user can confirm whether or not the face image of the user captured at the time of the authentication process is an easily recognizable image based on this display.

However, the configuration is not limited to this, and the degree of matching of the feature information may be displayed in a level manner on the main display unit 101 by using a numerical value, a graph, or the like. Further, the comparison result may be displayed for at least one authentication process, without being limited to a configuration in which the comparison result is displayed for all authentication processes.

When a predetermined time elapses after the start of the automatic authentication and the time is exceeded, a message indicating that the authentication has failed is displayed on the main display unit 101 (see fig. 9 (e)). When the user selects the re-authentication key 118 included in this display screen, the display of the main display unit 101 returns to the viewfinder screen shown in fig. 9(b), and automatic re-authentication is started. On the other hand, in fig. 9(e), when the user selects the terminal password number input key 119 displayed on the main display portion 101, the display of the main display portion 101 is switched to the terminal password number input screen facilitating the input of the terminal password number (see fig. 9 (f)).

When the character string input on the terminal password number input screen of fig. 9(f) matches the terminal password number set in advance, the display of the main display portion 101 is switched to a screen for switching on/off of the security mode (see fig. 9 (d)).

On the other hand, in the terminal password number input screen of fig. 9(f), when the character string input by the operation of the operation key 201 does not match the terminal password number set in advance, this presentation is displayed on the main display unit 101 (see fig. 9 (g)). When the character string input on the terminal password number input screen is 3 bits or less, a prompt that 4 to 8 bits of character string must be input is displayed on the main display unit 101 (see fig. 9 (h)). After the display shown in fig. 9(g) or 9(h) is performed, if a time-out occurs after a predetermined time has elapsed or any of the operation keys 201 is operated, the display of the main display unit 101 returns to the management function screen shown in fig. 9 (a).

Example 2

Although the configuration in which the sub-camera 105 is started and the automatic authentication is performed based on the selection of the "privacy mode" option in the management function screen is described in embodiment 1, in embodiment 2, a configuration may be adopted in which: before the operation (final operation) of selecting the privacy mode in which the face authentication is necessary, the sub-camera 105 is started at the time of the selection operation of the management function, the sub-camera 105 is brought into a state in which the image can be captured, and the automatic authentication is directly started based on the operation of selecting the privacy mode.

Fig. 10 is a diagram showing an example of display of the main display unit 101 when performing face authentication according to embodiment 2 of the present invention. In the menu screen shown in fig. 10(a), when the "management function" option is selected from the options displayed on the menu screen, a management function setting screen for setting the management function is displayed (see fig. 10(b)), and by selecting and operating the "security mode" option from the options displayed on the management function screen, a screen for switching on/off of the security mode can be displayed (see fig. 10 (e)). That is, in the present embodiment, by a series of operations of hierarchy, on/off of the security mode can be switched.

In the menu screen of fig. 10(a), when the "management function" option is selected from the options displayed on the menu screen, the sub-camera 105 is started. Thereafter, the display of the main display unit 101 is switched to the management function setting screen (see fig. 10(b)), and when the "privacy mode" option is selected and operated on the management function setting screen, the viewfinder screen is displayed on the main display unit 101, and then the automatic authentication is started (see fig. 10 (c)).

Since the time required for the initialization process of the sub-camera 105, that is, the time required from the start of the sub-camera 105 to the entry into the image-enabled state is about 0.5 seconds, the sub-camera 105 generally enters the image-enabled state before the option of "privacy mode" is selected and operated from the options displayed on the management function screen shown in fig. 10 (b). Accordingly, in response to selection of the "privacy mode" option in the operation management function screen, automatic authentication can be directly started. The processing after the start of the automatic authentication and the display of the main display unit 101 (fig. 10(d) to (i)) are the same as those of the processing after the start of the automatic authentication and the display of the main display unit 101 (fig. 9(c) to (h)) in embodiment 1, and therefore, the description thereof will be omitted here.

In the present embodiment, after the "management function" option in the menu screen of fig. 10(a) is selected and operated and the sub-camera 105 is started, the next operation in the series of operations is executed within a predetermined time, that is, the operation for selecting the "privacy mode" option in the management function screen of fig. 10(b) is not performed, and when time out occurs, the power supply to the sub-camera 105 is stopped to stop the driving of the sub-camera 105.

Further, even when the "management function" option in the menu screen of fig. 10(a) is selected and operated and the sub-camera 105 is started, if an operation different from the next operation in the series of operations is performed, for example, if the clear key included in the operation key 201 is operated and the menu screen is returned, the power supply to the sub-camera 105 is stopped, and the driving of the sub-camera 105 is stopped.

With this configuration, after the sub-camera 105 is started, it is possible to prevent the sub-camera from being left for a long time without starting the authentication process. Since the sub-camera 105 consumes a relatively large amount of drive power, when the next selection operation is not performed within a predetermined time, the drive of the sub-camera 105 is stopped, and power consumption can be reduced.

However, the selection of the "confidential mode" which is the final operation in the series of operations for determining the processing requiring the authentication processing may be performed not only by starting the sub-camera 105 in accordance with the selection of the last selection operation, that is, the "management function", but also by starting the sub-camera 105 in accordance with the previous selection operation in the series of operations.

Example 3

Although the configuration in which the sub-camera 105 is started and the automatic authentication is performed in accordance with the selection of the "privacy mode" option in the management function screen is described in embodiment 1, in embodiment 3, the sub-camera 105 is started at the time of the selection operation of the management function before the selection operation (final operation) of the privacy mode in which the face authentication is necessary, and the automatic authentication is started once the sub-camera 105 is in the photographable state.

Fig. 11 is a diagram showing an example of display of the main display section 101 when face authentication is performed according to embodiment 3 of the present invention. In the menu screen shown in fig. 11(a), when an option to operate the "management function" is selected from the options displayed on the menu screen, a management function setting screen for setting the management function is displayed (see fig. 11(b)), and when an option to operate the "security mode" is selected from the options displayed on the management function screen, a screen for switching on/off of the security mode can be displayed (see fig. 11 (c)). That is, in the present embodiment, the on/off of the security mode can be switched by a series of operations of the hierarchy.

When the "management function" option is selected and operated from among the options displayed on the menu screen in fig. 11(a), the sub-camera 105 starts automatic authentication after the initialization processing is completed and the shooting enabled state is entered. That is, according to the end of the initialization process of the sub-camera 105, the user automatically extracts feature information from the photographed face image without performing a photographing operation, and compares the feature information with the feature information of the user stored in advance in the feature information extracting section 413 of the memory 410, and automatically performs an authentication process.

In the present embodiment, as in the case of embodiment 1, when the degree of coincidence of the feature information is smaller than the authentication threshold value and the authentication fails, the feature information is automatically extracted again from the image captured by the sub-camera 105 at that time, the feature information is compared with the feature information of the user stored in the memory 410, the authentication process is continuously repeated until a time-out elapses after a predetermined time (for example, 3 seconds) elapses, and the authentication process is ended at the time when the authentication succeeds. However, in the present embodiment, unlike embodiment 1, the main display portion 101 does not display the viewfinder screen while the authentication process is repeatedly performed.

Since the time required for the 1-time authentication process is about 0.4 seconds, if the authentication is successful in the 1 st authentication process, the automatic authentication is normally completed until the user selects and operates the "confidential mode" option from among the options displayed on the management function screen shown in fig. 11 (b). Therefore, when the authentication is successful, the display of the main display portion 101 is directly switched to a screen for switching on/off of the security mode in accordance with a selection operation of the "security mode" option in the management function screen (refer to fig. 11 (c)).

However, in the case where authentication fails and authentication processing is performed a plurality of times in the authentication processing of the 1 st time, if the authentication processing has not been completed at the time when the user selects the operation of the "privacy mode" option, the next processing may be performed after waiting for the completion of the authentication processing after selecting the operation of the "privacy mode" option.

When authentication fails in all the authentication processes repeated within a predetermined time, a terminal password number input screen urging input of a terminal password number is displayed on the main display unit 101 after selecting an operation "confidential mode" option from options displayed on the management function screen shown in fig. 11(b) (see fig. 11 (d)).

When the character string input on the terminal password number input screen in fig. 11(d) matches the terminal password number set in advance, the display of the main display unit 101 is switched to a screen for switching on/off of the security mode (see fig. 11 (c)).

On the other hand, in the terminal password number input screen of fig. 11(d), when the character string input by the operation of the operation key 201 does not match the terminal password number set in advance, this presentation is displayed on the main display unit 101 (see fig. 11 (e)). When the character string input on the terminal password number input screen is 3 bits or less, a message indicating that a 4-8-bit character string should be input is displayed on the main display unit 101 (see fig. 11 (f)). After the display shown in fig. 11(e) or 11(f) is performed, when a predetermined time elapses and a timeout occurs or any one of the operation keys 201 is operated, the display of the main display unit returns to the menu screen shown in fig. 11 (a).

However, the selection of the "confidential mode" as the final operation in the series of operations for deciding to perform the process requiring the authentication process is not limited to the starting of the still camera 105 according to the selection of the last selection operation, that is, the "management function", but the starting of the still camera 105 may be performed according to the previous selection operation in the series of operations.

In addition, a viewfinder screen may be displayed in the main display portion 101 during the automatic authentication. In this case, it is preferable that a viewfinder screen is displayed in a part of the display screen of the main display unit 101 without hindering a series of operations.

Fig. 12 is a timing chart for comparing the time required until the authentication process is completed, (a) in the case of the mobile phone of the related art, (b) in the case of the mobile phone of embodiment 1, (c) in the case of the mobile phone of embodiment 2, and (d) in the case of the mobile phone of embodiment 3.

In embodiments 1 to 3, the sub-camera 105 is started based on the selection operation of the "confidential mode" (fig. 12(b)) or the selection operation of the "management function" (fig. 12(c), (d)) by the user, and the authentication process can be automatically performed after the sub-camera 105 enters the image-enabled state after the initialization process is completed (fig. 12(b), (d)) or after the selection operation of the "confidential mode" (fig. 12 (c)). That is, the user does not need to perform a photographing operation after the sub-camera 105 is started, but automatically extracts feature information from an image photographed by the sub-camera 105 and compares the feature information with feature information of the user stored in advance to perform an authentication process. Therefore, as shown in the related art shown in fig. 12(a), it is possible to avoid the trouble that the shooting operation is performed after the sub-camera 105 is started up until it enters the photographable state, and therefore, the labor and time required for the authentication process are reduced.

Further, as in the conventional example shown in fig. 12(a), since the time delay TL that occurs after the user confirms that the sub-camera 105 is activated and can perform shooting until the shooting operation is performed can be reduced and the authentication process can be performed in a shorter time after the sub-camera 105 is activated, the time required for the authentication process is shortened.

In embodiment 1 shown in fig. 12(b), the sub-camera 105 is automatically started based on the selection operation of the final operation, i.e., the "confidential mode", in embodiments 2 and 3 shown in fig. 12(c) and (d), based on the selection operation of the "management function" performed before the final operation, and the authentication process is performed, wherein the series of operations is a process for determining that the authentication process is required to be performed. Therefore, the sub-camera 105 can be started earlier based on the operation performed before the final operation.

In particular, in embodiments 2 and 3 shown in fig. 12(c) and (d), since the sub-camera 105 is started up by the selection operation based on the "management function" performed before the final operation, the sub-camera 105 enters the image-enabled state until the final processing is performed, and the waiting time TS until the sub-camera 105 is in the image-enabled state can be omitted (see fig. 12 (a)).

In example 3 shown in fig. 12(d), since the sub-camera 105 performs the authentication process automatically once in the image-pickup enabled state, the authentication process can be terminated in advance until the final operation is performed, and the time required for the authentication process can be further shortened.

As in embodiments 1 to 3, if the authentication process is automatically performed again when the authentication fails in the authentication process, the authentication process is performed again without the user performing the photographing operation even if the authentication process 1 is not determined to be a genuine user when the genuine user performs the operation. That is, since the authentication process is automatically repeated until it is determined to be a genuine user, the labor and time required for the authentication process can be reduced, and the time required for the authentication process can be shortened.

When the face image is captured during the authentication process, it is preferable not to generate shutter sound. With the above configuration, when the authentication process is performed in an environment where people are around the bus, for example, it is possible to prevent the sound of the shutter from being mistaken for a candid photograph or the like. In the present embodiment, the photographing operation is not performed at the time of the authentication processing, so that it is possible to prevent the photographing operation from being mistaken for the candid photograph or the like.

However, if it is not determined that the user is a genuine user within the predetermined time, the authentication process may be repeated before the authentication is successful, without being limited to the timeout. The authentication process is not limited to being continuously performed, and may be performed only 1 time after the sub-camera 105 is started and in a photographable state.

The face authentication is not limited to the case of switching on/off the secret mode, and may be performed in other functions with high security and privacy, such as card setting, incoming call restriction, lock setting, password number change, sender number notification/non-notification setting, setting reset, software update, display of the own station number in detail, and various setting operations related to other data changes.

The authentication process is not limited to face authentication based on a face image, and may be performed based on a fingerprint image of a user if the authentication process is an image representing the user. In this case, the feature information of the fingerprint image may be extracted and compared, thereby performing the authentication process.

The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the claims.

The present application claims priority from japanese patent application (patent application 2005-) -220085, 29/7/2005 in accordance with the paris convention, the entire contents of which are incorporated herein by reference.

Claims (5)

1. A portable information terminal device is characterized by comprising:

an imaging unit that images an object;

a shooting unit starting part which supplies power to the shooting unit based on the operation of a user and performs initialization processing of the shooting unit;

an imaging control unit that controls an imaging operation performed by the imaging unit;

a feature information storage unit that stores feature information extracted from the face image as registered feature information in advance;

an authentication processing unit that performs authentication processing based on a captured image of the imaging unit using the registration feature information; and

an operation input unit for selecting any one of a plurality of selection items displayed on a screen based on a user operation,

the authentication processing means starts an authentication process based on a series of operations consisting of a plurality of the user operations,

the photographing unit activating means activates the photographing unit based on an operation performed before a selection operation for which face authentication is necessary among the series of operations,

the shooting control means automatically starts shooting in accordance with the end of the initialization processing of the shooting means by the shooting means starting means.

2. A portable information terminal device according to claim 1, comprising:

and a continuous authentication unit that controls the imaging control unit and the authentication processing unit to perform imaging again when authentication in the authentication processing fails, and performs authentication processing based on the captured image.

3. A portable information terminal device according to claim 2, comprising:

an authentication result display unit for displaying the result of the authentication process,

the authentication processing means includes: a feature extraction unit that extracts feature information from the captured image; a matching degree calculating unit that compares the feature information extracted by the feature extracting unit with the registered feature information stored in the feature information storing unit and calculates a digitized matching degree; and a coincidence degree judging section for comparing the coincidence degree with an authentication threshold value,

the authentication result display means displays a display corresponding to the degree of coincidence in each authentication process.

4. A portable information terminal device according to claim 1, comprising:

and an imaging unit stopping unit that stops power supply to the imaging unit when an interval between the user operations in the series of operations exceeds a predetermined time after the imaging unit is started.

5. A portable information terminal device according to claim 1, comprising:

and an imaging unit stopping unit that stops power supply to the imaging unit when an operation different from the series of operations is performed after the imaging unit is started.