JP7531190B2 - Facial recognition device and facial recognition method - Google Patents

Description

This disclosure relates to a facial recognition device and a facial recognition method that processes information regarding payment of usage fees based on the results of facial recognition of a subject.

Conventionally, for example, a gate device is known that includes a reading unit that receives an identifier transmitted from a wireless tag (remote IC card) carried by a user, an acquisition unit that acquires facial features registered in correspondence with the identifier received from the wireless tag, an imaging unit that images the user, an extraction unit that extracts facial features from image data captured by the imaging unit, a face matching unit that checks whether the facial features extracted by the extraction unit match those acquired by the acquisition unit, and an opening/closing control unit that opens or closes the gate exit depending on the result of the matching by the face matching unit (see Patent Document 1).

International Publication No. WO2018/181968

The above-mentioned conventional technology makes it possible to narrow down the subjects of facial recognition (i.e., registered facial features) using the identifier transmitted from the user's wireless tag. However, since users passing through the gate must always carry their IC card, it becomes difficult to securely manage personal information (i.e., information that can identify a specific individual) contained in the IC card. Furthermore, the above-mentioned conventional technology does not take into consideration a mechanism for storing both the facial image data and the facial feature data on a cloud server or the like and managing the data securely.

In addition, in the above-mentioned conventional technology, the gate device executes a process to verify whether the facial features extracted by the extraction unit match those acquired by the acquisition unit, but no consideration is given to a mechanism for easily performing face authentication processing when a user makes a payment at a facility.

Therefore, the main objective of this disclosure is to provide a facial recognition device and a facial recognition method that can safely manage the facial images of users who use a facility while easily performing facial recognition when users make payments.

The facial recognition device disclosed herein is a facial recognition device that works in cooperation with a payment terminal device, which is an external device, and makes a request for facial recognition processing to a face matching server that performs facial authentication of a facility user when the user pays a facility fee, and completes information processing regarding the payment of the fee based on the authentication result.The facial recognition device includes a memory unit that stores information regarding the matching group to which the device belongs, a facial image acquisition unit that photographs the user and acquires a facial image of the user for authentication when the fee is paid, a transmission unit that transmits the facial recognition processing request to the face matching server whose matching group matches the facial image for authentication to match the facial image for authentication with the facial image of a user whose matching group matches, a receiving unit that receives the authentication result returned from the face matching server , and a control signal transmission unit that transmits the authentication result to the external device as a control signal, and is configured such that a processor of the external device receives the control signal and reflects it in information processing regarding the payment.

In addition, the facial recognition method disclosed herein is a facial recognition method that links a facial recognition device and a payment terminal device, which is an external device, and makes a facial recognition processing request from the facial recognition device to a face matching server that performs facial authentication of facility users when the user pays a facility fee at the facility, and completes information processing regarding the payment of the fee based on the authentication result.The facial recognition method stores information regarding the matching group to which the facial recognition device belongs, and when the fee is paid, photographs the user and obtains a facial image for authentication of the user, sends the facial recognition processing request to the face matching server whose matching group matches the facial image for authentication to match the facial image for authentication with the facial image of a user whose matching group matches, receives the authentication result returned from the face matching server , and sends the authentication result to the external device as a control signal, and a processor of the external device receives the control signal and reflects it in information processing regarding the payment.

According to the present disclosure, it is possible to safely manage the facial images of users who use a facility while easily performing facial authentication when the users make a payment.

FIG. 1 is an overall configuration diagram of a face authentication system according to an embodiment of the present invention. An explanatory diagram showing examples of events that occur at a facility An overview of match groups FIG. 1 is an explanatory diagram showing an overview of a face matching inquiry performed by the face authentication device 1. An explanatory diagram showing an overview of the face matching unit A block diagram showing a schematic configuration of a face recognition device 1. FIG. 2 is a block diagram showing a schematic configuration of a management terminal 2. FIG. 2 is a block diagram showing a schematic configuration of a registration device 4. A block diagram showing a schematic configuration of a face management server 5. FIG. 1 is a block diagram showing a schematic configuration of a face matching server 6. A block diagram showing a schematic configuration of an external device. FIG. 1 is an explanatory diagram showing an overview of a face image extraction process; An explanatory diagram showing an overview of face recognition inquiry based on matching group information FIG. 13 is an explanatory diagram showing an example of a registration screen displayed on the registration device 4. A sequence diagram showing the steps of the user registration process FIG. 1 is a sequence diagram showing a first example of a process for face authentication of a user. FIG. 11 is a sequence diagram showing a second example of a process for face authentication of a user. FIG. 1 is an explanatory diagram showing an example of an authentication screen displayed on the face authentication device 1. FIG. 11 is a sequence diagram showing a third example of a process for face authentication of a user. FIG. 13 is an explanatory diagram showing a login screen displayed on the management terminal 2. Screen displayed on the management terminal 2 when registering, referencing, updating, and deleting associations between the face authentication device 1 and a matching group Screen displayed on the management terminal 2 when registering, referencing, updating, and deleting associations between the face authentication device 1 and a matching group FIG. 13 is an explanatory diagram showing an example of a user management screen displayed on the management terminal 2. FIG. 13 is an explanatory diagram showing an example of a conference room search screen displayed on the management terminal 2. FIG. 13 is an explanatory diagram showing an example of an attendee display screen displayed on the management terminal 2. FIG. 13 is an explanatory diagram showing an example of a personal relationship search screen displayed on the management terminal 2.

The first invention made to solve the above problem is a face authentication device that cooperates with a payment terminal device, which is an external device, and makes a request for processing the face authentication to a face matching server that performs face authentication of a user of a facility when the user pays a usage fee at a facility, and completes information processing regarding the payment of the usage fee based on the authentication result.The face authentication device includes a memory unit that stores information regarding the matching group to which the device belongs, a face image acquisition unit that photographs the user and acquires a face image for authentication of the user when the usage fee is paid, a transmission unit that transmits the face authentication processing request to the face matching server whose matching group matches the face authentication image to match the face image for authentication with the face image of a user whose matching group matches , a receiving unit that receives the authentication result returned from the face matching server , and a control signal transmission unit that transmits the authentication result to the external device as a control signal, and is configured so that a processor of the external device receives the control signal and reflects it in information processing regarding the payment.

With this system, facial authentication of users is performed by a facial authentication server, and the processor of the external device receives the control signal and reflects it in the information processing related to the payment, so facial images of users who use the facility can be safely managed while easily authenticating their faces when they make payments at the facility.

The second invention is also configured to include a power supply unit that receives power from the payment terminal device.

This makes it possible to receive power supply with a simple configuration. Furthermore, if the payment terminal device is configured to be connected to an uninterruptible power supply (or emergency power supply), the payment terminal device can continue to operate normally even if a power outage occurs within the facility, making it possible to maintain the security of the facility.

The third invention further includes a storage unit that stores the face image for authentication, and the face image for authentication stored in the storage unit is deleted after the face authentication process is completed.

As a result, even if a facial recognition device is attached to a payment terminal device used by unspecified persons, it is possible to prevent facial images from being stored in the facial recognition device for a long period of time, and to manage users' facial images more safely.

In addition, a fourth invention is a facial recognition method in which a facial recognition device and a payment terminal device, which is an external device, are linked, and the facial recognition device makes a facial recognition processing request to a face matching server that performs facial authentication of facility users when the user pays a facility fee at the facility, and completes information processing regarding the payment of the fee based on the authentication result. The method is configured to store information regarding the matching group to which the facial recognition device belongs, and when the fee is paid, the user is photographed and a facial image of the user for authentication is obtained, and a facial authentication processing request for matching the facial image for authentication with a facial image of a user whose matching group is sent to the face matching server whose matching group is received , the authentication result returned from the face matching server is received, and the authentication result is sent to the external device as a control signal, and a processor of the external device receives the control signal and reflects it in information processing regarding the payment.

With this system, facial authentication of users is performed by a facial authentication server, and the processor of the external device receives the control signal and reflects it in the information processing related to the payment, so that facial images of users who use the facility can be safely managed while facial authentication can be easily performed when users enter and leave the facility.

The following describes an embodiment of the present disclosure with reference to the drawings.

Figure 1 is a diagram showing the overall configuration of a face authentication system according to this embodiment.

This face authentication system includes a face authentication device 1, a management terminal 2, a face authentication server 3 (cloud server), and a registration device 4 (face authentication registration device). The face authentication server 3 includes a face management server 5 (face authentication management server) and a face matching server 6.

The face authentication device 1, management terminal 2, registration device 4, face management server 5, and face matching server 6 are connected via a network such as the Internet. A face authentication system is constructed for each face authentication service provider (business operator). Multiple face authentication devices 1 are installed at each location where face authentication is required, such as building entrances and room entrances. The registration device 4 is placed at a reception counter that receives visitors to the facility, etc. The required number of face matching servers 6 are provided depending on the number of face authentication devices 1, etc.

The facial recognition device 1 is equipped with a camera 11, which captures images of users (visitors to the facility, employees of companies or stores within the facility, etc.). The facial recognition device 1 also has a display 12, which displays the facial recognition results acquired from the face matching server 6 and notifies the user of the facial recognition results.

The management terminal 2 is operated by an administrator of the face authentication system, and is configured as a PC with a management application installed that manages the operations of the face authentication device 1, the registration device 4, the face management server 5, and the face matching server 6. This management application allows the administrator to perform various management tasks. The management application is implemented as a Web application. For the administrator, multiple administrator groups are set up, each of which has different access rights to each piece of information in the face authentication system.

The registration device 4 acquires information used in the process of facial authentication of users (hereinafter referred to as authentication information). The authentication information may include not only the facial image for registration of each user, but also specific information other than the facial image that is used to identify the user.

Examples of the specific information that can be used include information displayed on the user's business card, information displayed on a payment card (e.g., a credit card or cash card) held by the user, and information indicating the user's location within the facility. The specific information may include information that can permanently identify the user, such as the user's name. However, the specific information may also be information that can temporarily identify the user, such as a user number assigned to a user of the facility, the number of a conference room (including any space available for interviews) used by the user within the facility, or the number of the room in which the user is staying at a lodging facility, etc.

The face management server 5 centrally manages user information (including personal information). Specifically, the face management server 5 associates and accumulates data of facial images for registration and specific information for each user, and manages them. The face management server 5 acquires the facial images for registration and the specific information from the registration device 4 when registering a user. The face management server 5 may also acquire at least a portion of the information contained in the facial images for registration and the specific information from the management terminal 2 (i.e., the administrator).

During face authentication, the face matching server 6 acquires face image data for authentication of the person to be authenticated from the face authentication device 1, generates face feature data of the person from the face image data, and performs face matching by comparing the face feature data of the person to the face feature data of a registered person (registered user) stored in the device, thereby performing face authentication to determine whether the person is a registered person or not.

Furthermore, prior to face authentication, when a user is registered, the face matching server 6 acquires face image data for registration of the user from the face management server 5 (or the registration device 4), generates face feature amount data of the user from the face image data, and stores the data in its own device. In some cases, the face matching server 6 can also acquire a photographed image of the user from the management terminal 2 (i.e., the administrator), and acquire face image data from the photographed image.

In this embodiment, the face management server 5 (face image management unit) and the face matching server 6 (face image matching unit) are located in physically different information processing devices, but they may also be located in a single information processing device.

In addition, in this embodiment, the management terminal 2 and the face management server 5 are provided, but the management terminal 2 and the face management server 5 can be configured as a single information processing device. For example, by installing a management application on the face management server 5, the face management server 5 can also serve as the management terminal 2. In addition, an administrator (e.g., an employee of a company or store within the facility) with limited access rights to the information of the face authentication system can access the face management server 5 from his/her own information processing terminal (e.g., a PC or tablet) and refer to information that needs to be kept secret less than the user's personal information (e.g., information about a conference attended by the user, which will be described later, and information about the user's personal connections).

In addition, in this embodiment, facial feature matching is performed, but the facial matching is not limited to facial feature matching, and a matching method that applies machine learning or the like may be used. Furthermore, this embodiment can also be applied to biometric authentication other than facial recognition.

Next, we will explain examples of events that occur due to actions of users of a facility to which a face recognition system is applied. Figure 2 is an explanatory diagram showing examples of events that occur in the facility.

As shown in FIG. 2, a user 10 of the facility can use a registration device 4 located at the reception desk to register as a user of the facility (i.e., to register information to be used in face matching processing). For registered users who revisit the facility within a specified period of time, registration using the registration device 4 can be omitted.

In the facility, face recognition is performed for each of the various events (see the area within the two-dot chain line in Figure 2) that occur as a result of the actions of people (including registered users) within the facility.

For example, a registered user 10 can pass through a security gate that manages the passage of the user 10 within a facility by undergoing facial authentication in front of the security gate. A facial authentication device 1 that works in conjunction with a gate device 16 (i.e., controls the open/close state of the gate) can be installed at the security gate.

For example, a registered user 10 can enter a conference room in a facility by undergoing facial authentication in front of the conference room. A facial authentication device 1 can be placed in front of the conference room, which works in conjunction with a locking device that locks the door of the conference room (i.e., controls the locking and unlocking of the door). For example, in a lodging facility such as a hotel, a facial authentication device 1 can be installed that works in conjunction with a locking device on the door of the room where the user is staying, instead of a conference room.

For example, a registered user 10 can undergo facial recognition in a restaurant in the facility, allowing automatic payment of the restaurant's charges. The restaurant is equipped with a camera 11 capable of photographing the face of the user 10, and the facial recognition machine 1 works in conjunction with a payment terminal device (or payment system) used to pay the restaurant charges.

For example, a registered user 10 can undergo facial recognition at a gym within the facility, allowing automatic payment of the gym's usage fee. The gym is equipped with a camera 11 capable of photographing the user's face, and the facial recognition machine 1 works in conjunction with a payment terminal device (or payment system) used to pay the gym usage fee.

Events that occur within a facility are not limited to the above examples, but may include various other occurrences, such as payment of fees for the facility's parking lot, payment of fees for the laundry service within the facility, and payment of fees for the bathhouse within the facility.

In addition, for example, if employees of a company that provides services within a facility are registered as users, security in the back yard can be ensured by placing the facial recognition device 1 at the entrance/exit of the facility's back yard (e.g., a warehouse, workshop, etc.).

Next, we will explain the comparison group. Figure 3 is an explanatory diagram showing an overview of the comparison group.

In this embodiment, users are grouped according to the installation location of the face authentication device 1 where the users undergo face authentication, and a matching group is set for each user. Similarly to the users, face authentication devices 1 are grouped according to the installation location of the face authentication device 1, and a matching group is set for each face authentication device 1. Furthermore, face matching servers 6 corresponding to the face authentication devices 1, i.e., face matching servers 6 that accept inquiries (requests) for face matching from the face authentication devices 1, are grouped in the same way as the face authentication devices 1, and a matching group is set for each face matching server 6.

For example, in a face recognition device 1 installed at the entrance of a multi-tenant building housing multiple companies, users belonging to multiple companies undergo face recognition. However, information on users belonging to different organizations cannot be managed in the same database. In such cases, therefore, a matching group is formed for each of the multiple companies, and the registered information of users is stored on a matching group basis, and the face recognition device 1 makes a face matching inquiry to the face matching server 6 of the matching group for each company.

A user can only be authenticated by a face recognition device 1 that corresponds to the matching group to which the user belongs. Therefore, even a registered user cannot be authenticated by a face recognition device 1 that does not belong to the user's matching group, for example, a face recognition device 1 in a building that the user is not permitted to enter. For example, if a user who works at one business location visits another business location, the user cannot be authenticated by the face recognition device 1 at that business location.

Next, we will explain the face matching inquiry from the face authentication device 1 to the face matching server 6. Figure 4 is an explanatory diagram showing an overview of the face matching inquiry made by the face authentication device 1.

The face authentication device 1 makes a face matching inquiry to a face matching server 6 that belongs to the same matching group as the device itself, and sends a face matching request (face authentication processing request) to the face matching server 6 that is the inquiry destination (request destination). The face authentication device 1 also holds the network address (IP address) of the face matching server 6 as matching group information, and makes a face matching inquiry to the face matching server 6 that corresponds to the device itself based on this network address.

The face recognition inquiry method includes a simultaneous inquiry shown in FIG. 4(A), a random inquiry shown in FIG. 4(B), and a sequential inquiry (switching inquiry destination) shown in FIG. 4(C).

As shown in FIG. 4(A), in the case of a simultaneous inquiry, the face authentication device 1 makes a face matching inquiry to the face matching servers 6 corresponding to each of the multiple matching groups at the same time. For example, if the face authentication device 1 is installed at the entrance of a multi-tenant building and users of multiple matching groups undergo face authentication, the face authentication device 1 sends a face matching request to the face matching servers 6 of each group at the same time.

In order to reduce the load on one face matching server 6, it is also possible to split and store the facial feature amount data of users belonging to one matching group among multiple face matching servers 6. In this case, too, the face authentication device 1 simultaneously queries all face matching servers 6 in the same matching group as itself and that have different registered contents.

As shown in FIG. 4(B), in the case of a random inquiry, the face authentication device 1 randomly selects a face matching server 6 to be queried for face matching from among the multiple face matching servers 6 in the same matching group as itself. The multiple face matching servers 6 have the same registered contents. In other words, the same user is the target of matching, and the facial feature amount data of the same user is stored. In this inquiry method, inquiries from the face authentication device 1 are distributed to each face matching server 6, so the load on each face matching server 6 can be reduced and the load can be distributed.

As shown in FIG. 4(C), in the case of sequential inquiry, the face authentication device 1 sequentially selects the inquiry destination for face matching from among the multiple face matching servers 6 in the same matching group as itself. Specifically, an order (priority) is assigned to the face matching servers 6, and the face authentication device 1 selects the face matching server 6 according to that order, and if there is no response from the face matching server 6 with a higher order, the transmission destination is switched to the face matching server 6. That is, the face authentication device 1 selects the face matching server 6 with the first order in the matching group and requests face authentication from that face matching server 6. Then, if there is no response from the face matching server 6, the face authentication device 1 selects another face matching server 6 with the next order in the same matching group and requests face authentication from that face matching server 6. This type of control allows for redundancy (backup).

Next, we will explain the face matching unit. Figure 5 is an explanatory diagram showing an overview of the face matching unit.

The face matching server 6 can be provided with multiple face matching units that perform face matching. The face matching units have a face matching process that executes a face matching program, and a face matching database in which information (facial feature data) of users to be compared in face matching is registered.

This face matching unit is provided for each matching group, and performs face matching in response to a face matching request from a face authentication device 1 that belongs to the matching group. Therefore, one face matching server 6 can handle multiple matching groups.

Here, the face authentication device 1 holds the network address (IP address) of the face matching server 6 as matching group information, and can make a face matching inquiry to the face matching server 6 corresponding to the device itself based on this network address. Furthermore, in this embodiment, the face authentication device 1 holds a network address (IP address) for each face matching unit, and can make a face matching inquiry to the face matching unit corresponding to the device itself based on this network address.

In addition, since a face matching database is provided for each matching group, the facial feature data of the user can be stored on a group-by-group basis. The face matching process executes a matching process between the facial feature data stored on a group-by-group basis and facial feature data generated from facial image data acquired from a face authentication device 1 of the same group.

Here, there are various modes of correspondence between the face recognition device 1 and the face matching unit.

In the example shown in FIG. 5(A), the face matching server 6 is provided with one face matching unit of the matching group to which the face authentication device 1 belongs, and the face authentication device 1 and the face matching server 6 have a one-to-one correspondence.

In the example shown in FIG. 5(B), the face authentication device 1 and the face matching server 6 have a one-to-one correspondence, but the face matching server 6 is provided with two face matching units for the matching group to which the face authentication device 1 belongs.
In this case, if the face recognition device 1 is installed at the entrance of a multi-tenant building, information of users who belong to different organizations (e.g., user A, user B) can be managed in different matching databases.

In the example shown in FIG. 5(C), the face authentication device 1 and the face matching server 6 correspond in a one-to-two relationship, and the face authentication device 1 makes face matching inquiries (user matching requests) to the two face matching servers 6. In this case, by providing two or more identical face matching units, i.e., face matching units that perform face matching using the facial feature amount data of the same user, on different face matching servers 6, it is possible to distribute the load on the face matching servers 6 and handle life and death situations.

Next, we will explain the general configuration of the face recognition device 1. Figure 6 is a block diagram showing the general configuration of the face recognition device 1.

The face recognition device 1 includes a camera 11 (face image acquisition unit), a display 12 (display unit), a communication unit 13 (transmission unit, reception unit), a memory unit 14, a control unit 15, and a control signal transmission unit 17.

The camera 11 constantly captures images of a specified shooting area, and when a person enters the shooting area, the person is photographed, and a captured image of the person (including registered users) can be obtained. To save power, a human presence sensor (not shown) may be provided to detect the approach of a person and activate the camera 11.

The display 12 displays the progress of the facial recognition and the facial recognition results, so that the person undergoing facial recognition can check them. A speaker may be provided as an output unit that outputs the authentication results of the subject, so that the facial recognition results are notified by voice. Also, the display 12 may be omitted depending on the installation location of the facial recognition device 1 (for example, when the facial recognition device 1 is installed at a security gate).

The communication unit 13 communicates with the face matching server 6 via the network. In this embodiment, it transmits face image data to the face matching server 6. It also receives the authentication result of the user from the face matching server 6. The communication unit 13 also communicates with the face management server 5 via the network. In this embodiment, it receives matching group information and the like from the face management server 5.

The memory unit 14 stores data of the captured image (face image) of the user, matching group information, a control program executed by the processor constituting the control unit 15, etc.

The control unit 15 includes a startup processing unit 21, a face image extraction unit 22, a face matching request unit 23, an authentication result notification unit 24, an authentication result adjustment unit 25 (history information generation unit), and an operation status monitoring unit 26. The control unit 15 is configured with a processor, and each unit of the control unit 15 is realized by the processor executing a program stored in the storage unit 14.

When the face authentication device 1 is started, the startup processing unit 21 acquires the matching group information from the face management server 5 and stores it in the memory unit 14. Note that the acquisition of the matching group information is not limited to this, and the matching group information may be distributed from the face management server 5 to the target face authentication device 1 in conjunction with an update of the matching group information, etc.

The face image extraction unit 22 acquires a photographed image of the subject of face authentication from the camera 11 (camera image capture), detects the person's face from the photographed image (face detection), determines whether the size of the detected face is appropriate (face size check), cuts out the face area from the photographed image (face cut-out), and acquires data of the subject's face image (face image for authentication). Note that the face image data may be data of only the image of the face area, but may also be a combination of data of the photographed image (image of a specified photographed area) and position information of the face area on the photographed image data (face frame information).

The face matching request unit 23 transmits a face matching request via the communication unit 13 to the face matching server 6 whose matching group matches that of the own device.

The authentication result notification unit 24 notifies the user of the face authentication result by displaying the face authentication result obtained from the face matching server 6 via the communication unit 13 on the display 12.

The authentication result adjustment unit 25 performs control for linking with the external device 16 based on the user's face matching result obtained from the face matching server 6. Examples of the external device 16 that can be used include a gate device (see FIG. 2) that manages the passage of users through the facility, a door locking device that manages entry and exit to a conference room or a specified area in the facility, and a payment terminal device for making payments for services received or purchased items by users within the facility. When linking with a door locking device of a conference room, the authentication result adjustment unit 25 can generate entry history information that associates multiple users who have entered the conference room.

The face authentication device 1 and the external device 16 are directly connected by a communication cable or the like, or are connected so as to be able to communicate with each other via a known network. The authentication result adjustment unit 25 is configured as a connection application, and can notify the user's face matching result to a server of an external system including the external device 16.

The operation status monitoring unit 26 monitors the operation status of the device itself and notifies the face management server 5 of the operation status of the device itself.

The control signal transmission unit 17 transmits a control signal to the external device 16 to control its operation.

Next, we will explain the management terminal 2. Figure 7 is a block diagram showing the general configuration of the management terminal 2.

The management terminal 2 includes a communication unit 31, a display 32 (display unit), an input device 33 (operation unit), a memory unit 34, and a control unit 35.

The communication unit 31 communicates with the face management server 5 via the network. In this embodiment, it receives screen information and the like from the face management server 5, and transmits corresponding operation information of the administrator to the face management server 5.

The display 32 displays various screens. The input device 33 is a mouse, keyboard, etc., and is used to operate the screens displayed on the display 32.

The memory unit 34 stores programs (management applications) and the like that are executed by the processor constituting the control unit 35.

The control unit 35 includes a GUI control unit 38. The control unit 35 is configured with a processor, and each part of the control unit 35 is realized by the processor executing a program (management application) stored in the memory unit 34.

The GUI control unit 38 displays various operation screens delivered from the face management server 5 on the display 32. In addition, in response to the input operation of the administrator using the input device 33, it acquires input information and controls the screen. In this embodiment, the GUI control unit 38 performs display input control on a screen related to login, specifically, a login screen. In addition, the GUI control unit 38 performs display input control on a screen related to user management, specifically, a screen related to registration (individual registration, batch registration), reference, update, and deletion of user information. In addition, the GUI control unit 38 performs display input control on a screen related to matching group management, specifically, a screen related to registration (individual registration, batch registration), reference, update, and deletion of matching groups. In addition, the GUI control unit 38 performs display input control on a screen related to authentication device management, specifically, a screen related to registration, reference, update, and deletion of association between authentication devices and matching groups. In addition, the GUI control unit 38 performs display input control on a screen for reference of authentication logs (face authentication history information).

Next, we will explain the general configuration of the registration device 4. Figure 8 is a block diagram showing the general configuration of the registration device 4.

The registration device 4 includes a camera 41A for photographing a face, a camera 41B for acquiring information (personal information acquisition unit), a display 42 (display unit), a communication unit 43, a memory unit 44, and a control unit 45.

The facial camera 41A can capture an image of a user by, for example, capturing a picture of the user's face when the user stops by the facility's reception desk.

The information acquisition camera 41B can capture an image of a business card by photographing the business card presented by the user. However, the information acquisition camera 41B is not limited to business cards; it can also capture an image of the user's belongings that display information that can identify the user (for example, the user's payment card, or a card showing information about the user's place of stay) and capture the image. To ensure that the information acquisition camera 41B captures images reliably, the registration device 4 can be provided with a placement stand (placement surface) 116 (see FIG. 14C) on which the user can easily place business cards, etc. within the capture area of the camera 41B.

In addition, in the registration device 4, a single camera may be used to configure the cameras 41A and 41B for photographing faces and acquiring information. Furthermore, the timing of photographing with the camera 41A for photographing faces and the camera 41B for acquiring information can be determined by the user's operation on the registration device 4 (e.g., pressing a photographing button). Alternatively, the timing of photographing may be determined by the receptionist at the facility.

Furthermore, the registration device 4 may be provided with an information reading device (e.g., an RFID reader) capable of reading specific information about a user from an information recording medium (e.g., an RFID tag) carried by the user via wireless communication.

The display 42 can display the registration procedure (including the user's personal information) for the user (see Figs. 14(A)-(F)). In addition, a speaker can be attached to the display 42 so that the registration procedure and the like can be guided to the user by voice. In addition, by configuring the display 42 as a touch panel, it becomes possible to obtain information input by the user through touch operations.

The communication unit 43 communicates with the face management server 5 via the network. In this embodiment, it transmits face image data and specific information to the face management server 5.

The memory unit 44 stores data of the captured image (face image) of the user, specific information about the user, and a control program executed by the processor constituting the control unit 45.

The control unit 45 includes a face image extraction unit 46, a specific information acquisition unit 47, a registration request unit 48, a GUI control unit 49, and an operation status monitoring unit 50. The control unit 45 is configured with a processor, and each unit of the control unit 45 is realized by the processor executing a program stored in the storage unit 44.

The face image extraction unit 46 can acquire a photographed image of the subject of face authentication from the face photographing camera 41A, detect the person's face from the photographed image, and determine whether the user's face image has been acquired appropriately. If the acquired face image is not appropriate, the face image extraction unit 46 can display a message on the display 42 urging the user to retake the image. The face image extraction unit 46 may have the functions of camera image capture, face detection, face size check, and face cut-out, similar to the face image extraction unit 22 of the face authentication device 1 described above.

The specific information acquisition unit 47 has an OCR (optical character recognition) function and generates specific information (here, information written on a business card) from characters and symbols extracted from the image captured by the information acquisition camera 41B.

The registration request unit 48 sends a request for user registration to the face management server 5 via the communication unit 43.

The GUI control unit 49 displays various guidance screens for the user distributed from the face management server 5 on the display 42. It also acquires input information and controls the screen in response to input operations by the user using the input function of the display 42 (here, a touch panel).

The operation status monitoring unit 50 monitors the operation status of the device itself and notifies the face management server 5 of the operation status of the device itself.

Next, we will explain the general configuration of the face management server 5. Figure 9 is a block diagram showing the general configuration of the face management server 5.

The face management server 5 includes a communication unit 51, a memory unit 52, and a control unit 53.

The communication unit 51 communicates with the management terminal 2 via the network. The communication unit 51 also communicates with the face authentication device 1 via the network. The communication unit 51 also communicates with the registration device 4. The communication unit 51 also communicates with the face matching server 6.

The memory unit 52 stores a face information database, a database relating to association information between the face authentication device 1 and a matching group, a database relating to association information between the face matching server 6 and a matching group, an administrator access log, a control program executed by a processor constituting the control unit 53, and the like.

In the face information database, a facial image for registration is stored as information about each registered user. In addition, the face information database stores specific information about each user in association with the facial image for registration. In addition, matching groups and the like are registered in the face information database. Note that the facial image and specific information of the user may be stored in an encrypted state to protect privacy. Furthermore, the face information database may store the facial image and specific information of the user in a state in which part of the user's facial image and specific information has been replaced with anonymized information.

The control unit 53 includes an administrator access management unit 61, a user management unit 62, a matching group management unit 63, a device management unit 64, a face matching server management unit 65, a face information management unit 66, a database management unit 67, an operation status monitoring unit 68, an authentication log presentation unit 69, an encryption unit 70, an anonymized information generation unit 80, and an image generation unit 90. The control unit 53 is configured with a processor, and each unit of the control unit 53 is realized by the processor executing a program stored in the memory unit 52. Each unit of the control unit 53 is configured as a Web API (Web Application Programming Interface).

The administrator access management unit 61 permits or denies access (login) by an administrator depending on the access authority of the administrator who accesses its own device from the management terminal 2. The administrator access management unit 61 monitors the status of access (login) from the management terminal 2 to the face management server 5 and face matching server 6, and when it detects access from the management terminal 2, it records information about the access (such as the administrator who accessed and the date and time) as an administrator access log (history information). As a result, the face management server 5 provides the administrator access log to the management terminal 2 in response to a request from the management terminal 2 to view the administrator access log, allowing the administrator to view the administrator access log.

The administrator access management unit 61 also manages access to the device from the management terminal 2, and when the administrator operates the management terminal 2 to access the device, the information at that time is stored in the storage unit 52 as an administrator access log (history information). In addition, in response to a reference request from the management terminal 2, the administrator access log is presented to the management terminal 2.

The user management unit 62 manages information about users, such as face images for registration and specific information, and performs necessary processing related to users in response to requests from the management terminal 2. In this embodiment, requests for registration, reference, update, and deletion related to users are made from the management terminal 2, and the user management unit 62 performs necessary processing in response to the requests. The user management unit 62 also manages information about meetings attended by users and information about the users' personal connections.

The matching group management unit 63 manages information related to matching groups, and performs necessary processing related to matching groups in response to requests from the management terminal 2. In this embodiment, requests for registration, reference, update, and deletion related to matching groups are made from the management terminal 2, and the necessary processing is performed in response to the requests. In addition, matching group information, i.e., information required for the face authentication device 1 to request face authentication from the face matching server 6 that matches its own matching group, is generated for each face authentication device 1, and the matching group information is provided to the face authentication device 1.

The device management unit 64 manages information related to the face authentication device 1 and the registration device 4, and performs necessary processing related to the face authentication device 1 and the registration device 4 in response to a request from the management terminal 2. In this embodiment, a request is made from the management terminal 2 to register, refer to, update, and delete associations between the face authentication device 1 and a matching group, and the device management unit 64 performs the necessary processing in response to the request.

The face matching server management unit 65 manages information related to the face matching server 6, and performs necessary processing related to the face matching server 6 in response to a request from the management terminal 2. In this embodiment, a request is made from the management terminal 2 to register, refer to, or delete associations between the face matching server 6 and matching groups, and the necessary processing is performed in response to the request.

The facial information management unit 66 synchronizes the facial information so that the facial information (facial image, identification information, etc.) of the user stored in its own device and the facial information (facial feature data of the user) of the user stored in the face matching server 6 are maintained in a consistent state. The facial information management unit 66 also copies the facial information (facial feature data of the user).

The database management unit 67 manages the database installed on the device itself and backs up and restores the database.

The operation status monitoring unit 68 monitors the operation status of its own device, and also receives notifications of operation status from the face authentication device 1, the registration device 4, and the face matching server 6, and displays the operation status of the face authentication device 1, the registration device 4, its own device (face management server 5), and the face matching server 6 on the screen of the management terminal 2 in response to the administrator's operation on the management terminal 2.

The authentication log presentation unit 69 acquires the authentication log from the face matching server 6 in response to a reference request from the management terminal 2, and presents the authentication log to the management terminal 2.

The encryption unit 70 can encrypt at least a portion of the user's facial image and specific information. The encrypted information is stored in the storage unit 52. Note that the face management server 5 may delete from the storage unit 52 the original information (information before encryption processing) corresponding to the information encrypted by the encryption unit 70.

The anonymized information generating unit 80 can perform a process of anonymizing at least a portion of the user's facial image and specific information. As an anonymization process, the anonymized information generating unit 80 can, for example, replace a portion of the user's facial image and specific information with dummy information. In addition, in the anonymization process, dummy information may be used for information missing from the user's facial image and specific information (for example, information that the user has refused to provide). The anonymized information is stored in the storage unit 52. Note that the encryption unit 70 may encrypt only the original information (information before the anonymization process) corresponding to the information anonymized by the anonymized information generating unit 80.

The image generation unit 90 generates a management screen that enables the administrator to check or change the status of information management by the user management unit 62 and the device management unit 64. The management screen is distributed to the management terminal 2, etc.

Next, the general configuration of the face matching server 6 will be described. FIG. 10 is a block diagram showing the general configuration of the face matching server 6.

The face matching server 6 includes a communication unit 71, a memory unit 72, and a control unit 73.

The communication unit 71 communicates with the face authentication device 1 via the network. In this embodiment, it receives face image data and the like from the face authentication device 1. It also transmits the user authentication result and the like to the face authentication device 1. The communication unit 71 also communicates with the face management server 5 via the network. In this embodiment, it receives requests for various processes and the like from the face management server 5, and transmits corresponding responses and the like to the face management server 5.

The memory unit 72 stores a face matching database, association information between users and features, information about the matching group of the device itself, an authentication log, a control program executed by the processor constituting the control unit 73, and the like.

In the face matching database, the facial feature data of each registered user is registered as information about the user. A face matching database is provided for each matching group, and the facial feature data of users is stored on a group basis. Here, the facial feature data of registered users may be backed up and stored in non-volatile memory such as an HDD or SSD provided in the face matching server 6 in case of unexpected loss from the face matching database.

The control unit 73 includes a matching group management unit 81, an image quality check unit 82, a facial image extraction unit 83, a facial feature generation unit 84, a facial feature management unit 85, a facial feature matching unit 86, an authentication log management unit 87, a database management unit 88, and an operation status monitoring unit 89. The control unit 73 is configured with a processor, and each unit of the control unit 73 is realized by the processor executing a program stored in the memory unit 72. Each unit of the control unit 73 is configured as a Web API.

The matching group management unit 81 manages the matching group to which the device belongs, and performs registration and deletion processes related to the matching group in response to requests from the face management server 5.

The image quality check unit 82 determines whether the image of the face area in the captured image satisfies a predetermined quality. Specifically, it detects from the target image whether a mask and sunglasses are being worn, and calculates the face recognition suitability (evaluation value based on the face orientation and facial expression).

The facial image extraction unit 83 extracts a facial image from the captured image of the user acquired by the registration device 4 during user registration. If necessary, the facial image extraction unit 83 detects the face of a person from the captured image (face detection), determines whether the size of the detected face is appropriate (face size check), cuts out the face area from the captured image (face cut-out), and acquires the facial image of the person. Note that the facial image extraction unit 83 can also extract a facial image from the captured image of the subject acquired by the face authentication device 1 during face authentication.

The facial feature generation unit 84 detects facial feature points from the data of the facial image for registration and the facial image for authentication during user registration and facial authentication, respectively, to generate facial feature data.

When a user is registered, the facial feature amount management unit 85 registers the facial feature amount data of the user generated by the facial feature amount generation unit 84 in a face matching database corresponding to the user's matching group. When updating or deleting, the facial feature amount management unit 85 deletes the facial feature amount data registered in the face matching database in response to a request from the face management server 5. When a program related to the facial feature amount generation or face matching algorithm is upgraded, the facial feature amount data registered in the face matching database is updated in response to a request from the face management server 5 to correspond to the new program.

During face authentication, the facial feature matching unit 86 compares the facial feature data of the subject generated from the data of the face image for authentication obtained from the face authentication device 1 with the facial feature data of a registered person (registered user) stored in the device to determine whether the subject is a registered person or not. This facial feature matching unit 86 corresponds to a matching group associated with the device, and when multiple matching groups are associated with the device, the device has multiple facial feature matching units 86 for each matching group. This facial feature matching unit 86 (face matching process) is combined with a face matching database to form a face matching unit.

The facial feature matching unit 86 calculates the similarity (matching score) between the subject and the registrant. By comparing this similarity with a predetermined threshold value, it is possible to determine whether or not facial authentication has been successful. In addition to notifying the facial authentication device 1 of the success/failure determination result as the matching result, the facial authentication device 1 may also be notified of the person ID and similarity of a highly similar registrant as the matching result.

When a facial authentication request is made from the facial authentication device 1 to the authentication log management unit 87, the authentication log management unit 87 accumulates the information at that time, i.e., the device ID of the facial authentication device 1 that made the facial authentication request, the facial matching result, and other information, as an authentication log, in the memory unit 72. In addition, in response to a request from the face management server 5, it provides the authentication log to the face management server 5.

The database management unit 88 manages the database installed on the device itself and backs up and restores the database.

The operation status monitoring unit 89 monitors the operation status of the device itself and notifies the face management server 5 of the operation status of the device itself.

In this embodiment, the main functions of the face matching server 6 are face image extraction, face feature generation, and face feature matching, but these functions can be configured in separate information processing devices independent of each other. For example, the face image extraction function can be configured in a separate information processing device independent of other functions such as face feature generation and face feature matching.

Next, the general configuration of the external device 16 will be described. Figure 11 is a block diagram showing the general configuration of the external device 16.

The external device 16 includes a control signal receiving unit 91, a drive unit 92, a power supply unit 93, and a control unit 94.

The control signal receiving unit 91 receives a control signal (operation command from the face recognition device 1 in cooperation with the external device 16) transmitted from the control signal transmitting unit 17 of the face recognition device 1.

The drive unit 92 is controlled by the control unit 94 based on a control signal from the face recognition device 1. For example, if the external device 16 is a gate device, the drive unit 92 supplies power to open (or close) a door of the gate device. Also, if the external device 16 is a door lock device, the drive unit 92 supplies power to lock (or unlock) the lock device. Note that if the external device 16 is a device that does not have a driven part (for example, a payment terminal device), the drive unit 92 can be omitted.

The power supply unit 93 supplies power to each part of the external device 16. The power supply unit 93 is also electrically connected to a power supply unit 97 of the face authentication device 1, and is also capable of supplying power to the power supply unit 97. The power supply unit 97 supplies power to each part of the face authentication device 1. By being connected to the power supply unit 97 of the face authentication device 1 via a LAN cable, power can be supplied to the power supply unit 97 via PoE power supply.

The power supply unit 93 can also receive power from an uninterruptible power supply (or emergency power supply) that is independent of the power supply line for the entire facility. This allows, for example, a gate device as an external device 16 to operate normally even if a power outage occurs within the facility, which has the advantage of maintaining the security of the facility.

The control unit 94 controls the operation of each part of the external device 16. The control unit 94 can also execute the processes required by the external device 16. The control unit 94 is composed of a processor, and is realized by having the processor execute a program stored in a storage unit (memory) not shown.

Next, we will explain the facial image extraction process. Figure 12 is an explanatory diagram showing an overview of the facial image extraction process.

As described above, the facial image extraction unit 22 of the facial authentication device 1 performs a facial image extraction process, i.e., processes for detecting the face, checking the face size, and cutting out the face to generate facial image data from the captured image data of the user, and this facial image extraction process is also performed in the facial image extraction unit 83 of the face matching server 6.

When a user is registered, data of a captured image of the user is sent from the registration device 4 to the face matching server 6 via the face management server 5, and a face image extraction process is performed by the face image extraction unit 83 of the face matching server 6. In some cases, data of a captured image of the user may be sent from the management terminal 2 to the face matching server 6 via the face management server 5.

On the other hand, during face authentication, it is advisable to perform face image extraction processing only in the face authentication device 1 and not in the face matching server 6. In this case, the face authentication device 1 is given a highly accurate face detection function. Also, the face authentication device 1 constantly captures images of the shooting area with the camera 11, and sends face image information (captured image data and face frame information) to the face matching server 6 when a face is detected. This distributes the load of the face image extraction processing to multiple face authentication devices 1, and reduces the load on the face matching server 6. In addition, the amount of communication can be reduced, and therefore the network load can be reduced. And, since face authentication responses can be performed at high speed, face authentication of subjects who appear one after another can be performed efficiently.

In other words, by sharing the user's face authentication process between the face authentication device 1 and the face matching server 6, there is no need to install a large number of expensive authentication devices that centralize all of the user's face authentication process in the face authentication device 1, as in the past. Furthermore, even if updates to the face feature data occur, no large-scale maintenance work on the authentication devices is required, and the work can be done on the face matching server 6. Therefore, according to this embodiment, it is possible to build a face authentication system that is inexpensive and has excellent operability.

However, the face recognition device 1 may be configured not to have the function of extracting a face image, i.e., the face recognition device 1 may not be provided with a face image extraction unit 22.

Figure 13 is an explanatory diagram showing an overview of face recognition inquiries using matching group information.

The face management server 5 holds information about the matching group to which the user belongs, association information between the face authentication device 1 and the matching group, and association information between the face matching server 6 and the matching group. In addition, the matching group management unit 63 of the face management server 5 generates matching group information for each face authentication device 1 based on the association information between the face authentication device 1 and the matching group and the association information between the face matching server 6 and the matching group.

This matching group information is information required for the face authentication device 1 to request face authentication from the face authentication server 6 that matches its own matching group. This matching group information includes identification information (group number) of the matching group to which the face authentication device 1 belongs, and the destination information of the face authentication server 6 that requests face authentication from the face authentication device 1, that is, the face authentication server 6 that corresponds to the matching group with the face authentication device 1. Here, the destination information is specifically the network address (e.g., IP address) of the face authentication server 6, and this destination information identifies the face authentication server 6 that is the destination of the face authentication request, and associates the face authentication device 1 with the face authentication server 6 that is the destination of the face authentication request. Note that if the face authentication device 1 belongs to multiple matching groups, the addresses (IP addresses) of each of the corresponding multiple face authentication servers 6 are included in the matching group information.

When the face authentication device 1 is started, the startup processing unit 21 acquires matching group information from the face management server 5 as operation setting information and stores it in the device itself. When the face authentication device 1 detects a person's face, it sends a face matching request to the face matching server 6 that matches the matching group to which the device itself belongs. This face matching request includes information on the matching group to which the device itself belongs.

In addition, in the face authentication device 1, the timing for obtaining setting information related to matching group information and the like from the face management server 5 may be at startup, or may be periodically at a specified timing or at specified intervals, or a configuration may be adopted in which matching group information is distributed from the face management server 5.

The face matching server 6 holds association information between the face matching process of its own device and the matching group. When the face matching server 6 receives a face recognition request from the face authentication device 1, it identifies the face authentication process corresponding to the face authentication device 1 based on the association information between the matching group and the face authentication process and the matching group acquired from the face authentication device 1, and causes that face authentication process to perform face matching. As a result, face matching processing is performed by the face authentication process corresponding to the specified matching group.

In this embodiment, the face matching server 6 provides a face matching database for each matching group, and stores face feature data on a matching group basis. Therefore, when matching faces, face matching can be performed using face feature data registered in the face matching database corresponding to the matching group of the face authentication device 1. On the other hand, the face matching server 6 may perform matching regardless of the matching group of the face authentication device 1, and then apply a filter to the matching results. In other words, face matching may be performed using the face feature data of all users stored in the face matching server 6, and only the matching results with users who belong to the matching group of the face authentication device 1 may be extracted. In this case, it is not necessary to provide a separate face matching database for each matching group.

Next, we will explain the registration operation by the user on the registration device 4. Figure 14 is an explanatory diagram showing an example of a registration screen displayed on the registration device 4.

When a user starts using a facility (or if the user does not revisit the facility within a specified period after registration), the user can perform a registration operation for authentication using the registration device 4. In this embodiment, the user can perform the registration operation by following the guidance screen displayed on the display 42 (here, a touch panel) of the registration device 4. Note that some of the registration operation for authentication described below (such as pressing a button) may be performed by a receptionist with the user's consent.

An initial screen showing a registration start button 101 is displayed on the display 42 of the registration device 4 (see FIG. 14(A)). When the user presses (touches) the registration start button 101, a screen is displayed that explains how personal information will be handled in the face authentication system and asks the user for their consent (see FIG. 14(B)).

The user can consent to the handling of personal information in the face authentication system by pressing the consent button 102 on the screen in FIG. 14(B). This causes the registration device 4 to start acquiring authentication information for the user. Then, a screen (first screen) showing a method for acquiring specific information used to identify the user is displayed on the display 42 (see FIG. 14(C)). The user can cancel the registration operation by pressing the stop button 103. Note that the screen requesting the user to give consent may be omitted, as consent may not be required depending on the user. Consent may also be substituted by the operation of the capture button 104, capture button 107, or registration button 111 described below. Consent may also be obtained through a predetermined procedure before use of the registration device 4.

In the example shown in FIG. 14(C), a screen (first screen) is displayed showing how to photograph the user's business card (here, guidance for placing the user's business card within the photographing area of the information acquisition camera 41B). When the user places the business card on the placement stand 116 of the registration device 4 and presses the photographing button 104, the registration device 4 photographs the business card. The user can stop photographing the business card by pressing the stop button 105.

After the business card has been photographed, the registration device 4 displays a screen (first screen) showing a method for photographing the user's face image for registration (guidance for positioning the user's face within the photographing area of the face photographing camera 41A) (see FIG. 14(D)). At this time, the registration device 4 can prompt the user to align the face by blinking the photographing frame 106 corresponding to the photographing area. A live image (a real-time moving image) photographed by the face photographing camera 41A is displayed within the photographing frame 106. When the user positions his/her face within the photographing frame and presses the photographing button 107, the registration device 4 photographs the user's face. At this time, the registration device 4 can generate a shutter sound when the photographing button 107 is pressed. The user can stop photographing the face by pressing the stop button 108.

After the face has been photographed, the registration device 4 then displays a screen (first screen) for the user to check the photographed face image (see FIG. 14 (E)). The photographed face image (still image) is displayed on this screen, and if the user determines that there is no problem with the photographed face image, he or she presses the registration button 111. On the other hand, if the user determines that there is a problem with the photographed face image, he or she can press the re-photograph button 112. This causes the registration device 4 to display the screen shown in FIG. 14 (D), making it possible to photograph the face again. The user can also cancel photographing the face by pressing the cancel button 113.

When the face is photographed, the acquired face image and specific information (here, including the business card image and the business card information extracted from the business card image by OCR) are sent to the face management server 5. In this case, for a user who does not wish to provide both a face image and a business card image, at least one of the face image and the business card image may be sent to the face management server 5. In this way, the face management server 5 executes the registration process for the user (see FIG. 15). When the registration process of the face management server 5 is normally completed, the registration device 4 displays a screen (second screen) indicating that registration has been completed (see FIG. 14(F) and "Screen display of processing result" in FIG. 15). When the user presses the end button 115, the registration operation in the registration device 4 is completed. Note that the order of obtaining the face image and the business card image may be reversed.

After the registration operation is completed, the registration device 4 can delete the facial image and specific information stored in the memory unit 44. This prevents personal information from being stored for a long period of time in the registration device 4 used by unspecified persons, and enables the user's personal information to be managed more safely.

Instead of registering users with the above-mentioned registration device 4, the administrator can prepare a file of a facial image for the user to be registered in advance and then register the user using the management terminal 2. In addition, in order to complement the registration with the registration device 4, the administrator can also modify the authentication information registered by the registration device 4 or add new information from the management terminal 2.

Next, we will explain the user registration process that is executed in response to the user's registration operation described above. Figure 15 is a sequence diagram showing the steps of the user registration process.

When a user performs a registration operation on the registration device 4 as described above, the face management server 5 starts the registration process for that user as shown in FIG. 15. At this time, the face management server 5 receives the user's operation information from the registration device 4 along with a captured image of the face and specific information. In this registration process, the face management server 5 first sends a request for facial image extraction to a face matching server 6 whose matching group matches the target user. This request includes the captured image of the user's face obtained from the registration device 4. At this time, if there are multiple face matching servers 6 whose matching group matches the target user, one face matching server 6 is selected and a request for facial image extraction is sent to that one face matching server 6.

When the face matching server 6 receives a request for face image extraction from the face management server 5, it performs face image extraction processing. In this face image extraction processing, it performs processes such as face detection and face cropping on the photographed image of the user's face obtained from the face management server 5, and extracts the user's face image. It then sends a face image extraction response to the face management server 5. This response includes the user's face image.

When the face management server 5 receives a response to the facial image extraction from the face matching server 6, it sends a facial registration request to the face matching server 6 whose matching group matches the target user. This request can include the facial image of the user obtained from a specific face matching server 6. In this case, if there are multiple face matching servers 6 whose matching group matches the target user, the facial registration request is sent to all (two in FIG. 15) face matching servers 6, including the facial image of the user obtained from the specific face matching server 6.

The face management server 5 may omit the request for face image extraction to the face matching server 6 described above, and may add the photographed image of the user's face acquired from the registration device 4 together with the operation information to the face registration request to the face matching server 6.

When the face matching server 6 receives a face registration request from the face management server 5, it performs face registration processing. In this face registration processing, facial features are generated from the user's face image and the facial features are registered in a database. At this time, a facial registration ID is assigned in association with the user's facial features. A face registration response is then sent to the face management server 5. This response includes a result indicating whether or not the face registration was completed successfully, and the facial registration ID. Note that upon completion of registration of the facial features in the database, the user's face image in the face matching server 6 is deleted. Furthermore, the facial feature data may be backed up and stored in non-volatile memory such as an HDD or SSD managed separately from the face matching DB of the face matching server 6.

The face management server 5 receives a face registration response from the face matching server 6, and if the process has been completed successfully, performs registration processing for the user. In this registration process, the face image of the user acquired from the registration device 4 is registered in the face information database. The face management server 5 also associates the specific information of the user acquired from the face matching server 6 with the face image and registers it in the face information database. The face management server 5 also registers the face registration ID issued by the face matching server 6 in the face information database as user information.

Next, the process of facial authentication of a user in the facial authentication device 1 will be described. FIG. 16 shows a first example of the process of facial authentication for a user. Here, an example is shown in which a gate device (security gate) that manages the passage of users is used as the external device 16 that cooperates with the facial authentication device 1.

When the face authentication device 1 detects a person's face from the image captured by the camera 11 and acquires the face image, it sends a face matching request to the face matching server 6. This face matching request includes the device ID of the face authentication device 1 that made the request, the matching group of the face authentication device 1, data of the captured image of the person to be face authenticated (face image for authentication), face frame information, and matching conditions.

When the face matching server 6 receives the face matching request, first, the face feature generation unit 84 generates face features of the subject from the face image of the subject acquired from the face authentication device 1. Next, the face feature matching unit 86 of the face authentication unit corresponding to the matching group of the face authentication device 1 matches the face features of the subject with the face features of the user registered in the face authentication database. At this time, the face matching process is performed in a face authentication process whose matching group matches the face authentication device 1 that made the request. When the face matching process is completed, the face matching server 6 sends a face matching response to the face authentication device 1 that made the request. This face matching response includes the matching result (success or failure), the matching score, and the user code.

The facial feature matching unit 86 calculates a matching score that indicates the degree of similarity between the subject of facial authentication and the registered user, and if this matching score is equal to or greater than a predetermined reference value, it regards the subject of facial authentication as the registered user, and generates a matching result indicating that facial matching was successful. On the other hand, if the matching score is not equal to or greater than the reference value for all users, it determines that the subject of facial authentication is not the registered user, and generates a matching result indicating that facial matching was unsuccessful.

The processing conditions (face matching parameters) for face matching performed by the face matching server 6 may be added to the matching request. This allows the face authentication device 1 to instruct the content of the face matching processing performed by the face matching server 6. For example, a threshold value for the matching score may be specified as a processing condition for face matching, and matching results with a matching score equal to or greater than the specified threshold may be included in the response. Also, the number of matching results may be specified as a processing condition for face matching, and a specified number of matching results starting from the highest matching score may be included in the response.

In addition, in the face matching server 6, when face matching is completed, the authentication log management unit 87 stores information such as the matching results obtained in the face matching in a database as an authentication log (face recognition history information). At this time, in addition to storing only the face recognition result (success or failure) as the authentication log, the matching score may also be included in the authentication log. Also, the subject's facial image obtained from the face recognition device 1 may be included in the authentication log. In this case, it is recommended that the subject's facial image is encrypted and stored.

If many valid matching results with matching scores exceeding the threshold are obtained, a predetermined number of matching results may be narrowed down to those with the highest matching scores and saved as an authentication log. Also, if face matching does not end normally or if no valid matching results with matching scores exceeding the threshold are obtained, only the information included in the request from the face recognition device 1 may be saved as an authentication log.

When the face authentication device 1 receives a face matching response indicating success of face authentication from the face matching server 6, it transmits a door opening/closing control signal to the associated gate device. In other words, the face authentication device 1 converts the face matching response into a door opening/closing control signal, and thereby functions as a control device (gate opening/closing control device) that controls the operation of the door of the associated gate device. As a result, the gate device performs an operation to open the door (i.e., allow the user to pass) by the drive unit 92. Note that the face authentication device 1 deletes the user's face image stored in the memory unit 14 upon completion of transmission of the control signal (opening/closing control signal). Here, the face authentication device 1 is configured to convert the face matching response into a door opening/closing control signal, but if the opening/closing control of the door of the gate device is performed by an entrance/exit management system constructed as an external system, it transmits a control signal to the entrance/exit management system so that the face authentication result is reflected in the opening/closing control of the gate device.

Figure 17 shows a second example of the face authentication process for a user. Figure 18 is an explanatory diagram showing an example of an authentication screen (screen at the time of authentication) displayed on the face authentication device 1. Here, an example is shown in which a lock device installed on the door of a conference room is used as the external device 16 that cooperates with the face authentication device 1. Note that detailed explanations of matters similar to those in the first example described above regarding the second example of face authentication process will be omitted.

The facial recognition device 1 is installed, for example, on a wall of a corridor facing a conference room. When the facial recognition device 1 senses the approach of a person (i.e., detects a user) using a motion sensor or the like, it displays a standby screen (initial screen) for authentication on the display 12 (see FIG. 18(A)). Here, the authentication screen (notification screen) displayed on the display 12 includes an image display area 121 that displays a live image (real-time moving image) captured by the camera 11 of a specified shooting area, and an information display area 122 that displays information indicating the progress of the facial recognition process.

After that, when the face authentication device 1 detects a person's face from the image captured by the camera 11 and acquires a face image, it sends a face matching request to the face matching server 6. At this time, the information display area 122 displays characters 123 and a figure 124 indicating that authentication of the user is in progress (see FIG. 18(B)). Note that in the information display area 122, display of either the characters 123 or the figure 124 may be omitted. In addition, in the case of a face authentication device 1 that does not have a display, for example, an LED lamp may be used, and the LED may be made to blink during authentication and turned on when authentication is successful, to notify the progress of face authentication. The progress of face authentication may also be notified by the display color of the LED. The display color of the LED may also be changed depending on the success or failure of the user's face authentication.

When the face matching server 6 receives the face matching request, it generates face features and executes face matching processing, as in the first example described above. When the face matching processing is completed, the face matching server 6 transmits a face matching response to the face authentication device 1 that originated the request.

When the face authentication device 1 receives a face matching response from the face matching server 6 indicating successful face authentication, it displays text 127 and a figure 128 in the information display area 122 of the authentication screen indicating that authentication of the user has been completed (see FIG. 18(C)). At the same time, the face authentication device 1 transmits a control signal (a command to unlock the door) to the lock device of the linked door. In other words, the face authentication device 1 functions as a control device that controls the operation of the linked lock device. As a result, the lock device performs an operation to unlock (i.e., allow the user to enter). Note that, upon completion of transmission of the control signal, the face authentication device 1 deletes the user's face image stored in the memory unit 14.

The face recognition device 1 can also be linked to an external device that only processes information, without any mechanical operation, such as the gate device or door lock device described above. Figure 19 shows a third example of face recognition processing for a user.

Here, an example is shown in which a payment terminal device is used as the external device 16 that works in conjunction with the face recognition device 1. Note that with regard to the third example of face recognition processing, detailed explanations of matters similar to those in the first or second example described above will be omitted.

In this third example of face recognition processing, when the face recognition device 1 receives a face matching response indicating successful face recognition from the face matching server 6, it transmits a control signal to the linked payment terminal device (or payment system). As a result, the payment terminal device executes a process to link the authenticated user with the data of the usage fee for the service used by the corresponding user (e.g., the provision of a meal at a restaurant), and as a result, the usage fee is added to the facility's fee billing data for the corresponding user.

Next, we will explain the management operations performed by the administrator of the face authentication system. Figure 20 is an explanatory diagram showing the login screen displayed on the management terminal 2.

When an administrator launches a management application on the management terminal 2 and accesses the face management server 5, a login screen is displayed. On this login screen, the administrator can enter his or her own user ID and password. When the administrator enters a user ID and password on this login screen and operates the login button, user authentication is performed on the face management server 5. If login is successful, various management screens can be displayed. If login is unsuccessful, an error is displayed.

Next, we will explain how to register, refer to, update, and delete associations between the face authentication device 1 and a matching group in this system. Figures 21 and 22 are explanatory diagrams showing the management screens displayed on the management terminal 2 when registering, referencing, updating, and deleting associations between the face authentication device 1 and a matching group.

When the management terminal 2 logs in to the face management server 5, it is possible to display the reference screen shown in FIG. 21(A), the registration screen shown in FIG. 21(B), and the deletion screen shown in FIG. 22 when registering, referencing, updating, and deleting the association between the face authentication device 1 and the matching group.

The reference screen shown in FIG. 21(A) displays a list of association settings between registered face recognition devices 1 and matching groups. By viewing this reference screen, the administrator can check the registered contents of the matching group.

This reference screen is provided with a list display section 211 and a face authentication machine designation section 212. The list display section 211 displays the device ID of each authentication machine, the matching group (number) associated with each authentication machine, and an explanation of the matching group. The face authentication machine designation section 212 allows the target face authentication machine 1 (device ID) to be selected using a pull-down menu. This allows the display content of the list display section 211 to be narrowed down to a specific face authentication machine 1, and only the matching group associated with the specific face authentication machine 1 is displayed in the list display section 211. Note that if a face authentication machine 1 is not designated in the face authentication machine designation section 212, information on all face authentication machines 1 is displayed in the list display section 211.

In addition, on this reference screen, sorting can be performed by specifying items (device ID, face authentication device 1, description). It is also possible to specify the number of items to be displayed in the list display section 211 (the number of face authentication devices 1) and the display range of the face matching server 6 and matching group. It is also possible to specify search conditions for each item (device ID, face authentication device 1, description) and perform a search.

In the example shown in FIG. 21(A), the facial recognition devices 1 are grouped according to the location where they are installed (e.g., the location of the business establishment).

The registration screen shown in FIG. 21(B) allows you to set up an association between the face authentication device 1 and a matching group. You can register information that associates the authentication device with a matching group.

This registration screen is provided with an authentication machine designation section 213, a matching group designation section 214, and a registration button 215. In the authentication machine designation section 213, the target face authentication machine 1 (device ID) can be selected from a pull-down menu. In the matching group designation section 214, the target face authentication machine 1 matching group can be selected from a pull-down menu. This pull-down menu displays matching groups that have already been registered. When the face authentication machine 1 and matching group are entered on this registration screen and the registration button 215 is operated, the face management server 5 performs a process of registering the matching group information based on the input contents.

Note that there may be cases where multiple matching groups are associated with one face authentication device 1. In this case, it is sufficient to repeat the registration operation on the registration screen to associate one matching group with one face authentication device 1. In addition, multiple matching group designation sections 214 may be provided on the registration screen.

The deletion screen shown in Figure 22 is used to delete the association setting between the face recognition device 1 and the matching group.

This deletion screen is provided with a list display section 216, a face authentication device designation section 217, and a delete button 218. In the list display section 216, the device ID of each authentication device, the matching group (number) associated with each face authentication device 1, and an explanation of the matching group are displayed. In addition, the list display section 216 is provided with a check box for each association setting. The association setting can be selected using the check box. When the registration item to be deleted is selected using the check box and the delete button 218 is operated, the face management server 5 performs a process of deleting the selected registration item. In the face authentication device designation section 217, a face authentication device 1 can be selected using a pull-down menu. This updates the display content of the list display section 216 to a state where it has been narrowed down to a specific face authentication device 1.

Here, by selecting Reference on the menu screen (not shown), the screen transitions to the reference screen shown in FIG. 21 (A). By selecting Registration on the menu screen, the screen transitions to the registration screen shown in FIG. 21 (B). By selecting Delete on the menu screen, the screen transitions to the deletion screen shown in FIG. 22. By selecting Update on the menu screen, the screen transitions to an update screen (not shown). By selecting Association of the Face Recognition Device 1 with a Matching Group on the reference screen, the screen transitions to an update screen (not shown).

In the example shown in FIG. 22, the deletion screen displays a list of registered associations, but deletion may also be possible on an individual edit screen. Also, the update screen (not shown) is similar to the registration screen (FIG. 21 (B)).

Figure 23 is an explanatory diagram showing an example of a user management screen displayed on the management terminal 2.

When managing information about a user, the management terminal 2 displays the user management screen shown in FIG. 23.

The user management screen shown in FIG. 23 allows an administrator to register, refer to, update, delete, and search for facial images and specific information (here, business card information) related to the user. The information (including images) entered into the user management screen includes information acquired by the registration device 4.

The user management screen includes an operation selection area 131, a user information area 132, and a business card information area 133.

In the operation selection area 131, the administrator can select the item to be executed (such as registering, referencing, updating, deleting, and searching information). In FIG. 23, business card registration is selected as the item to be executed. This allows the administrator to newly enter information written on the business card displayed on the user's management screen.

The user information area 132 includes the user's facial image and name, as well as information used for facial authentication. Here, the user information area 132 includes a facial image display section 141, a name input section 142, a user ID input section 143, an identification ID input section 144, a matching group input section 145, an access permission group input section 146, an additional information input section 147, an activation date input section 148, and an invalidation date input section 149.

The facial image display unit 141 displays the facial image of the user. The administrator can delete the facial image in the facial image display unit 141 by pressing the delete button 151. If the administrator is unable to acquire a facial image using the registration device 4, the administrator can select a prepared facial image (image file) as the display target by pressing the image selection button 152. Alternatively, the administrator can display the facial image selected using the image selection button 152 on the facial image display unit 141 in place of the facial image acquired by the registration device 4. The facial image prepared by the administrator is not limited to the actual facial image of the user, and may be a dummy image (e.g., an avatar image of the user).

The user's first and last name are input in the name input section 142. The user ID (such as an employee number) is input in the user ID input section 143. An ID capable of identifying the user (such as a series of unique numbers assigned in the order of user registration) is input in the identification ID input section 144. A match group is input in the match group input section 145. An access permission group indicating an administrator group that can access the user's information is input in the access permission group input section 146. Additional information can be input in the additional information input section 147. The date on which the face matching of the person can be enabled is input in the activation date input section 148. The date on which the face matching can be disabled is input in the invalidation date input section 149. By setting these activation and invalidation dates, services for multiple users can be started or ended simultaneously.

The business card information area 133 includes photographed images of business cards registered by the registration device 4 and information extracted from the photographed images of business cards. Here, the business card information area 133 includes a business card image display section 161, an affiliation information input section 162, an activation date input section 163, and an invalidation date input section 164.

The business card image display section 161 displays the user's business card image. As with the face image described above, the administrator can delete the business card image or select a prepared business card image (image file) by pressing the delete button 171 or the image selection button 172.

The affiliation information input section 162 is used to input the name of the company to which the user belongs, the name of the department of the user, the job title, telephone number, fax number, email, postal code, and address. The activation date input section 163 is used to input the date on which the information on the person's business card can be activated. The invalidation date input section 164 is used to input the date on which the information on the person's business card can be invalidated.

When the administrator presses the registration button 167 after completing input into the desired input section, the information that has been entered (added or changed) becomes valid.

The user management screen can also manage other specific information (for example, information displayed on a payment card or information indicating the user's location within the facility) in addition to or instead of the information on the business card.

In addition, the administrator can search for the necessary information by selecting a search tab (e.g., match group search) in the operation selection area 131 of the user management screen. On such an information search screen, each input section on the user management screen described above is displayed as blank. For example, the administrator can input the match group to be searched for into the blank match group input section 145, thereby searching for users who belong to the input match group.

When an administrator enters search criteria in the required fields on the search screen and presses a search button (not shown), the screen transitions to a list display screen reflecting the search results (e.g., a screen showing a list of extracted users).

Next, the use of information about users will be described. FIG. 24 is an explanatory diagram showing an example of a conference room search screen (conference search screen) displayed as a management screen on the management terminal 2, etc. FIG. 25 is an explanatory diagram showing an example of an attendee display screen displayed as a management screen on the management terminal 2.

The conference room search screen shown in FIG. 24 is intended for the administrator to use (reference) information about the conference acquired in the face recognition process. By accessing the face management server 5 from the management terminal 2, the administrator can add information about the conference held at the facility that cannot be acquired by the face recognition device 1 as appropriate.

The conference room search screen displays information such as the date, time, and room number (conference room number) for past conferences held in the facility's conference rooms. The conference room search screen can be generated using entry history information acquired by face recognition device 1 linked to the locking device of the conference room door. When the administrator selects the desired conference (clicking on the check box for the conference shown at the top in Figure 24) and presses the select button 173, the screen transitions to a search result display screen reflecting the search results.

As a search result display screen, for example, an attendee display screen including an attendee list is displayed on the management terminal 2, as shown in FIG. 25.

The attendee list includes an attendee information area 175 and a respondent information area 176. The attendee information area 175 displays facial images and specific information of the attendees of the meeting (here, the name of the company to which the user belongs and the user's name). The respondent information area 176 displays information (here, the department and name) about the person who responded to the attendee (for example, an employee of a company located in the facility).

Figure 26 is an explanatory diagram showing an example of a personal connection search screen displayed as a management screen on the management terminal 2.

The personal connection search screen shown in FIG. 26 is intended for the administrator to use (reference) information about the personal connections of the user 10 acquired in the face recognition process (here, information about human connections estimated from information about the attendees of the meeting). By accessing the face management server 5 from the management terminal 2, the administrator can add necessary information about the user's personal connections as appropriate.

The connection search screen includes a correlation diagram display area 181 and a list display area 182. The correlation diagram display area 181 displays a diagram that visualizes the connections related to a person selected by the administrator. The list display area 182 displays a connection list 185 related to the person (user 10) selected by the administrator. This connection list 185 displays the person's facial image, affiliation information (company name, department, etc.), a numerical value indicating the strength of the connection (connection strength), etc.

Such a personal connection search screen may be displayed, for example, by the administrator selecting (clicking) the area of the desired attendee on the attendee display screen of FIG. 25.

The use of the conference room search screen and personal connection search screen described above does not have to be limited to use via the management terminal 2. For example, by granting administrator access rights to facility users (e.g., employees of a company located in the facility), the users may be able to access the conference room search screen and personal connection search screen from their own information processing terminals (e.g., PCs, tablets, etc.).

As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to this, and can also be applied to embodiments in which modifications, substitutions, additions, omissions, etc. have been made. It is also possible to combine the components described in the above embodiments to create new embodiments.

In recent years, in the world of IoT (Internet of Things), attention has been focused on CPS (Cyber-Physical Systems), a new concept that creates new added value by linking information between physical space and cyberspace. In this context, the present embodiment can also adopt the CPS concept. That is, as a basic configuration of CPS, for example, an edge device placed in physical space and a cloud server placed in cyberspace are connected via a network, and face authentication processing can be distributed and processed by processors installed in the edge device and the cloud server. In this face authentication processing, the edge device acquires face image data of the user captured by the camera of the authentication machine. Next, the cloud server executes a process of generating feature data from the face image data received from the edge device via the network, compares it with the feature data of the user registered in advance, executes user authentication processing, and takes over the authentication result related to the authentication processing with the face authentication application software and displays it on a display or the like in an output format defined on the application. Here, each piece of data generated in the edge device or cloud server is preferably generated by a web application or the like installed on a standardized platform, and the use of such a standardized platform can improve the efficiency of building a system that includes a variety of different sensor groups and IoT application software.

The facial recognition device and facial recognition method disclosed herein have the effect of easily performing facial recognition when a user makes a payment at a facility while safely managing the facial images of the user who uses the facility, and are useful as a facial recognition device and facial recognition method that control information processing related to the payment of a subject based on the results of facial recognition of the subject.

1: Face authentication device 2: Management terminal 3: Face authentication server 4: Registration device 5: Face management server 6: Face matching server 10: User 11: Camera (face image acquisition unit)
12: Display (display unit)
14: Memory unit 15: Control unit (processor)
16: External device 25: Authentication result adjustment unit (history information generation unit)
41A: Face camera (personal information acquisition unit)
41B: Information acquisition camera (personal information acquisition unit)
42: Display (display unit)
45: Control unit (processor)
62: User management unit 64: Device management unit 90: Image generation unit 97: Power supply unit

Claims (4)

A face authentication device that cooperates with a payment terminal device, which is an external device, and issues a face authentication processing request to a face matching server that performs face authentication of a user of a facility when the user pays a usage fee at a facility, and completes information processing regarding the payment of the usage fee based on the authentication result,
A storage unit that stores information regarding a matching group to which the own device belongs;
a face image acquisition unit that photographs the user and acquires a face image for authentication of the user when the user pays the usage fee;
a transmission unit that transmits a processing request for the face authentication to the face matching server that matches the matching group, the face authentication processing request being used to match the face image for authentication with a face image of a user that matches the matching group ;
a receiving unit for receiving an authentication result returned from the face matching server ;
a control signal transmission unit that transmits the authentication result to the external device as a control signal,
A facial recognition machine characterized in that a processor of the external device receives the control signal and reflects it in information processing regarding the payment. The face recognition device according to claim 1, further comprising a power supply unit that receives power from the payment terminal device. A storage unit that stores the face image for authentication,
3. The face authentication device according to claim 1, wherein the face image for authentication stored in the storage unit is deleted after the face authentication process is completed. A face authentication method in which a face authentication device and a payment terminal device, which is an external device, are linked to a face authentication server that performs face authentication of a user of a facility, and a face authentication process request is made from the face authentication device to a face matching server that performs face authentication of a user of the facility when the user pays a usage fee at the facility, and information processing regarding the payment of the usage fee is completed based on the authentication result,
Store information regarding a matching group to which the face authentication device belongs;
When the user pays the usage fee, the user is photographed and a face image for authentication of the user is obtained;
Transmitting a processing request for the face authentication to the face matching server having the matching group, for matching the face image for authentication with a face image of a user having the matching matching group ;
Receive an authentication result returned from the face matching server ;
Transmitting the authentication result to the external device as a control signal;
A facial recognition method, characterized in that a processor of the external device receives the control signal and reflects it in information processing regarding the payment.

Images