CN115705726A - A face recognition method and device - Google Patents

Abstract

The embodiment of the application provides a face recognition method and device, relates to the field of terminals, and can improve the safety of face recognition. The method is applied to electronic equipment, the electronic equipment comprises a TOF camera module, the TOF camera module comprises an emitter used for emitting optical signals and an image sensor used for receiving reflected light and imaging, and the method comprises the following steps: receiving a first operation of a user, wherein the first operation is used for triggering face recognition; controlling the emitter to operate at a first light intensity; determining whether the transmitter is in a normal operating state; under the condition that the emitter is in a normal working state, controlling the emitter to work at a second light intensity, wherein the second light intensity is greater than the first light intensity; controlling an image sensor to collect image data; face recognition is performed based on the image data. Wherein determining whether the transmitter is in a normal operating state may be determined by the camera HAL in the electronic device by the camera driving module from operating state parameters of the transmitter obtained by the transmitter.

Description

A face recognition method and device

technical field

The present application relates to the field of terminals, and in particular to a face recognition method and device.

Background technique

At present, face recognition is widely used in scenarios where electronic devices perform identity authentication. The current face recognition adopts the technology of plane (2D) face feature detection, and the extracted face features are 2D features, which are vulnerable to false attacks (for example, impersonating through the owner's photo), and the security is not high.

Contents of the invention

Embodiments of the present application provide a face recognition method and device, which can improve the security of face recognition.

In the first aspect, the embodiment of the present application provides a face recognition method, which is applied to electronic equipment. The electronic equipment includes a time-of-flight TOF camera module, and the TOF camera module includes a transmitter for emitting optical signals and a transmitter for receiving reflected light. and an imaging image sensor, the method includes: receiving a user's first operation, the first operation is used to trigger face recognition; controlling the transmitter to work with a first light intensity; determining whether the transmitter is in a normal working state; In the working state, the transmitter is controlled to work with the second light intensity, and the second light intensity is greater than the first light intensity; the image sensor is controlled to collect image data; face recognition is performed based on the image data.

Based on the method provided by the embodiment of the present application, it is determined whether the transmitter is in a normal working state when it is working with the first light intensity, and if the transmitter is in the normal working state, then control the transmitter to work with the second light intensity. The second light intensity is greater than the first light intensity. Since the first light intensity is relatively small, the first light intensity will not cause damage to human eyes, which can ensure the safety of human eyes. If the transmitter can work normally and emit an optical signal with the first light intensity, it means that the transmitter is intact and not damaged, so that the electronic equipment can normally control the transmitter, and when the transmitter works with a larger second light intensity, it can also Ensure the safety of human eyes, and there will be no problem of emitting light signals that hurt human eyes due to damage. Moreover, when the emitter works with the second light intensity, it can ensure that the image sensor collects more accurate image data, so that face recognition can be performed more accurately.

In addition, the image data collected by the TOF camera module is 3D face data. Compared with 2D face data, face recognition based on 3D face data is safer and more accurate.

In a possible implementation, the light signal emitted by the transmitter when it works at the first current value is the first light intensity, the light signal emitted by the transmitter when it works at the second current value is the second light intensity, and the second current value is greater than the first current value. That is to say, the greater the current value of the transmitter, the greater the light intensity of the optical signal emitted by the transmitter.

In a possible implementation manner, determining whether the transmitter is in a normal working state includes: determining a first parameter of the transmitter, where the first parameter is used to indicate the working state of the transmitter; if the first parameter is used to indicate the working state of the transmitter If the state is a normal working state, it is determined that the transmitter is in a normal working state; if the first parameter is used to indicate that the working state of the transmitter is an abnormal working state, it is determined that the transmitter is in an abnormal working state. The first parameter of the transmitter may be generated after the transmitter operates at a first current value and sends an optical signal with a first light intensity.

In a possible implementation, the method further includes: when the transmitter is in an abnormal working state, controlling the transmitter to work with a third light intensity, where the third light intensity is 0; controlling the image sensor to collect image data; The data performs face recognition. Since the transmitter is in an abnormal working state, the transmitter may not work at this time, so that the problem of emitting light signals that are harmful to human eyes due to damage to the transmitter can be avoided.

In a possible implementation, performing face recognition based on image data includes: obtaining a grayscale image and a depth image based on the image data; performing face comparison based on the grayscale image, performing anti-counterfeiting detection based on the depth image, and obtaining a face recognition result . If the face comparison result satisfies the first preset condition and the anti-counterfeiting detection result satisfies the second preset condition, the face recognition result can be considered successful, and operations such as unlocking can be performed.

In a possible implementation manner, the first operation includes an operation for unlocking an electronic device, an operation for online payment, an operation for entering a face, or an operation for securely registering or logging into an application program. For example, the first operation may be an operation such as pressing a power button, clicking, or sliding, which is not limited in this application.

)中进行支付或转账操作)的人脸识别，用户在安全注册或登录应用程序的人脸安全验证(例如，用户在

中进行注册或登录操作)等场景中，本申请不做限定。In a possible implementation, the method further includes: determining whether to perform unlocking according to the face recognition result; if the face recognition result is successful, performing unlocking; if the face recognition result is failure, not performing unlocking or displaying that unlocking failed ; Or determine whether to execute the payment according to the face recognition result; if the face recognition result is successful, execute the payment; if the face recognition result is a failure, do not execute the payment or display payment failure; Face registration; if the face recognition result is successful, face registration will be performed; if the face recognition result is a failure, face registration will not be performed or face registration failure will be displayed; or whether to perform registration or login is determined according to the face recognition result; If the face recognition result is successful, perform registration or login; if the face recognition result is failure, do not perform registration or login or display registration or login failure. That is, this application can be applied in payment or transfer (for example, the user is in the payment application/finance application/chat application/shopping application (for example,

), face recognition for payment or transfer operations in ), and face security verification for users to securely register or log in to the application (for example, users in

In scenarios such as registration or login operations in the middle, this application does not make limitations.

In a possible implementation, the method further includes: when the transmitter is in an abnormal working state, prompting the user that the unlocking fails; or prompting the user that the payment fails; or prompting the user that the face registration fails; or prompting the user Registration or login failed. Because the transmitter is in an abnormal working state, the transmitter is not working at this time, so the face cannot be successfully recognized, but the problem of emitting light signals that are harmful to human eyes due to damage to the transmitter can be avoided.

In a possible implementation, the electronic device includes a camera hardware abstraction layer HAL and a camera driver module, the camera HAL includes a sensor node, and controlling the transmitter to work with the first light intensity includes: the sensor node determines that the camera module works in the first mode One working mode; the first working mode is used to instruct the transmitter to work at the first current value; the sensor node sends the configuration parameters of the first working mode to the camera driver module; the camera driver module writes the configuration parameters of the first working mode into the TOF In the register of the camera module; the camera driver module sends a message that the configuration parameter is written to the sensor node; in response to receiving the message that the configuration parameter is written, the sensor node sends the first start command to the camera driver module; the camera driver module sends the TOF The camera module sends a second startup command; the transmitter works with the first current value, and the light signal emitted by the transmitter when it works with the first current value is the first light intensity. In this way, the transmitter can be controlled to work with the first light intensity through the camera HAL (including the sensor node) and the camera driving module in the electronic device.

In a possible implementation manner, the electronic device further includes a first application, a face recognition software development kit SDK, a face recognition service, a face recognition control module, and a camera service. After receiving the user's first operation, the method further Including: the first application calls the face recognition SDK to perform face recognition; the first application corresponds to the first operation, and the first application includes a lock screen application, a shopping application, a chat application or a wealth management application; Send a face recognition request; the face recognition request carries the identity of the face recognition type, the resolution size of the image and the data stream format; the face recognition service sends the face recognition request to the face recognition control module; The recognition control module matches the camera module according to the face recognition request; the face recognition control module sends the first request to open the camera module to the camera service; the first request to open the camera module carries the security identification and the identification of the camera module ID, image resolution, and data stream format; the security ID is used to apply for secure memory; the camera service sends a second request to the camera HAL to open the camera module, and the second request carries the security ID, the ID of the camera module, and the image resolution and data stream format. In this way, the sensor nodes in the camera HAL can obtain information such as the security identification, the identification ID of the camera module, the resolution of the image, and the format of the data stream, and then determine the working mode of the camera module based on these information.

In a possible implementation manner, the sensor node determines that the working mode of the camera module is the first working mode, which specifically includes: the sensor node determines that the working mode of the camera module is The first working mode. In this way, the sensor node can determine that the working mode of the camera module is the first working mode.

In a possible implementation, after the camera service sends the second request to the camera HAL to open the camera module, the method further includes: the camera HAL creates a file for transmission according to the camera module ID, image resolution, and data stream format. The path of data flow and control flow; the camera HAL returns the result of creating the path to the camera service; the result of creating the path is success; the camera service returns the message that the camera module is opened to the face recognition control module; the face recognition control module sends a message to the camera The service sends a data request, and the data request is used to obtain the data stream; the camera service calls the camera HAL to obtain the data stream.

In a possible implementation manner, after the transmitter works at the first current value, the method further includes: the image sensor acquires an optical signal within the exposure time corresponding to the first working mode; based on the received optical signal, the image sensor acquires the second - image data.

In a possible implementation manner, after the image sensor acquires the first image data, the method further includes: the image sensor sends a request for acquiring the first parameter to the transmitter; the image sensor receives the first parameter from the transmitter; the image sensor receives the first parameter based on the first The image data and the first parameter are used to obtain the first original RAW Data; the first parameter is used to indicate the working state of the transmitter under the first current value.

In a possible implementation manner, the electronic device further includes an image processing module, a first memory, and a face recognition TA, and determining whether the transmitter is in a normal working state includes: the image sensor obtains the first image data and the first parameter based on the first After the original RAW Data, the image sensor sends the first RAW Data to the image processing module; the image processing module sends the first RAW Data to the first memory for storage, and the first memory corresponds to the first FD; the image processing module sends the first FD Send to the camera driver module; the camera driver module sends the first FD to the camera HAL; in response to receiving the first FD, the camera HAL sends a request to read the first parameter to the camera driver module, and the first parameter is used to indicate the transmitter. Working state: the camera driver module reads the first parameter from the register of the transmitter, and sends the first parameter to the sensor node of the camera HAL; the camera HAL determines whether the transmitter is in a normal working state according to the first parameter.

In a possible implementation manner, the camera HAL determines whether the transmitter is in a normal working state according to the first parameter, which specifically includes: if the first parameter is used to indicate that the working state of the transmitter is a normal working state, the camera HAL determines that the transmitter is in a normal working state. Normal working state; if the first parameter is used to indicate that the working state of the transmitter is an abnormal working state, the camera HAL determines that the transmitter is in an abnormal working state. In this way, face recognition can determine whether the transmitter is in a normal working state according to the first parameter in the first RAW Data.

In a possible implementation manner, controlling the transmitter to work with the second light intensity includes: if the working state of the transmitter is the normal working state, the camera HAL determines that the working mode of the camera module is the second working mode, and the second working mode The mode is used to indicate that the transmitter works at the second current value; the camera HAL sends the configuration parameters of the second working mode to the camera driver module; the camera driver module writes the configuration parameters of the second working mode into the register of the TOF camera module; The camera driver module sends a message that the configuration parameter writing is completed to the sensor node; in response to receiving the message that the configuration parameter is written, the sensor node sends the third startup command to the camera driver module; the camera driver module sends the fourth startup command to the TOF camera module command; the transmitter works with the second current value, and the light signal emitted when the transmitter works with the second current value is the second light intensity. In this way, the camera HAL can determine that the working mode of the camera module is the second working mode according to the normal working state of the transmitter. The transmitter can be controlled to work with the second light intensity through the camera HAL (including the sensor node) and the camera driver module in the electronic device.

In a possible implementation manner, after the transmitter works at the second current value, controlling the image sensor to collect image data specifically includes: the image sensor acquires the light signal within the exposure time corresponding to the second working mode; signal, the image sensor acquires the second image data.

In a possible implementation manner, after the image sensor acquires the second image data, the method further includes: the image sensor sends a request for acquiring the second parameter to the transmitter; the image sensor receives the second parameter from the transmitter; the image sensor based on the first The image data and the second parameter are used to obtain the second RAW Data; the second parameter is used to indicate the working state of the transmitter under the second current value.

In a possible implementation, the electronic device further includes a second memory and a face recognition trusted application TA, and performing face recognition based on image data includes: the image sensor sends the second RAW Data to the image processing module; the image processing module Send the second RAW Data to the second memory for storage; the second memory corresponds to the second FD; the face recognition TA reads the second RAW Data from the second memory according to the second FD; the face recognition TA reads the second RAW Data according to the second RAW Data The first grayscale image and the first depth image are obtained from the image data in the image data; the face recognition TA performs face comparison according to the first grayscale image, and conducts anti-counterfeiting detection according to the first depth image to obtain the face recognition result. In this way, the face recognition TA can determine the face recognition result according to the second RAW Data.

In a possible implementation manner, after the image processing module sends the second RAW Data to the second memory for storage, the method further includes: the image processing module sends the second FD to the camera driver module; the camera driver module sends the second FD Send to the camera HAL; the camera HAL sends the second FD to the camera service through the preset interface; the camera service sends the second FD to the face recognition control module; the face recognition control module sends the second FD to the face recognition TA. In this way, the face recognition TA can obtain the second FD, so that the second RAW Data can be read according to the second FD.

In a possible implementation, the method further includes: the face recognition TA sends the face recognition result to the face recognition control module; the face recognition control module sends the face recognition result to the face recognition service; The service sends the face recognition result to the face recognition SDK; the face recognition SDK sends the face recognition result to the first application; in response to receiving the face recognition result, the first application performs unlocking, and the face recognition result is successful. In this way, when the face recognition result is successful, it can be successfully unlocked.

In a possible implementation manner, when the transmitter is in an abnormal working state, controlling the transmitter to work with the third light intensity specifically includes: in response to receiving that the working state of the transmitter is an abnormal working state, the camera HAL determines that the camera The working mode of the module is the third working mode, the third working mode is used to instruct the transmitter to work at the third current value, and the third current value is 0; the camera HAL sends the configuration parameters of the third working mode to the camera driver module; The camera driver module writes the configuration parameters of the third working mode into the register of the TOF camera module; the camera driver module sends a message that the configuration parameters are written to the sensor node; in response to receiving the message that the configuration parameters are written, the sensor node sends The camera driver module sends the fifth startup command; the camera driver module sends the sixth startup command to the TOF camera module; the transmitter does not work. Because the transmitter is in an abnormal working state, the transmitter does not work at this time, which can avoid the problem of emitting light signals that are harmful to human eyes due to damage to the transmitter.

In a possible implementation manner, after the transmitter does not work, the method further includes: the image sensor receives an optical signal within an exposure time corresponding to the third working mode; based on the received optical signal, the image sensor acquires third image data.

In a possible implementation manner, the method further includes: the face recognition TA acquires a second grayscale image and a second depth image based on the third image data; performing face comparison based on the second grayscale image, and Perform anti-counterfeiting detection, and the face recognition result is failed. Since the transmitter is not working (no electricity, no light), the image data (third image data) obtained by the image sensor is usually a "black image" without a clear face image, so the face recognition result is a failure.

In a possible implementation, the method further includes: the face recognition TA sends the face recognition result to the face recognition control module; the face recognition control module sends the face recognition result to the face recognition service; The service sends the face recognition result to the face recognition SDK; the face recognition SDK sends the face recognition result to the first application; in response to receiving the face recognition result, the first application does not perform unlocking or displays that the unlocking failed, and the face recognition The result is failure. Because the transmitter is in an abnormal working state, the transmitter is not working at this time, so the face recognition result is a failure, and thus the unlocking fails, but this can avoid the problem of emitting light signals that are harmful to human eyes due to damage to the transmitter.

In a second aspect, the present application provides a chip system, which includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by wires. The system-on-a-chip described above can be applied to an electronic device including a communication module and a memory. The interface circuit is for receiving signals from the memory of the electronic device and sending the received signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instruction, the electronic device can execute the method described in the first aspect and any possible design manner thereof.

In a third aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium includes computer instructions. When the computer instructions are run on the electronic device (such as a mobile phone), the electronic device is made to execute the method described in the first aspect and any possible design manner thereof.

In a fourth aspect, the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the method described in the first aspect and any possible design manner thereof.

In the fifth aspect, the embodiment of the present application provides a face recognition device, including a processor, the processor is coupled to a memory, and the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the device realizes the above-mentioned The method described in the first aspect and any possible design manner thereof. The apparatus may be an electronic device or a server device; or may be a component of the electronic device or the server device, such as a chip.

In the sixth aspect, the embodiment of the present application provides a face recognition device. The device can be divided into different logical units or modules according to functions, and each unit or module performs different functions, so that the device can perform the above-mentioned first The method described in the aspect and any possible design manner thereof.

It can be understood that the chip system described in the second aspect provided above, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the devices described in the fifth aspect and the sixth aspect provide For the beneficial effects that can be achieved, reference can be made to the beneficial effects in the first aspect and any possible design manner thereof, and details will not be repeated here.

Description of drawings

FIG. 1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

Fig. 2 is a schematic diagram of the principle of a TOF imaging technology provided by the embodiment of the present application;

FIG. 3 is a schematic diagram of a software module architecture provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of interaction between software modules provided by an embodiment of the present application;

Fig. 5 is another schematic diagram of interaction between software modules provided by the embodiment of the present application;

FIG. 6 is a schematic diagram of signal interaction provided by an embodiment of the present application;

FIG. 7 is a schematic display diagram provided by an embodiment of the present application;

FIG. 8 is another schematic display diagram provided by the embodiment of the present application;

FIG. 9 is another schematic display diagram provided by the embodiment of the present application;

FIG. 10 is another schematic diagram of signal interaction provided by the embodiment of the present application;

FIG. 11 is a schematic diagram of a chip structure provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Wherein, in the description of the present application, unless otherwise specified, "at least one" refers to one or more, and "multiple" refers to two or more than two. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. Those skilled in the art can understand that words such as "first" and "second" do not limit the number and execution order, and words such as "first" and "second" do not necessarily limit the difference.

In order to make the description of the following embodiments clear and concise, a brief introduction of related concepts or technologies is given first:

Rich execution environment (REE), also known as rich execution environment, common execution environment or untrusted execution environment, refers to the system operating environment of the mobile terminal, in which operating systems such as Android, IOS, and Linux can run. REE has good openness and scalability but not high security.

A trusted execution environment (TEE), also called a secure side or a secure zone, is an area that requires authorization to access. TEE and REE coexist in the operating environment of electronic equipment. It is isolated from REE through hardware support, has security capabilities, and can resist software attacks that conventional REEs are vulnerable to. TEE has its own operating space and defines strict protection measures. Therefore, it has a higher security level than REE and can protect assets (assets) in TEE, such as data, software, etc., from software attacks and resist specific types of security. threaten.

The REE+TEE architecture refers to the architecture that provides services for applications through the combination of TEE and REE. That is to say, TEE and REE co-exist in electronic equipment. Exemplarily, the TEE can implement an operating mechanism isolated from the REE through the support of hardware. TEE has its own operating space and has a higher security level than REE, which can protect assets in TEE (such as data, software, etc.) from software attacks. Only authorized security software can be executed in TEE, and it also protects the confidentiality of resources and data of security software. Compared with REE, TEE can better protect the security of data and resources due to its protection mechanisms such as isolation and permission control.

TA, that is, a trusted application, is an application running in the TEE, which can provide security services for the CA running outside the TEE, such as entering passwords, generating transaction signatures, face recognition, etc.

CA, the client application. CA usually refers to the application running in REE. The CA can call the TA through a client (Client) application programming interface (application programming interface, API) and instruct the TA to perform corresponding security operations.

Software development kit (software development kit, SDK): In a broad sense, it refers to a collection of related documents, examples, and tools that assist in the development of a certain type of software.

RAW Data, that is, raw data, can be understood as "unprocessed and uncompressed data". In the embodiment of the present application, RAW Data may refer to the raw data that the TOF camera converts the captured light source signal into a digital signal. Some metadata (Metadata) generated by camera shooting is also recorded in RAWData.

Metadata, also known as intermediary data or relay data, is data used to describe data (data about data), mainly information describing data attributes (property). In the embodiment of the present application, the Metadata can indicate the working mode of the camera, the value of the photoelectric current, the working status of the TOF camera device, the exposure value and other information.

The TOF camera (TOF camera module) can include a transmitter (TX) and a receiver (RX). TX is used to emit infrared light or laser pulses, and RX is used to receive reflected light and image it. Since the TX can autonomously emit light signals for imaging, TOF images are not affected by most light in the environment. In this way, the application of TOF images in unlocking services can improve the security of face recognition.

Time of flight (TOF) imaging technology refers to a group of infrared light (or laser pulses) invisible to the human eye that is emitted outwards, reflected after encountering an object, reflected to the end of the camera, and calculated from emission to reflection. The time difference or phase difference of the camera, and the data are collected to form a set of distance and depth data, so as to obtain a three-dimensional 3D model imaging technology. That is to say, TOF imaging technology is based on the traditional 2D XY axis imaging, adding depth information from the Z axis direction, and finally generating 3D image information.

When using TOF imaging technology, it is necessary to project infrared light, laser, etc. to the face. For the safety of human eyes, it is necessary to detect the optical power of the light emitted by TX to ensure that the optical signal emitted by TX is within the safe range of human eyes. Injury to the human eye.

The embodiment of the present application provides a face recognition method, which uses a TOF camera to collect images. The TOF camera can first work in the eye-safe mode to determine whether the TX is abnormal. If the TX works normally, the TOF camera can be used to collect images normally, which can ensure The optical power of the light emitted by the TX is within the range of human eye safety. If TX is abnormal, TX can be turned off to avoid the optical signal emitted by TX from hurting human eyes.

FIG. 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

As shown in FIG. 1 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, and a battery 142 , antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193 , a display screen 194, and a subscriber identification module (subscriber identification module, SIM) card interface 195, etc.

Among them, the sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an environmental Light sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 100 . In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or A universal serial bus (universal serial bus, USB) interface, etc.

It can be understood that the interface connection relationship between the modules shown in this embodiment is only for schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 140 is configured to receive a charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also provide power for electronic devices through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 .

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (codedivision multiple access, CDMA), wideband code Wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM , and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou satellite navigation system (beidounavigation satellite system, BDS), a quasi-zenith satellite system (quasi- zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (liquid crystal display, LCD), light-emitting diode (light-emitting diode, LED), organic light-emitting diode (organic light-emitting diode, OLED), active matrix organic light-emitting diode or active matrix Organic light-emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED )wait.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor. The ISP is used for processing the data fed back by the camera 193 . Camera 193 is used to capture still images or video. Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The cameras 193 may include 1 to N number. For example, an electronic device may include 2 front cameras and 4 rear cameras. Wherein, the front camera may include a TOF camera. The TOF camera includes TX and RX, TX can be used to transmit optical signals (infrared light or laser pulses), and RX can be used to receive imaging. TX can be, for example, an infrared light transmitter. RX can be, for example, a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor.

Exemplarily, as shown in (a) in Figure 2, the light signal (infrared light or laser pulse) can be continuously sent to the target (for example, user) through the optical transmitter (Tx) of the TOF camera, and the TOF camera The sensor end (Rx) of the sensor receives the light signal returned by the measured target, as shown in (b) in Figure 2, and the depth information of the measured target can be obtained according to the phase difference (delay) of the transmitted and received light signals.

Among them, Tx and Rx can exchange information through the bus. For example, Rx may send a configuration parameter to Tx through a bus (for example, a serial peripheral interface (Serial Peripheral Interface, SPI) bus), where the configuration parameter is used to indicate the address of the register corresponding to Tx and the value of the register. For example, the address of the register corresponding to Tx may be 0x11, and the storage space corresponding to 0x11 may store the current value. Tx can work at a corresponding current value based on a corresponding configuration parameter so as to emit an optical signal with a corresponding light intensity. Rx can acquire corresponding image data based on the reflected light of the light signal of corresponding intensity emitted by the transmitter. It should be noted that the Tx can emit light signals with different light intensities when working at different current values. For example, Tx works at the first current value and can emit an optical signal with the first light intensity. Tx works at the second current value, and can emit an optical signal with a second optical intensity. The second current value is greater than the first current value. The second light intensity is greater than the first light intensity. The image data acquired by the Rx based on the reflected light of the light signal with different intensities is also different. For example, when Tx works at the first current value and emits an optical signal of the first light intensity, Rx acquires the first image data within the corresponding exposure time; Tx works at the second current value and emits the second light intensity When the optical signal is light, Rx acquires the second image data within the corresponding exposure time; the second image data is different from the first image data.

When Tx works under the corresponding current value, it can judge its own working state, which can be normal or abnormal. Rx can request the working state of Tx to Tx through the bus, and Tx can feed back its own working state (for example, normal or abnormal) to Rx through the bus, so that Rx can obtain the working state of Tx. Rx may pack the working status of Tx, its own working status and the working modes of both into the first data packet (for example, Metadata). Rx may also pack Metadata and image data obtained based on reflected light into a second data packet (for example, RAW Data).

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card. The internal memory 121 may be used to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 . For example, in the embodiment of the present application, the processor 110 may execute instructions stored in the internal memory 121, and the internal memory 121 may include a program storage area and a data storage area. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The earphone interface 170D is used for connecting wired earphones.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 . The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like. The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on.

The methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure.

The software system of the above-mentioned electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through interfaces. In some embodiments, the Android system may include an application program layer, an application program framework layer, an Android runtime (Android runtime) and a system library, a hardware abstraction layer (hardware abstraction layer, HAL) and a kernel layer. It should be noted that the embodiment of the present application is illustrated with the Android system as an example. In other operating systems (such as Hongmeng system, IOS system, etc.), as long as the functions realized by each functional module are similar to the embodiments of the present application, the present application can also be implemented. scheme.

Wherein, the application program layer may include a series of application program packages.

As shown in FIG. 3 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, lock screen application, and setting application. Of course, the application layer may also include other application packages, such as payment applications, shopping applications, bank applications, etc., which are not limited in this application.

Wherein, the setting application has a function of recording a face, and the recorded face is used for face unlocking. The lock screen application has a function of unlocking in response to a user's unlocking operation (for example, pressing a power key). The screen lock application can perform unlocking processes such as face unlocking, fingerprint unlocking, and password unlocking. This embodiment of the present application mainly uses face unlocking as an example for illustration.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, it may include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service (Camera Service) and a face recognition service, etc., which are not limited in this embodiment of the present application.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), OpenGL ES, SGL, etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

OpenGL ES is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

SGL is a graphics engine for 2D graphics.

Android Runtime (Android Runtime) includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android. The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The HAL layer is the encapsulation of the Linux kernel driver, provides an interface upwards, and shields the implementation details of the low-level hardware.

The HAL layer can include Wi-Fi HAL, audio (audio) HAL, camera HAL (Camera HAL) and face recognition control module (Face CA), etc.

Wherein, the camera HAL is the core software framework of the Camera, and the camera HAL may include a sensor node (sensornode) and an image front end (image front end, IFE) node (IFE node). The sensor node and the IFE node are the components (nodes) in the image data and control instruction transmission path (also called the transmission pipeline) created by the camera HAL.

The face recognition control module is the core software framework/application of face recognition.

Face Trusted Application (Face TA): an application for face recognition running in the TEE environment. In this embodiment of the present application, Face TA is referred to as a face recognition TA.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

Among them, the camera driver is the driver layer of the Camera device, which is mainly responsible for the interaction with the hardware.

The hardware layer includes display, TOF camera, IFE module and secure memory (Secure Buffer), etc.

Among them, the secure memory refers to the memory with security protection function, which can be used to store the raw data collected by the TOF camera.

A TOF camera, also known as a TOF sensor (TOF sensor), can include a transmitter (TX) and a receiver (RX). TX is used to emit infrared light or laser pulses, and RX is used to receive reflected light and image it.

IFE module (IFE-Lite): It can be called an image pre-processing module, which can be used to forward image data, and the image data is not processed during the forwarding process.

The following describes the software modules involved in the face recognition method provided by the embodiment of the present application and the interaction between the modules. As shown in Figure 4, the lock screen application in the application layer can interact with the face recognition SDK, and the face recognition SDK can communicate with the face recognition in the framework layer by calling a preset application programming interface (application programming interface, API) interface Service interaction, the face recognition service can interact with the face recognition control module in the HAL layer, the face recognition control module can interact with the camera HAL in the HAL layer through the camera service in the framework layer, or the face recognition control module can Interact directly with the camera HAL in the HAL layer. The camera HAL can include sensor nodes and IFE nodes. The sensor node can interact with the camera driver module in the kernel layer, and the camera driver module can be used to drive the TOF camera in the hardware layer to collect images in the default working mode (for example, the human eye safety mode, please refer to the description in S112 below) data. The IFE module can store the image data collected by the TOF camera into a secure memory. The storage location of the image data collected by the TOF camera in the secure memory can be represented by a file descriptor (file descriptor, FD). The IFE module can send the FD (for example, FD1) of image data to the camera driver module, and the camera driver module can pass FD1 to the IFE node of the camera HAL. After the IFE node receives FD1, it can pass FD1 to the sensor node of the camera HAL. , after the sensor node receives FD1, it triggers the process of reading the working status of the TOF camera. That is, the sensor node can read the working state register of the TOF camera (that is, the register used to store the working state of the TOF camera) through the camera driver module, determine the human eye safety detection result according to the value of the working state register, and according to the human eye safety detection result Switch the working mode of the TOF camera. After the working mode of the TOF camera is switched, the camera HAL can continue to interact with the camera driver module, so that the camera driver module can drive the TOF camera to capture data from the switched working module (for example, face ID mode, please refer to the description in S112 below) Image data, the image data can be stored in the safety memory, the FD (for example, FD2) corresponding to the image data can be passed to the face recognition TA through the IFE module, the camera driver module, the IFE node, the camera service and the face control module, The face recognition TA can read image data from the safety memory according to FD2 and process it, and feed back the processing result (face recognition success or face recognition failure) to the face recognition control module. The face recognition control module can feed back the processing results to the lock screen application through the face recognition service and face recognition SDK, so that the lock screen application can determine whether to unlock (if the face recognition is successful, unlock; if the face recognition fails, not unlock, i.e. unlocking failed). Wherein, the solid line arrows in FIG. 4 may be used to represent the control flow, and the dotted line arrows may be used to represent the data flow.

Specifically, as shown in Figure 5, the sensor node in the camera HAL can be used to select the working mode of the TOF camera, including eye-safe mode (first working mode), face ID mode (second working mode) and TX off mode (the third working mode), etc., for details of various working modes, please refer to the description in S112 below. The default initial working mode of the TOF camera can be eye-safe mode. When the TOF camera works in the eye-safe mode, it can read the eye-safe safety current value (first current value) calibrated by the production line from the memory, update the eye-safe mode configuration according to the current value, and set the human eye The configuration parameters of the safe mode are sent to the camera driver module. The camera driver module can be used to drive the TOF camera in the hardware layer to collect image data in a default working mode (eg, eye-safe mode, for details, refer to the description in S112 below). The storage location of the image data collected by the TOF camera in the secure memory can be represented by FD. The IFE module can send the FD (for example, FD1) of image data to the camera driver module, and the camera driver module can pass FD1 to the IFE node of the camera HAL. After the IFE node receives FD1, it can pass FD1 to the sensor node of the camera HAL. , the sensor node can read the working status register of the TOF camera (that is, the register used to store the working status of the TOF camera) through the camera driver module, and determine the human eye safety detection result according to the value of the working status register. After the human eye safety detection result is obtained, the mode switching process can be performed. Specifically, if the eye safety detection result is successful, the TOF camera can switch to the face ID mode. If the human eye safety detection result fails, the TOF camera can switch to the TX off mode. In this way, the safety of human eyes can be guaranteed.

For ease of understanding, the method provided in the embodiment of the present application will be specifically introduced below in conjunction with the accompanying drawings.

As shown in Figure 6, the embodiment of the present application provides a method for eye safety detection and face recognition based on TOF images, and the process is as follows:

S101. The lock screen application invokes the face recognition SDK to perform face recognition.

When the user's unlocking operation (the first operation) is detected, the lock screen application invokes the face recognition SDK to perform face recognition. Wherein, the user's unlocking operation includes the user picking up the mobile phone, or pressing the power button, or operating (clicking, sliding, etc.) on the screen, or pulling out the charging cable and other operations.

At the same time, the lock screen application can register a callback with the face recognition SDK. The function of registering this callback is that after the face recognition SDK obtains the face recognition result, it can return the face recognition result to the lock screen application.

S102. The face recognition SDK sends a face recognition request to the face recognition service.

Wherein, the face recognition request carries the identification of the face recognition type, the resolution size of the image and the data stream format. Wherein, the face recognition type includes a 2D face recognition type (for example, may correspond to an identifier 0) and a 3D face recognition type (for example, may correspond to an identifier 1).

Exemplarily, the face recognition type carried in the face recognition request may be 1 (that is, the 3D face recognition type), the resolution of the image may be 1280x2898 pixels (pixel), and the data stream format may be the original image format ( raw image format, RAW) 16.

At the same time, the face recognition SDK can register a callback with the face recognition service. The function of registering this callback is that when the face recognition service obtains the face comparison result, it can return the face recognition result to the face recognition SDK.

S103. The face recognition service sends a face recognition request to the face recognition control module.

For the face recognition request, please refer to the description of S102, which will not be repeated here.

That is to say, the face recognition SDK can notify the face recognition control module to perform face recognition through the face recognition service. The face recognition service can send the face recognition request received from the face recognition SDK to the face recognition control module.

At the same time, the face recognition service can register a callback to the face recognition control module. The function of registering this callback is that when the face recognition control module obtains the face comparison result, it can return the face comparison result to the face recognition service. .

S104. In response to receiving the face recognition request, the face recognition control module matches the camera according to the face recognition request.

Specifically, the face recognition control module can obtain the identity of the face recognition type, image resolution and data stream format from the face recognition request, and determine the matching camera by querying the camera capabilities from the camera service.

It should be understood that, during the boot process of the electronic device, the camera service may send a camera capability query request to the camera HAL, and the camera capability query request is used to query the camera capability supported by the electronic device. After receiving the camera capability query request, the camera HAL may send the camera capability supported by the electronic device to the camera service, and the camera service may store the received camera capability supported by the electronic device. Among them, the camera capabilities supported by the electronic device include the camera identification (identity, ID) of each camera, the maximum supported resolution size, the format of the data stream, and whether the camera supports collecting depth information, etc.

Exemplarily, assuming that there are three cameras installed on the mobile phone, the capability information of the three cameras may be shown in Table 1:

Table 1

camera ID installation location Supported maximum resolution data stream format depth information 1 rear 4096x3072pixel YUY no 2 Front 3264x2448 pixel YUY no 3 Front 1280x2898 pixel RAW16 Yes

Among them, the camera whose camera ID is 3 is a TOF camera, which supports collecting depth information. Cameras with camera IDs 1 and 2 are common cameras and do not support collecting depth information. Of course, more front or rear cameras can be installed on the mobile phone, for example, two front cameras and four rear cameras can be installed on the mobile phone.

The face recognition control module can send a camera capability query request to the camera service, and the camera service can send the capability of the camera supported by the electronic device to the face recognition control module, and the face recognition control module can determine according to the capability of the camera supported by the electronic device For a matched camera, for example, it may be determined that the matched camera is a camera with an ID of 3 (that is, a TOF camera).

It should be noted that Table 1 is only an example, and the data stream formats corresponding to each camera may include multiple types. For example, the camera whose camera ID is 1 may not only correspond to the YUY data stream format, but may also correspond to the RAW16 data stream format, which is not limited in this application.

S105. The face recognition control module sends a request to open the camera (Camera) to the camera service.

Exemplarily, the face recognition control module may send a request for opening the Camera to the camera service through a vendor native development kit (vendor native development kit, VNDK) interface. Wherein, the request for opening the Camera carries information such as a security identifier, a camera ID, a size of a resolution, and a data stream format. Wherein, the security identifier is used to indicate that the data is stored in the security Buffer. That is to say, the security mark can be used to apply for a piece of security memory, which is then used to store the data collected by the camera. For example, the security flag can be 1 or 0, 1 means storing data in a secure buffer, and 0 means storing data in a non-secure buffer.

Exemplarily, the security identifier carried in the request for opening the Camera may be 1 (that is, the data is stored in a security buffer), the resolution of the image may be 1280x2898 pixel, the data stream format may be RAW16, and the camera ID may be 3.

At the same time, the face recognition control module can register a callback to the Camera service, and the registered callback is used to notify the face recognition control module that the camera is opened after the camera service is completed.

S106. In response to receiving the request for opening the Camera, the camera service sends a request for opening the Camera to the camera HAL, and the request for opening the Camera carries information such as security identification, camera ID, resolution size, and data stream format.

When the camera service calls the camera HAL, the camera service can send information such as the security ID, camera ID, image resolution, and data stream format to the camera HAL. The camera HAL can cache information such as security identification, camera ID, image resolution, and data stream format for a preset time.

At the same time, the camera service can register a callback with the camera HAL, and the callback is used for the camera HAL to notify the camera service of the result of creating the channel.

S107. The camera HAL creates a corresponding channel according to the camera ID, image resolution and data stream format.

The camera HAL can select available nodes according to the camera ID, resolution and data stream format, and then create corresponding paths according to the available nodes. Exemplarily, if the resolution is 1280x2898 pixel, the data stream format is RAW16, and the camera ID is 3, then it can be determined to select the sensor node and the IFE node. This is because the sensor node and the IFE node can support the transmission of the data collected by the camera whose ID is 3 with a resolution of 1280x2898 pixel and a data stream format of RAW 16.

Wherein, the path corresponding to the sensor node may be: a path composed of sensor node-camera driver-TOF camera-IFE module-secure memory. The path corresponding to the IFE node may be: a path composed of an IFE module (carrying an FD)-camera driver-IFE node. The camera HAL can connect the output port of the sensor node and the input port of the IFE node at the HAL layer. Thus, the path corresponding to the sensor node and the path corresponding to the IFE node can form a closed-loop path. After the path is created, the hardware in the path is powered on (that is, the hardware circuit is powered on) and waits for data requests.

S108. The camera HAL returns the path creation result to the camera service.

Wherein, the result of creating the path can be success or failure. If the result of creating the path is a failure, the camera HAL notifies the camera service of the path creation failure. If the result of creating the path is successful, the camera HAL notifies the camera service that the path is successfully created, and S109 and its subsequent steps can be continued.

S109. In response to receiving the notification that the path is successfully created, the camera service returns a message that the camera is opened to the face recognition control module.

It can be understood that the completion of opening the camera refers to the completion of preparatory work (for example, camera parameter configuration, power-on and other preparatory work) before the camera takes pictures or takes pictures.

S110. In response to receiving the message that the opening of the camera is completed, the face recognition control module sends a data request to the camera service.

Among them, the data request is used to request to obtain the data stream of the camera.

S111. In response to receiving the data request sent by the face recognition control module, the camera service calls the camera HAL to obtain a data stream.

S112. The camera HAL selects the working mode of the camera through the sensor node.

Specifically, the sensor node can select the camera working mode corresponding to the sensor node through the camera resolution and data stream format cached in S106. Exemplarily, the sensor node may select the camera working mode corresponding to the sensor node by looking up a table (for example, Table 2).

Table 2

Among them, the eye safety mode (EyeSafe Mode) refers to the mode in which the Tx of the TOF camera works under a small current (current less than the preset threshold, the first current value). The eye safety mode is used to check whether the TOF camera has damage. Face ID mode (Face ID Mode) is a mode in which the Tx of the TOF camera works at a normal current (the second current value, within the preset threshold range). The Face ID mode is used for safe face unlocking and safe payment, etc. Scenes. Wherein, the second current value is greater than the first current value. TX off mode (Tx OFF Mode) refers to the mode in which the Tx of the TOF camera is not powered (so that it does not emit light). The TX off mode is used to detect that the Tx device of the TOF camera is damaged or cannot work normally. This is because the use of power on when the Tx device is damaged may have an adverse effect on the human eye. Therefore, when the Tx device of the TOF camera is detected to be damaged, the TX off mode is used to prevent the Tx device from being powered on, thereby avoiding damage to the human eye. .

Certainly, the working mode of the camera may include more, which is not limited in this application.

According to Table 2, when the maximum resolution of the image is 1280x2898 pixel and the data stream format is Raw, the working modes of the camera can include eye safety mode, face ID mode, TX off mode, etc. The sensor node can default the initial working mode of the camera to be the human eye safety mode. When the working mode of the camera is eye-safe mode, the sensor node can read the eye-safe current value calibrated by the production line (that is, the current value that is not harmful to human eyes) from the memory (for example, oeminfo), and according to the eye-safe current value Update the eye-safe mode setting of the TOF camera. For example, the address of the current register of the TOF camera can be obtained by means of table lookup, and the eye-safe current value can be written into the current register of the TOF camera. It should be understood that the sensor node may store the addresses of the registers of the TOF camera, and the addresses of the registers of the TOF camera may be shown in Table 3.

table 3

Register ID storage data type address 1 electric current 0x1 2 resolution 0x2 3 data stream format 0x3 4 Working status of TOF camera device 0x4 5 Working mode of TOF camera 0x5

Exemplarily, by looking up Table 3, it can be determined that the address of the register corresponding to the current value is 0x1, so that the eye-safe current value can be written into the storage space corresponding to 0x1.

S113. The sensor node sends the configuration parameters of the eye safety mode to the camera driver (Camera Driver) module at the Kernel layer.

Exemplarily, the configuration parameters of the eye-safe mode may be: the current value is 700 mA, the exposure time of the IR grayscale image is 10 μs, and the exposure time of the depth image is 10 μs.

S114a, the camera driver module writes (updates) the configuration parameters of the eye safety mode into the register of the TOF camera.

That is, the camera driver module can send configuration parameters of the eye safety mode to the TOF camera.

Exemplarily, the camera driver module may write the configuration parameters of the eye safety mode into the RX register of the TOF camera through an integrated circuit bus (inter-integrated circuit, I2C). The address corresponding to the RX register can be 0x01. There may be multiple registers corresponding to RX, which are not limited in this application. That is, the configuration parameters of the eye safety mode can be sent to the RX of the TOF camera through I2C. Wherein, the configuration parameters of the eye safety mode include configuration parameters for RX and TX. For example, the configuration parameter for TX may be the first current value. A configuration parameter for RX may be exposure time. RX can write the configuration parameters corresponding to TX into the registers corresponding to TX through the SPI bus. The address of the register corresponding to TX may be 0x11. There may be multiple registers corresponding to TX, which is not limited in this application.

S114b. The camera driver module sends a start (stream on) command/instruction (second start command) to the TOF camera.

The stream on command is used to drive the TOF camera for data collection.

It should be noted that before S114b and after S114a, the camera driver module can also send a message that the configuration parameters are written to the sensor node; in response to receiving the message that the configuration parameters are written, the sensor node sends a startup command to the camera driver module (section a start command).

S115. In response to receiving the stream on command, the TOF camera collects RAW Data1 based on the human eye safety mode.

Specifically, in response to receiving the stream on command, RX can send a request for a light-emitting signal to TX, and TX works at a corresponding current value (first current value) to send an optical signal of the first light intensity; (for example, 10 us) to receive an optical signal, and the optical signal received by the RX includes reflected light of the optical signal with the first light intensity. Based on the received light signal, the RX acquires first image data.

That is to say, RAW Data 1 (raw data 1) refers to the image data obtained by the Rx of the TOF camera receiving reflected light and imaging when the Tx of the TOF camera is working at the eye safety current value calibrated by the production line to emit light signals to the face (first image data). Wherein, when Tx works at the eye-safe current value calibrated by the production line, the emitted light signal is the first light intensity.

Among them, RAW Data contains Metadata. Exemplarily, Metadata saves the working mode of the current TOF camera (for example, eye-safe mode), the size of the light current value (for example, the eye-safe current value calibrated by the production line), and the working status of the TOF camera device (for example, normal or anomalies) and map exposure values (eg, 10μs) and other information.

S116. The TOF camera sends the RAW Data 1 collected based on the human eye safety mode to the IFE module.

Exemplarily, the TOF camera can transmit the RAW Data 1 collected by the TOF camera to the IFE module through a mobile industry processor interface (mobile indμstryprocessor interface, MIPI). The IFE module may also be called an image pre-processing module (IFE-Lite), and the IFE module may not process the RAW Data 1.

S117. The IFE module sends the RAW Data 1 to a secure memory (Secure Buffer) for storage.

The storage location of the RAW Data 1 collected by the TOF camera based on the eye-safe mode in the secure memory can be represented by FD1.

Exemplarily, when FD1 is 69, it may indicate that the storage location is XX secure memory; when FD1 is 96, it may indicate that the storage location is YY non-secure memory (ordinary memory).

S118. The IFE module sends FD1 to the camera driver module.

S119. The camera driver module sends FD1 to the IFE node.

S120. The IFE node sends FD1 to the sensor node of the camera HAL.

S121. In response to receiving FD1, the sensor node sends a request for reading the working status register to the camera driver module, and the request carries the address of the working status register.

The request for reading the register is used to request the camera driver module to read the value of the register storing the working state of the TOF camera.

S122. The camera driver module reads the value of the working status register according to the address of the working status register.

The working status register can be set on the TOF camera, so the camera driver module can read the value of the corresponding register from the TOF camera according to the address of the working status register.

It should be noted that the working status of the TOF camera is stored in the working status register. The working status of the TOF camera can be represented by 0 or 1, 0 can indicate that the working status is abnormal, and 1 can indicate that the working status is abnormal.

S123. The camera driver module sends the value of the working status register to the sensor node of the camera HAL.

S124. The sensor node of the camera HAL calculates an eye safety detection result according to the value of the working state register.

If the working status of the device is normal (for example, the value of the working status register is 0), the eye safety detection result is safe/normal (or the human eye safety detection is successful). If the working status of the device is abnormal (for example, the value of the working status register is 1), the eye safety detection result is unsafe/abnormal (or the human eye safety detection fails).

S125. The sensor node of the camera HAL determines the working mode of the TOF camera based on the human eye safety detection result.

If the human eye safety detection result is safe (normal), the working mode of the TOF camera is determined to be the face ID mode; if the human eye safety detection result is unsafe (abnormal), then the working mode of the TOF camera is determined to be the TX off mode.

It should be noted that configuration parameters corresponding to the face ID mode and the TX off mode may be stored in the sensor node.

Exemplarily, the configuration parameters corresponding to the face ID mode may be: the current value (second current value) is 2800mA, the exposure time of the IR grayscale image is 500 μs, the depth is yes, and the image exposure time is 800 μs. The configuration parameters corresponding to the TXOFF mode can be: the current value (the third current value) is 0mA, the exposure time of the IR grayscale image is 10μs, the depth is no, and the image exposure time is 10μs.

It should be noted that, the embodiment of the present application does not limit the execution sequence of S101-S125. In some embodiments, after S101-S107 is executed, S112-S124 may be executed directly, S108-S111 may be executed after S124, and S125 may be executed after S111. Of course, S101-S125 can also have other combinations, so as to ensure that the sensor node of the camera HAL can obtain the human eye safety detection results, so as to determine the working mode of the TOF camera, and this application will not repeat them here.

The following takes the working mode of the TOF camera determined by the sensor node as the face ID mode as an example. After S125, S126-S142 can also be included:

S126. The sensor node sends the configuration parameters of the face ID mode to the camera driver module.

S127. The camera driver module writes the configuration parameters of the face ID mode into the registers of the TOF camera, so as to drive the TOF camera to collect data based on the face ID mode.

That is, the camera driver module can send configuration parameters of the face ID mode to the TOF camera.

Exemplarily, the camera driver module can write the configuration parameters of the face ID mode into the register of the TOF camera through I2C. That is, the camera driver module can send configuration parameters of the face ID mode to the TOF camera through I2C.

S128. The TOF camera collects RAW Data 2 based on the face ID mode.

Wherein, RAW Data 2 may be the image data (second image data) obtained by receiving the reflected light and imaging the Rx of the TOF camera when the Tx of the TOF camera works at the second current value (for example, 2800mA) to emit light signals to the face. When the Tx of the TOF camera works at the second current value, the light signal emitted is the second light intensity. The second light intensity is greater than the first light intensity.

Among them, RAW Data 2 contains Metadata. Exemplarily, Metadata saves the working mode of the current TOF camera (for example, face ID mode), the size of the light current value (for example, 2800mA), the working state of the TOF camera device (for example, normal) and the exposure time of the image (for example, 800μs) and other information.

S129. The TOF camera sends RAW Data 2 to the IFE module.

Exemplarily, the TOF camera can transmit the RAWData2 collected by the TOF camera based on the face ID mode to the IFE module through MIPI.

S130. The IFE module sends the RAW Data 2 to a safe memory for storage.

The storage location of the RAW Data 2 collected by the TOF camera based on the face ID mode in the secure memory can be represented by FD2.

FD2 in this step may be the same as or different from FD1 in S117. When FD2 in this step is the same as FD1 in S117, that is, the RAW Data 2 collected by the TOF camera based on the face ID mode and the RAW Data 1 collected by the TOF camera based on the eye-safe mode in S117 are stored in the same safe memory. The RAW Data 1 collected by the TOF camera based on the human eye safety mode in S117 can be deleted, so that the RAW Data 2 collected by the TOF camera based on the face ID mode can be stored in the safe memory again. When FD2 is different from FD1, the RAW Data 2 collected by the TOF camera based on the face ID mode and the RAW Data 1 collected by the TOF camera based on the human eye safety mode in S117 can be stored in different secure memories.

S131. The IFE module sends FD2 to the camera driver module.

S132. The camera driver module sends FD2 to the IFE node.

S133. The IFE node sends the FD2 to the camera service through the interface of the camera HAL.

S134. The camera service sends FD2 to the face recognition control module.

S135. The face recognition control module sends FD2 to the face recognition TA.

S136. The face recognition TA reads the RAW Data 2 from the safety memory according to the FD2.

S137. The face recognition TA obtains a face recognition result according to the RAW Data 2 .

Specifically, the face recognition TA can obtain the working mode of the TOF camera from the Metadata in RAW Data 2, for example, it can be a face ID mode. Then, the face recognition TA can process the second image data in RAW Data 2 through the TOF algorithm to obtain the first grayscale image and the first depth image, and then perform face recognition based on the first grayscale image through the face ID algorithm , based on the first depth map and anti-counterfeiting detection, the face recognition result is obtained.

It should be noted that the face recognition TA also stores the face information entered by the user before, and the TOF algorithm can convert the face information entered by the user into a grayscale image and a depth image. If the currently collected face information (the RAW Data collected by the TOF camera based on the face ID mode, that is, RAW Data 2) corresponds to the previously entered face information (that is, the RAW Data collected by the electronic device when the user performs the face entry operation) It can be considered as the same user (that is, the operation of entering the face and the operation of unlocking are the same user), and if the currently collected face information includes depth information, it can be considered that the current user is Authentic and credible (not disguised by photos, videos, etc.), at this time, the face of the current user can be considered safe, that is, the face recognition result is successful. If the currently collected face information (the RAW Data collected by the TOF camera based on the face ID mode, that is, RAW Data 2) corresponds to the previously entered face information (that is, the RAW Data collected by the electronic device when the user performs the face entry operation) or, if the currently collected face information does not include depth information, it is considered that the current user's face is not safe, that is, the face recognition result is a failure.

S138. The face recognition TA sends the face recognition result to the face recognition control module.

S139. The face recognition control module sends the face recognition result to the face recognition service.

The face recognition control module may transmit the face recognition result (success or failure) to the face recognition service based on the callback registered by the face recognition service before (in S103).

S140. The face recognition service transmits the face recognition result to the face recognition SDK.

The face recognition service transmits the face recognition result (success or failure) to the face recognition SDK based on the callback registered by the face recognition SDK before (in S102).

S141. The face recognition SDK transmits the face recognition result to the lock screen application.

The face recognition SDK transmits the face recognition result (success or failure) to the lock screen application based on the callback registered by the previous (in S101 ) lock screen application.

S142. The screen lock application determines whether to unlock according to the face recognition result.

If the face recognition result is successful, the lock screen application can be successfully unlocked, so that the electronic device can display a desktop or an application (system application or third-party application) interface. If the face recognition result is failure, the lock screen application will not be unlocked, that is, face unlocking fails. After the face unlock fails, the lock screen application can disable the face recognition function within a period of time (for example, 5 minutes) during which the face recognition fails.

Exemplarily, if the user sets face unlocking, as shown in (a) in Figure 7, when the user picks up the mobile phone for face recognition, in response to the user's operation of picking up the mobile phone, as shown in (b) in Figure 7 ), the mobile phone can display a lock screen interface 701, and the mobile phone can display an unlock icon 702 and a prompt text "recognizing face" 703 on the lock screen interface 701 during the face recognition process.

If the face recognition is successful, as shown in (a) in Figure 8, an interface 704 can be displayed, which can include an unlock icon 705 (in an open state, which can visually prompt the user that the face unlock is successful) and the prompt text "up "Slide to enter" 706, in response to the user's sliding up operation, the mobile phone can display the interface of the desktop or application (system application or third-party application). Or, as shown in (b) in FIG. 8 , if the face recognition is successful, the mobile phone can be directly unlocked without any additional operation by the user, that is, the desktop 707 can be displayed immediately (or the interface of the application can be directly displayed).

If the face recognition fails, as shown in (a) in Figure 9, an interface 708 can be displayed, which can include an unlock icon 709 (in a closed shape, which can visually prompt the user that the user's face has not been unlocked successfully) and the prompt text " Unrecognized, double-click the screen to try again" 710, in response to the user's double-click operation, the mobile phone can perform face recognition again (that is, collect the user's face information again for comparison and anti-counterfeiting judgment). Or, in response to the user's upward sliding operation on the interface 708, as shown in (b) in Figure 9, the mobile phone can display an interface 711, and after entering the interface 711, the mobile phone can perform face recognition again, and the interface 711 can include a face If the identification icon 712 and the prompt text "face recognition in progress" 713 are still not recognized successfully, as shown in (c) in Figure 9, the mobile phone can display an interface 714, which can include the prompt text "unrecognized successfully, Click here to try again" 715, the user can click on the corresponding position to re-trigger face recognition, or can also enter the password through the soft keyboard 716 to unlock, avoiding the problem of poor user experience caused by failure to identify successfully.

)中进行支付或转账操作)的人脸识别，用户在安全注册或登录应用程序的人脸安全验证(例如，用户在

中进行注册或登录操作)等场景中，本申请不做限定。即可以将锁屏应用替换为购物应用、聊天应用、支付应用、银行应用或理财应用等，本申请不做限定。It should be noted that, the above-mentioned embodiment takes the method flow of face unlocking by the lock screen application as an example to illustrate the selection of the working mode of the TOF camera, and the selection of the working mode of the TOF camera can also be applied to payment or transfer (for example, The user is in the payment application/finance application/chat application/shopping application (for example,

), face recognition for payment or transfer operations in ), face security verification for users to securely register or log in to the application (for example, users in

In the scenarios such as registering or logging in), this application does not limit it. That is, the lock screen application may be replaced with a shopping application, a chat application, a payment application, a banking application, or a wealth management application, which is not limited in this application.

The following takes the working mode of the TOF camera determined by the sensor node as the TX off mode as an example. As shown in Figure 10, after S125, S150-S167 can also be included:

S150. The sensor node of the camera HAL determines that the working mode of the TOF camera is the TX off mode based on the human eye safety detection result.

S151. The sensor node sends configuration parameters of the TX off mode to the camera driver module.

S152. The camera driver module writes (updates) the configuration parameters of the TX off mode into the register of the TOF camera, so as to drive the TOF camera for data collection.

That is, the camera driver module can send configuration parameters of the TX off mode to the TOF camera.

Exemplarily, the camera driver module can write the configuration parameters of the TX off mode into the register of the TOF camera through I2C. That is, send the configuration parameters of the TX off mode to the TOF camera through I2C.

S153, the TOF camera collects RAW Data 3 based on the TX off mode.

Among them, RAW Data 3 can be the image data (third image data) obtained by imaging the Rx of the TOF camera after receiving reflected light (no emitted light or ambient emitted light) when the Tx of the TOF camera is not powered on and does not emit light, usually without clarity. A "black map" of face images.

Among them, RAW Data 3 contains Metadata. Exemplary, Metadata saves the working mode (for example, TX off mode) of current TOF camera, the size (for example, 0mA) of light current value (third current value), TOF camera device working state (for example, abnormal) and graph Exposure time (for example, 10μs) and other information.

S154, the TOF camera transmits RAW Data 3 to the IFE module.

Exemplarily, the TOF camera can transmit the RAW Data 3 collected by the TOF camera to the IFE module through MIPI.

S155. The IFE module sends the RAW Data 3 to a safe memory for storage.

The storage location of the RAW Data 3 collected by the TOF camera based on the TX off mode in the secure memory can be represented by FD3.

FD3 in this step may be the same as or different from FD1 in S117. When FD3 in this step is the same as FD1 in S117, that is, the RAW Data 3 collected by the TOF camera based on the TX off mode and the RAW Data 1 collected by the TOF camera based on the eye-safe mode in S117 are stored in the same safe memory. The RAW Data 1 collected by the TOF camera in S117 based on the eye-safe mode can be deleted, so that the RAW Data 3 collected by the TOF camera based on the TX off mode can be stored in the safe memory again. When FD3 is different from FD1, the RAW Data 3 collected by the TOF camera based on the TX off mode and the RAW Data 1 collected by the TOF camera in the S117 based on the eye-safe mode can be stored in different secure memories.

S156. The IFE module sends FD3 to the camera driver module.

S157. The camera driver module sends FD3 to the IFE node.

S158. The IFE node sends the FD3 to the camera service through the interface of the camera HAL.

S159. The camera service sends the FD3 to the face recognition control module.

S160. The face recognition control module sends FD3 to the face recognition TA.

S161. The face recognition TA reads the RAW Data 3 from the safety memory according to the FD3.

S162. The face recognition TA obtains a face recognition result according to the RAW Data 3 .

Specifically, the face recognition TA can obtain the current Tx off mode from the Metadata data in RAW Data 3 collected by the TOF camera based on the Tx off mode, and then use the TOF algorithm to obtain the second grayscale image and the second depth based on the third image data In the figure, face recognition is performed based on the second grayscale image through the face ID algorithm, anti-counterfeiting detection is performed based on the second depth image, and the face recognition result is obtained.

It should be noted that when the TOF camera works in the TX off mode, the face recognition result is a failure. This is because the TOF camera cannot emit light in the TX off mode, so the TOF camera cannot collect a clear face image. Even if the currently unlocked user is an authenticated user (ie the owner), the face recognition result is still a failure.

S163. The face recognition TA transmits the face recognition result (failed) to the face recognition control module.

That is, the face recognition TA can notify the face recognition control module that the face recognition result is failure.

S164. The face recognition control module transmits the face recognition result (failed) to the face recognition service.

The face recognition control module transmits the face recognition result (failed) to the face recognition service based on the callback registered by the face recognition service before. That is, the face recognition control module notifies the face recognition service that the face recognition result is failure.

S165. The face recognition service transmits the face recognition result (failed) to the face recognition SDK.

The face recognition service passes the face recognition result (failure) to the face recognition SDK based on the callback registered by the face recognition SDK. That is, the face recognition service can notify the face recognition SDK that the face recognition result is a failure.

S166. The face recognition SDK transmits the face recognition result (failed) to the lock screen application.

The Face Recognition SDK passes the face recognition result (failure) to the lock screen application based on the callback registered by the previous lock screen application. That is, the face recognition SDK can notify the lock screen application that the face recognition result is a failure.

S167. The screen lock application decides not to unlock according to the face recognition result (failed).

Since the face recognition result is failed, the lock screen application is not unlocked.

Exemplarily, if the face recognition fails, as shown in (a) in FIG. 9 , an interface 708 can be displayed, and the interface 708 can include an unlock icon 709 (closed, which can visually prompt the user that the face is not unlocked successfully) And prompt text "recognized successfully, double-tap the screen to try again" 710, in response to the user's double-tap operation, the mobile phone can perform face recognition again (that is, collect the user's face information again for comparison and anti-counterfeiting judgment). Or, in response to the user's sliding up operation on the interface 708, the mobile phone can display an interface 711. After entering the interface 711, the mobile phone can perform face recognition again. The interface 711 can include a face recognition icon 712 and a prompt text "Face recognition is in progress Recognition" 713, if the recognition is still not successful, the mobile phone can display an interface 714, which can include the prompt text "not recognized successfully, click here to try again" 715, the user can click on the corresponding position to re-trigger face recognition, or can Enter the password through the soft keyboard 716 to unlock, avoiding the problem of poor user experience caused by failure to identify successfully.

Some embodiments of the present application provide an electronic device, and the electronic device may include: a touch screen, a memory, and one or more processors. The touch screen, memory and processor are coupled. The memory is used to store computer program code comprising computer instructions. When the processor executes the computer instructions, the electronic device can execute various functions or steps performed by the electronic device in the foregoing method embodiments. For the structure of the electronic device, reference may be made to the structure of the electronic device 100 shown in FIG. 1 .

The embodiment of the present application also provides a chip system (for example, a system on a chip (SoC)), as shown in FIG. 11 , the chip system includes at least one processor 1101 and at least one interface circuit 1102 . The processor 1101 and the interface circuit 1102 may be interconnected through wires. For example, interface circuitry 1102 may be used to receive signals from other devices, such as memory of an electronic device. For another example, the interface circuit 1102 may be used to send signals to other devices (such as the processor 1101 or a touch screen of an electronic device). Exemplarily, the interface circuit 1102 can read instructions stored in the memory, and send the instructions to the processor 1101 . When the instructions are executed by the processor 1101, the electronic device may be made to execute various steps in the foregoing embodiments. Of course, the chip system may also include other discrete devices, which is not specifically limited in this embodiment of the present application.

The embodiment of the present application also provides a TOF camera, which can be used to implement the human eye safety mode, face ID mode, and TX off mode in the above embodiments, and the electronic device installed with the TOF camera can execute the above method. Each function or step performed by the electronic device in the example.

An embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes computer instructions, and when the computer instructions are run on the above-mentioned electronic device, the electronic device is made to execute the electronic device in the above-mentioned method embodiment. each function or step.

An embodiment of the present application further provides a computer program product, which, when running on an electronic device, causes the electronic device to perform the various functions or steps performed by the electronic device in the foregoing method embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be assigned by Completion of different functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. The above-mentioned storage medium includes: U disk, mobile hard disk, read only memory (read only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk, and other various media that can store program codes.

The above content is only the specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered within the protection scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (29)

1. A face recognition method is applied to electronic equipment, the electronic equipment comprises a time of flight (TOF) camera module, the TOF camera module comprises a transmitter used for transmitting optical signals and an image sensor used for receiving reflected light and imaging, and the method comprises the following steps:

receiving a first operation of a user, wherein the first operation is used for triggering face recognition;

controlling the transmitter to operate at a first light intensity;

determining whether the transmitter is in a normal operating state;

under the condition that the emitter is in a normal working state, controlling the emitter to work at a second light intensity, wherein the second light intensity is greater than the first light intensity;

controlling the image sensor to acquire image data;

performing the face recognition based on the image data.

2. The method of claim 1,

the light signal emitted when the emitter works at a first current value is the first light intensity, the light signal emitted when the emitter works at a second current value is the second light intensity, and the second current value is larger than the first current value.

3. The method of claim 1 or 2, wherein said determining whether the transmitter is in a normal operating state comprises:

determining a first parameter of the transmitter, the first parameter being indicative of an operating state of the transmitter;

if the first parameter is used for indicating that the working state of the transmitter is a normal working state, determining that the transmitter is in the normal working state; and if the first parameter is used for indicating that the working state of the transmitter is an abnormal working state, determining that the transmitter is in the abnormal working state.

4. The method according to any one of claims 1-3, further comprising:

under the condition that the emitter is in an abnormal working state, controlling the emitter to work at a third light intensity, wherein the third light intensity is 0;

controlling the image sensor to acquire image data;

performing the face recognition based on the image data.

5. The method of any of claims 1-4, wherein the performing the face recognition based on the image data comprises:

acquiring a gray scale map and a depth map based on the image data;

and comparing the human face based on the gray image, and performing anti-counterfeiting detection based on the depth image to obtain a human face recognition result.

6. The method according to any one of claims 1 to 5,

the first operation comprises an operation for unlocking the electronic equipment, an operation for online payment, an operation for entering a human face, or an operation for safely registering or logging in an application program.

7. The method of claim 5 or 6, further comprising:

determining whether to execute unlocking according to the face recognition result;

if the face recognition result is successful, unlocking is executed;

if the face recognition result is failure, unlocking is not executed or unlocking failure is displayed; or

Determining whether to execute payment according to the face recognition result;

if the face recognition result is successful, executing payment;

if the face recognition result is failure, not executing payment or displaying payment failure; or

Determining whether to execute face input according to the face recognition result;

if the face recognition result is successful, executing face input;

if the face recognition result is failure, not executing face input or displaying the face input failure; or

Determining whether to execute registration or login according to the face recognition result;

if the face recognition result is successful, performing registration or login;

and if the face recognition result is failure, not executing registration or login or displaying registration or login failure.

8. The method of claim 4, further comprising:

when the emitter is in an abnormal working state, prompting the user that the unlocking is failed; or prompt the user for payment failure; or prompting the user that the face input fails; or prompt the user for registration or login failure.

9. The method of any of claims 1-8, wherein the electronic device comprises a camera Hardware Abstraction Layer (HAL) and a camera driver module, wherein the camera HAL comprises a sensor node, and wherein controlling the transmitter to operate at a first light intensity comprises:

the sensor node determines that the working mode of the camera module is a first working mode; the first operating mode is used for indicating the transmitter to operate at a first current value;

the sensor node sends the configuration parameters of the first working mode to the camera driving module;

the camera driving module writes the configuration parameters of the first working mode into a register of the TOF camera module;

the camera driving module sends a message of completing configuration parameter writing to the sensor node;

in response to receiving the message that the writing of the configuration parameters is completed, the sensor node sends a first starting command to the camera driving module;

the camera driving module sends a second starting command to the TOF camera module;

the emitter works at the first current value, and the light signal emitted when the emitter works at the first current value is the first light intensity.

10. The method of claim 9, wherein the electronic device further comprises a first application, a face recognition Software Development Kit (SDK), a face recognition service, a face recognition control module, and a camera service, and wherein after receiving the first operation by the user, the method further comprises:

the first application calls the face recognition SDK to perform face recognition; the first application corresponds to the first operation, and the first application comprises a screen locking application, a shopping application, a chat application or a financing application;

the face recognition SDK sends a face recognition request to the face recognition service; the request of the face recognition carries the identification of the face recognition type, the resolution of the image and the data stream format;

the face recognition service sends the face recognition request to the face recognition control module;

the face recognition control module is matched with a camera module according to the face recognition request;

the face recognition control module sends a first request for opening a camera module to the camera service; the first request for opening the camera module carries a security identifier, an identifier ID of the camera module, the resolution of an image and a data stream format; the security identification is used for applying for a secure memory;

and the camera service sends a second request for opening a camera module to the camera HAL, wherein the second request carries the security identifier, the identifier ID of the camera module, the resolution of the image and the data stream format.

11. The method according to claim 10, wherein the determining, by the sensor node, that the working mode of the camera module is the first working mode specifically includes:

and the sensor node determines that the working mode of the camera module is a first working mode according to the resolution of the image, the data stream format and a preset rule.

12. Method according to claim 10 or 11, wherein after the camera service sends the camera HAL a second request to open a camera module, the method further comprises:

the camera HAL creates a path for transmitting data stream and control stream according to the ID of the camera module, the resolution of the image and the data stream format;

the camera HAL returns the result of creating a path to the camera service; the result of the creation of the path is successful;

the camera service returns a message of completing opening of the camera module to the face recognition control module;

the face recognition control module sends a data request to the camera service, wherein the data request is used for acquiring a data stream;

the camera service invokes the camera HAL to acquire a data stream.

13. The method of any of claims 9-12, wherein after the transmitter is operated at the first current value, the method further comprises:

the image sensor acquires an optical signal within the exposure time corresponding to the first working mode;

based on the received light signal, the image sensor acquires first image data.

14. The method of claim 13, wherein after the image sensor acquires the first image data, the method further comprises:

the image sensor sends a request for acquiring a first parameter to the transmitter;

the image sensor receiving the first parameter from the transmitter;

the image sensor obtains a first original RAW Data based on the first image Data and the first parameter; the first parameter is used for indicating the working state of the transmitter under the first current value.

15. The method of claim 14, wherein the electronic device further comprises an image processing module and a first memory, and wherein determining whether the transmitter is in a normal operating state comprises:

after the image sensor obtains first original RAW Data based on the first image Data and the first parameter, the image sensor sends the first RAW Data to the image processing module;

the image processing module sends the first RAW Data to the first memory for storage, and the first memory corresponds to the first FD;

the image processing module sends the first FD to the camera driving module;

the camera driving module sends the first FD to the camera HAL;

in response to receiving the first FD, the camera HAL sends a request to the camera drive module to read a first parameter indicating an operating status of the transmitter;

the camera driver module reads the first parameter from the register of the transmitter and sends the first parameter to the sensor node of the camera HAL;

the camera HAL determines whether the transmitter is in a normal operating state based on the first parameter.

16. The method as claimed in claim 15, wherein the determining, by the camera HAL, whether the transmitter is in a normal operating state according to the first parameter comprises:

if the first parameter is used for indicating that the working state of the transmitter is a normal working state, the camera HAL determines that the transmitter is in the normal working state;

and if the first parameter is used for indicating that the working state of the transmitter is an abnormal working state, the camera HAL determines that the transmitter is in the abnormal working state.

17. The method of claim 15 or 16, wherein said controlling said emitter to operate at a second light intensity comprises:

if the working state of the emitter is a normal working state, the camera HAL determines that the working mode of the camera module is a second working mode, and the second working mode is used for indicating that the emitter works at a second current value;

the camera HAL sends the configuration parameters of the second working mode to the camera driving module;

the camera driving module writes the configuration parameters of the second working mode into a register of the TOF camera module;

the camera driving module sends a message of completing configuration parameter writing to the sensor node;

in response to receiving the message that the writing of the configuration parameters is completed, the sensor node sends a third starting command to the camera driving module;

the camera driving module sends a fourth starting command to the TOF camera module;

the emitter works at the second current value, and the optical signal emitted when the emitter works at the second current value is the second light intensity.

18. The method of claim 17, wherein controlling the image sensor to acquire image data after the transmitter operates at the second current value comprises:

the image sensor acquires an optical signal within the exposure time corresponding to the second working mode;

based on the received light signal, the image sensor acquires second image data.

19. The method of claim 18, wherein after the image sensor acquires the second image data, the method further comprises:

the image sensor sends a request for acquiring a second parameter to the transmitter;

the image sensor receiving the second parameter from the transmitter;

the image sensor obtains second RAW Data based on the first image Data and the second parameter; the second parameter is used for indicating the working state of the transmitter under the second current value.

20. The method of claim 19, wherein the electronic device further comprises a second memory and a face recognition Trusted Application (TA), and wherein performing the face recognition based on the image data comprises:

the image sensor sends the second RAW Data to the image processing module;

the image processing module sends the second RAW Data to the second memory for storage; the second memory corresponds to a second FD;

the face recognition TA reads the second RAW Data from the second memory according to the second FD;

the face recognition TA acquires a first gray scale image and a first depth image according to the image Data in the second RAW Data;

and the face identification TA compares the face according to the first gray image and performs anti-counterfeiting detection according to the first depth image to obtain a face identification result.

21. The method of claim 20, wherein after the image processing module sends the second RAW Data to a second memory for storage, the method further comprises:

the image processing module sends the second FD to the camera driving module;

the camera driving module sends the second FD to a camera HAL;

the camera HAL sends the second FD to a camera service through a preset interface;

the camera service sends the second FD to the face recognition control module;

and the face recognition control module sends the second FD to a face recognition TA.

22. The method according to claim 20 or 21, further comprising:

the face recognition TA sends the face recognition result to the face recognition control module;

the face recognition control module sends the face recognition result to a face recognition service;

the face recognition service sends the face recognition result to a face recognition SDK;

the face recognition SDK sends the face recognition result to the first application;

and responding to the received face recognition result, and the first application executes unlocking, wherein the face recognition result is successful.

23. The method according to claim 15 or 16, wherein the controlling the emitter to operate at a third light intensity in case the emitter is in an abnormal operating state specifically comprises:

in response to receiving that the working state of the transmitter is an abnormal working state, the camera HAL determines that the working mode of the camera module is a third working mode, where the third working mode is used to indicate that the transmitter works at a third current value, and the third current value is 0;

the camera HAL sends the configuration parameters of the third working mode to the camera driving module;

the camera driving module writes the configuration parameters of the third working mode into a register of the TOF camera module;

the camera driving module sends a message of completing configuration parameter writing to the sensor node;

in response to receiving the message that the writing of the configuration parameters is completed, the sensor node sends a fifth starting command to the camera driving module;

the camera driving module sends a sixth starting command to the TOF camera module;

the transmitter is not operating.

24. The method of claim 23, wherein after the transmitter is not operating, the method further comprises:

the image sensor receives an optical signal within the exposure time corresponding to the third working mode;

based on the received light signal, the image sensor acquires third image data.

25. The method of claim 24, further comprising:

the face recognition TA acquires a second gray map and a second depth map based on the third image data;

and comparing the face based on the second gray image, and performing anti-counterfeiting detection based on the second depth image to obtain a face recognition result which is a failure.

26. The method of claim 25, further comprising:

the face recognition TA sends the face recognition result to the face recognition control module;

the face recognition control module sends the face recognition result to a face recognition service;

the face recognition service sends the face recognition result to a face recognition SDK;

the face recognition SDK sends the face recognition result to the first application;

and responding to the received face recognition result, the first application does not execute unlocking or display unlocking failure, and the face recognition result is failure.

27. An electronic device, characterized in that the electronic device comprises: a wireless communication module, memory, and one or more processors; the wireless communication module, the memory and the processor are coupled;

wherein the memory is to store computer program code comprising computer instructions; the computer instructions, when executed by the processor, cause the electronic device to perform the method of any of claims 1-26.

28. A computer-readable storage medium comprising computer instructions;

the computer instructions, when executed on an electronic device, cause the electronic device to perform the method of any of claims 1-26.

29. A chip system, comprising one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line;

the chip system is applied to an electronic device comprising a communication module and a memory; the interface circuit is configured to receive signals from the memory and to transmit the signals to the processor, the signals including computer instructions stored in the memory; the electronic device, when the processor executes the computer instructions, performs the method of any of claims 1-26.

Images