TWI545508B - Method for performing a face tracking function and an electric device having the same - Google Patents

Description

Method for performing face tracking function and electronic device thereof

The present invention relates to a method for performing a face tracking function and an electronic device thereof, and more particularly to a method for performing a face tracking function and an electronic device thereof under a video call (Video Call) Device.

In recent years, the resolution of the front camera lens of electronic devices has become higher and higher, and the use of video Talk has become more and more popular. Therefore, how to provide users with more diverse video call applications is a technology development issue that the industry values.

In accordance with an embodiment of the present invention, a method of performing a face tracking function in an electronic device is provided. The electronic device has a touch panel, a camera lens and a processor. The method includes the following steps. A touch signal is received by the touch panel. Under a video call, the processor enters a face tracking mode based on the touch signal. Performing face tracking on a captured image captured by the camera lens to obtain at least one Region of Interesting (ROI) in the captured image, each region of interest (ROI) having a face An image of the ministry. A target frame is generated by the processor by combining at least one region of interest (ROI). The target picture is transmitted by the processor to another electronic device such that the target picture is displayed as a video call picture on another electronic device.

According to another embodiment of the present invention, an electronic device for performing a face tracking function is provided. The electronic device includes a touch panel, a camera lens and a processor. The touch panel is for receiving a touch signal. Camera lens is Used to capture a screenshot. The processor is configured to enter a face tracking mode under a video call based on the touch signal, and perform face tracking on the captured image captured by the camera lens to obtain at least one region of interest in the captured image. (ROI), each region of interest (ROI) has an image of a face. The processor is further configured to generate a target image by combining at least one region of interest (ROI), and transmit the target image to another electronic device, so that the target image is displayed as a video call screen on another electronic device. .

In accordance with yet another embodiment of the present invention, a non-transitory storage medium for storing a code is provided. When the code is executed by an electronic device, the electronic device performs a method to perform a face tracking function. The electronic device has a touch panel, a camera lens and a processor. The method includes the following steps. A touch signal is received by the touch panel. Under a video call, the processor enters a face tracking mode based on the touch signal. Performing face tracking on a captured image captured by the camera lens to obtain at least one region of interest (ROI) in the captured image, each region of interest (ROI) having an image of a face . A target picture is generated by the processor by combining at least one region of interest (ROI). The target picture is transmitted by the processor to another electronic device such that the target picture is displayed as a video call picture on another electronic device. In order to better understand the above and other aspects of the present invention, the following detailed description of the embodiments and the accompanying drawings are as follows:

102~110‧‧‧Steps

200‧‧‧Electronic devices

202‧‧‧Touch panel

204‧‧‧ Processor

206‧‧‧ camera lens

208‧‧‧Communication module

402‧‧‧Floating Control Interface

404‧‧‧General pattern

406‧‧‧Face tracking pattern

410‧‧‧ fill light pattern

502‧‧‧ screen

602‧‧‧ screen

702‧‧‧Video Call (VT) App

704‧‧•Video Conversation Enhancement (VTE) Floating Control Interface

706‧‧‧ Camera Service Program

708‧‧•Video Conversation Enhancement (VTE) native app

710‧‧‧ Camera Hardware Adaptation Layer (HAL)

712‧‧‧ Camera sensor

714‧‧‧Image signal processor

716‧‧‧Face detector

718‧‧‧Face Tracking Engine

720‧‧‧Cut splicing unit

722‧‧‧Scaler

802‧‧‧ picture buffer

804, 806‧‧‧ blocks

ROI1~ROI4‧‧‧ area

F _HQ ‧‧‧High resolution screen

F _T1 , F _T2 ‧‧‧ target screen

FIG. 1 is a flow chart of a method for performing a face tracking function in an electronic device according to an embodiment of the invention.

FIG. 2 is a functional block diagram of an electronic device.

Figure 3 shows an example of a white list.

FIG. 4 is a diagram showing an example of a floating control interface of an electronic device according to an embodiment of the invention.

5A-5E illustrate corresponding screens displayed on an electronic device A.

6A to 6E are diagrams showing respective screens displayed on an electronic device B.

Figure 7 illustrates an example of a software/hardware architecture of an electronic device having a face tracking function.

8A-8C illustrate the operation of a face detector, a face tracking engine, and a cutting and splicing unit.

Figure 9 illustrates another example of a target picture having three regions of interest.

In the following detailed description, numerous specific details are set forth It is apparent that one or more embodiments may be practiced without these specific details. In other embodiments, the general structures and devices are merely schematic representations for simplicity.

FIG. 1 is a flow chart of a method for performing a face tracking function in an electronic device according to an embodiment of the invention. The functional block diagram of the electronic device 200 is as shown in FIG. The electronic device 200 has a touch panel 202 , a processor 204 , and an imaging lens 206 . This method includes the following steps. First, in step 102, the touch panel 202 receives a touch signal. Next, in step 104, under a video call, the processor 204 enters a face tracking mode based on the touch signal. Next, in step 106, the processor 204 performs face tracking on a captured image captured by the camera lens 206 to obtain at least one region of interest (ROI) in the captured image. Each region of interest (ROI) has an image of a face. Thereafter, in step 108, the processor 204 generates a target frame by combining at least one region of interest (ROI). Finally, in step 110, the processor 204 transmits the target picture to another electronic device to cause the target picture to be displayed as a video call picture on another electronic device.

Therefore, even if the application currently on the video call by the electronic device 200 does not provide the face tracking function, the electronic device 200 according to the present embodiment can perform the face tracking function and transmit the target screen to another electronic device as a Video call screen. In other words, the target picture can replace the picture of its video call, and the target picture can be transmitted by the same picture as one of the video calls. And transmitted to another electronic device to implement the face tracking function.

When the electronic device 200 executes a communication program to perform a video call with another electronic device, the touch panel 202 can further display a floating control interface. The floating control mask has a face tracking control project. When the face tracking control item is selected, the touch panel 202 will receive the touch signal, and the processor 204 correspondingly enters a face tracking mode under a video call based on the touch signal.

In addition, the touch panel 202 has the ability to touch and/or display. If the touch panel 202 does not have the capability of display, the electronic device 200 may have a display (not shown). The screen can be displayed on the screen of the touch panel 202 or on the screen of the display. The electronic device 200 can further have a communication module 208 for communicating with other electronic devices through wireless communication or through the Internet.

The electronic device 200 can be a mobile device such as a smart phone or tablet. The operating system of the electronic device 200 can be, for example, an Android operating system. The processor 204 determines whether the electronic device 200 is performing a video call by determining whether the communication program for the video call is executed in the foreground (Foreground) and whether the camera lens 206 is turned on. When the communication program for the video call is executed in the foreground and the camera lens 206 is turned on, the processor 204 determines that the electronic device 200 is performing the video call. The communication program can be one of a plurality of applications that can perform a video call in a White List. The whitelist also contains other candidate communication programs. An example of a whitelist is presented in Figure 3.

If the application currently executed by the electronic device 200 in the foreground is included in the white list of FIG. 3 and the camera lens 206 has been turned on, it is determined that the electronic device 200 is performing a video call, and the floating control interface is displayed. On the screen. The floating control interface is from a separate application that is different from these communication programs, and the floating control interface is not included in the whitelist. The floating control interface can be used as a user interface.

Referring to FIG. 4, an example of a floating control interface of an electronic device 200 in accordance with an embodiment of the present invention is illustrated. Floating control interface 402 has a A Face Tracking pattern 406 is used as the above-described face tracking control item. The floating control interface 402 may further include, for example, a Normal Mode pattern 404 and a Light Filling pattern 410, but is not limited thereto. Depending on the functional requirements of the electronic device 200, other patterns may be added to the floating control interface 402 or some patterns may be removed.

When the user presses or selects the face tracking pattern 406, the electronic device 200 will perform the following operations. The operation as shown in FIGS. 5A to 5E is performed by an electronic device A as an example. The electronic device A is, for example, making a video call with an electronic device B, and the corresponding picture is displayed on the electronic device B as shown in FIGS. 6A to 6E. Referring to FIG. 5A, it is assumed that the picture taken locally by the imaging lens 206 of the electronic device A is a picture 502, and the picture 502 is further transmitted to the electronic device B and displayed as one shown in FIG. 6AA. Screen 602.

If the application currently executed in the foreground of the electronic device A is included in the white list of FIG. 3 and the camera lens 206 has been turned on, the floating control interface 402 will be displayed on the screen of the electronic device A in a superimposed manner, such as Figure 5A shows. When the user of the electronic device A selects the face tracking pattern 406 of the floating control interface 402, the processor 204 will enter the face tracking mode, and the electronic device A will activate the face tracking function, as shown in FIG. 5B. The electronic device A will perform face tracking on the captured screen to obtain at least one region of interest (ROI).

If an ROI is obtained, the ROI will be amplified and the target frame will be generated based on the amplified ROI. That is, if the user of the electronic device A selects the area ROI1, the electronic device A will only enlarge the area ROI1 to form a target picture, and further transfer the target picture to the electronic device B, as shown in FIG. 6B. The advantage of this method is that the electronic device A can still detect and track the face corresponding to the area ROI1, so that the electronic device B can continuously see the face corresponding to the area ROI1 without It will be affected by the movement of the face corresponding to the area ROI1.

If more than one ROI is obtained, these ROIs are amplified, and the target picture is generated by combining the enlarged ROIs. For example, if electricity When the user of the sub-device A selects the areas ROI1 and ROI2, the electronic device A will enlarge both the areas ROI1 and ROI2 to form a target picture, and further transmit the target picture to the electronic device B, as shown in FIG. 6C. Shown. Each region ROI1 and ROI2 can each occupy one-half of the target picture. In this way, the electronic device A can transmit only the regions ROI1 and ROI2 of interest to the user to the electronic device B, so that the user of the electronic device B can see the enlarged regions ROI1 and ROI2.

If the user of the electronic device A selects the regions ROI1, ROI2 and ROI3, the electronic device A will enlarge the regions ROI1, ROI2 and ROI3 to form a target image, and further transmit the target image to the electronic device B, such as Figure 6D shows. Each of the regions ROI1, ROI2, and ROI3 may, for example, occupy an area that is one-third of the target picture. If the user of the electronic device A selects the regions ROI1, ROI2, ROI3 and ROI4, the electronic device A will enlarge the regions ROI1, ROI2, ROI3 and ROI4 to form a target image, and further transmit the target image to the electronic device. Device B, as shown in Figure 6E. Each of the regions ROI1, ROI2, ROI3, and ROI4 may, for example, occupy an area of one quarter of the target picture.

Referring to FIG. 7, an example of a software/hardware architecture of an electronic device 200 having a face tracking function is illustrated. After the communication program initiates the video call function, the floating control interface 402 will be automatically executed and displayed on the screen of the electronic device 200. The communication program can be, for example, a Video Talk (VT) application 702, and the floating control interface 402 can be, for example, a Video Talk Enhancement (VTE) floating control interface 704. The VT application 702 is included in the white list of Figure 3, such as Skype or Line. Whether the video call function is enabled by the communication program is determined by judging whether the VT application 702 is executed in the foreground and the camera lens 206 has been turned on (this means that the user is using the video call function). After the VT application 702 is executed in the foreground and the camera lens 206 has been turned on (representing the user's function of using the video call), the video call enhancement (VTE) floating control interface 704 (eg, the floating control shown in FIG. 4) The interface 402) will be automatically executed and displayed on the screen of the electronic device 200.

During a video call in the normal mode, the VT application 702 will then call a Camera Service 706 to initiate a camera function, and the VTE Floating Control Interface 704 will call a VTE Native Application. 708. When the user of the electronic device A selects the face tracking pattern 406 of the floating control interface 402, the electronic device A will activate the face tracking function. At the same time, the VTE Floating Control Interface 704 will immediately call the VTE Native Application 708. The VTE native application 708 then notifies the camera service program 706 that the electronic device A will enter the face tracking mode. The VTE native application 708 will control the camera service program 706 to obtain a target screen that presents only one or more ROIs to perform a face tracking function, and display the target screen on the screen of the electronic device 200 and further transmit Go to electronic device B.

An example of the method of generating the target screen described above will be described below. First, an image is obtained by a camera sensor 712. The image is processed by an Image Signal Processor (ISP) 714 to obtain a high resolution frame F _HQ . The resolution of the high-resolution screen F _{HQ is} , for example, the highest resolution that can be generated by the camera sensor 712 or the resolution greater than the resolution of the target screen. The resolution of the target screen depends on the resolution selected or used by the VT application 702.

Then, when the user selects the face tracking function, the high-resolution screen F _HQ is transmitted to a face detector 716. The face detector 716 detects the position of the face image of at least one high resolution picture F _HQ . Then, the face detector 716 outputs location information of at least one facial image to a face tracking engine 718. The face tracking engine 718 selects a position in a picture buffer based on the position information of the face image, wherein the picture buffer is used to store the picture to be displayed. The position information of at least one face image from the face detector 716, the position in the picture buffer determined by the face tracking engine 718, and the high-resolution picture F _HQ are input to a cut splicing unit 720. . The cutting and splicing unit 720 further cuts and splices at least one region of interest selected by the user, thereby correspondingly forming a target image F _T1 . Face detector 716, face tracking engine 718, and splicing splicing unit 720 can be implemented by, for example, a graphics processing unit (GPU), a central processing unit (CPU), or other processor.

Then, the target screen F _T1 is transmitted to the camera service program 706 through a camera hardware adaptation layer (HAL) 710, and further displayed on the screen of the electronic device 200 by using the VTE application 702. Then, the target screen F _T1 is transmitted by the electronic device A to the electronic device B at the other end of the video call.

If the user does not select the face tracking function and only selects the normal mode, the high-resolution screen F _HQ is transmitted to a scaler 722 to reduce the resolution of the high-resolution screen F _HQ . One of the lower resolutions of the target picture F _T2 , and further transmits the target picture F _T2 to the camera hardware adaptation layer 710. Then, the target screen F _T2 is transmitted by the electronic device A to the electronic device B at the other end of the video call.

The VT application 702 and the VTE floating control interface 704 described above are, for example, Java applications written in the Java language. The camera service program 706, the VTE native application 708, and the camera hardware adaptation layer 710 are, for example, native applications. The VTE native application 708 can apply the Binder mechanism of Android's Inter-Process Communication (IPC) mechanism. The above architecture can be implemented in an Android operating system or other operating system.

The operations of the face detector 716, the face tracking engine 718, and the cut splicing unit 720 will be described in more detail with the examples shown in Figs. 8A to 8C. After the high resolution picture F _HQ is transmitted to the face detector 716, the face detector 716 detects the position of at least one of the face images in the high resolution picture F _HQ . As shown in FIG. 8A, the positions of these facial images are detected and labeled as a plurality of regions of interest ROI1, ROI2, ROI3, and ROI4. Suppose the user selects two regions of interest, for example, ROI1 and ROI2 are selected; then, the face detector 716 records the location information of the facial images, such as the coordinates of ROI1 and ROI2 in the high-resolution picture F _HQ (for example) The coordinates of the rectangular ROI1 and ROI2 at the two corners of the diagonal are output to the face tracking engine 718.

As shown in FIG. 8B, the face tracking engine 718 amplifies at least one of ROI1 and ROI2, or scales at least one of ROI1 and ROI2 such that the sizes of ROI1 and ROI2 are suitable for combining the two. The face tracking engine 718 also selects a location in the picture buffer 802 for ROI1 and ROI2 for storing the picture to be displayed on the screen of the electronic device 200. The face tracking engine 718 selects a position in the picture buffer 802 based on the coordinates of ROI1 and ROI2 and the size of the zoom or enlargement of ROI1 and ROI2. It is assumed that the face tracking engine 718 selects blocks 804 and 806 in the picture buffer 802 for ROI1 and ROI2. The face tracking engine 718 then outputs, for example, the addresses of the blocks 804 and 806 in the picture buffer 802 to the tiling unit 720. High resolution pictures and high resolution pictures F _HQ F _HQ of ROI1 and ROI2 is also input to the coordinates of the cut splice unit 720. Based on the coordinates of ROI1 and ROI2 in the high-resolution picture F _HQ , the splicing splicing unit 720 cuts ROI1 and ROI2 from the high-resolution picture F _HQ and stores the ROI1 and ROI2 in the block 804 of the picture buffer 802 and In 806, the image of the combined ROI1 and ROI2 is formed as a target image F _T1 for output, and displayed on the screen of the electronic device 200 as shown in FIG. 8C. Wherein the target screen F _T1 (i.e. ROI1 and ROI2 combined image) is lower than the high-resolution screen resolution lines F _HQ resolution.

Please refer to FIG. 9, which illustrates another example of a target picture having three regions of interest. When the user selects three regions of interest ROI1~ROI3, the three regions of interest ROI1~ROI3 in the target screen F _T1 can be arranged as shown in FIG. 9 in addition to the manner shown in FIG. 6D. The way to arrange. In Fig. 9, the more important region of interest ROI1 is displayed in a larger area. Other different numbers of regions of interest may also determine the size of the display area according to the importance of the region or the distance from the camera lens.

In addition, the electronic device 200 can also provide a Light Filling function. A front flash can be controlled depending on the position of the face or the brightness in the environment The color temperature of the captured picture is changed by performing the fill light function, or the light intensity at the time of capturing is changed. For example, the color of the light of the front flash can be modulated to change the color temperature of the captured picture, or the light intensity of the front flash can be modulated to change the light intensity at the time of the capture.

Furthermore, the electronic device 200 can provide a Pan Focus function, or the camera lens 206 can focus on a target that is separated from the electronic device 200 by a distance of 0.5 meters to 2.5 meters. Therefore, the image of the target (such as the face) can be clearly captured even at a distance of 0.5 m to 2.5 m, which contributes to the face tracking function.

Important areas of interest may be other objects or objects of interest to the user in addition to the facial images described above. In addition, although the above embodiment is described in the Android operating system, the embodiment of the present invention is not limited thereto. The embodiments of the present invention can also be applied to other operating systems.

In addition, according to still another embodiment of the present invention, a non-transitory storage media is provided for storing a code. When the code is executed by the electronic device 200, the electronic device 200 performs the above-described method to perform the face tracking function described above.

The advantage of using the VTE floating control interface 704 is that the face tracking function and/or the ability to enlarge and splicing important regions of interest into a new target image can be achieved without the need to modify the VT application 702 (eg, Skype and Line, etc.) The application code, especially when the VT application 702 does not support the aforementioned functions. That is, through the VTE floating control interface 704, the VTE native application 708, the face detector 716, the face tracking engine 718, and the splicing unit 720 provided by an original equipment manufacturer (OEM). It allows the various video calling application software in the whitelist to match the face tracking function and/or the function of zooming in and stitching the region of interest into a new target image. Since it is not necessary to modify the code of the communication program (VT application 702) (such as Skype and Line) for the video call, the above purpose can be achieved, so that more video call functions can be provided to the user and the view can be improved. The quality of the call, the better experience in use and the increased market competitiveness of the product.

Although the present invention has been disclosed above by way of example, it is not intended to limit the invention. A person skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the scope of the invention is defined by the scope of the appended claims.

102~110‧‧‧ Process steps

Claims (18)

A method for performing a face tracking function in an electronic device, the electronic device having a touch panel, a camera lens and a processor, the method comprising the steps of: receiving a touch signal by the touch panel; In a video call, the processor enters a face tracking mode based on the touch signal; the processor performs face tracking on a captured image captured by the camera lens to obtain the captured image. a plurality of regions of interest (ROI), each of the regions of interest (ROI) having an image of a face; the processor generating a target image by combining the regions of interest (ROI); The processor transmits the target picture to another electronic device to display the target picture as a video call picture on the other electronic device. The method of claim 1, further comprising: displaying, by the touch panel, a floating control interface when the electronic device executes a communication program to perform the video call with the other electronic device; The floating control mask has a face tracking control item; wherein the receiving step comprises receiving the touch signal by the touch panel when the face tracking control item is selected. The method of claim 2, wherein the method further comprises: determining whether the electronic device is performing the video call by determining whether the communication program is executed in the foreground and whether the camera lens is turned on. The communication program series is in a white list. The method of claim 3, wherein the plurality of candidate communication program series are in the white list. The method of claim 1, wherein the generating step comprises amplifying the regions of interest (ROI), and the target image is generated by combining the enlarged regions of interest (ROI). The method of claim 1, wherein before the step of performing face tracking, the method further comprises performing face detection on the captured image captured by the camera lens to determine the at least one The position of the face in the captured picture. An electronic device, configured to perform a face tracking function, the electronic device includes: a touch panel for receiving a touch signal; a camera lens for capturing a captured image; and a processor for using Entering a face tracking mode under a video call based on the touch signal, and performing face tracking on the captured image captured by the camera lens to obtain a plurality of regions of interest in the captured image (ROI), each of the regions of interest (ROI) has an image of a face; wherein the processor is further configured to generate a target image by combining the regions of interest (ROI) and used to transmit the image The target screen is displayed to another electronic device such that the target screen is displayed as a video call screen on the other electronic device. The electronic device of claim 7, wherein the touch panel is further configured to display a floating control interface when the electronic device executes a communication program to perform the video call with the other electronic device. The floating control mask has a face tracking control item; wherein when the face tracking control item is selected, the touch panel receives the touch signal. The electronic device of claim 8, wherein the processor is further configured to determine whether the communication program is executed in the foreground and whether the camera lens is Is turned on to determine whether the electronic device is performing the video call, wherein the communication program series is in a white list. The electronic device of claim 9, wherein the plurality of candidate communication program series are in the white list. The electronic device of claim 8, wherein the floating control mask has a face tracking pattern as the face tracking control item. The electronic device of claim 7, wherein the processor is further configured to enlarge the regions of interest (ROI), and the target image is combined with the enlarged regions of interest (ROI). produce. The electronic device of claim 7, wherein the processor is further configured to perform face detection on the captured image captured by the camera lens to determine the at least one face before performing face tracking. The position of the part in the capture screen. A non-transitory storage medium for storing a code, wherein when the code is executed by an electronic device, the electronic device performs a method for performing a face tracking function, the electronic device having a touch panel and a camera a lens and a processor, the method comprising the steps of: receiving a touch signal by the touch panel; in a video call, the processor enters a face tracking mode based on the touch signal; Face tracking is performed on a captured image captured by the camera lens to obtain a plurality of regions of interest (ROI) in the captured image, each of the regions of interest (ROI) having an image of a face Generating a target picture by the processor by combining the regions of interest (ROI); The target picture is transmitted by the processor to another electronic device such that the target picture is displayed as a video call picture on the other electronic device. The non-transitory storage medium of claim 14, wherein the method further comprises: when the electronic device executes a communication program to perform the video call with the other electronic device, the touch panel displays a floating control interface, wherein the floating control mask has a face tracking control item; wherein the receiving step comprises receiving the touch signal by the touch panel when the face tracking control item is selected. The non-transitory storage medium of claim 15, wherein the method further comprises: determining whether the electronic device is executing by determining whether the communication program is executed in the foreground and whether the camera lens is turned on. Video call, where the communication program series is in a white list. The non-transitory storage medium of claim 14, wherein the generating step comprises amplifying the regions of interest (ROI), and the target image is enlarged by combining the regions of interest (ROI) And produced. The non-transitory storage medium of claim 14, wherein before the step of performing face tracking, the method further comprises performing face detection on the captured image captured by the camera lens. Determining the position of the at least one face in the captured picture.