CN113747058B - Image content shielding method and device based on multi-camera - Google Patents

Abstract

Embodiments of the present application provide a method and device for masking image content based on multiple cameras. The method can be applied to electronic devices capable of shooting, such as mobile phones and tablet computers. By implementing the multi-camera-based image content masking method provided in the embodiment of the present application, electronic devices such as mobile phones and tablet computers can collect images with visual difference through two or more cameras. Then, the electronic device can cut and replace the images captured by different cameras based on the above-mentioned visual difference, so as to shield the content that the user does not expect to appear in the images, so as to improve the shooting experience of the user.

Description

Image content shielding method and device based on multi-camera

technical field

The present application relates to the field of terminals, in particular to a method and device for masking image content based on multiple cameras.

Background technique

At present, users have a strong demand for taking selfies with electronic devices such as mobile phones, which has also brought about the emergence of selfie sticks. The selfie stick effectively solves the problem of limited composition when taking selfies, especially in outdoor sports, multi-person selfies or full-body selfies. However, using a selfie stick to take pictures also creates new problems: the selfie stick appears in selfie photos, which affects user experience.

Contents of the invention

The present application provides a method and device for masking image content based on multiple cameras. The method can be applied to electronic devices capable of shooting, such as mobile phones and tablet computers. By implementing the above method, electronic devices such as mobile phones and tablet computers can splice a group of images with visual difference collected by two or more cameras, so as to obtain images with specific image content shielded, so as to improve the user's shooting experience.

In a first aspect, the present application provides a method for masking image content, the method is applied to an electronic device equipped with a camera, and the method includes: acquiring a group of image frames, a group of image frames at least including a first image, a second image , the first image and the second image are respectively collected by the first camera and the second camera of the electronic device at the same time, and the first camera and the second camera are both the front camera or the rear camera of the electronic device; using the second image Replace the first object in the first image with the content of the image in to obtain a third image; display the third image.

By implementing the method provided in the first aspect, the electronic device can simultaneously use multiple front cameras or rear cameras to collect a group of image frames with visual difference. The images captured by multiple front cameras or rear cameras at the same time naturally have parallax. Using the images caused by parallax to fill in the content to be blocked can preserve the image features blocked by the content to be blocked as much as possible and improve the shielding effect. At the same time, using parallax image stitching to mask some image content can reduce the amount of calculation and improve processing efficiency.

With reference to the embodiment of the first aspect, in some embodiments, using the image content in the second image to replace the first object in the first image to obtain the third image specifically includes: using the image content in the first region in the second image The third image is obtained by replacing the image content of the second area in the first image, the second area is an area displaying the first object in the first image, and the first area corresponds to the second area.

With reference to the embodiments of the first aspect, in some embodiments, the first area corresponds to the second area, including: the first area and the second area have the same size and the same position; or, the first area is larger than the second area, and the second area The central position of the first area is the same as the central position of the second area, and the first area covers the second area.

Implementing the method provided by the above embodiment, the first area may be exactly the same as the area corresponding to the first object, that is, the same as the second area, so that the electronic device may use the image content of the first area to fill the first object to be masked. The first area can also be larger than the second area. In this way, the first area can not only fill the first object in the second area, but also avoid edge jaggedness caused by incomplete shielding, and improve the shielding effect.

With reference to the embodiment of the first aspect, in some embodiments, before acquiring a group of image frames, the method further includes: detecting a first user operation.

By implementing the methods provided in the foregoing embodiments, the electronic device can detect a user's operation, and in response to the operation, the electronic device can execute the image masking algorithm provided in this application. In this way, the electronic device can execute the image masking algorithm according to the needs of the user, thereby improving the application pertinence of the image masking function and reducing invalid calculations of the electronic device.

With reference to the embodiment of the first aspect, in some embodiments, the first user operation is: an image capture operation.

By implementing the methods provided in the foregoing embodiments, the electronic device may implement the image content shielding method provided in the present application after detecting the user's shooting operation.

With reference to the embodiment of the first aspect, in some embodiments, the image capturing operation includes: an image capturing operation detected by the electronic device, and/or, an image capturing operation detected by the second device; an image capturing operation detected by the electronic device Including: the operation acting on the shooting control, and, the operation acting on the button, the shooting control is displayed on the user interface provided by the electronic device, and the second device is connected to the first device.

With reference to the embodiments of the first aspect, in some embodiments, the imaging of the second device in the first image is the first object.

With reference to the embodiment of the first aspect, in some embodiments, displaying the third image specifically includes: saving the third image in response to the first user operation; detecting the second user operation, displaying the third image or the third image thumbnail.

Implementing the method provided by the above embodiment, in response to the first user operation, the electronic device can save the third image, then, the electronic device can detect the second user operation, and in response to the operation, the electronic device can display the third image.

With reference to the embodiment of the first aspect, in some embodiments, the second user operation includes: an operation of displaying the third image in the gallery application, or an operation of invoking the third image by a third-party application program.

Implementing the method provided by the above embodiment, in the scene of browsing the gallery application program, the electronic device can display the third image; in the process of calling the image stored in the gallery and sending it to other electronic devices, the electronic device can display the third image .

With reference to the embodiment of the first aspect, in some embodiments, the first user operation is an operation of opening a shooting preview interface.

By implementing the methods provided in the foregoing embodiments, the electronic device may implement the image content shielding method provided in the present application after detecting the user's operation of invoking the camera for shooting.

With reference to the embodiment of the first aspect, in some embodiments, the operation of opening the shooting preview interface includes: the operation of entering the shooting preview interface from the first application program icon, and the first application program is an application program provided by the electronic device; or, from The first control provided by the third-party application program enters the operation of the shooting preview interface.

Implementing the methods provided in the foregoing embodiments, the electronic device may implement the image content shielding method provided in the present application in a scenario where it is detected that the user opens the camera application. In the scenario of running other application programs, after detecting that the camera of the electronic device is invoked, the electronic device can implement the image content shielding method provided by the present application.

With reference to the embodiment of the first aspect, in some embodiments, after the first user operation is detected, the method further includes: displaying a shooting preview interface, the shooting preview interface includes a preview window, and the preview window is used to display the camera of the electronic device in real time The collected image; displaying the third image specifically includes: displaying the third image in a preview window.

By implementing the method provided in the foregoing embodiments, the electronic device can display the masked image in the preview window when displaying the shooting preview interface. In this way, the user can see the shooting effect after masking processing before shooting.

With reference to the embodiments of the first aspect, in some embodiments, the second device is a selfie stick, and the first object is the selfie stick in the image.

By implementing the method provided in the foregoing embodiments, the electronic device can shield the selfie stick appearing in the image in the preview window or in the captured photo or video, thereby improving the user's shooting experience.

With reference to the embodiments of the first aspect, in some embodiments, the selfie stick includes a stick body and a clamping part, and the first object is the selfie stick in the image, specifically including: the first object is the stick body of the selfie stick in the image.

In a second aspect, the present application provides an electronic device, which includes one or more processors and one or more memories; wherein, one or more memories are coupled with one or more processors, and one or more The memory is used to store computer program codes. The computer program codes include computer instructions. When one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation manner of the first aspect.

In a third aspect, the present application provides a computer-readable storage medium, including instructions. When the above-mentioned instructions are run on an electronic device, the above-mentioned electronic device executes the method described in the first aspect and any possible implementation manner of the first aspect. method.

In a fourth aspect, the present application provides a computer program product containing instructions. When the above-mentioned computer program product is run on an electronic device, the above-mentioned electronic device is executed as described in the first aspect and any possible implementation manner of the first aspect. method.

It can be understood that the above-mentioned electronic device provided in the second aspect, the computer storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that it can achieve can refer to the beneficial effects in the corresponding method, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of an electronic device provided by an embodiment of the present application;

Fig. 2 is a group of images with visual difference provided by the embodiment of the present application;

3A-3C are a set of user interfaces provided by the embodiment of the present application;

FIG. 4 is a flow chart of masking image content provided by an embodiment of the present application;

Fig. 5 is a schematic diagram of an electronic device provided by an embodiment of the present application acquiring a set of images with visual difference;

FIG. 6A is a schematic diagram of a selfie stick in an electronic device recognition image provided by an embodiment of the present application;

FIG. 6B and FIG. 6C are structural diagrams of encoders and decoders provided by the embodiments of the present application;

FIG. 7 is a schematic diagram of masking image content provided by an embodiment of the present application;

FIG. 8 is a hardware structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

With the improvement of the shooting capabilities of mobile smart terminals such as mobile phones and tablet computers, especially the depth of field cameras (depth of field, DOF), TOF (Time of flight) cameras and other cameras with deep ability (Deep ability) on mobile smart terminals Applications, mobile smart terminals such as mobile phones and tablet computers can already capture images containing depth information, such as RGBD images. The aforementioned camera capable of deep sensing may be referred to as a deep sense camera.

Images captured by the depth-sensing camera may include depth information of objects in the image. The image processing module can generate a 3D model of the selfie stick according to the above depth information, and then determine the area of the selfie stick in the selfie image. Then, the image processing module can shield the area of the selfie stick, so that the selfie stick does not appear in the image obtained by the user, thereby improving the shooting experience of the user.

However, the method of shielding the selfie stick with a depth-sensing camera requires a large computational cost. In particular, if the method is applied to video-type image processing, the calculation amount of electronic equipment will increase significantly. Correspondingly, the time cost of calculation is also significantly increased. For users, long computing time greatly affects user experience.

Therefore, in order to prevent content that users do not expect to appear in pictures or videos, such as selfie sticks, from affecting the user's shooting experience, and at the same time reduce the amount of calculations for image processing and the time cost of calculations, the embodiment of the present application provides a Image content masking method based on multiple cameras. The method can be applied to electronic devices capable of shooting, such as mobile phones and tablet computers.

By implementing this method, electronic devices such as mobile phones and tablet computers (electronic device 100 ) can collect images through two or more cameras. Because the spatial positions of different cameras are different, the images collected by different cameras at the same moment have visual difference. Furthermore, the electronic device 100 can cut and replace the images collected by different cameras based on the above-mentioned visual difference, so as to shield some content in the images that the user does not expect to appear, and improve the user's shooting experience.

At the same time, since the image to be processed is an ordinary image that does not include depth information, the calculation amount of the electronic device 100 for masking processing is significantly reduced, and the time cost is correspondingly significantly reduced. Therefore, the method is also applicable in video-type image processing.

Taking mobile phone A as the electronic device 100 as an example, FIG. 1 exemplarily shows a schematic diagram of mobile phone A. As shown in FIG.

As shown in Figure 1, mobile phone A includes two cameras, namely cam1 and cam2. The above cam1 and cam2 may be a group of front cameras of mobile phone A, or a group of rear cameras.

In this embodiment of the application, cam1 and cam2 may be standard lenses. When the front camera is enabled for shooting, both cam1 and cam2 can collect images and generate a time-series image frame stream. The image frame stream generated by cam1 can be called stream1; the image frame stream generated by cam2 can be called stream2. When cam1 and cam2 are standard lenses, the images captured by cam1 and cam2 are RGB images.

Preferably, the above cam1 and cam2 are color cameras. Not limited to color cameras, cam1 and cam2 can also be black and white cameras. For example, cam1 may be a color camera, and cam2 may be a black and white camera; or cam1 may be a black and white camera, and cam2 may be a color camera. In addition, not limited to cam1 and cam2, cell phone A may include more cameras. Here, more cameras include: more front cameras, and/or, more rear cameras. Further, the above-mentioned more cameras may be: standard lens, wide-angle lens, telephoto lens, periscope zoom lens, depth-of-field lens, etc. This embodiment of the present application does not limit this.

Correspondingly, for different types of cameras, the images collected by the cameras include different information. For example, when the mobile phone A also includes a depth-of-field lens, the mobile phone A can obtain RGBD images through the depth-of-field lens. An image of the RGBD type can record depth information of an object in the image.

The distance between cam1 and cam2 is D. Based on the distance D, there is a visual difference between the images collected by cam1 and cam2, refer to the principle of binocular vision. FIG. 2 exemplarily shows a schematic diagram of visual differences in images collected by different cameras.

Figure 2 includes (left) and (right) graphs. (Left) is a frame of image collected by cam1; (right) is a frame of image collected by cam2 at the same time. The contents in the (left) and (right) figures both include the selfie stick 10 and the character 20 .

In the image (left), the selfie stick 10 is positioned to the left of the center of the image, covering the user's right leg. In the (right) image, the selfie stick 10 is in the center of the image, between the user's left and right legs. The above selfie stick 10 is in different positions in the (left) and (right) figures, which means that there is a visual difference in the images collected by different cameras at the same time.

Based on the above parallax, after acquiring multiple frames of images with parallax, mobile phone A can stitch the above multiple frames of images with parallax, and fuse the above multiple frames of images with parallax into an image that shields the selfie stick . The above fused image may be referred to as an output image. After obtaining the output image, mobile phone A can store the image for further operation by the user. The above operations include editing, browsing, and so on.

It can be understood that when the distance D is small, that is, cam1 and cam2 are close to each other, the difference between the viewing ranges of cam1 and cam2 is very small. At this time, the visual difference between the images collected by cam1 and cam2 at the same time is relatively small. When the distance D is large, the viewing ranges of cam1 and cam2 differ greatly. At this time, the visual difference between the images collected by cam1 and cam2 at the same time is relatively large.

That is to say, for mobile phone A, when the distance between cam1 and cam2 installed in mobile phone A is relatively large, the visual difference between the captured images is more significant. Therefore, the effect of shielding selfie sticks based on parallax is better.

In other embodiments, when the mobile phone A has more cameras, the mobile phone A can acquire more image frames (at the same moment), and there is a visual difference between any two of the above image frames. In this way, mobile phone A can fuse more of the above images, thereby improving the shielding effect.

It can be understood that when the aforementioned cam1 and cam2 are both front-facing cameras, more cameras here refer to more front-facing cameras; more image frames refer to images collected by the above-mentioned more front-facing cameras. Conversely, when the aforementioned cam1 and cam2 are both rear cameras, more cameras here refer to more rear cameras; more image frames refer to images collected by the above more rear cameras.

Linux或者其它操作系统的便携式电子设备。Not limited to mobile phones, the electronic device 100 also includes tablet computers, smart watches, smart bracelets, sports cameras, augmented reality (augmented reality, AR) devices, virtual reality (virtual reality, VR) devices and other portable handheld devices capable of shooting. Exemplary embodiments of the electronic device 100 include, but are not limited to

Portable electronic devices with Linux or other operating systems.

3A-3C exemplarily show user interfaces in a group of camera-taking scenarios. Next, with reference to the user interface shown in FIG. 3A-FIG. 3C , a scenario of implementing the method for masking image content based on multiple cameras provided by the embodiment of the present application is introduced. In the embodiment described in FIG. 3A-FIG. 3C, the shooting scene is a selfie scene, and the image content that the user desires to mask is a selfie stick.

The electronic device 100 may detect a user operation of invoking the camera to display a shooting preview interface, and in response to the operation, the electronic device 100 may display the shooting preview interface shown in FIG. 3A .

提供的拍照控件、视频通话控件等等。The above-mentioned user operation of invoking the camera to display the shooting preview interface includes: an operation acting on the camera application icon of the electronic device 100, and an operation acting on the shooting control provided by the third-party application. The shooting controls provided by the above-mentioned third-party applications are, for example,

Provided photo control, video call control and so on.

FIG. 3A shows a user interface 31 using the electronic device 100 to display a shooting preview interface. As shown in FIG. 3A , the user interface 31 includes a preview window 311 , a mode bar 312 , a shooting button 313 , and an image preview window 314 .

The preview window 311 can be used to display the images collected by the camera in real time. Here, the image displayed on the preview window 311 may be referred to as an original image. It can be understood that the original image has also undergone some basic processing of the image processing module, such as color temperature calibration, linear correction, noise removal, dead point removal, interpolation, automatic exposure control, etc. However, the above-mentioned basic processing operations are all necessary processing for converting optical signals into electrical signals and imaging them on electronic devices such as mobile phones. Therefore, in the embodiment of the present application, the image that undergoes the above basic image processing and is displayed in the preview window 311 may be referred to as an original image.

The mode bar 312 can be used to select a shooting mode. The shooting modes mentioned above include: night view module, portrait mode, camera mode, video recording mode, professional mode and so on. In the user interface in the photographing scene shown in FIGS. 3A-3C , the electronic device 100 is processing a photographing mode.

The shooting button 313 can be used to receive a user's operation of taking a photo. When a user operation for controlling shooting by the electronic device 100 is detected, in response to the operation, the electronic device 100 may determine that the image frame displayed in the preview window 311 at this time is a captured image.

After determining that the image frame displayed in the preview window 311 is a captured image, the electronic device 100 may store the captured image in a designated storage space. Then, the electronic device 100 may display the thumbnails of the above-mentioned captured pictures in the image preview window 314 . At the same time, the electronic device 100 can receive user operations acting on the image preview window 314, and provide users with user operations for browsing the captured pictures.

The user interface 31 may also include other controls, for example, a plurality of controls provided in the area 315 for the user to adjust the shooting picture, a switch button 316 and the like. The switching button 316 can be used to change the camera (front camera or rear camera) of the electronic device 100 . It can be understood that the user interface 31 shown in FIG. 3A is a possible user interface, and should not be construed as a limitation to this embodiment of the present application.

When a user operation to control the shooting of the electronic device 100 is detected, in response to the operation, the electronic device 100 may display the user interface 32 shown in FIG. 3B . In the embodiment of the present application, in the scene where the user uses a selfie stick to take a selfie, the user operations for controlling the shooting of the electronic device 100 include: the operation acting on the shooting button 313 in FIG. 3A , and the operation on the shooting button on the selfie stick operate.

In other embodiments, the above-mentioned control of the electronic device 100 to shoot also includes a shooting operation on a physical button of the electronic device 100 and the like.

At this time, the image preview window 314 may display thumbnails of captured pictures generated in response to the user's shooting operation at the previous moment. Then, the electronic device 100 may detect a user operation on the image preview window 314 . In response to this operation, the electronic device 100 may display the user interface 33 shown in FIG. 3C.

As shown in FIG. 3C , the user interface 33 is a user interface on the electronic device 100 for displaying captured photos. User interface 33 includes area 331 . Area 331 can be used to display images taken by the user, including single-frame images and multi-frame images. Wherein, a single-frame image is, for example, a photo; and a multi-frame image includes a video, a GIF, and the like.

In addition, the user interface 33 may also include other controls, such as a control 332, a control 333, and so on. In response to actions on control 332, cell phone A may display more pictures from the album. In response to an operation acting on the control 333 , the mobile phone A may display detailed information of the image in the area 331 . The above detailed information includes, for example, shooting time, shooting location, image size (storage space usage), resolution, and storage path, etc. This embodiment of the present application does not limit this.

Comparing the image displayed in the preview window 311 with the captured image displayed in the region 331 , the former image includes the selfie stick 10 and the person 20 ; the latter image only includes the person 20 . That is to say, after detecting the user's shooting operation, mobile phone A can identify the selfie stick 10 in the original image (ie, the displayed image in the preview window 311), and remove it from the original image, so that the user obtains The selfie stick is not included in the photo, which improves the user's shooting experience.

In other embodiments, after the user's shooting operation is detected and the masked image is generated, the electronic device 100 does not immediately display the masked image. At this point, the electronic device 100 may detect the user's operation of invoking the image, such as an operation of displaying the above image in a gallery, an operation of sending the above image to other electronic devices, and so on. During the execution of the above operation of invoking the image, the electronic device 100 may display the image or a thumbnail of the image.

It is not limited to the shooting scene described in FIG. 3A-FIG. 3C , the multi-camera-based image content masking method provided in the embodiment of the present application can also be applied to the scene of shooting video.

In the scene of shooting a video, after the user's operation to start shooting is detected, the image displayed in the preview window 311 will be recorded and stored (video) by the mobile phone A. After the shooting is completed, mobile phone A can display a summary of the captured video in the image preview window 314 . Here, the video summary includes: a cover image of the video, and/or a sign indicating a video type image file, such as a time bar and a play icon.

After detecting the user's operation on the image preview window 314 , the mobile phone A may display the video taken by the user in the area 331 . During the process of shooting a video, the image displayed in the preview window 311 may include the selfie stick 10 and the person 20 . However, in the video displayed by mobile phone A in area 331 , the image of the video may not include the selfie stick 10 . That is, after the shooting is completed, mobile phone A can identify whether the selfie stick 10 is included in the video. If the selfie stick 10 is recognized, the mobile phone A can remove the selfie stick 10 in the video, and then generate a new video. In this video, the selfie stick 10 is not included in the video.

Although mobile phone A captures the selfie stick during video recording, the generated video file does not include the selfie stick. In this way, the user can directly obtain the selfie video with the selfie blocked.

Further, in the scene of taking pictures or in the scene of shooting video, mobile phone A can also display the processed image in the preview window 311 . Specifically, after the camera captures the image, the mobile phone A can perform shielding processing on the image captured by the camera. Then, the masked image is displayed in the preview window 311 . In this way, the video generated after the shooting is completely consistent with the image stream displayed in the preview window 311 during the shooting process, that is, the user can see the shooting effect after shielding the self-timer from the preview window 311 during the shooting process, Thereby improving the user's shooting experience.

In the following, the flow of the multi-camera-based image content masking method provided by the embodiment of the present application is introduced with reference to FIG. 4 .

S101: The electronic device 100 detects that the user is using a camera to take pictures.

Here, shooting with a camera includes taking a selfie with a front camera and general shooting with a rear camera. Usually, the appearance of the selfie stick in the shooting screen often occurs in the selfie scene.

Therefore, the following mainly takes a selfie scene as an example to introduce a scene in which the electronic device 100 determines that the user is taking a selfie with a camera. In the embodiment of the present application, the selfie scene specifically includes taking a picture (image frame) and taking a video (image frame stream).

In the Selfie scenario, the working state of the electronic device 100 includes: running an application program with a shooting function, and the application is calling the front camera of the electronic device 100 . Therefore, the electronic device 100 can determine whether the user is taking a selfie according to whether the front camera is in a working state.

A working front camera may indicate a selfie scene, but not necessarily a selfie stick selfie scene. Therefore, further, the electronic device 100 may also detect whether a selfie stick is connected. When detecting that the selfie stick is connected, the electronic device 100 may confirm that the user is taking a selfie with the selfie stick.

Specifically, the electronic device 100 is provided with an external interface; the selfie stick includes a button and a connection line for controlling taking pictures. The above external interfaces include 3.5mm headphone jack, Type-C interface, Lighting interface and so on. This embodiment of the present application does not limit this. The connecting wire of the selfie stick can be connected with the above-mentioned external interface. Then, the user can control the shooting task of the electronic device 100 through the shooting button on the selfie stick.

Therefore, when the front camera is turned on, when it is detected that other devices are connected to the external interface, the electronic device 100 can determine that the user is using the selfie stick to take a selfie. Referring to FIG. 3A , when the electronic device 100 displays the user interface 31 shown in FIG. 3A , the electronic device 100 may determine that the user is taking a selfie.

In shooting scenes other than the selfie scene, the electronic device 100 may also implement the multi-camera-based image content masking method provided in the embodiment of the present application. For example, in the scene of taking a single-person photo, the user can choose to remove other people who happen to appear in the viewfinder. That is to say, the shooting scene is not limited to the selfie scene, but also can be a normal shooting scene using the rear camera. Secondly, the shielded object is not limited to the selfie stick, and may also be other preset image content that can be recognized by the electronic device 100 . The embodiment of the present application does not limit this.

S102: The electronic device 100 acquires a group of image frames with parallax.

After recognizing a scene where a selfie stick is used to take a selfie, the electronic device 100 may enable the selfie stick shielding method provided in the embodiment of the present application. That is to say, it is optional to implement the multi-camera selfie stick shielding method provided in the embodiment of the present application.

In some embodiments, the electronic device 100 can implement the above-mentioned method in any shooting scene to identify and shield the selfie stick in the image. In other embodiments, the electronic device 100 may further refine the shooting scene. After identifying the preset detailed scene, the electronic device 100 implements the above method to identify and block the selfie stick in the image. For example, it is determined in S101 that the user is in the scene of taking a selfie, or further, it is determined that the user is in the scene of taking a selfie with a selfie stick, and so on.

In this way, the electronic device 100 can specifically enable this function to avoid abuse of computing resources.

After the multi-camera selfie stick shielding method provided in the embodiment of the present application is enabled, the electronic device 100 can acquire a group of image frames with visual difference captured by different cameras at the same moment. Specifically, taking cam1 and cam2 in a working state in a selfie scene as an example, FIG. 5 exemplarily shows a schematic diagram of capturing images by cam1 and cam2 to generate an image frame stream.

As shown in FIG. 5 , stream1 may represent the image frame stream collected and generated by cam1; stream2 may represent the image frame stream collected and generated by cam2. stream1 and stream2 respectively include a plurality of consecutive image frames, and the plurality of image frames in stream1 and stream2 are aligned in time sequence. Here, time sequence alignment means that any image frame in stream1 has the same time stamp as the image frame at the corresponding position in stream2. That is, the image frames at the same position in stream1 and stream2 are respectively collected by cam1 and cam2 at the same time.

Specifically, stream1 includes image frames T1, T2, T3, T4, and T5; stream2 includes image frames S1, S2, S3, S4, and S5. Wherein, T1 and S1 are images collected by cam1 and cam2 at the same time respectively. The relationship between T2, T3, T4, T5 and S2, S3, S4, S5 can refer to the introduction of T1 and S1 above, and will not be repeated here.

The above T1 and S1 can be referred to as a group of image frames with parallax. Similarly, T2 and S2, T3 and S3, T4 and S4, T5 and S5 are respectively a group of image frames with visual difference. It can be understood that when more cameras are used for shooting, the electronic device 100 acquires more image frames with parallax.

In this way, the electronic device 100 can conveniently and quickly acquire images with visual difference without requiring the user to adjust the position of the mobile phone A or move its own position.

Because in a non-moving shooting state, there is generally no large gap between the image contents of two or more consecutive frames. Therefore, the images with visual difference acquired by the electronic device 100 may also include images acquired by cameras at different times.

In other embodiments, the electronic device 100 may also acquire multiple consecutive frames of images as a group of image frames, and then fuse the multiple frames of images to obtain an image shielding the selfie stick or other image content.

The above-mentioned continuous multi-frame images include: two or more consecutive frames of images in stream1, and two or more consecutive frames of images in stream2. Specifically, referring to FIG. 5 , the above-mentioned continuous multi-frame images are, for example: T1, T2, S1, S2, or include: T1, T2, T3, S1, S2, S3.

In this way, the electronic device 100 can acquire more images with visual difference, thereby further improving the shielding effect of shielding other image contents such as the selfie stick.

Next, taking the group of image frames T1 and S1 as an example, a method for the electronic device 100 to mask part of the image content in the image by using the above image frames is introduced. Referring to Figure 2, the image content of T1 can refer to the (left) picture in Figure 2; the image content of S1 can refer to the (right) picture in Figure 2.

S103: The electronic device 100 locates the selfie stick in the image frame T1.

After acquiring T1 and S1, the electronic device 100 first locates the selfie stick in the image frame T1. Specifically, the electronic device 100 may use an image segmentation algorithm to perform image segmentation on T1 respectively to obtain an edge image (mask). The above-mentioned edge image refers to an image in which image content is marked and distinguished by using edge contours of different objects in the image as boundaries.

FIG. 6A exemplarily shows a schematic diagram of dividing an image frame T1 by the electronic device 100 . As shown in FIG. 6A, the image frame on the left is the image frame T1 (image frame T1 in stream1) acquired by the electronic device 100 from the image frame stream collected by cam1; The edge image (T1-mask) obtained after segmentation.

Using an image segmentation algorithm, the electronic device 100 may determine contents included in T1, and determine edges of each contents. Taking T1 as an example, the image content in T1 includes: a person 10 and a selfie stick 20 . Through image segmentation, the content of the edge image T1-mask of T1 obtained by the electronic device 100 includes: a selfie stick 601 , a portrait 602 and a background 603 .

In the embodiment of the present application, the image segmentation algorithm used by the electronic device 100 is an image semantic segmentation algorithm. The algorithm includes self-encoder (encoder) and self-decoder (decoder). The above autoencoders and autodecoders are built based on deep neural networks. Among them, the autoencoder is a data compression algorithm that can be used to extract the features of the input image. The autodecoder is the reverse reconstruction of the autoencoder, which is the reverse decoding of the deep feature space.

Figure 6B shows the structure of the autoencoder. As shown in FIG. 6B , the autoencoder includes a front end 60 and a back end 61 . The front end 60 includes a three-layer convolutional network (conv2). The backend is jointly composed of Block1 and Block2. Among them, Block1 includes a two-layer convolutional network (conv2), and Block2 includes a three-layer convolutional network (conv2).

In the process of image segmentation, the electronic device 100 may first input T1 into an autoencoder of the image segmentation algorithm to extract image features of T1. The image frames generated by cam1 and cam2 are three-channel RGB images. After the split reconstruction and feature extraction of the autoencoder, the above three-channel RGB image can be converted into a 196-channel high-dimensional feature.

Fig. 6C shows the structure of the self-decoder. As shown in FIG. 6C , the self-decoder includes a front end 62 and a back end 63 . The front end 62 includes three layers of upsampling modules (upsamples). The upsampling module consists of a three-layer convolutional network (conv2) and a layer of linear interpolation. The backend 63 is a one-layer convolutional network (conv2).

After reconstructing T1 through self-encoding, the electronic device 100 can input the reconstructed T1 to the self-decoder, and reversely reconstruct the characterized T1, thereby restoring the above-mentioned 196-channel high-dimensional features into 3-channel RGB image, and mark the segmentation results in this image.

Referring to FIG. 6A , after the image frame T1 is processed by the image segmentation algorithm, the electronic device 100 can obtain the edge image shown by T1-mask. The image content displayed in the T1-mask includes: a selfie stick 601 , a portrait 602 and a background 603 . Further, the electronic device 100 can locate the selfie stick 601 in the picture.

It can be understood that the self-encoder shown in Figure 6B and the self-decoder shown in Figure 6C are both a possible self-encoder structure and a possible self-decoder structure, and should not constitute a limitation of the embodiment of the present application . That is to say, in other embodiments, the autoencoder and the autodecoder may have different hierarchical structures. Here, different hierarchical structures include different numbers of convolutional layers and different ways of combining convolutional layers.

In addition, not limited to the image semantic segmentation algorithm used in the embodiment of the present application, the electronic device 100 can also use other image segmentation algorithms to locate the selfie stick in the image frame, such as VGGnet based on feature coding, ResNet, convolutional neural network based on region selection Network (Region-based Convolutional Neural Network, R-CNN) and traditional region growing algorithms, segmentation methods based on edge detection, etc. The embodiment of the present application does not limit this.

S104: The electronic device 100 uses the image frame S1 to replace the selfie stick in T1 to obtain a masked image.

The selfie stick 501 in the above T1 may be called a shielding object, that is, the image content that the electronic device 100 needs to remove. The pixel area marked in S1 corresponding to the pixel position of the selfie stick in T1 may be called a replacement material, that is, the pixel area used to replace the selfie stick in T1.

FIG. 7 exemplarily shows a schematic diagram of electronic device 100 replacing a shielded object with a replacement material. Next, the above replacement process will be described with reference to FIG. 7 .

First, after determining the selfie stick in T1, the electronic device 100 can cut S1 according to the pixel position of the selfie stick in T1, and determine the pixel point area of the replacement material.

As shown in FIG. 7 , according to the position of the selfie stick 601 in T1 , the electronic device 100 may mark the pixel at the same position in S1 , referring to S1 ′ in the image. S1 is an image frame with visual difference collected by cam2 at the same moment as T1. The foregoing embodiments have already introduced S1 in detail, so details are not repeated here. Thus, the electronic device 100 may determine the area 60 . The area 60 may represent a replacement material, and the pixels included in the area 60 correspond one-to-one to the pixels marked as the selfie stick 601 in T1.

Then, the electronic device 100 may replace the pixel marked as the selfie stick 601 in T1 with the image content marked as the replacement material in S1. For example, the electronic device 100 can delete the image content marked as the selfie stick 601 in T1, and then move the image content in the area 60 in S1 to T1, so that the electronic device 100 can obtain the image frame without the selfie stick 601 , as shown in T1'.

At this time, the content in the image frame T1' includes: the person 602, the background 603, and does not include the selfie stick 601, that is, the selfie stick 601 in T1 is removed.

It can be understood that T1 and S1 are images collected by the camera of the exemplary electronic device 100 . In the embodiment of the present application, there is no limitation on which camera of the electronic device 100 cam1 and cam2 are. Therefore, in other embodiments, the electronic device 100 may also use the image content in T1 to replace the content to be blocked in S1.

Preferably, in order to avoid jagged edges, the size of the replacement material may be larger than the size of the masking object. The edge aliasing phenomenon refers to the phenomenon that the area of the masked object positioned by the image segmentation algorithm does not completely coincide with the position of the actual object in the original image, and the masked object is not completely eliminated.

In the embodiment of this application, the size of the replacement material can be extended outward by 10-20 pixels based on the size of the masking object, so that the replacement material can completely cover the masking object in the original image, thereby avoiding edge aliasing , to enhance the shielding effect.

It can be understood that when the distance D between cam1 and cam2 is small, the visual difference between the image frames captured by cam1 and the image frames captured by cam2 is small. This may result in the image content in the replacement material including masked objects. In this way, even if the masking object is filled with a replacement material, the masking object in the original image cannot be completely removed.

Therefore, further, the electronic device 100 may also perform self-filling on the replaced image. Here, self-filling refers to filling dirty pixels with pixels near the dirty pixels. The above-mentioned dirty pixels refer to the masking objects left after the replacement, that is, the pixels where the masking objects have not been completely removed.

S105: The electronic device 100 displays the masked image.

In the scene of taking pictures, after detecting the user's operation on the shooting button, the electronic device 100 may implement the methods shown in S102-S104, and then, the electronic device 100 may display the masked image.

Combining with the user interface shown in Fig. 3A-Fig. 3C. When the electronic device 100 detects the user's operation on the shooting button 313 , in response to the operation, the electronic device 100 implements the methods shown in S102 - S104 to generate an image shielding the selfie stick 10 . Further, in response to a user operation acting on the image preview window 314, the electronic device 100 may display the above-mentioned image of the shielded selfie stick 10, as shown in the user interface 33 in FIG. 3C.

By implementing the method shown in S101-S105, the electronic device 100 can use multiple cameras to collect images during the shooting process. Then, the electronic device 100 can shield the selfie stick in the image according to the visual difference of the images collected by the above-mentioned multiple cameras, so as to obtain an image without the selfie stick, thereby satisfying the user's desire to take selfie images without the selfie stick. .

In order to improve the accuracy of image content replacement, before using the image content in S1 to replace the selfie stick 10 in T1, the electronic device 100 may also determine the similarity between the image content in S1 and T1.

If the image contents in S1 and T1 are obviously different, and the electronic device 100 still uses the image in S1 to fill in the image content in T1 that needs to be masked, this will not only fail to achieve the purpose of masking, but will make the image after processing outperform the image before processing It further reduces the user shooting experience.

For example, when cam2 is blocked by an object, such as a finger, most of S1 is the time image content of the finger. At this time, if S1 is used to fill the masked image content in T1, the replacement material obtained by the electronic device 100 from S1 may not match the masked image content, for example, use the finger image content in S1 to fill in the selfie in T1 In this way, not only the shielding purpose cannot be achieved, but the image after processing will reduce the user's shooting experience even more than the image before processing.

Therefore, if the image contents in S1 and T1 are obviously different, the electronic device 100 may consider that S1 does not have the ability to replace T1, and further, the electronic device 100 may fill the masked image content with similar image content near the masked image content. image.

It is not limited to the scene of using a selfie stick to take a selfie, and the multi-camera-based image content masking method provided in the embodiment of the present application can also be used in other shooting scenes. For example, in the scene of taking a single-person photo, the user can choose to remove other people who happen to appear in the viewfinder. The embodiment of the present application does not limit this.

In the embodiment of this application:

A group of image frames collected at the same time by multiple front cameras (or multiple rear cameras) of the electronic device 100 may be referred to as a group of image frames. Wherein, cam1 can be called the first camera, and cam2 can be called the second camera; the image frame collected by cam1, such as T1 shown in Figure 5, can be called the first image, and the image frame collected by cam2, such as shown in Figure 5 S1, may be referred to as the second image. An image obtained by filling part of the image content in the first image with the second image may be referred to as a third image, such as T1 ′ shown in FIG. 7 .

The selfie stick may be referred to as a second device, and the selfie stick in the image may be referred to as a first object.

The area used to fill the first image in the second image may be referred to as a first area, such as the area 60 shown by S1' in FIG. 7 . The area corresponding to the masked image content in the first image may be referred to as a second area, for example, the area corresponding to the selfie stick 10 in the image frame T1 in FIG. 6A .

Referring to FIG. 3A , the operations acting on the shooting control 313 may be referred to as operations on the shooting controls detected by the electronic device. Operations acting on physical keys such as a power key and a volume key of the electronic device 100 may be referred to as operations on keys detected by the electronic device. The user's operation on the shooting button on the selfie stick may be referred to as an image shooting operation detected by the second device. The above-mentioned photographing operation may be referred to as a first user operation.

Referring to FIG. 3B , an operation acting on the control 314 may be referred to as a second user operation. The user interface shown in FIG. 3C may be referred to as displaying a third image. The third image displayed in control 314 may be referred to as displaying a thumbnail of the third image.

FIG. 8 shows a schematic structural diagram of the electronic device 100 .

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor (ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or A universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example: the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . The MIPI interface includes a camera serial interface (camera serial interface, CSI), a display serial interface (displayserial interface, DSI), and the like. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves and radiate them through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (codedivision multiple access, CDMA), wideband code Wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM , and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou satellite navigation system (beidounavigation satellite system, BDS), a quasi-zenith satellite system (quasi- zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light emitting diode). AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

In the embodiment of the present application, the electronic device 100 displays the user interface (FIG. 3A-FIG. 3C), the preview image captured by the camera, and the photo or video taken by the user, etc., can be completed through the GPU, the display screen 194, and the application processor.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

In the embodiment of the present application, the electronic device 100 includes two or more cameras 193 .

In the embodiment where the electronic device 100 displays the original image in the preview window, the image captured by the camera 193 and processed by the ISP may be directly sent to the GPU for display on the display screen 194 . Then, the electronic device 100 performs masking processing on the original image according to the specific shooting content of the user.

In the embodiment in which the electronic device 100 displays the masked image in the preview window, the electronic device 100 can process the image collected by the camera 193 before sending it for display, so that the image displayed by the GPU, the display screen 194, etc. does not include Image of masked object.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The internal memory 121 may include one or more random access memories (random access memory, RAM) and one or more non-volatile memories (non-volatile memory, NVM).

Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous dynamic random access memory, SDRAM), double data rate synchronous Dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM, for example, the fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc. Non-volatile memory may include magnetic disk storage devices, flash memory (flash memory).

According to the operating principle, flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the potential order of storage cells, it can include single-level storage cells (single-level cell, SLC), multi-level storage cells (multi-level cell, MLC), triple-level cell (TLC), quad-level cell (QLC), etc., can include universal flash storage (English: universal flash storage, UFS), An embedded multimedia memory card (embedded multi media Card, eMMC), etc.

The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other running programs, and can also be used to store data of users and application programs.

The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be loaded into the random access memory in advance for the processor 110 to directly read and write.

The external memory interface 120 can be used to connect an external non-volatile memory, so as to expand the storage capacity of the electronic device 100 . The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and video are stored in an external non-volatile memory.

In the embodiment of the present application, image data such as photos and videos taken by the user can be stored in the non-volatile memory connected to the external memory interface 120, so as to be used by the user at any time.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the receiver 170B can be placed close to the human ear to receive the voice.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 may be provided with two microphones 170C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be the USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 180A may be disposed on display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may be comprised of at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.

The gyro sensor 180B can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y and z axes) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a clamshell machine, the electronic device 100 can detect opening and closing of the clamshell according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

The distance sensor 180F is used to measure the distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to make a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.

The ambient light sensor 180L is used for sensing ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature treatment strategy. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the electronic device 100 may reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

The touch sensor 180K is also called "touch device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

In the embodiment of the present application, the electronic device 100 detects whether the touch sensor 180K is used for a user operation on the display screen 194 (click operation, double-click operation, long-press operation, etc.).

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the vocal part acquired by the bone conduction sensor 180M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M, so as to realize the heart rate detection function.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calling and data communication. In some embodiments, the electronic device 100 adopts an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

By implementing the multi-camera-based image content masking method provided in the embodiment of the present application, electronic devices such as mobile phones and tablet computers can collect images with visual difference through two or more cameras. Then, the electronic device can cut and replace the images captured by different cameras based on the above-mentioned visual difference, so as to shield the content that the user does not expect in the images, so as to improve the shooting experience of the user.

In particular, a group of image frames captured by two or more cameras naturally have parallax, so that users do not need to adjust the position of the electronic device or themselves, so that the image content can be shielded by parallax.

The term "user interface (UI)" in the specification, claims and drawings of this application is a medium interface for interaction and information exchange between an application program or an operating system and a user, and it realizes the internal form of information Conversion to and from a form acceptable to the user. The user interface of the application program is the source code written in specific computer languages such as java and extensible markup language (XML). Such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, and scroll bars. (scrollbar), images and text. The properties and contents of the controls in the interface are defined through labels or nodes. For example, XML specifies the controls contained in the interface through nodes such as <Textview>, <ImgView>, and <VideoView>. A node corresponds to a control or property in the interface, and after the node is parsed and rendered, it is presented as the content visible to the user. In addition, the interfaces of many applications, such as hybrid applications, usually include web pages. A web page, also called a page, can be understood as a special control embedded in an application program interface. A web page is a source code written in a specific computer language, such as hypertext markup language (GTML), cascading style Tables (cascading style sheets, CSS), java scripts (JavaScript, JS), etc., and the source code of the web page can be loaded and displayed as user-recognizable content by a browser or a web page display component similar in function to the browser. The specific content contained in the web page is also defined by the tags or nodes in the source code of the web page. For example, GTML defines the elements and attributes of the web page through <p>, <img>, <video>, and <canvas>.

A commonly used representation form of a user interface is a graphical user interface (graphic user interface, GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. Visual interface elements.

As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also include Plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in this application refers to and includes any and all possible combinations of one or more of the listed items. As used in the above embodiments, depending on the context, the term "when" may be interpreted to mean "if" or "after" or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "in determining" or "if detected (a stated condition or event)" may be interpreted to mean "if determining..." or "in response to determining..." or "on detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)".

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state hard disk), etc.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments are realized. The processes can be completed by computer programs to instruct related hardware. The programs can be stored in computer-readable storage media. When the programs are executed , may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk, and other various media that can store program codes.

Claims (15)

1. An image content masking method, wherein the method is applied to an electronic device with a camera, the method comprising:

acquiring a group of image frames, wherein the group of image frames at least comprises a first image and a second image, the first image and the second image are respectively acquired by a first camera and a second camera of the electronic equipment at the same time, and the first camera and the second camera are a front camera or a rear camera of the electronic equipment at the same time;

obtaining a background, a portrait and a self-timer rod in the first image through a neural network model; the neural network model includes a self-encoder and a self-decoder; the self-encoder comprises a front end formed by three layers of convolution networks and a rear end formed by combining two types of convolution blocks, wherein the two types of convolution blocks comprise a block1 and a block2, the block1 comprises two layers of convolution networks, and the block2 comprises three layers of convolution networks; the self-decoder comprises a front end formed by three layers of up-sampling modules and a rear end formed by one layer of convolution network, wherein the up-sampling modules comprise three layers of convolution network and one layer of linear interpolation;

Replacing the image content of a second area in the first image with the image content of a first area in the second image to obtain a third image, wherein the second area is the area where the self-timer stick is located, the first area covers the second area and a plurality of pixel points extending outwards from the second area, and the third image does not comprise the self-timer stick;

and displaying the third image.

2. The method of claim 1, wherein prior to acquiring a set of image frames, the method further comprises: a first user operation is detected.

3. The method of claim 2, wherein the first user operation is: and (5) image shooting operation.

4. A method according to claim 3, wherein the image capturing operation comprises:

an image capturing operation detected by the electronic device and/or an image capturing operation detected by the second device;

the image capturing operation detected by the electronic device includes: and an operation acting on a shooting control, wherein the shooting control is displayed on a user interface provided by the electronic device, and the second device is connected with the electronic device.

5. The method of claim 4, wherein the imaging of the second device in the first image is the selfie stick.

6. The method according to any one of claims 2-5, wherein said displaying said third image, in particular comprises:

saving the third image in response to the first user operation;

and detecting a second user operation, and displaying the third image or a thumbnail of the third image.

7. The method of claim 6, wherein the second user operation comprises: and displaying the third image in the gallery application, or calling the third image by a third party application program.

8. The method of claim 2, wherein the first user operation is: and opening a shooting preview interface.

9. The method of claim 8, wherein the operation of opening the shooting preview interface comprises:

entering the shooting preview interface from a first application program icon, wherein the first application program is an application program provided by the electronic equipment;

or entering the operation of the shooting preview interface from a first control provided by a third party application program.

10. The method according to claim 8 or 9, wherein after detecting the first user operation, the method further comprises: and displaying the shooting preview interface, wherein the shooting preview interface comprises a preview window, the preview window is used for displaying the replaced image in real time, and the replaced image comprises the third image.

11. The method of claim 4, wherein the second device is a self-timer stick device.

12. The method according to claim 11, wherein the self-timer stick device comprises a stick body and a grip, and the second device is imaged as the self-timer stick in the first image, comprising in particular: imaging of the rod body of the self-timer rod device in the first image is the self-timer rod.

13. An electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories for storing computer program code comprising computer instructions that, when executed by the one or more processors, cause the method of any of claims 1-12 to be performed.

14. An electronic device comprising a chip system and one or more memories, the chip system comprising one or more processors, the one or more memories to store computer program code comprising computer instructions to invoke the computer instructions to cause performance of the method of any of claims 1-12.

15. A computer readable storage medium comprising instructions which, when run on an electronic device, cause the method of any of claims 1-12 to be performed.

Images