CN112083864B - Method, device and equipment for processing object to be deleted - Google Patents

Abstract

The invention discloses a method, a device and equipment for processing an object to be deleted, wherein the method comprises the following steps: photographing through the double cameras, recording depth information of a viewfinder, and storing the photographed pictures; displaying the selected deleting frame according to the operation and framing the object to be deleted in the photo; and removing the photo data in the deletion frame according to the depth information, and filling and repairing the removed area through the environment data. The method provided by the embodiment of the invention can be used for rapidly processing the object to be deleted, is not limited by time and space, has no requirement on any processing skill on a user, and can solve the problems that the processing mode of the object to be deleted in the existing photo is time-consuming and has the skill requirement. In addition, the invention also discloses a device and equipment for processing the object to be deleted.

Description

A method, device and apparatus for processing objects to be deleted

Technical Field

The present invention relates to the technical field of image processing, and in particular to a method, device and equipment for processing an object to be deleted.

Background Art

Users often take photos or videos with their mobile phone cameras and then share them through corresponding social software; this reflects more timeliness and usually requires quick sharing.

If intruders (such as people, cars, small animals, etc.) suddenly appear in the shooting frame while taking photos or recording videos, or some objects that affect the appearance (such as trash cans, garbage, toilets, etc.) cannot be removed from the shooting frame, these intruders and objects that affect the appearance are all objects to be deleted, which may affect the user's shooting environment and reduce the overall effect.

At present, the processing method for objects to be deleted in photos is usually through post-processing by third parties or computer software. If it can only be processed by computer, it is necessary to use PS (image processing software) to remove the objects to be deleted, and then use the tools in PS (such as clone stamp) to paint the removed parts into the corresponding environment. Although this can achieve the desired effect, it takes a long time to process and requires users to master the corresponding PS processing skills. The processing effect is related to the processing skills and proficiency of the user, and sometimes the processing effect will be greatly reduced.

Therefore, the prior art still needs to be improved and developed.

Summary of the invention

In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a method, device and apparatus for processing objects to be deleted, so as to solve the problem that the existing method for processing objects to be deleted in photos is relatively time-consuming and requires skills.

In order to achieve the above object, the present invention adopts the following technical solutions:

In a first aspect, an embodiment of the present invention provides a method for processing an object to be deleted, which comprises the following steps:

Take photos through dual cameras and record the depth information of the objects, and store the photos taken;

The selected deletion frame is displayed according to the operation and frames the object to be deleted in the photo;

The photo data within the deletion box is removed according to the depth of field information, and the removed area is filled and repaired using the environmental data.

In the method for processing an object to be deleted, the step of displaying the selected deletion frame and framing the object to be deleted in the photo according to the operation includes:

When a photo in the gallery is detected to be clicked, a function bar is displayed below the photo;

When the edit icon in the detection function bar is clicked, a delete box selection bar is displayed below the photo;

According to the user's operation, the object to be deleted is framed with the selected deletion frame and saved.

In the method for processing the object to be deleted, the deletion boxes provided in the deletion box selection bar include: a nine-square diagram, a vertical circle and a vertical rectangle.

In the method for processing objects to be deleted, after the icon of the nine-square diagram is clicked, the photo is divided into 9 areas by dividing lines. After detecting that a certain area is selected, the size range of the selected area is adjusted according to the user's drag operation to frame all the objects to be deleted in the area.

In the method for processing the objects to be deleted, after the vertical circular icon is clicked, a circular frame is displayed in the center of the photo, and the position and size of the circular frame are adjusted according to the user's dragging operation to frame all the objects to be deleted in the circular frame.

In the method for processing the object to be deleted, after the vertical rectangular icon is clicked, a square grid is displayed in the center of the photo, and the position and size of the square grid are adjusted according to the user's dragging operation.

In the method for processing an object to be deleted, the step of removing the photo data in the deletion frame according to the depth of field information comprises:

Calculate the distance between each object in the photo and the lens, separate the corresponding focal plane according to the distance, find the focal plane where the deletion box is located, and fill the area in the deletion box with white.

In the method for processing the object to be deleted, the step of filling and repairing the removed area by using the environmental data comprises:

Identify each pixel on the edge of the white area in the deletion box, calculate the value of the corresponding pixel based on the environmental data around each pixel, and fill the edge;

Each pixel point adjacent to the edge is taken as a new edge, and the value of each pixel point on the new edge is calculated and filled according to the calculated value of each pixel point on the edge and the environmental data;

Return to the step of identifying each pixel point on the edge of the white area in the deletion box, until the values of each pixel point in the entire white area are calculated and the filling and patching are completed by calculating along each edge and shrinking inward.

In a second aspect, an embodiment of the present invention provides a device for processing an object to be deleted, comprising:

A shooting unit, used to take photos through dual cameras and record depth information of the objects being photographed, and store the taken photos;

A selection unit, used to display a selected deletion frame and frame the object to be deleted in the photo according to the operation;

The removal and repair unit is used to remove the photo data within the deletion frame according to the depth of field information, and fill and repair the removed area through the environmental data.

In a third aspect, an embodiment of the present invention further provides a device for processing objects to be deleted, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to include the method for executing the one or more programs.

In a fourth aspect, an embodiment of the present invention further provides a non-temporary computer-readable storage medium, which enables the device to execute the described method when instructions in the storage medium are executed by a processor of the device.

Compared with the prior art, the present invention provides a method, device and apparatus for processing objects to be deleted, the method comprising the steps of: recording the depth of field information in the framed photo through a dual camera and storing the taken photo; displaying the selected deletion frame according to the operation and framing the object to be deleted in the photo; removing the photo data in the deletion frame according to the depth of field information, and filling and repairing the removed area with data. The object to be deleted can be processed quickly, without being restricted by time and space, and does not require any processing skills from the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for processing an object to be deleted provided by the present invention;

FIG2 is a schematic diagram of a function bar provided by the present invention;

FIG3 is a schematic diagram of a deletion box provided by the present invention;

FIG4 is a schematic diagram of a nine-square diagram provided by the present invention;

FIG5 is a schematic diagram of a vertical circle provided by the present invention;

FIG6 is a schematic diagram of focal planes of objects at different positions provided by the present invention;

FIG7 is a schematic diagram of a vertical rectangle provided by the present invention;

FIG8 is a schematic diagram of the result of processing the object to be deleted provided by the present invention;

FIG. 9 is a schematic diagram of the structure of the device for processing objects to be deleted provided by the present invention.

DETAILED DESCRIPTION

The present invention provides a method, device and apparatus for processing objects to be deleted. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

Since dual-camera (two cameras) hardware configuration has become a standard configuration for existing mobile terminals (a mobile phone is used as an example in this embodiment), this embodiment uses the dual-camera blur function to record the depth of field displacement map and the depth of field information before and after taking a photo or recording a video. After taking a photo or recording a video, the object to be deleted is removed from the gallery.

Please refer to Figure 1, which is a flow chart of the method for processing objects to be deleted provided by the present invention. As shown in Figure 1, the method for processing objects to be deleted includes the following steps:

S10. Taking photos through the dual cameras and recording depth information of the objects being photographed, and storing the taken photos.

In this step, the dual camera is started during shooting, the dual camera mode is turned on to record the depth of field information of the object, and the taken photos are stored in the gallery. Shooting includes taking photos and taking videos, and photos are taken as an example here. The depth of field information is the depth of field displacement map.

Video is composed of multiple frames of data. Taking H.264 video encoding as an example, it is divided into I frames, P frames and B frames during encoding.

I frame, or intra-coded frame, is an independent frame that carries all its own information and can be decoded independently without referring to other images. It can be simply understood as a static picture.

P frame is an inter-frame prediction coding frame, which needs to refer to the previous I frame for coding.

B frame is a bidirectional predictive coding frame, which records the difference between the current frame and the previous and next frames.

I frames only need to consider the current frame, P frames record the difference with the previous frame; B frames record the difference between the previous frame and the next frame, which can save more space.

Therefore, when processing a video, the editing page is entered during playback, and the data of the I frame is mainly displayed. The user can refer to the method of removing objects from photos and remove the data of the object to be deleted in the I frame. Then, the P frame and B frame of the saved video data are re-encoded with reference to the data of the I frame to eliminate the object to be deleted in the entire video.

S20. Display the selected deletion frame according to the operation and frame the object to be deleted in the photo.

After the detected photo is opened, a deletion frame is displayed according to the user's operation to facilitate the user to select the object to be deleted; due to the different contours of the object to be deleted, this embodiment provides three types of deletion frames, and the step S20 specifically includes

S21. When a photo in the detection gallery is clicked, a function bar is displayed below the photo.

As shown in Figure 2, the function bar includes a share icon on the left, an edit icon in the middle, and a delete icon on the right. When the user clicks the edit icon, the process of removing the object to be deleted is entered.

S22. When the edit icon in the detection function bar is clicked, a delete box selection bar is displayed below the photo.

This embodiment provides three shapes of deletion frames for objects to be deleted, namely, a nine-square grid, a vertical circle, and a vertical rectangle, as shown in FIG3 , so that users can easily select the most appropriate deletion frame for objects to be deleted with different contours or positions.

S23. Frame the object to be deleted with the selected deletion frame according to the user operation and save it.

After clicking the icon of the nine-square diagram (the leftmost one in the delete box selection bar), as shown in FIG4, the photo is divided into 9 areas by dividing lines. After selecting an area (a bold selection box is displayed at the edge of the area), the size of the selected area can be freely dragged to make the entire object to be deleted within the area and the area is minimized. After completion, click Save, and only the border of the selected area will remain on the photo, and the other borders will disappear.

However, there may be objects to be deleted that are not all in one area, such as the barrel shown in FIG4, which is located in two areas. In this case, the vertical circle can be used to directly select the object to be deleted.

As shown in FIG5 , after the vertical circular icon is clicked, a circular frame will be displayed in the center of the photo. The user can freely drag the circular frame to move its position and change its size, such as turning it into an ellipse, as long as the entire object to be deleted can be enclosed in the circular frame.

As shown in FIG. 6 , after the vertical rectangular icon is clicked, a square grid will be displayed in the center of the photo, and the user can freely drag the square grid to move its position and change its length and width.

Users can select the most appropriate deletion box according to the outline of the object to be deleted. Users can select the corresponding operation according to the operation bar above, such as cancel, arrow to the left to return to the previous step, arrow to the right to return to the next step, and save these four operations to retain the staged operations.

After selecting Save, you can start removing the objects to be deleted in the deletion box.

S30, removing the photo data in the deletion box according to the depth of field information, and filling and repairing the removed area with the environmental data.

This step is to remove all photo data in the deletion box (including the object to be deleted and part of the environmental data around it). In the process of removing the object to be deleted, the object to be deleted is first removed through the depth of field information (i.e., the depth of field displacement map) to make the removal of the object to be deleted more accurate; the deleted deletion box is first filled with white, and then the data is backfilled through machine learning, that is, machine learning is performed in combination with the environmental data of the adjacent area of the removed area, and the data of the removed area is filled and repaired, so that the backfilled data is closer to the original image data, making the overall photo more realistic; then save it.

Among them, the photo data in the deletion frame is removed according to the depth of field information, and the dual-camera principle is adopted, that is, the left and right cameras, when imaging, the left camera is set to shoot the background part, and the right camera is set to shoot the subject (usually a person, or the object closest to the lens). Dual-camera is to calculate the distance between the object and the lens based on the triangulation positioning principle of the human eye. Specifically, based on the dual-camera principle, according to the focal length of the dual-camera module, the distance between the main camera (i.e., the right camera) and the auxiliary camera (i.e., the left camera) and the distance of the subject of the view, the distance between the object and the lens at any position in the entire photo can be calculated. This is a prior art and will not be described in detail here.

Calculating the displacement information of different objects in the entire photo (subject person or object, object to be deleted, car, animal, etc.) can provide a theoretical numerical basis for the subsequent object removal. As shown in Figure 7, objects at different positions have different focal planes and focal lengths. The focal length a of focal plane A, the focal length b of focal plane B, and the focal length c of focal plane C are all different. Therefore, the dual-camera principle can be used to remove images with different focal planes.

By calculating the statistical data of the dual cameras, the object displacement information (the distance from the lens or the focal length) of different focal planes described in Figure 7, namely the depth of field displacement map, is obtained. According to the depth of field displacement map, all data in the deletion box can be removed, that is, the distance between each object in the photo and the lens is calculated, the corresponding focal plane is separated according to the distance, the focal plane where the deletion box is located is found, and the area in the deletion box is filled with white, thus completing the removal of the object to be deleted.

The above-mentioned machine learning is a multidisciplinary cross-disciplinary major that covers knowledge of probability theory, statistics, approximate theory and complex algorithm knowledge. It uses computers as tools and is committed to simulating human learning methods in real time, and divides existing content into knowledge structures to effectively improve learning efficiency.

Smart filling in machine learning, namely AI (artificial intelligence) smart filling, refers to the data information nearby for filling. As the configuration of mobile phones is getting higher and higher, AI intelligent operators are becoming more and more advanced. To fill the removed data, a mean weighted solution can be used. Based on machine learning and the environmental data around the removed area, the removed area is backfilled and repaired. The details are as follows:

When filling data, a to-be-filled data X represents the value of a pixel in the white area. The to-be-filled data is calculated by referring to the environmental data in 8 directions around (i.e., adjacent to) the to-be-filled data (i.e., the values of 8 pixels corresponding to the 8 directions), and the corresponding pixel value can be obtained. The values of each pixel on the edge of the white area are calculated first, and the edge is filled and is no longer white after filling. The white part adjacent to the edge forms a new edge, and the values of each pixel on the new edge can be obtained and filled according to the calculated values of each pixel on the edge and the environmental data. And so on, the calculation is continuously performed along the edge and the edges are shrunk inwards, and the size of the white area is gradually reduced until the entire white area is filled.

When calculating, the environmental data in the eight adjacent directions are divided into two weights, among which the environmental data in the 0-degree direction of the data X to be filled is X1 (equivalent to the environmental data on the right of the data X to be filled is X1), the environmental data in the 90-degree direction of the data X to be filled is X2 (equivalent to the environmental data on the top of the data X to be filled is X2), the environmental data in the 180-degree direction of the data X to be filled is X3 (equivalent to the environmental data on the left of the data X to be filled is X3), the environmental data in the 270-degree direction of the data X to be filled is X4 (equivalent to the environmental data on the bottom of the data X to be filled is X4), and the data weights in these four directions are higher; the environmental data in the 45-degree direction of the data X to be filled is X5 (equivalent to the environmental data in the upper right corner of the data X to be filled is X5), the environmental data in the 135-degree direction of the data X to be filled is X6 (equivalent to the environmental data in the upper left corner of the data X to be filled is X6), the environmental data in the 225-degree direction of the data X to be filled is X7 (equivalent to the environmental data in the lower left corner of the data X to be filled is X7), and the environmental data in the 270-degree direction of the data X to be filled is X8 (equivalent to the environmental data in the lower left corner of the data X to be filled is X8). The environmental data of the 315-degree direction of X is X8 (equivalent to the environmental data at the lower right corner of the data to be filled X is X8). The data weights of these four directions are relatively low and can be defaulted to 1. Then a data to be filled X can be calculated according to the following formula:

X= [(X1+ X2+ X3+ X4) *N +(X5+ X6+ X7+ X8) *1]/(N+1), N≥2. N is the data weight of the four directions of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and its value can be adjusted according to needs.

By repeating the above calculation, the values of all pixels in the area to be filled can be calculated based on the environmental data in the adjacent 8 directions (i.e., the values of the pixels corresponding to the 8 directions), so as to achieve backfilling of the removed area through the environmental data, and complete the data backfilling of the removed object area (i.e., the area to be filled, that is, the white area in the deleted box). It should be understood that when calculating along the edge, the pixels on the edge will use the pixels in the white area in one or several directions, and the values of the pixels in the white area are the values of the environmental data corresponding to white.

After the processing is completed, the photo can be saved and the photo shown in Figure 8 can be obtained. After that, the user can share the photo and upload it to the corresponding social software.

Based on the above method for processing an object to be deleted, an embodiment of the present invention further provides a device for processing an object to be deleted, as shown in FIG9 , which includes:

A shooting unit 10 is used to take pictures through dual cameras and record depth information of the objects being photographed, and store the taken pictures;

A selection unit 20, used to display a selected deletion frame and frame the object to be deleted in the photo according to the operation;

The removal and repair unit 30 is used to remove the photo data in the deletion frame according to the depth of field information, and fill and repair the removed area through the environmental data.

Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the device to which the scheme of the present application is applied. The specific device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

Based on the above-mentioned method for processing objects to be deleted, an embodiment of the present invention also provides a device for processing objects to be deleted, including a memory, which may be a hard disk or a memory, and the memory stores one or more algorithm processing programs of the method for processing objects to be deleted, and the algorithm processing program can be configured to be executed by one or more processors. The one or more algorithm processing programs include the method for executing the described method.

The device may be an electronic device with dual cameras, such as a smart phone or a tablet computer. In some embodiments, the processor may be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code stored in the memory or process data, such as executing the method for processing the object to be deleted.

The embodiment of the present invention also provides a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, and the above instructions can be executed by a processor of a device to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

In summary, in a method, device and apparatus for processing objects to be deleted provided by the present invention, depth of field information in the framed photos is recorded through dual cameras and the taken photos are stored; a selected deletion frame is displayed according to the operation and the object to be deleted in the photo is framed; the photo data in the deletion frame is removed according to the depth of field information, and the removed area is filled and repaired with data; in this way, the object to be deleted in the photo or video can be removed, thereby highlighting the subject of the photo, eliminating the influence of the framing object to be deleted on the framing effect of the subject, thereby improving the picture quality of the photo or video to reflect the effect of the core subject framing object.

At the same time, users only need to select the appropriate deletion frame to select the object to be deleted and automatically delete and repair it. It is not limited by time, place and space, and does not require any processing skills from users. The photos or videos taken can be shared quickly. In addition, the combination of machine learning and dual-camera depth of field displacement map makes the removal effect more accurate. The data of the removal area is filled by the mean weighting method, and the repaired effect can be well integrated with the surrounding area, ensuring the smoothness of the overall effect.

Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, an optical disk, etc.

It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims (8)

1. A method for processing an object to be deleted, characterized in that it comprises the following steps: Take photos through dual cameras and record the depth information of the objects, and store the photos taken; The selected deletion frame is displayed according to the operation and frames the object to be deleted in the photo; Remove the photo data in the deletion box according to the depth of field information, and fill and repair the deleted area with the environmental data; The steps to fill and repair the removed area using environmental data include: Identify each pixel on the edge of the white area in the deletion box, calculate the value of the corresponding pixel based on the environmental data around each pixel, and fill the edge; Each pixel point adjacent to the edge is taken as a new edge, and the value of each pixel point on the new edge is calculated and filled according to the calculated value of each pixel point on the edge and the environmental data; Return to the step of identifying each pixel point on the edge of the white area in the deletion box, until the values of each pixel point in the entire white area are calculated and the filling and patching are completed by calculating along each edge and shrinking inward; X＝[(X1+X2+X3+X4)*N+(X5+X6+X7+X8)*1]/(N+1), N≥2; N is the data weights in the four directions of 0, 90, 180, and 270 degrees; X is the data to be filled, and the environmental data at the directions of 0, 90, 180, 270, 45, 135, 225, and 315 degrees of the data to be filled X are X1, X2, X3, X4, X5, X6, X7, and X8 respectively; The pixels on the edge will use the pixels in the white area; The step of removing the photo data in the deletion frame according to the depth of field information comprises: Calculate the distance between each object in the photo and the lens, separate the corresponding focal plane according to the distance, find the focal plane where the deletion box is located, and fill the area in the deletion box with white; Data backfilling via machine learning. 2. The method for processing an object to be deleted according to claim 1, wherein the step of displaying the selected deletion frame and framing the object to be deleted in the photo according to the operation comprises: When a photo in the gallery is detected to be clicked, a function bar is displayed below the photo; When the edit icon in the detection function bar is clicked, a delete box selection bar is displayed below the photo; According to the user's operation, the object to be deleted is framed with the selected deletion frame and saved. 3. The method for processing an object to be deleted according to claim 2 is characterized in that the deletion boxes provided in the deletion box selection bar include: a nine-square diagram, a vertical circle and a vertical rectangle. 4. The method for processing objects to be deleted according to claim 3 is characterized in that after the icon of the nine-square diagram is clicked, the photo is divided into 9 areas by dividing lines. After detecting that a certain area is selected, the size range of the selected area is adjusted according to the user's drag operation to frame all the objects to be deleted in the area. 5. The method for processing objects to be deleted according to claim 3 is characterized in that after the vertical circular icon is clicked, a circular frame is displayed in the center of the photo, and the position and size of the circular frame are adjusted according to the user's drag operation to frame all the objects to be deleted in the circular frame. 6. The method for processing objects to be deleted according to claim 3 is characterized in that after the vertical rectangular icon is clicked, a square grid is displayed in the center of the photo, and the position and size of the square grid are adjusted according to the user's dragging operation. 7. A device for processing an object to be deleted, characterized by comprising: A shooting unit, used to take photos through dual cameras and record depth information of the objects being photographed, and store the taken photos; A selection unit, used to display a selected deletion frame and frame the object to be deleted in the photo according to the operation; The removal and repair unit is used to remove the photo data in the deletion frame according to the depth of field information, and fill and repair the removed area through the environmental data; calculate the distance between each viewfinder and the lens in the photo, separate the corresponding focal plane according to the distance, find the focal plane where the deletion frame is located, and fill the area in the deletion frame with white; perform data backfilling through machine learning; identify each pixel point on the edge of the white area in the deletion frame, calculate the value of the corresponding pixel point according to the environmental data around each pixel point, and fill the edge; Each pixel point adjacent to the edge is taken as a new edge, and the value of each pixel point on the new edge is calculated and filled according to the calculated value of each pixel point on the edge and the environmental data; Return to the step of identifying each pixel point on the edge of the white area in the deletion box, until the values of each pixel point in the entire white area are calculated and the filling and patching are completed by calculating along each edge and shrinking inward; X＝[(X1+X2+X3+X4)*N+(X5+X6+X7+X8)*1]/(N+1), N≥2; N is the data weights in the four directions of 0, 90, 180, and 270 degrees; X is the data to be filled, and the environmental data at the directions of 0, 90, 180, 270, 45, 135, 225, and 315 degrees of the data to be filled X are X1, X2, X3, X4, X5, X6, X7, and X8 respectively; The pixels on the edge will use the pixels in the white area. 8. A device for processing objects to be deleted, characterized in that it includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including for executing the method described in any one of claims 1-6.