CN111462164A - Foreground segmentation method and data enhancement method based on image synthesis - Google Patents
Foreground segmentation method and data enhancement method based on image synthesis Download PDFInfo
- Publication number
- CN111462164A CN111462164A CN202010169798.8A CN202010169798A CN111462164A CN 111462164 A CN111462164 A CN 111462164A CN 202010169798 A CN202010169798 A CN 202010169798A CN 111462164 A CN111462164 A CN 111462164A
- Authority
- CN
- China
- Prior art keywords
- image
- foreground
- color
- background
- depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The application is applicable to the technical field of digital image processing, and provides a foreground segmentation method and a data enhancement method based on image synthesis. The foreground segmentation method comprises the following steps: acquiring a depth image and a color image of a background scene with a foreground; carrying out image segmentation on the color image to obtain a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area; fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image; and extracting a color foreground image corresponding to the third foreground binary image from the color image. The method and the device improve the accuracy of foreground segmentation.
Description
Technical Field
The application belongs to the technical field of digital image processing, and particularly relates to a foreground segmentation method and a data enhancement method based on image synthesis.
Background
Foreground extraction is an important branching area of image processing. With the rapid development of image processing technology, the requirement for foreground extraction is higher and higher, so that overfitting in foreground extraction also becomes a hot spot problem.
Data enhancement is an effective solution to overfitting. The existing data enhancement method comprises the following steps: randomly changing image brightness, contrast or color, randomly cropping or flipping the image, etc. However, the data generated by these data enhancement methods have limited diversity.
In addition, the quality of data enhancement usually depends on the accuracy of image segmentation, if the segmentation accuracy is not high, the quality of data enhancement is not high, and especially when the colors of the foreground and the background are close, the segmentation accuracy is not high, and further the efficiency and the applicability of foreground extraction are low.
Disclosure of Invention
The embodiment of the application provides a foreground segmentation method and a data enhancement method based on image synthesis, which can solve the technical problem of low foreground segmentation accuracy in the related technology.
In a first aspect, an embodiment of the present application provides a foreground segmentation method, including:
acquiring a depth image and a color image of a background scene with a foreground;
carrying out image segmentation on the color image to obtain a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image;
and extracting a color foreground image corresponding to the third foreground binary image from the color image.
In the embodiment of the application, two different foreground binary images are obtained by performing image segmentation on the color image and matching the depth image with the background model. And fusing the two different foreground binary images to obtain a new foreground binary image. And extracting a corresponding color foreground image of the new foreground binary image from the color image. The color and depth information of the image are comprehensively considered, so that the accuracy of foreground segmentation is improved.
In a second aspect, an embodiment of the present application provides a method for data enhancement based on image synthesis, including:
acquiring a depth image and a color image of a background scene with a foreground;
carrying out image segmentation on the color image to obtain a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image;
extracting a color foreground image corresponding to the third foreground binary image from the color image;
and pasting the color foreground image to a pre-collected background image to generate a composite image and a label.
In a third aspect, an embodiment of the present application provides a foreground segmentation apparatus, including:
the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a depth image and a color image of a background scene with foreground;
the segmentation and matching unit is used for carrying out image segmentation on the color image and acquiring a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
a fusion unit, configured to fuse the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image;
and the extraction unit is used for extracting a color foreground image corresponding to the third foreground binary image from the color image.
In a fourth aspect, an embodiment of the present application provides an apparatus for data enhancement based on image synthesis, including:
the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a depth image and a color image of a background scene with foreground;
the segmentation and matching unit is used for carrying out image segmentation on the color image and acquiring a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
a fusion unit, configured to fuse the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image;
an extracting unit, configured to extract a color foreground image corresponding to the third foreground binarized image from the color image;
and the synthesis unit is used for pasting the color foreground image to a pre-collected background image to generate a synthesized image and a label.
In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the first aspect or the second aspect when executing the computer program.
In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to the first aspect or the second aspect.
In a seventh aspect, an embodiment of the present application provides a computer program product, which, when run on an electronic device, causes the electronic device to perform the method of the first aspect or the second aspect.
It is to be understood that, the beneficial effects of the second to seventh aspects may be referred to the relevant description of the first aspect, and are not repeated herein.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a foreground segmentation method according to an embodiment of the present disclosure;
FIG. 2 is a schematic flow chart of a data enhancement method based on image synthesis according to another embodiment of the present application;
fig. 3 is a schematic structural diagram of a foreground segmentation apparatus according to an embodiment of the present application;
FIG. 4 is a schematic structural diagram of an apparatus for enhancing data based on image synthesis according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
The foreground segmentation method and the data enhancement method based on image synthesis provided by the embodiment of the application can be applied to electronic devices such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, a super-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), a server and the like, and the embodiment of the application does not limit the specific types of the electronic devices at all. The server includes, but is not limited to, an independent server, a distributed server, a server cluster or a cloud server, etc.
Fig. 1 shows a schematic flow chart of a foreground segmentation method provided by the present application, which is applicable to a situation that a foreground segmentation needs to be performed on an image. The method can be applied to medium electronic equipment. The method is executed by a foreground segmentation apparatus, which is configured in an electronic device and can be implemented by software and/or hardware. As shown in fig. 1, the foreground segmentation method includes steps S110 to S140.
S110, a depth image and a color image of a background scene with foreground are obtained.
In one possible implementation, the electronic device may acquire the depth image and the color image acquired by the acquisition device from the acquisition device, such as a depth camera and a color camera, which are in communication connection.
In another possible implementation, the electronic device may acquire the depth image and the color image through a self-configured acquisition device, for example, the electronic device is configured with a depth camera and a color camera, and the depth camera and the color camera respectively acquire the depth image and the color image.
In another possible implementation, the electronic device may retrieve pre-stored depth images and color images from a communicatively coupled memory.
The depth camera is used for collecting depth images, and the color camera is used for collecting color images. Among them, the depth camera may be a depth camera based on structured light, binocular, or time of Flight (TOF) technology. Color images include, but are not limited to, RGB images.
It should be noted that, in the embodiment of the present application, a depth image and a color image of a background scene with a foreground are acquired. The foreground and background scenes can be set according to specific requirements, and the selection of the foreground and the background scenes are not limited by the application.
It should be further noted that the acquisition device includes a depth camera and a color camera to acquire a depth image and a color image, the acquisition frequencies of the depth image and the color image may be the same or different, and corresponding settings are performed according to specific requirements, which is not limited in this application.
S120, carrying out image segmentation on the color image to obtain a first foreground binary image of the segmented foreground region; and matching the depth image with a pre-established background model to obtain a second foreground binary image of the area with failed matching.
In the embodiment of the application, the electronic device performs image segmentation on a color image comprising a foreground and a background, divides a foreground area and a background area, and binarizes the divided foreground area and the divided background area to obtain a first foreground binary image.
Exemplarily, after the image segmentation, the pixel value of each pixel point of the segmented foreground region is marked as 1, and the pixel value of each pixel point of the segmented background region is marked as 0, at this time, the pixel value of each pixel point in the first foreground binarized image is 1.
In the embodiment of the present application, a background model is established in advance, and the background model may be a background scene with a single color, such as a green screen scene, which has no prospect, and depth data of each pixel point of a background image of the background scene is obtained, so as to establish the background model. It should be noted that, the depth image and the background model are usually acquired by the same acquisition device, and thus the pixel size of the background model is the same as that of the depth image. When not the same, the background model is cropped to have the same pixel size as the depth image.
The depth data of each pixel point of the depth image is obtained and matched with the depth data of the background model, and the foreground area and the background area of the depth image are divided according to whether the depth data of the pixel points are matched, namely whether the depth data of the pixel points are consistent. If the depth data of the pixel point of the depth image are successfully matched, namely the depth data are consistent, the pixel point is indicated to belong to the background; if the depth data of the pixel point of the depth image fails to be matched, namely the pixel point is inconsistent, the pixel point is indicated to belong to the foreground. And after the depth data of each pixel point of the depth image is matched with the depth data of the background model, obtaining a second foreground binary image of the area with failed matching in the depth image.
Illustratively, the depth data of each first pixel point of the depth image is matched with the depth data of a second pixel point at a corresponding pixel position on the background model. If the depth data matching between the first pixel point and the second pixel point at the same pixel position fails, namely the depth data matching is inconsistent, the first pixel point in the depth image is a foreground pixel point, and the pixel values of the pixel points can be marked as 1; if the depth data of the first pixel point and the second pixel point at the same pixel position are successfully matched, namely, the depth data are consistent, the first pixel point in the depth image is a background pixel point, and the pixel values of the pixel points can be marked as 0. Therefore, according to the matching result, a second foreground binary image of the foreground region with failed matching is obtained, each pixel point in the second foreground binary image is a foreground pixel point, and the pixel value of each pixel point can be marked as 1.
It should be understood that, in the process of acquiring depth data, a phenomenon that part of the pixel points cannot acquire depth data occurs, and in order to improve the recall rate of the foreground and obtain a high-precision image, the part of the pixel points that cannot acquire depth data can be classified as a foreground region.
In the embodiment of the application, because the background model considers the depth information of the background at the same time, the defect that the foreground cannot be accurately detected when the background color and the foreground color are the same when the color image is segmented is overcome, the accuracy of segmenting the foreground and the background can be improved, and then the pixel points of the foreground and the background in the depth image can be divided according to the depth image and the depth data of the pre-established background model so as to accurately determine the foreground area of the depth image.
And S130, fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image.
In some embodiments of the present application, a logical and operation may be performed on the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image.
Illustratively, the first foreground binary image and the second foreground binary image are subjected to logical and operation as shown in the following table one, so as to obtain a high-precision third foreground binary image.
Watch 1
When the pixel values of the pixel points input by the first foreground binary image and the second foreground binary image are both 0, or the pixel value of one of the input pixel points of the two images is 0 and the pixel value of the other input pixel point is 1, the pixel values of the background part of the third foreground binary image are obtained, and only when the pixel values of the pixel points input by the first foreground binary image and the second foreground binary image are both 1, the pixel value of the foreground part of the third foreground binary image can be obtained.
It should be noted that the edges of the first foreground binarized image and the second foreground binarized image may not be aligned, in which case, the edges of the two images are aligned by way of zero padding. After the two are aligned, performing logical AND operation on corresponding pixel points, namely two pixel points at the same pixel coordinate position of the two to obtain a third foreground binary image.
And S140, extracting a color foreground image corresponding to the third foreground binary image from the color image.
In this embodiment of the application, step S130 obtains a third foreground binarized image, and the foreground region in the color image can be determined according to the third foreground binarized image. Therefore, in step S140, a color foreground image corresponding to the third foreground binary image is extracted from the color image, i.e., a color foreground image of a foreground region in the color image is extracted.
In the embodiment of the application, two different foreground binary images are obtained by performing image segmentation on the color image and matching the depth image with the background model. And fusing the two different foreground binary images to obtain a new foreground binary image. And extracting a corresponding color foreground image of the new foreground binary image from the color image. The color and depth information of the image are comprehensively considered, so that the accuracy of foreground segmentation is improved.
In another embodiment of the present application, a data enhancement method based on image synthesis is provided, and this embodiment implements a data enhancement scheme based on image synthesis on the basis of the foreground segmentation method embodiment in fig. 1. This embodiment adds an image synthesis process to the embodiment shown in fig. 1. As shown in fig. 2, the data enhancement method based on image synthesis includes steps S210 to S250. It should be understood that the same points in this embodiment as those in the embodiment shown in fig. 1 are not repeated here, and please refer to the foregoing description.
S210, obtaining a depth image and a color image of a background scene with foreground.
S220, carrying out image segmentation on the color image to obtain a first foreground binary image of the segmented foreground region; and matching the depth image with a pre-established background model to obtain a second foreground binary image of the area with failed matching.
And S230, fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image.
And S240, extracting a color foreground image corresponding to the third foreground binary image from the color image.
And S250, pasting the color foreground image to a pre-collected background image, and performing edge smoothing on the color foreground image pasted in the background image to generate a synthetic image and a label.
In the embodiment of the present application, the background image is a background image acquired in advance for synthesizing an image. The background image may be a background color image. The size of the background image and the color image may be the same or different, and the present application does not limit this.
In one possible implementation, after the color foreground image is extracted, the color foreground image is pasted to the background image, and at this time, the color foreground image becomes a foreground region of the background image. And performing edge smoothing on the foreground area in the background image to further generate a composite image.
In the embodiment of the application, the color foreground image is pasted to any area in the background image, namely the pasting area, so that pixel value assignment of pixel points of the background image is essentially realized. That is, the pixel values of the pixels in the color foreground image are assigned to the pixels in the pasting region on the background image.
Illustratively, a bounding rectangle of a color foreground image in the color image, i.e. a bounding rectangle of a connected domain of the foreground, is determined, and coordinates (x _ form, y _ form) of a pixel at the upper left corner of the bounding rectangle are obtained, where the width of the bounding rectangle is w _ form and the height of the bounding rectangle is h _ form.
And (4) optionally selecting a target position on the background image, wherein the pixel coordinates of the target position are (w _ rand, h _ rand), and the upper left corner of the circumscribed rectangle corresponds to the target position.
Traversing each pixel point of a circumscribed rectangle in the color image, wherein the pixel coordinate of any pixel point in the circumscribed rectangle in the color image is (k + x _ form, j + y _ form), wherein 0< ═ k < ═ w _ form, and 0< ═ j < ═ h _ form. For the pixel point with the pixel coordinate of (k + x _ form, j + y _ form), if the pixel value of the third foreground binary image is 1, the pixel value of the pixel point with the pixel coordinate of (k + w _ rand, j + h _ rand) in the background image is assigned to the pixel value of the pixel point with the pixel coordinate of (k + x _ form, j + y _ form) in the color image. If the pixel value of the third foreground binary image is 0, the pixel value of the pixel point with the pixel coordinate of (k + w _ rand, j + h _ rand) in the background image is unchanged.
It should be understood that in other examples, other shapes may be selected for the circumscribing area. Any pixel point of the color foreground image in the color image can be selected to correspond to the target position on the background image. This is merely an example description and should not be construed as a specific limitation of the present application.
As an example and not by way of limitation, any pixel point at the edge of the foreground region is selected, the pixel point is taken as an anchor point, a window region, for example, a pixel region with the size of 3 × 3, is selected, the pixel point in the window region is subjected to pixel average value calculation, and the calculated pixel average value is set as the pixel value of the anchor point, so as to better generate a synthesized image.
Alternatively, on the basis of the embodiment shown in fig. 2, in other embodiments of the present application, in addition to generating the composite image, a label of the composite image may be generated. The generated synthetic image and the label thereof can be used as training data of tasks such as target detection, image segmentation and the like so as to enhance the generalization capability of the model. In the embodiment of the application, different tags can be generated according to different tasks.
In a possible implementation manner, on the basis of the embodiment shown in fig. 2, after the color foreground image corresponding to the third foreground binarized image is pasted to the background image, the pixel values of the pixels in the background image may have been modified, and the modified pixel value of each pixel in the background image is the label.
In another possible implementation manner, the generation of the label for the image segmentation task is taken as an example for explanation. Acquiring a background binary image with the same size as the background image, wherein the pixel value of each pixel of the background binary image is 0, projecting the third foreground binary image acquired in the step S230 onto the background binary image, wherein the position of the third foreground binary image projected onto the background binary image is the same as the position of the color foreground image stuck onto the background image in the step S250, the pixel values of the pixel points in the projected background binary image may change, and the pixel values of each pixel point on the background binary image are labels at this moment. And projecting the third foreground binary image onto the background binary image, and substantially performing pixel value assignment on the background binary image by using the third foreground binary image.
Illustratively, traversing each pixel point of a third foreground binary image foreground connected domain circumscribed rectangle, wherein the pixel coordinate of any pixel point in the circumscribed rectangle is (k + x _ form, j + y _ form), where 0< ═ k < ═ w _ form, and 0< ═ j < ═ h _ form. If the pixel value of the pixel point with the pixel coordinate of (k + x _ form, j + y _ form) in the third foreground binary image is greater than 0, the pixel value of the pixel point with the pixel coordinate of (k + w _ rand, j + h _ rand) in the background binary image is assigned to be 1; otherwise, the value is not changed.
Optionally, on the basis of the embodiment shown in fig. 2, in other embodiments of the present application, before pasting the color foreground image onto the pre-captured background image, the method further includes: and synchronously zooming or rotating the third foreground binary image and the color foreground image. By means of the setting, the diversity of the image data is improved, and if the image data are used in an application scene of a training model, the quantity of the training data can be increased, and the generalization capability of the model is further enhanced.
Optionally, on the basis of the embodiment shown in fig. 2, in other embodiments of the present application, step S250 may also be repeated multiple times, the same foreground image may be pasted to different positions of the same background image, and the number of times of repetition may be operated according to an actual situation, which is not limited here. It should be noted that steps S210 to S250 may be repeatedly executed to obtain a plurality of different foreground images, a plurality of different foreground images may be pasted to the background image, and the number of times of repetition may be operated according to the actual situation, which is not limited herein
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 3 shows a structural block diagram of a foreground segmentation apparatus provided in the embodiment of the present application, corresponding to the foreground segmentation method described in the foregoing embodiment, and only shows portions related to the embodiment of the present application for convenience of description.
Referring to fig. 3, the foreground segmentation apparatus includes:
an acquisition unit 31 for acquiring a depth image and a color image of a background scene having a foreground;
a segmentation and matching unit 32, configured to perform image segmentation on the color image, and obtain a first foreground binary image of the segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
a fusion unit 33, configured to fuse the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image;
and an extracting unit 34, configured to extract a color foreground image corresponding to the third foreground binarized image from the color image.
Fig. 4 shows a block diagram of an apparatus for enhancing data based on image synthesis according to an embodiment of the present application, which corresponds to the method for enhancing data based on image synthesis according to the foregoing embodiment, and only shows portions related to the embodiment of the present application for convenience of description.
Referring to fig. 4, the apparatus for data enhancement based on image synthesis includes:
an acquisition unit 41 configured to acquire a depth image and a color image of a background scene having a foreground;
a segmentation and matching unit 42, configured to perform image segmentation on the color image, and obtain a first foreground binary image of the segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
a fusion unit 43, configured to fuse the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image;
an extracting unit 44, configured to extract a color foreground image corresponding to the third foreground binarized image from the color image;
and a synthesizing unit 45, configured to paste the color foreground image to a pre-acquired background image, and generate a synthesized image and a label.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
Fig. 5 is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 5, the electronic apparatus 5 of this embodiment includes: a processor 50, a memory 51 and a computer program 52 stored in said memory 51 and executable on said processor 50, such as a program for image foreground segmentation or data enhancement based on image synthesis. The processor 50 executes the computer program 52 to implement the steps in the above-mentioned method for image foreground segmentation or data enhancement based on image synthesis, such as steps S110 to S140 shown in fig. 1 or steps S210 to S250 shown in fig. 2. Alternatively, the processor 50 executes the computer program 52 to implement the functions of the modules/units in the device embodiments, such as the functions of the units 31 to 34 shown in fig. 3 or the functions of the units 41 to 45 shown in fig. 4.
Illustratively, the computer program 52 may be partitioned into one or more modules/units that are stored in the memory 51 and executed by the processor 50 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 52 in the electronic device 5.
The electronic device 5 may be a smart phone, a computer, a tablet, a server, or the like. The electronic device 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that fig. 5 is merely an example of an electronic device 5 and does not constitute a limitation of the electronic device 5 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the electronic device may also include input-output devices, network access devices, buses, etc.
The Processor 50 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or a memory of the electronic device 5. The memory 51 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the electronic device 5. The memory 51 is used for storing the computer program and other programs and data required by the electronic device. The memory 51 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The embodiments of the present application provide a computer program product, which when running on a mobile terminal, enables the mobile terminal to implement the steps in the above method embodiments when executed.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/electronic device, a recording medium, computer Memory, Read-Only Memory (ROM), random-access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A method of foreground segmentation, comprising:
acquiring a depth image and a color image of a background scene with a foreground;
carrying out image segmentation on the color image to obtain a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image;
and extracting a color foreground image corresponding to the third foreground binary image from the color image.
2. A method of image synthesis based data enhancement, comprising:
acquiring a depth image and a color image of a background scene with a foreground;
carrying out image segmentation on the color image to obtain a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
fusing the first foreground binary image and the second foreground binary image to obtain a third foreground binary image;
extracting a color foreground image corresponding to the third foreground binary image from the color image;
and pasting the color foreground image to a pre-collected background image to generate a composite image and a label.
3. The method of claim 1 or 2, wherein said matching the depth image to a pre-established background model comprises:
acquiring depth data of each pixel point of the depth image, matching the depth data with depth data of a pre-established background model, and determining that the pixel point in the depth image belongs to a background if matching is successful; and if the matching fails, determining that the pixel points in the depth image belong to the foreground.
4. The method as claimed in claim 1 or 2, wherein said fusing the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image, comprises:
and performing logical AND operation on the first foreground binary image and the second foreground binary image to obtain a third foreground binary image.
5. The method of claim 2, wherein pasting the colored foreground image onto a pre-captured background image further comprises:
and performing edge smoothing processing on the color foreground image pasted in the background image.
6. The method of claim 2, wherein the tag comprises:
acquiring a pixel value of each pixel point of the synthetic image as a label; or
And acquiring a background binary image with the same size as the background image, projecting the third foreground binary image onto the background binary image, and taking the pixel value of each pixel point of the projected background binary image as a label.
7. An apparatus for foreground segmentation, comprising:
the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a depth image and a color image of a background scene with foreground;
the segmentation and matching unit is used for carrying out image segmentation on the color image and acquiring a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
a fusion unit, configured to fuse the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image;
and the extraction unit is used for extracting a color foreground image corresponding to the third foreground binary image from the color image.
8. An apparatus for image synthesis based data enhancement, comprising:
the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a depth image and a color image of a background scene with foreground;
the segmentation and matching unit is used for carrying out image segmentation on the color image and acquiring a first foreground binary image of a segmented foreground region; matching the depth image with a pre-established background model to obtain a second foreground binary image of a matching failure area;
a fusion unit, configured to fuse the first foreground binarized image and the second foreground binarized image to obtain a third foreground binarized image;
an extracting unit, configured to extract a color foreground image corresponding to the third foreground binarized image from the color image;
and the synthesis unit is used for pasting the color foreground image to a pre-collected background image to generate a synthesized image and a label.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010169798.8A CN111462164A (en) | 2020-03-12 | 2020-03-12 | Foreground segmentation method and data enhancement method based on image synthesis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010169798.8A CN111462164A (en) | 2020-03-12 | 2020-03-12 | Foreground segmentation method and data enhancement method based on image synthesis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111462164A true CN111462164A (en) | 2020-07-28 |
Family
ID=71682806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010169798.8A Pending CN111462164A (en) | 2020-03-12 | 2020-03-12 | Foreground segmentation method and data enhancement method based on image synthesis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462164A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113344942A (en) * | 2021-05-21 | 2021-09-03 | 深圳瀚维智能医疗科技有限公司 | Human body massage region segmentation method, device and system and computer storage medium |
| CN113362365A (en) * | 2021-06-17 | 2021-09-07 | 云从科技集团股份有限公司 | Video processing method, system, device and medium |
| CN113592753A (en) * | 2021-07-23 | 2021-11-02 | 深圳思谋信息科技有限公司 | Image processing method and device based on industrial camera shooting and computer equipment |
| CN113850784A (en) * | 2021-09-27 | 2021-12-28 | 联想(北京)有限公司 | An image processing method, device, equipment and storage medium |
| CN114842063A (en) * | 2021-01-30 | 2022-08-02 | 华为技术有限公司 | Depth map optimization method, device, equipment and storage medium |
| CN115187627A (en) * | 2022-06-13 | 2022-10-14 | 奥比中光科技集团股份有限公司 | A method, device and electronic device for obtaining foreground mask |
| CN116823888A (en) * | 2023-06-29 | 2023-09-29 | 合肥工业大学 | Visual sorting method based on rotating target detection |
| CN119863485A (en) * | 2025-03-24 | 2025-04-22 | 北京开源芯片研究院 | Code identification method, device, equipment and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6661918B1 (en) * | 1998-12-04 | 2003-12-09 | Interval Research Corporation | Background estimation and segmentation based on range and color |
| CN102999901A (en) * | 2012-10-17 | 2013-03-27 | 中国科学院计算技术研究所 | Method and system for processing split online video on the basis of depth sensor |
| CN104361577A (en) * | 2014-10-20 | 2015-02-18 | 湖南戍融智能科技有限公司 | Foreground detection method based on fusion of depth image and visible image |
| CN105590309A (en) * | 2014-10-23 | 2016-05-18 | 株式会社理光 | Method and device for segmenting foreground image |
| CN108447060A (en) * | 2018-01-29 | 2018-08-24 | 上海数迹智能科技有限公司 | Front and back scape separation method based on RGB-D images and its front and back scene separation device |
| CN110060205A (en) * | 2019-05-08 | 2019-07-26 | 北京迈格威科技有限公司 | Image processing method and device, storage medium and electronic equipment |
-
2020
- 2020-03-12 CN CN202010169798.8A patent/CN111462164A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6661918B1 (en) * | 1998-12-04 | 2003-12-09 | Interval Research Corporation | Background estimation and segmentation based on range and color |
| CN102999901A (en) * | 2012-10-17 | 2013-03-27 | 中国科学院计算技术研究所 | Method and system for processing split online video on the basis of depth sensor |
| CN104361577A (en) * | 2014-10-20 | 2015-02-18 | 湖南戍融智能科技有限公司 | Foreground detection method based on fusion of depth image and visible image |
| CN105590309A (en) * | 2014-10-23 | 2016-05-18 | 株式会社理光 | Method and device for segmenting foreground image |
| CN108447060A (en) * | 2018-01-29 | 2018-08-24 | 上海数迹智能科技有限公司 | Front and back scape separation method based on RGB-D images and its front and back scene separation device |
| CN110060205A (en) * | 2019-05-08 | 2019-07-26 | 北京迈格威科技有限公司 | Image processing method and device, storage medium and electronic equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114842063A (en) * | 2021-01-30 | 2022-08-02 | 华为技术有限公司 | Depth map optimization method, device, equipment and storage medium |
| CN113344942A (en) * | 2021-05-21 | 2021-09-03 | 深圳瀚维智能医疗科技有限公司 | Human body massage region segmentation method, device and system and computer storage medium |
| CN113344942B (en) * | 2021-05-21 | 2024-04-02 | 深圳瀚维智能医疗科技有限公司 | Human body massage region segmentation method, device and system and computer storage medium |
| CN113362365A (en) * | 2021-06-17 | 2021-09-07 | 云从科技集团股份有限公司 | Video processing method, system, device and medium |
| CN113592753A (en) * | 2021-07-23 | 2021-11-02 | 深圳思谋信息科技有限公司 | Image processing method and device based on industrial camera shooting and computer equipment |
| CN113592753B (en) * | 2021-07-23 | 2024-05-07 | 深圳思谋信息科技有限公司 | Method and device for processing image shot by industrial camera and computer equipment |
| CN113850784A (en) * | 2021-09-27 | 2021-12-28 | 联想(北京)有限公司 | An image processing method, device, equipment and storage medium |
| CN115187627A (en) * | 2022-06-13 | 2022-10-14 | 奥比中光科技集团股份有限公司 | A method, device and electronic device for obtaining foreground mask |
| CN116823888A (en) * | 2023-06-29 | 2023-09-29 | 合肥工业大学 | Visual sorting method based on rotating target detection |
| CN119863485A (en) * | 2025-03-24 | 2025-04-22 | 北京开源芯片研究院 | Code identification method, device, equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462164A (en) | Foreground segmentation method and data enhancement method based on image synthesis | |
| CN107633526B (en) | Image tracking point acquisition method and device and storage medium | |
| CN109753953B (en) | Method and device for positioning text in image, electronic equipment and storage medium | |
| US8494297B2 (en) | Automatic detection and mapping of symmetries in an image | |
| CN111563908B (en) | An image processing method and related device | |
| CN114359048B (en) | Image data enhancement method, device, terminal device and storage medium | |
| CN114677394B (en) | Matting method, matting device, image pickup apparatus, conference system, electronic apparatus, and medium | |
| CN113298187B (en) | Image processing method and device and computer readable storage medium | |
| US11657511B2 (en) | Heuristics-based detection of image space suitable for overlaying media content | |
| CN111598076B (en) | Method and device for detecting and processing date in label image | |
| CN112233049B (en) | Image fusion method for improving image definition | |
| CN111191580B (en) | Synthetic rendering method, apparatus, electronic device and medium | |
| CN110599525A (en) | Image compensation method and apparatus, storage medium, and electronic apparatus | |
| CN107633498B (en) | Image dark state enhancement method and device and electronic equipment | |
| CN114079815B (en) | Subtitle protection method, system, terminal equipment and storage medium | |
| CN115937065A (en) | Foreign object detection method, device, equipment and storage medium of display module | |
| CN114677392A (en) | Matting method, image pickup apparatus, device, conference system, electronic apparatus, and medium | |
| CN119359923A (en) | A three-dimensional modeling method, device, terminal equipment and storage medium based on point cloud fusion | |
| CN111401365B (en) | OCR image automatic generation method and device | |
| CN105894068B (en) | FPAR card design and rapid identification and positioning method | |
| CN115965647A (en) | Background image generation method, image fusion method, device, electronic equipment and readable medium | |
| CN115578715B (en) | Image generation methods and vehicles | |
| CN117557777A (en) | Sample image determining method and device, electronic equipment and storage medium | |
| CN111899204B (en) | Vehicle loss detection data synthesis method, device and storage medium | |
| CN115205111A (en) | Image splicing method and device, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 11-13 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant after: Obi Zhongguang Technology Group Co.,Ltd. Address before: 12 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant before: SHENZHEN ORBBEC Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200728 |