CN112150532B - Image processing method, device, electronic device and computer readable medium - Google Patents
Image processing method, device, electronic device and computer readable medium Download PDFInfo
- Publication number
- CN112150532B CN112150532B CN202010867321.7A CN202010867321A CN112150532B CN 112150532 B CN112150532 B CN 112150532B CN 202010867321 A CN202010867321 A CN 202010867321A CN 112150532 B CN112150532 B CN 112150532B
- Authority
- CN
- China
- Prior art keywords
- image
- processed
- cost
- cube
- pixel point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The application provides an image processing method, an image processing device, electronic equipment and a computer readable medium, which relate to the technical field of image processing and comprise the steps of acquiring a main shot image to be processed and a secondary shot image to be processed, and respectively determining the main shot image and the secondary shot image as images to be processed; the method comprises the steps of calculating a cost cube of an image to be processed to obtain a target cost cube, wherein the target cost cube represents matching cost values of pixel points in the image to be processed under each preset parallax level, determining a filtering window of each pixel point in the image to be processed, filtering a cost map under each preset parallax level in the target cost cube through the filtering window, and calculating parallax maps of a main shooting image and a sub shooting image based on the filtered target cost cube.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method, an apparatus, an electronic device, and a computer readable medium for image processing.
Background
With the rapid development of software, hardware and algorithm industries, three-dimensional reconstruction algorithms in the field of computer vision are increasingly applied in a plurality of fields, such as intelligent robots, mobile phone camera applications, and the like. Therefore, a binocular stereo matching algorithm for estimating scene depth based on a main image and a sub image acquired by double cameras in three-dimensional reconstruction is researched by a plurality of students and applied to a plurality of industries. The conventional binocular stereo matching algorithm flow generally comprises four modules of cost cube calculation, cost aggregation, optimal parallax calculation and parallax post-processing of a main image and a sub image. Cost aggregation plays a very important role in disparity map computation as a module to enhance the reliability of the cost cube.
There are two common cost aggregation methods, local and global. The local method is implemented by averaging (i.e. filtering) the cost value weights of other points within a certain window radius of the current point. The local approach can be seen as using a guard-side smoothing filter to denoise the cost cube. But as the parallax level d increases, the image size increases and the performance of the local method becomes a bottleneck to be implemented on the mobile device. Global methods, in which the idea of cost aggregation is usually embodied in the form of energy terms in the optimization of an energy function, obtain a final disparity map by optimizing the energy function, are difficult to use on mobile terminal devices.
Disclosure of Invention
Accordingly, an object of the present invention is to provide an image processing method, apparatus, electronic device and computer readable medium, so as to alleviate the technical problem that the existing cost aggregation algorithm is difficult to support large-size and large-parallax-level computation on a mobile device with poor performance.
In a first aspect, an embodiment of the present invention provides an image processing method, including acquiring a main shot image to be processed and a sub shot image to be processed, determining the main shot image and the sub shot image as images to be processed respectively, calculating a cost cube of the images to be processed to obtain a target cost cube, where the target cost cube represents a cost map of pixels in the images to be processed under each preset parallax level, determining a filtering window of each pixel in the images to be processed, performing filtering processing on the cost map under each preset parallax level in the target cost cube through the filtering window, and calculating parallax maps of the main shot image and the sub shot image based on the filtered target cost cube.
Further, the target cost cube comprises a first cost cube corresponding to the main shooting image and a second cost cube corresponding to the auxiliary shooting image, the calculating of the cost cube of the image to be processed comprises the steps of carrying out three-dimensional correction operation on the image to be processed to enable the image to be processed to be transformed into a space with parallel optical axes, and the calculating of the cost cube of the image to be processed after the three-dimensional correction operation is carried out to obtain the first cost cube and the second cost cube.
Further, calculating the cost cubes of the images to be processed after the three-dimensional correction operation to obtain the first cost cubes and the second cost cubes comprises calculating the cost cubes of the images to be processed after the three-dimensional correction operation by a target cost calculation method to obtain the first cost cubes and the second cost cubes, wherein the target cost calculation method comprises any one of a SAD image matching algorithm and a Census similarity algorithm.
Further, determining the filter window of each pixel point in the image to be processed comprises determining the filter window corresponding to each pixel point based on the color position relation among the pixel points of the image to be processed.
Further, determining a filter window corresponding to each pixel point based on the color position relation among the pixel points of the image to be processed comprises the steps of performing spatial domain transformation on the image to be processed to obtain a spatial domain transformation result, determining the color position relation among each pixel point and the neighborhood pixel points of each pixel point based on the spatial domain transformation result, wherein the color position relation comprises color information among the pixel points and spatial information among the pixel points, calculating a similarity value among each pixel point and the neighborhood pixel points of each pixel point according to the color position relation, determining the window radius of the filter window according to the similarity value, and determining the filter window based on the window radius of the filter window.
Further, filtering the cost map under each preset parallax level in the target cost cube through the filtering window comprises the step of carrying out average filtering processing on the cost map under each preset parallax level in the target cost cube through the filtering window corresponding to each pixel point.
Further, calculating the disparity maps of the main shot image and the auxiliary shot image based on the target cost cube after the filtering process comprises determining a disparity value of each pixel point in the main shot image and the auxiliary shot image based on the target cost cube after the filtering process, and determining the disparity maps of the main shot image and the auxiliary shot image based on the disparity value of each pixel point.
Further, the cost map in the target cost cube comprises matching cost values of each pixel point in the image to be processed under each preset parallax level, and determining the parallax values of each pixel point in the main shooting image and the auxiliary shooting image based on the target cost cube after filtering processing comprises determining the matching cost values of each pixel point in the main shooting image and the auxiliary shooting image under each preset parallax level in the target cost cube, and determining the minimum matching cost value as the parallax value of the corresponding pixel point in the determined matching cost values.
Further, the method further comprises the step of optimizing the parallax map according to a preset optimization algorithm to obtain the optimized parallax map, wherein the preset optimization algorithm comprises at least one of parallax map consistency detection, occlusion region detection and edge protection smooth optimization.
In a second aspect, an embodiment of the present invention provides an image processing apparatus, which includes an acquisition unit configured to acquire a main shot image to be processed and a sub shot image to be processed, and determine the main shot image and the sub shot image as images to be processed, respectively, a first calculation unit configured to calculate a cost cube of the images to be processed, to obtain a target cost cube, where the target cost cube represents a cost map of a pixel point in the images to be processed under each preset parallax level, a determination unit configured to determine a filter window of each pixel point in the images to be processed, and perform a parallax filtering process on the cost map under each preset parallax level in the target cube through the filter window, and a second calculation unit configured to calculate the parallax maps of the main shot image and the sub shot image based on the filtered target cube.
In a third aspect, an embodiment of the present invention provides an electronic device, including a storage device, a processing device, and a computer program stored on the storage device and executable on the processing device, where the processing device implements the steps of the method according to any one of the first aspects when the processing device executes the computer program.
In a fourth aspect, embodiments of the present invention provide a computer readable medium having non-volatile program code executable by a processing apparatus, the program code causing the processing apparatus to perform the steps of the method of any one of the first aspects.
In the embodiment of the invention, firstly, a main shot image to be processed and a secondary shot image to be processed are acquired, the main shot image and the secondary shot image are respectively determined to be the images to be processed, then, a cost cube of the images to be processed is calculated to obtain a target cost cube, next, a filter window of each pixel point in the images to be processed is determined, filtering processing is carried out on a cost map under each preset parallax level in the target cost cube through the filter window, and the parallax maps of the main shot image and the secondary shot image are calculated based on the target cost cube after the filtering processing. As can be seen from the above description, a corresponding filtering window is determined for each pixel point in the image to be processed, and when filtering is performed through the filtering window, filtering processing can be performed on the cost map under each preset parallax level according to the filtering window, so that the cost maps under all preset parallax levels only need to calculate the filtering window once, by adopting the method, the performance of the image processing method can be improved, the possibility of using a large parallax level on some platforms with poor performance is increased, and therefore the technical problem that the existing cost aggregation algorithm is difficult to support large-size large parallax level calculation on mobile equipment with poor performance is solved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the invention;
FIG. 2 is a flow chart of a method of image processing according to an embodiment of the invention;
FIG. 3 is a schematic illustration of a primary and secondary view prior to correction in accordance with an embodiment of the present invention;
fig. 4 is a schematic diagram of an optical axis relationship between a main photographing device and a sub photographing device before correction according to an embodiment of the present invention;
FIG. 5 is a schematic illustration of a primary and secondary view after correction according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an optical axis relationship between a main photographing device and a sub photographing device after correction according to an embodiment of the present invention;
FIG. 7 is a schematic representation of the computation of a cost cube according to an embodiment of the invention;
FIG. 8 is a schematic diagram of a cost cube structure according to an embodiment of the invention;
FIG. 9 is a schematic diagram of an averaging filter process according to an embodiment of the present invention;
fig. 10 is a schematic diagram of a relationship between a preset disparity level and a matching cost value according to an embodiment of the present invention;
FIG. 11 is a flow chart of another method of image processing according to an embodiment of the invention;
Fig. 12 is a schematic diagram of an apparatus for image processing according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1:
first, an electronic device 100 for implementing an embodiment of the present invention, which can be used for running the method of image processing of the embodiments of the present invention, will be described with reference to fig. 1.
As shown in fig. 1, electronic device 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and an image collector 110, which are interconnected by a bus system 112 and/or other forms of connection mechanisms (not shown). It should be noted that the components and structures of the electronic device 100 shown in fig. 1 are exemplary only and not limiting, as the electronic device may have other components and structures as desired.
The Processing device 102 may be implemented in at least one of a digital signal Processing device (DSP, digital Signal Processing), a Field-Programmable gate array (FPGA), a Programmable logic array (PLA, programmable Logic Array), and an ASIC (Application SPECIFIC INTEGRATED Circuit), the Processing device 102 may be a central Processing unit (CPU, central Processing Unit) or other form of Processing unit having data Processing and/or instruction execution capabilities, and may control other components in the electronic apparatus 100 to perform desired functions.
The storage 104 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile and/or nonvolatile storage. The volatile storage may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile storage may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer readable storage medium that can be executed by the processing device 102 to perform client functions and/or other desired functions in embodiments of the present invention described below. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer readable storage medium.
The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, mouse, microphone, touch screen, and the like.
The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like.
The image collector 110 is configured to collect an image to be identified, where the parallax value of the pixel point in the image to be processed is obtained by the image processing method of the data collected by the camera.
Example 2:
According to an embodiment of the present invention, there is provided an embodiment of a method of image processing, it being noted that the steps shown in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.
Fig. 2 is a flowchart of a method of image processing according to an embodiment of the present invention, as shown in fig. 2, the method including the steps of:
step S202, acquiring a main shot image to be processed and a sub shot image to be processed, and determining the main shot image and the sub shot image as images to be processed, respectively, that is, the images to be processed in the present application include the main shot image to be processed and the sub shot image to be processed.
In the present application, the electronic device described in the above embodiment 1 may be a mobile terminal including a plurality of image capturing apparatuses, for example, including a main image capturing apparatus and a sub image capturing apparatus, and the main image to be processed is an image captured by the main image capturing apparatus, and the sub image to be processed is an image captured by the sub image capturing apparatus.
It should be noted that, for a mobile device, a main camera is generally included, and at least one auxiliary camera is included, and the number of main cameras and auxiliary cameras is not particularly limited in the present application.
Step S204, calculating a cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost graph of pixel points in the image to be processed under each preset parallax level, and the middle cost graph of the target cost cube under each preset parallax level contains matching cost values of each pixel point in the image to be processed under each preset parallax level.
The cost cube is mainly used for calculating the similarity (namely, the matching cost value in the application) of the pixel point of the main shooting image in (x, y) and the pixel point of the auxiliary shooting image in (x+d, y), wherein d is the number of preset parallax levels.
In the application, after the main shot image and the auxiliary shot image are acquired, cost calculation can be carried out on the main shot image and the auxiliary shot image respectively. For example, the similarity between the pixel point (x, y) in the main image and the pixel point (x, y) to the pixel point (x+d, y) in the sub-image may be calculated, respectively, to thereby obtain the target cost cube.
Step S206, determining a filtering window of each pixel point in the image to be processed, and performing filtering processing on the cost map under each preset parallax level in the target cost cube through the filtering window.
In the application, a filtering window is correspondingly determined for each pixel point (x, y) of the image to be processed, and the image to be filtered under each preset parallax level in the target cost cube (namely, the cost map under each preset parallax level) is filtered through the filtering window.
Step S208, a disparity map of the main shot image and the sub-shot image is calculated based on the target cost cube after the filtering process.
In the present application, parallax computation may be performed on a main image to be processed and a sub-image to be processed, to obtain corresponding parallax maps (i.e., left and right parallax maps) respectively, and then a final parallax map may be determined based on the left and right parallax maps.
As can be seen from the above description, a corresponding filtering window is determined for each pixel point in the image to be processed, and when filtering is performed through the filtering window, filtering processing can be performed on the cost map under each preset parallax level according to the filtering window, so that the cost maps under all preset parallax levels only need to calculate the filtering window once, by adopting the method, the performance of the image processing method can be improved, the possibility of using a large parallax level on some platforms with poor performance is increased, and therefore the technical problem that the existing cost aggregation algorithm is difficult to support large-size large parallax level calculation on mobile equipment with poor performance is solved.
In an alternative embodiment, in the present application, the target cost cube includes a first cost cube corresponding to the primary image and a second cost cube corresponding to the secondary image, based on which, in step S204, the cost cube of the image to be processed is calculated, and the process of obtaining the target cost cube includes the following steps:
First, a stereoscopic correction operation is performed on the image to be processed to transform the image to be processed into a space in which optical axes are parallel.
As is apparent from the above description, in the present application, an image to be processed is first acquired. For example, a main image to be processed and a sub-image to be processed are included. A schematic diagram of the main shot image to be processed and the auxiliary shot image to be processed is shown in fig. 3. As can be seen from fig. 4, the optical axes of the main camera and the sub-camera are not in a balanced state. At this time, it is necessary to perform a stereoscopic correction operation on the image to be processed, for example, a rotation operation may be performed on the image to be processed by a rotation matrix so that the image to be processed is transformed into a space in which optical axes are parallel. As shown in fig. 6, the optical axes of the main photographing device and the sub photographing device are not in a balanced state, and as shown in fig. 5, an effect diagram after performing a stereoscopic correction operation on the image to be processed as shown in fig. 3 is shown.
And secondly, calculating a cost cube of the image to be processed after the stereo correction operation, and obtaining the first cost cube and the second cost cube.
In the application, after the stereo correction operation is carried out on the image to be processed, the cost cube of the image to be processed after the stereo correction operation can be calculated to obtain the first generation cube and the second cost cube, wherein the calculation process of the first generation cube and the second cost cube is described as follows:
Specifically, in the application, a cost cube of an image to be processed after the three-dimensional correction operation can be calculated by a target cost calculation method to obtain a first cost cube and the second cost cube, wherein the target cost calculation method comprises any one of SAD (Sum of Absolute Differences) image matching algorithm and Census similarity algorithm. In the present application, other calculation methods may be used in addition to the above-described method, and the present application is not particularly limited thereto.
As shown in fig. 7, the left side image is a main shot image, denoted mainImg, and the right side image is a sub-shot image, denoted auxImg. As shown in fig. 7, the pixel points of the hatched portion in the main photographed image mainImg are denoted as (x, y), and the pixel points of the hatched portion in the sub-photographed image auxImg are denoted as (x, y) to (x+d, y), where d is the number of preset parallax levels.
When calculating the target cost cube, a cost cube with length W, width H and height dmax as shown in fig. 8 can be obtained by calculating the similarity between the left image pixel point (x, y) and the right image dmax pixel points (i.e., (x, y) to (x+d, y)), and in fig. 8, C (x, y, d) is the similarity (i.e., the matching cost value in the present application).
In the present application, after the target cost cube is calculated in the manner described above, a filter window for each pixel in the image to be processed may be determined.
In the application, for each pixel point in the image to be processed, the filter window corresponding to each pixel point can be determined according to the color position relation among the pixel points.
The color positional relationship may be understood as a color relationship between the current pixel point and its neighboring pixel point and a positional relationship between the current pixel point and its neighboring pixel point. If the image to be processed is an RGB image, the color relationship may be understood as a color value of each pixel, and if the image to be processed is a gray image, the color relationship may be understood as a gray value of each pixel.
In an optional embodiment, determining the filter window corresponding to each pixel based on the color positional relationship between the pixels of the image to be processed includes the following processes:
Step S1, carrying out spatial domain transformation on the image to be processed to obtain a spatial domain transformation result;
step S2, determining a color position relation between each pixel point and a neighborhood pixel point of each pixel point based on the spatial domain transformation result, wherein the color position relation comprises color information among the pixel points and spatial information among the pixel points;
Step S3, calculating a similarity value between each pixel point and a neighborhood pixel point of each pixel point according to the color position relation, and determining a window radius of the filter window according to the similarity value;
and step S4, determining the filter window based on the window radius of the filter window.
In the present application, the image to be processed may be subjected to spatial Domain transformation, thereby transforming the image to be processed into Domain transformation Domain.
Specifically, in the present application, the expression can be usedAnd performing spatial domain transformation on the image to be processed to obtain a spatial domain transformation result, wherein k represents a color channel of the image to be processed, k can be 1 (gray scale image) or 3 (RGB) or other, I' k (x) represents a gradient image of the image I to be processed in the channel k, sigma s represents a spatial distance parameter for controlling the mutual influence spatial distance between pixel points, for example, the spatial distance between the pixel points is controlled to be not influenced by each other beyond what spatial distance, sigma r represents a color distance parameter for representing the mutual influence color parameter between the pixel points, and represents how large the color difference between the pixel points is not influenced by each other, c represents a value range of the color channel k, and u belongs to data in the spatial domain.
In the application, after the spatial domain transformation is carried out on the image to be processed, the neighborhood pixel point of each pixel point can be determined in the spatial domain transformation result according to the controllable parameters. Thereby determining color information and spatial information between each pixel and the neighboring pixel.
It should be noted that, for a pixel, if the color of its neighboring pixel and its difference are too large, the influence weight of the pixel on it will be small, even 0, and if a neighboring pixel is far from it, the influence weight of the pixel on it will be small, even 0, then at this time, the neighboring pixel of each pixel can be determined by the controllable parameters.
After the color position relation is determined, a spatial similarity value and/or a color similarity value between each pixel point and the neighborhood pixel points can be calculated according to the color position relation, so that the window radius of the filter window corresponding to each pixel point is determined according to the spatial similarity value or the color similarity value. For example, the spatial similarity value or the color similarity value is directly determined to the window radius of the filter window.
It should be noted that, in the present application, the window radius may also be understood as the radius of the current pixel affected by the neighboring pixel. From the above description, it is known that if the color of a neighboring pixel point and the difference between the color and the color are too large, the influence weight of the point on the neighboring pixel point is small, even 0, and if a neighboring pixel point is far away from the neighboring pixel point, the influence weight of the point on the neighboring pixel point is small, even 0. Therefore, in the application, the field pixel point of the current pixel point can be determined through the adjustable parameter, so that the radius of the window is determined according to the phase velocity value between the current pixel point and the neighborhood pixel point.
In the application, after the filter window of each pixel point in the image to be processed is determined in the above-described manner, the cost map under each preset parallax level in the target cost cube may be subjected to mean filtering processing through the filter window corresponding to each pixel point.
Specifically, in the application, after the filtered window radius is calculated, traversing each preset parallax level, and carrying out window average filtering on the cost graph under each preset parallax level. That is, in the present application, for each cost map of each pixel point (x, y) in the image to be processed in each preset parallax level, the same filtering window is adopted to perform filtering processing, and the specific procedure is as follows:
for a cost map I (x, y) to be filtered at a certain preset parallax level, if an image is subjected to mean filtering by using a 3x 3 filtering window, the 3x 3 window can be disassembled into the structure shown in fig. 9.
Based on this, the above-mentioned mean filtering process can be described as averaging in the column direction and then in the row direction for a cost map I (x, y) to be filtered. In the present application, the window radius of the filter window calculated in the above step is the upper limit and the lower limit in the column direction in fig. 9, or the distance between the left boundary and the right boundary in the row direction and the current pixel point.
As can be seen from the above description, the cost map I (x, y) to be filtered at each preset parallax level may share a filtering boundary lookup table (i.e. filtering window) at the same time, so that the boundary may be directly looked up to implement the mean filtering of the variable window during filtering.
In the application, the complexity of variable window filtering of each cost cube slice (namely, cost graph) under d preset parallax levels is consistent with that of common average filtering, and the complexity is much smaller than that of a cost aggregation scheme based on a edge-preserving smooth filter by a local method. Therefore, the algorithm performance can be greatly improved by adopting the method, and the possibility of using large-size stereo image pairs and large parallax levels on some platforms with poor performance is increased.
In the present application, after filtering the cost map under each preset parallax level in the target cost cube in the above-described manner, the parallax maps of the main shot image and the sub shot image may be calculated based on the target cost cube after the filtering, which specifically includes the following procedures:
First, a disparity value for each pixel point in a main image and a sub-image is determined based on the target cost cube after filtering processing.
Specifically, in the present application, the matching cost value (for example, the similarity described in the above embodiment) of each pixel point in the image to be processed at each preset parallax level may be determined in the target cost cube. The matching cost value of any pixel under each preset parallax level is shown in fig. 10. At this time, the minimum matching cost value may be determined as the parallax value of the corresponding pixel point from among the determined matching cost values. For example, a minimum matching cost value Winner d is selected from a plurality of matching cost values shown in fig. 10 as the parallax value of the current pixel point in the image to be processed.
Then, a disparity map of the main image and the sub-image is determined based on the disparity value of each pixel.
In the application, after the parallax map of the image to be processed is obtained, the parallax map is usually provided with a shielding area and a noise area to be optimized, and the final parallax map is obtained by carrying out consistency detection, shielding area interpolation and edge protection smooth optimization on the left and right parallax maps, specifically, the parallax map can be optimized according to a preset optimization algorithm to obtain the optimized parallax map, wherein the preset optimization algorithm comprises at least one of parallax map consistency detection, shielding area detection and edge protection smooth optimization.
As can be seen from the above description, in the present application, an image processing manner is provided, and the method is a stereo matching cost aggregation performance optimization scheme with edge-preserving smoothing characteristics, and for all preset parallax levels d, only one filtering window radius needs to be calculated for a fixed image size, and then only a local cost aggregation method based on integral graph mean filtering is required to be completed. The whole cost aggregation process is equivalent to mean filtering of the images under d preset parallax levels. The window radius of the filter window is calculated according to the color position relation among the color image pixel points, so that the filter window has the edge protection smooth characteristic. Meanwhile, the method for carrying out average filtering on d preset parallax levels by calculating the filter radius once can greatly improve the algorithm performance, and the possibility of using large-size stereogram pairs and large parallax levels on some platforms with poor performance is increased.
Fig. 11 is a flowchart of another method of image processing according to an embodiment of the present invention, as shown in fig. 11, including the following processes:
And inputting a main shooting RGB image (hereinafter referred to as left view) and a secondary shooting RGB image (hereinafter referred to as right view), and calculating cost cubes of the left view and the right view after three-dimensional correction operation to obtain a left cost cube and a right cost cube respectively. Then respectively calculating the window radius of the filter window under the left view and the right view, wherein the main idea is to determine the window radius of each pixel point based on the space and color information among RGB image pixel points and adjustable parameters, then traverse each parallax level for the left cost cube and the right cost cube, and perform average value filtering based on the window radius of the filter window calculated in the last step. Finally, a parallax map can be calculated by using a method such as WTA, and then the parallax map is optimized by using a common post-processing method, so that the whole stereo matching process can be completed.
According to the method provided by the application, under the condition of ensuring the edge-protecting smooth characteristic, the performance can be optimized to be close to the cost aggregation performance based on the mean value filtering. Although the cost aggregation method is fast, the cost aggregation method does not have the edge-protecting smooth characteristic, and the quality of the cost cubes in the parallax discontinuous areas can be lost. Such as a guided filter-based cost aggregation method, a guard smoothing filter needs to be performed once for each parallax level, including pixel weight calculation and weighted average in a window, and although the guard smoothing characteristic is provided, the performance is still slower, and it is difficult to support large-size large parallax level calculation on a mobile device with poor performance. In the application, a corresponding filtering window is determined for each pixel point in the image to be processed, when filtering is carried out through the filtering window, filtering processing can be carried out on the cost map under each preset parallax level according to the filtering window, therefore, the cost map under all preset parallax levels only needs to be calculated once, the performance of the image processing method can be improved by adopting the method, the possibility of using large parallax levels on some platforms with poor performance is increased, and the technical problem that the existing cost aggregation algorithm is difficult to support large-size large parallax level calculation on mobile equipment with poor performance is solved.
Example 3:
The embodiment of the invention also provides an image processing device, which is mainly used for executing the image processing method provided by the embodiment of the invention, and the image processing device provided by the embodiment of the invention is specifically described below.
Fig. 12 is a schematic view of an apparatus for image processing according to an embodiment of the present invention, and as shown in fig. 12, the apparatus for image processing mainly includes:
An acquisition unit 10 for a main shot image to be processed and a sub shot image to be processed, and determining the main shot image and the sub shot image as images to be processed, respectively;
the first calculating unit 20 is configured to calculate a cost cube of the image to be processed, so as to obtain a target cost cube, where the target cost cube represents a cost map of pixel points in the image to be processed under each preset parallax level;
A determining unit 30, configured to determine a filter window of each pixel point in the image to be processed;
A filtering unit 40, configured to perform filtering processing on the cost map under each preset parallax level in the target cost cube through the filtering window;
a second calculation unit 50 for calculating a disparity map of the main shot image and the sub-shot image based on the target cost cube after the filtering process.
In the application, a corresponding filtering window is determined for each pixel point in the image to be processed, when filtering is carried out through the filtering window, filtering processing can be carried out on the cost map under each preset parallax level according to the filtering window, therefore, the cost map under all preset parallax levels only needs to be calculated once, the performance of the image processing method can be improved by adopting the method, the possibility of using large parallax levels on some platforms with poor performance is increased, and the technical problem that the existing cost aggregation algorithm is difficult to support large-size large parallax level calculation on mobile equipment with poor performance is solved.
Optionally, the target cost cube comprises a first cost cube corresponding to a main shooting image and a second cost cube corresponding to a secondary shooting image, a first calculating unit is used for carrying out three-dimensional correction operation on the image to be processed so as to transform the image to be processed into a space with parallel optical axes, and calculating the cost cubes of the image to be processed after the three-dimensional correction operation to obtain the first cost cube and the second cost cube.
Optionally, the first calculating unit is further configured to calculate a cost cube of the image to be processed after the stereo correction operation by using a target cost calculating method, so as to obtain the first cost cube and the second cost cube, where the target cost calculating method includes any one of a SAD image matching algorithm and a Census similarity algorithm.
Optionally, the determining unit is configured to determine a filter window corresponding to each pixel point based on a color position relationship between the pixel points of the image to be processed.
Optionally, the determining unit is further configured to perform spatial domain transformation on the image to be processed to obtain a spatial domain transformation result, determine a color position relationship between each pixel point and a neighborhood pixel point of each pixel point based on the spatial domain transformation result, where the color position relationship includes color information between pixel points and spatial information between pixel points, calculate a similarity value between each pixel point and the neighborhood pixel point of each pixel point according to the color position relationship, and determine a window radius of the filter window according to the similarity value, and determine the filter window based on the window radius of the filter window.
Optionally, the filtering unit is configured to perform average filtering processing on the cost map under each preset parallax level in the target cost cube through a filtering window corresponding to each pixel point.
Optionally, the second computing unit is used for determining a parallax value of each pixel point in the main shooting image and the auxiliary shooting image based on the target cost cube after filtering processing, and determining parallax graphs of the main shooting image and the auxiliary shooting image based on the parallax value of each pixel point.
Optionally, the cost map in the target cost cube comprises matching cost values of each pixel point in the image to be processed under each preset parallax level, and the second calculation unit is further configured to determine, in the target cost cube, the matching cost value of each pixel point in the main image and the auxiliary image under each preset parallax level, and determine, in the determined matching cost values, that the minimum matching cost value is the parallax value of the corresponding pixel point.
Optionally, the device is further used for optimizing the parallax map according to a preset optimization algorithm to obtain the optimized parallax map, wherein the preset optimization algorithm comprises at least one of parallax map consistency detection, occlusion region detection and edge protection smooth optimization.
It should be noted that the foregoing embodiments are merely illustrative embodiments of the present invention, and not restrictive, and the scope of the invention is not limited to the embodiments, and although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any modification, variation or substitution of some of the technical features of the embodiments described in the foregoing embodiments may be easily contemplated within the scope of the present invention, and the spirit and scope of the technical solutions of the embodiments do not depart from the spirit and scope of the embodiments of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (11)
1. A method of image processing, comprising:
Acquiring a main shot image to be processed and a secondary shot image to be processed, and respectively determining the main shot image and the secondary shot image as images to be processed;
Calculating a cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of pixel points in the image to be processed under each preset parallax level;
Determining a filter window of each pixel point in the image to be processed based on the color position relation among the pixel points of the image to be processed, wherein the color position relation comprises color information among the pixel points and space information among the pixel points;
Filtering the cost graph under each preset parallax level in the target cost cube through the filtering window;
And calculating the parallax map of the main shooting image and the auxiliary shooting image based on the target cost cube after filtering processing.
2. The method of claim 1, wherein the target cost cube comprises a first cost cube corresponding to the primary image and a second cost cube corresponding to the secondary image;
calculating a cost cube of the image to be processed, wherein obtaining a target cost cube comprises:
Performing three-dimensional correction operation on the image to be processed so as to transform the image to be processed into a space with parallel optical axes;
And calculating a cost cube of the image to be processed after the stereo correction operation, and obtaining the first cost cube and the second cost cube.
3. The method of claim 2, wherein computing a cost cube for the image to be processed after the stereo correction operation, the first cost cube and the second cost cube comprising:
calculating a cost cube of the image to be processed after the three-dimensional correction operation by a target cost calculation method to obtain the first cost cube and the second cost cube, wherein the target cost calculation method comprises any one of a SAD image matching algorithm and a Census similarity algorithm.
4. The method of claim 1, wherein determining a filter window for each pixel based on a color positional relationship between pixels of the image to be processed comprises:
performing spatial domain transformation on the image to be processed to obtain a spatial domain transformation result;
Determining the color position relation between each pixel point and the neighborhood pixel point of each pixel point based on the spatial domain transformation result;
calculating a similarity value between each pixel point and a neighborhood pixel point of each pixel point according to the color position relation, and determining a window radius of the filter window according to the similarity value;
The filter window is determined based on a window radius of the filter window.
5. The method according to claim 1, wherein filtering the cost map at each preset disparity level in the target cost cube through the filtering window comprises:
And carrying out mean value filtering processing on the cost map under each preset parallax level in the target cost cube through a filtering window corresponding to each pixel point.
6. The method according to any one of claims 1 to 5, wherein calculating a disparity map of the main shot image and the sub-shot image based on the target cost cube after filtering processing includes:
determining a disparity value of each pixel point in the main shooting image and the auxiliary shooting image based on the target cost cube after filtering;
and determining the parallax map of the main shooting image and the auxiliary shooting image based on the parallax value of each pixel point.
7. The method of claim 6, wherein the cost map in the target cost cube contains a matching cost value for each pixel point in the image to be processed at each preset disparity level;
Determining the parallax value of each pixel point in the main shooting image and the auxiliary shooting image based on the target cost cube after filtering processing comprises:
Determining the matching cost value of each pixel point in the main shooting image and the auxiliary shooting image under each preset parallax level in the target cost cube;
And determining the minimum matching cost value as the parallax value of the corresponding pixel point in the determined matching cost values.
8. The method according to any one of claims 1 to 5, further comprising:
Optimizing the parallax map according to a preset optimization algorithm to obtain the optimized parallax map, wherein the preset optimization algorithm comprises at least one of parallax map consistency detection, occlusion region detection and edge protection smooth optimization.
9. An apparatus for image processing, comprising:
An acquisition unit configured to determine a main image to be processed and a sub-image to be processed as images to be processed, respectively;
The first computing unit is used for computing a cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost graph of pixel points in the image to be processed under each preset parallax level;
The determining unit is used for determining a filter window of each pixel point in the image to be processed based on the color position relation among the pixel points of the image to be processed, wherein the color position relation comprises color information among the pixel points and space information among the pixel points;
The filtering unit is used for carrying out filtering processing on the cost graph under each preset parallax level in the target cost cube through the filtering window;
and a second calculation unit configured to calculate a disparity map of the main shot image and the sub-shot image based on the target cost cube after the filtering process.
10. An electronic device comprising storage means, processing means and a computer program stored on said storage means and executable on said processing means, characterized in that said processing means implement the steps of the method according to any of the preceding claims 1 to 8 when said computer program is executed by said processing means.
11. A computer readable medium having non-volatile program code executable by a processing device, the program code causing the processing device to perform the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010867321.7A CN112150532B (en) | 2020-08-25 | 2020-08-25 | Image processing method, device, electronic device and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010867321.7A CN112150532B (en) | 2020-08-25 | 2020-08-25 | Image processing method, device, electronic device and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112150532A CN112150532A (en) | 2020-12-29 |
| CN112150532B true CN112150532B (en) | 2025-03-07 |
Family
ID=73887580
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010867321.7A Active CN112150532B (en) | 2020-08-25 | 2020-08-25 | Image processing method, device, electronic device and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112150532B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862705B (en) * | 2021-01-23 | 2023-08-25 | 西安点云生物科技有限公司 | Device, equipment, method and storage medium for optimizing edge antialiasing of photo-cured slice image |
| CN114596216A (en) * | 2021-12-22 | 2022-06-07 | 上海顺久电子科技有限公司 | Image filtering method and related device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105528785A (en) * | 2015-12-03 | 2016-04-27 | 河北工业大学 | Binocular visual image stereo matching method |
| CN105761270A (en) * | 2016-03-15 | 2016-07-13 | 杭州电子科技大学 | Tree type filtering three-dimensional coupling method based on epipolar line linear distance transformation |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106815594A (en) * | 2015-11-30 | 2017-06-09 | 展讯通信（上海）有限公司 | Solid matching method and device |
| CN106709862B (en) * | 2016-12-13 | 2019-08-20 | 北京航空航天大学 | An image processing method and device |
| US11272163B2 (en) * | 2017-02-07 | 2022-03-08 | Sony Corporation | Image processing apparatus and image processing method |
| CN107274448B (en) * | 2017-07-11 | 2023-05-23 | 江南大学 | Variable weight cost aggregation stereo matching algorithm based on horizontal tree structure |
| CN107507232B (en) * | 2017-07-14 | 2020-06-16 | 天津大学 | Stereo matching method based on multi-scale iteration |
| CN108257169B (en) * | 2018-01-04 | 2021-11-23 | 北京大学深圳研究生院 | Binocular image stereo matching method and system and filtering method and system thereof |
| CN108665499B (en) * | 2018-05-04 | 2021-08-10 | 北京航空航天大学 | Near distance airplane pose measuring method based on parallax method |
| CN109410234A (en) * | 2018-10-12 | 2019-03-01 | 南京理工大学 | A kind of control method and control system based on binocular vision avoidance |
| CN109493373B (en) * | 2018-11-07 | 2020-11-10 | 上海为森车载传感技术有限公司 | Stereo matching method based on binocular stereo vision |
| KR102090666B1 (en) * | 2019-02-11 | 2020-03-18 | 주식회사 넥스트칩 | Method and apparatus for generating a disparity map for stereo images |
-
2020
- 2020-08-25 CN CN202010867321.7A patent/CN112150532B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105528785A (en) * | 2015-12-03 | 2016-04-27 | 河北工业大学 | Binocular visual image stereo matching method |
| CN105761270A (en) * | 2016-03-15 | 2016-07-13 | 杭州电子科技大学 | Tree type filtering three-dimensional coupling method based on epipolar line linear distance transformation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112150532A (en) | 2020-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Hyun Kim et al. | Dynamic scene deblurring | |
| EP2811457B1 (en) | Image processing method and apparatus | |
| US9338437B2 (en) | Apparatus and method for reconstructing high density three-dimensional image | |
| CN111415310B (en) | Image processing method and device and storage medium | |
| WO2019221013A2 (en) | Video stabilization method and apparatus and non-transitory computer-readable medium | |
| CN103473743B (en) | A kind of method obtaining image depth information | |
| CN107025660B (en) | A method and device for determining image parallax of binocular dynamic vision sensor | |
| KR20150037369A (en) | Method for decreasing noise of image and image processing apparatus using thereof | |
| Xiao et al. | Multi-focus image fusion based on depth extraction with inhomogeneous diffusion equation | |
| CN113192101A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| US10582179B2 (en) | Method and apparatus for processing binocular disparity image | |
| KR20140118031A (en) | Image processing apparatus and method thereof | |
| JP2018195084A (en) | Image processing apparatus, image processing method, program, and storage medium | |
| CN112150532B (en) | Image processing method, device, electronic device and computer readable medium | |
| WO2015128542A2 (en) | Processing stereo images | |
| CN118485702B (en) | High-precision binocular vision ranging method | |
| CN111105452A (en) | High-low resolution fusion stereo matching method based on binocular vision | |
| CN115578286A (en) | High dynamic range hybrid exposure imaging method and apparatus | |
| CN117173232A (en) | Depth image acquisition method, device and equipment | |
| CN114998358A (en) | Multi-focus image fusion method and device, computer equipment and storage medium | |
| WO2020113419A9 (en) | Image processing method and device | |
| CN104504667B (en) | image processing method and device | |
| CN116823863A (en) | An infrared image contour extraction method and device | |
| CN117058183A (en) | Image processing method and device based on double cameras, electronic equipment and storage medium | |
| JP2019191661A (en) | Image processing apparatus, imaging device, image processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20250206 Address after: No. 257, 2nd Floor, Building 9, No. 2 Huizhu Road, Liangjiang New District, Yubei District, Chongqing, China 401123 Applicant after: Force Map New (Chongqing) Technology Co.,Ltd. Country or region after: China Address before: 316-318, block a, Rongke Information Center, No.2, south academy of Sciences Road, Haidian District, Beijing Applicant before: MEGVII (BEIJING) TECHNOLOGY Co.,Ltd. Country or region before: China |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Methods, devices, electronic devices, and computer-readable media for image processing Granted publication date: 20250307 Pledgee: Chongqing Industrial Investment Master Fund Partnership Enterprise (Limited Partnership) Pledgor: Force Map New (Chongqing) Technology Co.,Ltd. Registration number: Y2025110000093 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Granted publication date: 20250307 Pledgee: Chongqing Industrial Investment Master Fund Partnership Enterprise (Limited Partnership) Pledgor: Force Map New (Chongqing) Technology Co.,Ltd. Registration number: Y2025110000093 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right |