WO2019071976A1 - Panoramic image saliency detection method based on regional growth and eye movement model - Google Patents

Abstract

A panoramic image saliency detection method based on regional growth and an eye movement model. The regional growth and a fixed prediction model are used to realize detection of an automatic salient object of a panoramic image. The method comprises: detecting an original image based on regional growth, and roughly extracting an area with a significant density difference compared with a neighbor area by means of a regional growth algorithm to obtain an area with a large density difference; obtaining a saliency value of a salient region by means of eye movement fixed point prediction; carrying out maximum value normalization and then carrying out summing; and enhancing the salient region more uniformly by using a method for optimizing a geodesic line. According to the method, the problems that an existing method has insufficient saliency detection precision and robustness and is not suitable for the panoramic image can be solved, thus the salient region in the panoramic image is more accurately displayed, and accurate and useful information is provided for the later application of target recognition, classification and the like.

Description

Panoramic image saliency detection method based on regional growth and eye movement model Technical field

The invention relates to the field of image processing, computer vision and robot vision technology, and in particular to a method for performing saliency detection of a panoramic image by using a region growing algorithm and an eye movement model.

Background technique

The inherent and powerful ability of the human eye is to quickly capture the most prominent areas of the scene and pass them to the advanced visual cortex. Attentional attention reduces the complexity of visual analysis, making the human visual system quite efficient in complex scenes. As a preprocessor, many applications benefit from saliency analysis, such as detecting anomalous patterns, splitting primitives, generating object proposals, and more. The concept of salience has not only been studied in early visual modeling, but also in fields such as image compression, object recognition and tracking, robot navigation, advertising, etc.

Early work on computational saliency was designed to simulate and predict people's gaze on images. Recently the field has expanded to include subdivisions of entire highlighted areas or objects.

Most of the work extracts a prominent area with significant features compared to the surrounding area based on the concept of center-surrounded contrast. In addition, additional prior knowledge of the spatial layout of foreground objects and backgrounds can be used: there is a high probability of belonging to the background, while foreground highlighted objects are usually located near the center of the image. These assumptions have been successfully employed to improve the performance of saliency detection of conventional images with conventional aspect ratios. Recently, panoramic images that produce a wide field of view have become popular in various media contents, and have attracted widespread attention in many practical applications. For example, when used in a wearable device such as a head mounted display, the virtual reality content exhibits a wide field of view. A look-around monitoring system for an autonomous vehicle uses a panoramic image by combining a plurality of images taken at different viewing positions. These panoramic images can be obtained directly by using a special device, or can be generated by combining several conventional images having small aspect ratios using image stitching techniques. However, the assumptions for detecting the saliency of a conventional image do not fully reflect the characteristics of the panoramic image. Therefore, it is difficult to implement efficient panoramic image processing in the prior art, and the accuracy and robustness of the saliency detection method of the existing panoramic image need to be improved.

Summary of the invention

In order to overcome the deficiencies of the prior art, the present invention provides a method for detecting the saliency of a panoramic image by using a region growing algorithm and an eye movement model, which can solve the saliency detection accuracy and robustness of the existing method, and is not applicable. The problem of panoramic images makes the saliency area in the panoramic image more accurate, providing accurate and useful information for later target recognition and classification applications.

The principle of the invention is that the panoramic image has different characteristics compared to conventional images. First, the width of the panoramic image is much larger than the height, so the background is distributed over the horizontally elongated area. Second, the background of a panoramic image is usually composed of several homogeneous regions, such as the sky, mountains, and ground. Moreover, a typical panoramic image may include multiple foreground objects having different features and sizes that are arbitrarily distributed throughout the image. For these features, it is difficult to design a global method of directly extracting a plurality of salient regions from the input panoramic image. The present invention finds that the spatial density mode is useful for images with high resolution. Therefore, the present invention firstly uses a spatial density pattern detection method based on a region-grown panoramic image to roughly extract a preliminary object. Embedding an eye-fixed model into the frame to predict visual attention is a method consistent with the human visual system. Then, the previously obtained saliency information is fused by maximum normalization to obtain a rough saliency map. Finally, geodesic optimization techniques are used to obtain the final saliency map.

The technical solution provided by the invention is:

A panoramic image saliency detection method based on regional growth and eye movement model uses a region growing and eye movement fixed point prediction model (referred to as an eye movement model) to realize automatic protruding object detection of a panoramic image; the following steps are included:

1) performing region-based growth detection on the original image, and roughly extracting regions having significantly different densities compared to their neighbors by a region growing algorithm;

Among them, the areas of significant differences can be divided into three categories: 1) areas with excessive density, 2) areas with insufficient density, and 3) areas surrounded by ridges or ditches. Specifically, the following processes are included:

11) At the beginning, the original image is segmented into M*N small regions and converted into a density matrix, where each unit (i, j) represents the count of objects in the (i, j)th small region; the original image passes The processing of the density matrix yields an intensity image.

12) Based on the density matrix as the intensity image processing, the image processing method is applied to image enhancement of the intensity image, and then the region growing algorithm is applied to extract significantly different regions, and the exact shape of the distinct regions can be returned, and only the precise shape is output. Rough rectangular bounding box;

For the sake of simplicity, the original color image can be converted to a grayscale image, and then the rough rectangular bounding box of the precise image extracted by the object proposal algorithm described above is applied to the grayscale image, and the resulting image can be regarded as a density map. The process of extracting significantly different regions based on the region growing algorithm performs the following processing:

(a) Increased density image: Apply morphological operations, including morphological expansion, erosion, openness, and close proximity to eliminate noise such as very small areas and connect separate homogeneous regions that are close together.

(b) Exclude different background areas: Subsequent steps use optimization methods such as average intensity values and total area of the extraction area to exclude undesirable results.

(c) Seed selection: In the implementation process, automatic seed selection and iteration provide thresholds.

(d) Threshold selection: adaptive threshold processing is selected.

2) Eye movement fixed point prediction, obtain the salient value of the highlighted area; including the following steps:

21) Use eye fixation models (eye movement models, fixed prediction models) to analyze which areas are more attractive to people's attention and get significant areas;

22) Using a fixed prediction model in the frequency domain to quickly scan the image and roughly locate the places that attract people's attention;

表示X的DCT变换；签名模型被定义为IS(X)： 23) Using the signature model, the spatial support of the foreground is substantially isolated by taking the sign of the mixed signal X in the transform domain, and then converted back to the spatial domain, ie, by reconstructing the reconstructed image

A DCT transform representing X; the signature model is defined as IS(X):

IS(X)=sign(DCT(X)) (Formula 1)

A saliency map is formed by smoothing the squared reconstructed image defined above, expressed as Equation 2:

Where g is the Gaussian kernel.

24) combining the extracted protruding area with the saliency image S _m generated by the image signature, and assigning the saliency value of the extracted protruding area by averaging the saliency of all the pixels therein;

The resulting significance value is expressed as S _p , and for the region p initially identified as significant, the significance value is defined as Equation 3:

Where A(p) represents the number of pixels in the p-th region.

3) normalization of maximum value;

The present invention utilizes map statistics to determine the importance of each path (steps 1), 2)); in the final integration phase, combining the results of the two paths, they are summed (MN) after Maxima normalization.

The Maxima normalization operator N _max (·) was originally proposed for integration from multiple feature channels (L. Itti, C. Koch and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," In IEEE Profile on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, Nov 1998.).

4) Optimize geodesic technology, the specific steps are as follows:

We find that the weight of the significance value may be sensitive to the geodesic distance. The present invention employs a solution that can more evenly enhance the area of the protruding object. First, the input image is segmented into a plurality of superpixels according to a linear spectral clustering method, and the posterior probability of each superpixel is calculated by averaging the posterior probability values Sp of all the pixels therein. For the jth superpixel, if its posterior probability is marked as S(j), the significance value of the qth superpixel is improved by the geodesic distance as in Equation 4:

Where J is the total number of _superpixels ; _wqj is the weighting value of the geodesic distance between the qth _superpixel and the jth _superpixel .

First, there is already an undirected weight map connecting all adjacent superpixels (a _k , a _k+1 ), and the weights of the undirected graphs d _c (a _k , a _k+1 ) are assigned to their significance. The Euclidean distance between the values; then, the geodetic distance between the two, the superpixel d _g (p, i), can be defined as the weight of the shortest path on the cumulative edge graph, expressed as Equation 5:

Then define the weight δ _pi as Equation 6:

In Equation 6, δ _pi is the weight value of the geodesic distance between the pth superpixel and the i th superpixel; σ _c is the deviation of d _c ; d _g (p, j) is between pixels p and j Geodesic distance.

After the above steps, the saliency of the panoramic image is detected.

Compared with the prior art, the beneficial effects of the present invention are:

The invention provides a method for detecting the saliency of a panoramic image by using a region growing algorithm and an eye movement model. Firstly, a spatial density pattern detecting method based on a region-grown panoramic image is used to roughly extract a preliminary object. The eye-fixed model is embedded in the frame to predict visual attention; and the previously obtained saliency information is merged by maximum normalization to obtain a rough saliency map. Finally, geodesic optimization techniques are used to obtain the final saliency map. The invention can solve the problem that the saliency detection precision and the robustness of the prior method are insufficient, and is not suitable for the panoramic picture, so that the saliency area in the panoramic image is more accurately displayed, and provides for the later target recognition and classification applications. Precise and useful information.

Compared with the prior art, the technical advantages of the present invention are embodied in the following aspects:

1) A saliency detection model for panoramic images based on combined region growth and eye fixation models was first proposed.

2) The spatial density pattern detection algorithm for regional growth is introduced into the field of significant detection for the first time.

3) A new high-quality panoramic data set (SalPan) is constructed, which has a novel ground truth annotation method, which can eliminate the ambiguity of significant objects.

4) The model proposed by the present invention is also applicable to the saliency detection of a conventional image.

5) The method of the present invention can also help to find the perceptual characteristics of the human visual system for large-scale visual content in a wide field of view.

DRAWINGS

FIG. 1 is a flow chart of a detection method provided by the present invention.

2 is an input panoramic image, another method detection image, a detection image of the present invention, and an artificially calibrated image to be obtained according to an embodiment of the present invention;

Wherein, the first behavior input image; the second to sixth behaviors are the detection result images obtained by other methods; the seventh behavior is the detection result image of the present invention; and the eighth behavior is to manually calibrate the desired image.

3 is a diagram showing the effect of saliency detection applied to a conventional image according to the present invention;

Wherein, the first behavior inputs a regular image, the second behavior is a detection result image of the present invention, and the third behavior manually calibrates the desired image.

Detailed ways

The invention is further described by the following examples in conjunction with the accompanying drawings, but not by way of limitation.

The invention provides a method for detecting the saliency of a panoramic image by using a region growing algorithm and an eye movement model. Firstly, a spatial density pattern detecting method based on a region-grown panoramic image is used to roughly extract a preliminary object. The eye-fixed model is embedded in the frame to predict visual attention; and the previously obtained saliency information is merged by maximum normalization to obtain a rough saliency map. Finally, using the geodesic optimization technique to obtain the final saliency map, the experimental comparison is shown in Figure 2.

FIG. 1 is a block diagram of a saliency detection method provided by the present invention, which includes four main steps. First, we use the region growing algorithm to automate the selection of significant object regions. Second, use the fixed-eye prediction model to estimate significant points. The previous significance information is then fused using the maximum normalization method. Finally, the final significance test results are obtained by geodesic optimization techniques. The detailed process is described as follows:

Step 1. Detection based on regional growth.

In this step, our goal is to roughly extract areas of significantly different density compared to their neighbors. We believe that areas of significant difference can be divided into three categories: 1) overdensity, 2) insufficient density, and 3) areas surrounded by ridges or ditches. Initially, the original image is segmented into M*N regions and converted into a density matrix, where each unit (i, j) represents the count of objects within the (i, j)th cell. Based on the density matrix as intensity image processing, image processing techniques such as image morphological operators and enhancement techniques are applied, and then region growing based algorithms are applied to extract significantly different regions. Compared to other techniques that only output a rough rectangular bounding box, the algorithm can return the exact shape of a distinctly different area. For the sake of simplicity, we convert the original color image into a grayscale image and then apply the object proposal algorithm to the grayscale image. Therefore, the resulting image can be regarded as a density map. Some of the issues involved in regional growth are as follows: (a) Increased density images. We apply morphological operations, including morphological expansion, erosion, openness, and close proximity to eliminate noise like very small areas and connect separate homogeneous regions that are close together. (b) Exclude different background areas. Some hints are used for post-processing steps such as average intensity values and total area of the extraction area to rule out poor results. (c) Seed selection. In the implementation process, automatic seed selection and iteration provide thresholds. Automatic selection seems to have achieved good results and was therefore adopted as a seed selection method in the proposed method. (d) Threshold. Use adaptive thresholding. The experimental results show that the algorithm based on region growth works well in detecting important areas with effective computing power. By estimating the density matrix, we can propose some significant regions that can be enhanced or re-estimated in the next step.

Step 2: Eye movement fixed point prediction.

表示X的DCT变换。图像签名IS(X)定义为式1： Whether a location is significant depends largely on its ability to attract attention. A large amount of recent work on eye fixation predictions has more or less revealed the nature of this problem. The fixed-eye prediction model simulates the mechanisms of the human visual system so that it can predict the probability that a location will attract attention. So in this step, we use eye-fixed models to help us ensure which areas are more attractive. Panoramic images typically have a wide field of view and are therefore computationally more expensive than conventional images. Based on the color contrast algorithm, local information is not suitable as a pre-processing step for panoramic images because these algorithms are time consuming and costly computationally intensive. Therefore, the present invention employs a more efficient method to help us scan images quickly and roughly locate where attention is drawn. The fixed prediction model in the frequency domain is computationally efficient and easy to implement. Therefore, the present invention uses the eye movement prediction model in the frequency domain as the signature model. The signature model roughly isolates the spatial support of the foreground by taking the sign of the mixed signal X in the transform domain, and then converts it back into the spatial domain, ie by reconstructing the reconstructed image

Indicates the DCT transform of X. The image signature IS(X) is defined as Equation 1:

IS(X)=sign(DCT(X)) (Formula 1)

Where sign() is a symbol function and DCT() is a DCT transform function.

The saliency map is formed by smoothing the squared reconstructed image defined above, expressed as Equation 2.

Where g is the Gaussian kernel.

Image Signature is a simple and powerful natural scene descriptor that can be used to approximate the spatial location of a sparse foreground hidden in a spectrally sparse background. Compared to other fixed-eye models, image signatures are more efficient and run faster than all other methods. In order to combine the salient regions proposed in the previous step with the salient images S _m generated by the image signature, we assign the saliency values of the proposed salient regions by averaging the saliency of all the pixels therein. For convenience, we represent the resulting saliency map as S _p . That is, for a region p that is initially marked as significant, its significance value is defined as Equation 3:

Where A(p) represents the number of pixels in the p-th region.

Step 3: The maximum value is normalized.

The saliency detection results of merging multiple models are considered to be a challenging task because candidate models are usually developed based on different cues or assumptions. Fortunately, in our case, the integration problem is easier because we only consider the output of the two paths. Since there is no prior knowledge or other top-down guidance, it is safer to use map statistics to determine the importance of each path. In the final integration phase, we combine the results of the two paths and summon them (MN) after Maxima normalization. The Maxima normalization operator N _max (·) was originally proposed for integration from multiple feature channels (L. Itti, C. Koch and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," In IEEE Profile on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, Nov 1998.).

Step 4: Geodesic technology optimization.

The final step in the proposed method is to use the geodesic distance to optimize the final result. First, the input image is segmented into a plurality of superpixels according to a linear spectral clustering method, and the posterior probability of each superpixel is calculated by averaging the posterior probability values Sp of all the pixels therein. For the jth superpixel, if its posterior probability is marked as S(j), the significant value of the qth superpixel is improved by the geodesic distance as in Equation 4:

Where J is the total number of _superpixels , and _wqj will be based on the weight of the geodesic distance between the qth superpixel and the jth superpixel. First, there is already an undirected weight map connecting all adjacent superpixels (a _k , a _k+1 ) and assigning their weight d _c (a _k , a _k+1 ) between their significance values. Euclidean distance.

Then, the geodetic distance superpixel d _g (p, i) between the two can be defined as the weight of the shortest path on the cumulative edge graph, expressed as Equation 5:

In this way, superpixels in the geodetic distance image between any two can be obtained.

Then define the weight δ _pi as Equation 6:

In Equation 6, δ _pi is the weight value of the geodesic distance between the pth superpixel and the i th superpixel; σ _c is the deviation of d _c ; d _g (p, j) is between pixels p and j Geodesic distance.

Through the above steps, we can get the final saliency test results, the experimental comparison chart is shown in Figure 2. At the same time, the method of the invention is also applicable to pictures of conventional size, and the experimental effect diagram is shown in FIG.

It is to be noted that the embodiments are disclosed to facilitate a further understanding of the invention, but those skilled in the art can understand that various alternatives and modifications are possible without departing from the spirit and scope of the invention and the appended claims. of. Therefore, the invention should not be limited by the scope of the invention, and the scope of the invention is defined by the scope of the claims.

Claims (5)

A method for detecting panoramic image saliency based on regional growth and eye movement model, using region growing and fixed prediction models to realize automatic protruding object detection of panoramic images; the following steps are included: 1) Performing region-based growth detection on the original image, and roughly extracting regions having significantly different densities compared with their neighbors by the region growing algorithm, and obtaining regions with significant density differences, that is, protruding regions; 2) Obtaining the salient value of the highlighted area by the fixed point prediction of the eye movement; the following steps are included: 21) using an eye-fixed model for analysis to obtain a significant region; 22) Using a fixed prediction model in the frequency domain to quickly scan the image and roughly locate the places that attract people's attention; 23) Using a signature model to reconstruct images by calculation

Indicates the DCT transform of X; X is the mixed signal in the transform domain; the signature model is defined as IS(X), expressed as Equation 1: IS(X)=sign(DCT(X)) (Formula 1) A saliency image S _{m is} formed by smooth reconstruction of the image, expressed as Equation 2:

Where g is the Gaussian kernel; 24) combining the extracted protruding area with the saliency image S _m generated by the image signature, and averaging the saliency of all the pixels therein, and assigning the saliency value of the extracted protruding area; 3) summing after the maximum value is normalized; 4) Using the optimized geodesic method to make the protruding area more uniform, the specific steps are as follows: First, the input image is segmented into a plurality of superpixels according to a linear spectral clustering method, and the posterior probability of each superpixel is calculated by averaging the posterior probability values Sp of all the pixels; for the jth superpixel, if The posterior probability is marked as S(j), and the significant value of the qth superpixel is optimized by the geodesic distance as shown in Equation 4:

Where J is the total number of _superpixels ; _wqj is the weighting value of the geodesic distance between the qth _superpixel and the jth _superpixel ; There is an undirected weight map connecting all adjacent superpixels (a _k , a _k+1 ), and the weight of the undirected graph d _c (a _k , a _k+1 ) is assigned between its significance values. The Euclidean distance; the geodetic distance superpixel d _g (p, i) between the two is defined as the weight of the shortest path on the cumulative edge graph, expressed as Equation 5:

Then, the weight δ _{pi is} defined as Equation 6:

In Equation 6, δ _pi is the weight value of the geodesic distance between the pth superpixel and the i th superpixel; σ _c is the deviation of d _c ; d _g (p, j) is between pixels p and j Geodesic distance After the above steps, the saliency of the panoramic image is detected. The panoramic image saliency detecting method according to claim 1, wherein in step 1), the region of significant difference in density comprises: an area of excessive density, an area of insufficient density, an area surrounded by a ridge or a ditch; and the extraction process includes the following step: 11) At the beginning, the original image is segmented into M*N small regions and converted into a density matrix, where each unit (i, j) represents the count of objects in the (i, j) small regions; Through the processing of the density matrix, an intensity image is obtained; 12) Based on the density matrix, the image processing method is applied to enhance the image, and then the region growing algorithm is applied to extract significantly different regions, and the precise shapes of the distinct regions are obtained, and only the rough rectangular bounding box is output. A panoramic image saliency detecting method according to claim 2, wherein the original color image is converted into a grayscale image, and then the object proposal algorithm is applied to the grayscale image, and the obtained image is regarded as a density map; In the process of extracting the salient region, the morphological operation method is used to eliminate the noise, and the separate homogenous regions close to each other are connected to increase the density image; the optimization method is used to eliminate the bad results to exclude different background regions; The selection method, in the implementation process, automatic seed selection and iteration provides a threshold; the threshold selection uses adaptive threshold processing. The panoramic image saliency detecting method according to claim 1, wherein in step 24), the saliency value of the protruding region is obtained, and the obtained saliency map is specifically expressed as S _p , for the region initially identified as significant. p, define its significance value as Equation 3:

Where A(p) represents the number of pixels in the p-th region. The panoramic image saliency detecting method according to claim 1, wherein the step 3) performs maximum normalization, specifically using Maxima normalization, and summing after Maxima normalization.

Images