CN110060291B - A Calculation Method of Stereo Apparent Distance Considering Human Factors - Google Patents

Abstract

The present invention relates to a stereo apparent distance calculation model based on human factors, which is characterized in that: extracting the parallax of the image on the screen, and calculating its stereo depth; secondly, extracting the region of interest of the image, and obtaining its parallax angle and width angle Compared with the front and back background, the comfort is calculated according to the objective visual comfort calculation model; then, through the subjective experiment, the apparent distance value of the subjective perception of the stereo image is obtained; finally, the calculated stereo depth and the objective distance are established through the subjective measurement value. The relationship between the comfort level and the apparent distance calculation model based on the three-dimensional comfort level are established; Depth, stereo vision comfort, etc., comprehensively consider multi-dimensional factors, making the model evaluation results more accurate and reliable.

Description

A Calculation Method of Stereo Apparent Distance Considering Human Factors

technical field

The invention relates to a method for calculating a stereoscopic apparent distance considering human factors, and belongs to the technical fields of computer vision and image processing.

Background technique

In recent years, stereoscopic technology and related hardware technologies have developed rapidly. Stereoscopic video and games have fully entered the lives of ordinary people with the help of mobile phones, tablets, TVs, movies, and various head-mounted displays. The three-dimensional effect in the process of watching stereoscopic video is the most important feature that the three-dimensional display is superior to the two-dimensional display. Therefore, the strength of the three-dimensional effect must be properly controlled. It is necessary to avoid the "false three-dimensional" picture without any visual impact, and it must not appear to make people watch. The patient is uncomfortable, or even unable to achieve three-dimensional fusion.

Stereoscopic depth information can be calculated based on many factors such as left and right eye stereoscopic images, shooting and playback environment parameters, etc. In fact, there is a difference between the three-dimensional effect viewed by the user through various media and the solved three-dimensional depth. At present, in the entire life cycle of the stereo, only the viewer can understand the stereo effect according to the viewing experience effect after watching it finally, and it is impossible to realize timely stereo control.

The apparent distance of the stereo is the spatial distance of the stereo information perceived by the viewer. Unlike the "stereo depth", which can be directly calculated, the factors affecting the apparent distance are complex and diverse, except for the factors involved in the "stereo depth" calculation model In addition, it also includes the viewer's own factors, that is, human factors. Since the imaging of stereoscopic images on the retina of the human eye is different from the normal stereoscopic objects viewed by the human eye, the visual comfort will be low due to problems such as "adjustment-aggregation" during the viewing process, that is, the viewing experience is poor, and a certain amount of fatigue will occur The sense of perception causes a certain discrepancy between the perceived apparent distance and the calculated stereo depth value. Therefore, human factors must be considered in order to calculate the stereo apparent distance more accurately.

Contents of the invention

The object of the present invention is to provide a human-based stereoscopic apparent distance calculation model, because of the human perception of space, there is a significant error between the apparent distance and the actual depth perceived by humans in the virtual environment; although in the real world The accuracy of the apparent distance estimation is about 94%, but it drops to about 80% on average in the virtual environment, that is, it is underestimated or compressed by 20%, and the perceived apparent distance is affected by factors such as parallax, color and brightness. In view of this, the present invention is based on the three-dimensional depth calculation model, and uses the objective comfort model calculation result as human factor feedback to establish a three-dimensional apparent distance calculation model, and automatically completes the calculation of the apparent distance, obtaining a more accurate the apparent distance.

The technical scheme of the present invention is realized in the following way: a human factor-based stereoscopic apparent distance calculation model is characterized in that: extracting the parallax of the image on the screen, and calculating its stereoscopic depth; secondly, extracting the region of interest of the image, obtaining Its parallax angle, width angle, and front-background contrast are calculated according to the objective visual comfort calculation model; then, through subjective experiments, the apparent distance value of the subjective perception of the stereoscopic image is obtained; finally, the solution is established through the subjective measurement value. Based on the relationship between the calculated stereo depth and objective comfort, an apparent distance calculation model based on stereo comfort is established; the specific steps are as follows:

Step 1, extract the screen parallax when the stereoscopic image pair is projected, and calculate its stereoscopic depth according to the following formula:

Among them, SD represents the stereoscopic depth of the solution, e represents the eye-pupil distance of the viewer (usually 6.5cm), V represents the distance between the viewer and the playback screen, and Z represents the parallax of the stereo image pair on the screen during playback;

Step 2. Obtain the foreground area parallax angle, width angle, and foreground area contrast of the stereoscopic image respectively, and calculate its objective visual comfort according to the following formula:

VC(D,w,c)＝4.8736-0.7084D+0.1912ln(w)-0.0208Dln(w)+0.0015c ² -0.0572c

(0.50≤D≤2.00, 0.25≤w≤4.00)

Among them, D represents the disparity angle of the foreground area, w represents the width angle of the foreground area, and c represents the contrast between the foreground area and the background area. It is required that the parallax angle is within the range of 0.5°～2.0°, and the width angle is within the range of 0.25°～4.0°; VC is the objective stereoscopic vision comfort;

Step 3. Conduct a subjective experiment on the 21 sets of stereoscopic images that were actually shot, artificially measure the apparent distance perceived by the experimenters subjectively, organize the data, and obtain the average value of the data.

Step 4. Calculate the difference dif between the perceived apparent distance and the stereoscopic depth of each group of images, analyze the relationship between the difference and the comfort level VC, and perform curve fitting to obtain a linear relationship as follows:

dif＝0.13-0.0353*VC

Wherein, dif is the difference between perceived apparent distance (GTPD) and stereo depth (SD): dif=GTPD-SD, VC is objective stereo vision comfort;

Step 5. Arrange the formulas, and finally get the relationship between apparent distance, stereoscopic depth and stereoscopic comfort, and obtain the calculation model of apparent distance:

OPD＝SD+0.13-0.0353*VC

In the formula, OPD represents the objective apparent distance calculated by the model, SD represents the stereo depth calculated by the model, and VC represents the comfort value calculated by the objective stereo vision comfort model.

The positive effect of the present invention is to realize the objective estimation of the apparent distance of the stereoscopic image, and establish the objective estimation by analyzing the calculation result of the stereoscopic depth calculation model, the calculation result of the objective visual comfort model and the subjectively perceived apparent distance. The solution model for stereo apparent distance. It can effectively save the labor cost of traditional manual evaluation and the problem of relying on the evaluator; the starting point of the model is based on the human eye, which makes the model more objective and plays a better guiding role in stereo shooting; combined with stereo Depth, stereo vision comfort, etc., comprehensively consider multi-dimensional factors, making the model evaluation results more accurate and reliable.

Description of drawings

Fig. 1 is a schematic flow chart of the present invention.

Figure 2 is a binocular camera system.

Figure 3 is the right view of the experimental group.

Figure 4 is the experimental environment.

Figure 5 is a map of the region of interest in the experimental group.

Fig. 6 is a trend graph of stereo depth and apparent distance.

Fig. 7 shows the relationship between the difference between the apparent distance and the three-dimensional depth and the three-dimensional comfort.

FIG. 8 is a trend diagram of perceived apparent distance, stereoscopic depth and calculated apparent distance.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments: In this embodiment, 21 groups of real-shot stereo image pairs are obtained by using Daheng industrial cameras, and the model is verified using the stereo image library provided by the Korean Academy of Advanced Sciences. The process flow As shown in Figure 1, the specific steps are as follows:

Step 1. Use two Daheng MER-310-12UC industrial cameras (as shown in Figure 2) for image acquisition, and the resolution of the cameras is 2048×1536. And two cameras with the same model have the same basic parameter standards such as white balance, gain, and mode. When collecting data, first connect the computer to transmit the signal, place the device on a stable horizontal tripod stand, align and correct after assembly, and watch the dual-machine picture on the monitor through the software equipped with Daheng Camera. Synchronize random information such as the aperture, focal length, and focus of the two machines, and use the interpupillary distance of the human eye (5.5-7.5cm, 6.5cm is selected in this experiment) as the shooting baseline for synchronous shooting. In the process of stereo image shooting in this experiment, the scene layout is mostly divided into two layers, the picture is relatively simple, and the better stereo effect is highlighted through layer comparison. As shown in Figure 3, it is the right view of the 21 groups of images collected.

Step 2. Select 15 experimenters aged between 19 and 28. Make sure that the age of the experimenter is within a reasonable range, with normal eyesight and no visual physical diseases, such as color blindness, color weakness and other eye diseases. In addition, there is no adverse reaction when wearing the time-division stereoscopic glasses during viewing, and the perceived position of the stereoscopic image can be accurately and stably judged and the perceived apparent distance can be determined during viewing. They are also distinguished according to whether they have certain experience in observing stereoscopic images and whether they understand relevant professional knowledge. Since the tested personnel with relevant experience can better perform subjective evaluation, while other observers lack relevant experience, they need to arrange a series of training first. Through the early training, the subjective evaluation results can be determined more accurately.

Step 3. Use NVIDIA Quadro K620 graphics card, Samsung 2233RZ 3D display screen and NVIDIA Wireless 3D Vision - time-division stereoscopic glasses to watch stereoscopic images. The viewing distance is 1.5 meters, about 5 times the height of the display, and the horizontal (50cm) and vertical (30cm) viewing angles are 18.96° and 8.58°, respectively. The experimental environment is based on the settings recommended by ITU-R BT.500-11 and ITU-R BT.1438. In order to ensure the eye comfort of the viewers during the viewing process, eye rest can be performed between every two pairs of image playback during the experiment. In the process of watching the experiment, the ambient light is dimmed like the movie viewing environment of the theater, as shown in Figure 4, and the experiment is carried out for the testees in the dark room.

Step 4. In order to obtain the perceived apparent distance, experiments are carried out using subjective evaluation methods. In related research, there are mainly three types of measurement methods: verbal estimation, visual imagination and perception matching. Lexical estimates refer to the direct representation of depth estimates in meters. The act of visual imagination involves having the subject observe an object and imagine walking towards it. Imaginary time is recorded at their usual walking pace for depth judgment. In this embodiment, the method of language estimation is used in the experiment to directly quantify and estimate the perceived apparent distance. Obtain the apparent distance of each image perceived by the viewer through artificial measurement, take the average of the obtained apparent distance, organize the data and obtain the apparent distance of some experimental group image pairs shown in Table 1 value.

Table 1 Apparent distance of some image pairs

Step 5. Calculate the stereoscopic depth corresponding to each image pair in the experimental group through basic parameters such as parallax during projection, viewer interpupillary distance, and viewing distance. According to the following formula, the stereoscopic depth value of a certain frame of the image during playback can be calculated:

Among them, SD represents the calculated stereo depth, e represents the eye-pupil distance of the viewer (generally 6.5 cm), V represents the distance between the viewer and the playback screen, and Z represents the parallax of the stereo image pair on the screen during playback. As shown in Table 2, it is the partial stereo depth calculation value corresponding to the image pair of the experimental group (shown in Figure 3).

Table 2 Stereo Depth of Some Image Pairs

Step 6. Obtain the disparity map of the images in the experimental group, and calculate the plane saliency map of each image according to the GBVS algorithm. Then, calculate and fuse the disparity map and the plane saliency map according to the following formula to obtain the stereo saliency map:

IIS(x,y)＝ω ₁ IS _R (x,y)+ω ₂ D _R (x,y)

In the formula, IS _R (x, y) is a plane saliency map, D _R (x, y) is a disparity map, and ω ₁ and ω ₂ are their weights. In this embodiment, ω ₁ =ω ₂ =0.5. In order to obtain the region of interest, the stereo saliency map is subjected to threshold segmentation processing to obtain the mask image IIM(x, y). The specific method is as follows:

In the formula, C(x, y) is the pixel value at (x, y), and T is its segmentation threshold. If C(x,y)>T, the pixel belongs to the region of interest, corresponding to the white region in the mask image, otherwise it belongs to the black region. Use the preset ROI mask to multiply the right view and the disparity map to obtain the ROI image and the ROI disparity map respectively, and use the ROI as the foreground area. Invert the mask image of the ROI and multiply it with the right view to obtain the background area. As shown in FIG. 5 , it is a map of the region of interest corresponding to the image of the experimental group ( FIG. 3 ).

Step 7. According to the following formula, the objective visual comfort corresponding to the images of the experimental group can be calculated:

VC(D,w,c)＝4.8736-0.7084D+0.1912ln(w)-0.0208Dln(w)+0.0015c ² -0.0572c

(0.50≤D≤2.00, 0.25≤w≤4.00)

Among them, D represents the disparity angle of the foreground area, w represents the width angle of the foreground area, and c represents the contrast between the foreground area and the background area. It is required that the parallax angle is within the range of 0.5°-2.0°, and the width angle is within the range of 0.25°-4.0°. D, w, c are calculated as follows:

Among them, the average parallax value of the foreground area is D _f , o _f represents the foreground area, |o _f | represents the total number of pixels in the area of o _f , D is the average parallax angle of the foreground area, k is the projection magnification, D represents the parallax, L is the distance from the screen when viewing.

表示前景区域内第n条水平线长度。w为宽度角，W表示平均宽度，k为放映放大率，L为观看时距屏幕距离。where N _f represents the number of horizontal lines in the foreground area,

Indicates the length of the nth horizontal line in the foreground area. w is the width angle, W is the average width, k is the projection magnification, and L is the distance from the screen when viewing.

为基于空间加权的区域对比度(即客观舒适度模型中的c)，S_r(r_k,r_i)是区域r_k与r_i之间的空间距离，σ_s控制空间权值的强度。σ_s越大空间权值的影响越小，则背景区域的影响越显著。两区域间的空间距离被定义为区域重心间的欧几里得距离。其中，像素坐标为归一化后结果，取

Quantize each color channel to 16 different values, reducing the number of colors by a factor of 4096. Then the RGB space is replaced with the Lab space to obtain the color distance set between the foreground and foreground regions. d _r (r ₁ , r ₂ ) is the color distance between regions r ₁ and r ₂ . where f(c _k,i ) represents the frequency of the i-th color c _k,i appearing in all colors nk in the k-th region r _k . d(c _1,i ,c _2,j ) represents the color distance between the i-th color in area 1 and the j-th color in area 2 in Lab space.

is the regional contrast based on spatial weighting (ie c in the objective comfort model), S _r ( _rk , _ri ) is the spatial distance between region r _k and r _i , and σ _s controls the strength of the spatial weight. The larger the σ _s, the smaller the influence of the spatial weight, and the more significant the influence of the background area. The spatial distance between two regions is defined as the Euclidean distance between the centers of gravity of the regions. Among them, the pixel coordinates are the result after normalization, take

As shown in Table 3, the table shows the visual comfort (VC) values corresponding to some experimental images.

Table 3 Visual comfort value of each image

Step 8. In order to analyze the relationship between the stereoscopic depth and the apparent distance, the trend graph of the stereoscopic depth and the apparent distance shown in the figure can be observed: the apparent distance perceived by the human eye is largely affected by the stereoscopic depth, but There will be a certain numerical discrepancy between it and the stereoscopic depth obtained by the model solution, so the human factor is taken into consideration in the calculation of the apparent distance. Since the stereoscopic comfort directly reflects the viewer's viewing experience of the stereoscopic image, and is also one of the important criteria for evaluating the quality of the stereoscopic image, the present embodiment incorporates the stereoscopic comfort into the apparent distance as a feedback factor of the human factor control link. in the process of quantification. According to the stereo depth calculation model and the stereo comfort objective evaluation model, the stereo depth (VD) and stereo visual comfort (VC) of each image pair are calculated, combined with the apparent distance (GTPD) value obtained from the subjective evaluation for analysis.

Step 9. In order to obtain the relationship among the stereoscopic depth, stereoscopic comfort and apparent distance, first calculate the difference between the stereoscopic depth and the apparent distance (dif=GTPD-SD). As shown in Figure 7, with the increase of visual comfort, the difference between the apparent distance actually perceived by the human eye and the calculated stereo depth is continuously decreasing, and the relationship between the stereo comfort and the difference can be further analyzed. By establishing a scatter plot and making a fitting curve, the following formula is obtained:

dif＝0.13-0.0353*VC

Step 10, arrange the formula to obtain the stereoscopic apparent distance calculation model as shown in the following formula:

OPD＝SD+0.13-0.0353*VC

Step 11. In this embodiment, use five commonly used objective parameters as evaluation indicators to analyze the correlation between the model estimated value and the subjective evaluation value, use the model of the present invention to calculate the apparent distance of these images, and then compare the corresponding objective perception Pearson Linear Correlation Coefficient (PLCC), Kendall Rank-order Correlation Coefficient (KRCC), SROCC Correlation Coefficient (SpearmanRank OrderCorrelation Coefficient), mean absolute error (Mean Absolute Error, MAE) and root mean square error (Root Mean Squared Error, RMSE). And select some images with good visual comfort and poor visual comfort in the IVY image library as the test set, and compare the corresponding objective perceived apparent distance, the corresponding stereo depth and the Pearson correlation between the subjective perceived apparent depth measurement value coefficient, Kendall correlation coefficient, SROCC correlation coefficient, average absolute value error and root mean square error, the results verify that the apparent distance calculation value of the stereoscopic image obtained by the method of the present invention is closer to the visual perception of the human eye than the stereo depth calculation model result. At distance, the performance is better.

Claims (1)

1. A three-dimensional apparent distance resolving method based on human factors is characterized by comprising the following steps: extracting the parallax of the image on a screen, and calculating the stereo depth of the image; secondly, extracting an interested area of the image, acquiring a parallax angle and a width angle of a foreground area of the interested area and a contrast ratio of the foreground area and a background area, and calculating the comfort level of the interested area according to an objective stereoscopic vision comfort level calculation model; then, obtaining an apparent distance value of the subjective perception of the three-dimensional image through a subjective experiment; finally, establishing a correlation between the resolved stereo depth and the objective stereo vision comfort level through the subjective perception of the apparent distance value, and establishing an apparent distance resolving model based on the objective stereo vision comfort level; the method comprises the following specific steps:

step 1, extracting the screen parallax of the stereo image pair during projection, and calculating the stereo depth according to the following formula:

wherein,SDa calculated stereo depth is represented by the depth of the stereo,eindicating the distance of the viewer's eye pupil, taken at 6.5cm,Vindicating the distance of the viewer from the playback screen,Zrepresenting the disparity of the on-screen stereoscopic image pair when played;

step 2, respectively obtaining a foreground region parallax angle and a width angle of the stereo image and the contrast of the foreground region and a background region, and calculating the objective stereo vision comfort level according to the following formula:

wherein,Dthe parallax angle of the foreground region is represented,wthe width angle of the foreground region is represented,crepresenting the contrast of the foreground region and the background region; requiring the parallax angle of the foreground region to be in range

Inner, the width angle of the foreground region is in the range

Internal;VCcomfort level for objective stereoscopic vision;

step 3, carrying out subjective experiments on the 21 groups of real-shot stereo images, artificially measuring the apparent distance subjectively perceived by experimenters, sorting data, and obtaining a data average value;

step 4, calculating the difference value between the subjective perception apparent distance and the three-dimensional depth of each group of imagesdifAnalyzing the difference and the objective stereoscopic vision comfort levelVCFitting a curve to obtain a linear relation as follows:

wherein,

，GTPD isThe apparent distance of the subjective perception is,SD isThe depth of the three-dimensional space is increased,VCcomfort level for objective stereoscopic vision;

step 5, a formula is arranged, the incidence relation of the objective apparent distance, the three-dimensional depth and the objective three-dimensional visual comfort level is finally obtained, and a resolving model of the apparent distance is obtained:

in the formula,OPDrepresenting the objective apparent distance calculated by the model,SDthe depth of the solid of the model solution is represented,VCrepresenting the comfort value resolved by the objective stereoscopic vision comfort model.

Images