CN116402878A - Light field image processing method and device - Google Patents

Abstract

The disclosure relates to the technical field of image processing, and particularly provides a light field image processing method and device. A light field image processing method comprises the steps of obtaining target test point information and a light field image group sent by acquisition equipment, carrying out view point fusion on foreground images in all light field images based on the target view point information to obtain foreground view point images, carrying out view point fusion on background images in all light field images based on the target view point information and a pre-generated background view point diagram of each view point position to obtain background view point images, and generating target light field images according to the foreground view point images and the background view point images. In the embodiment of the disclosure, the parallax images of the background images are generated in advance in a front background segmentation mode, so that the parallax images aiming at complex backgrounds do not need to be calculated in real time, the data volume of view image synthesis is greatly reduced, the image processing speed and the image processing precision are improved, and the real-time light field video communication can be realized.

Description

Light field image processing method and device

technical field

The present disclosure relates to the technical field of image processing, and in particular to a light field image processing method and device.

Background technique

Light Field (Light Field) can record higher-dimensional light data, so as to obtain higher-precision 3D information than traditional 2D imaging and traditional 3D imaging represented by binocular stereo vision. Light field video can accurately perceive dynamic environments, Make users experience an immersive viewing experience.

However, light field video has a large amount of data and slow processing speed, and it is difficult to realize real-time light field video presentation, which limits the application scenarios of light field video.

Contents of the invention

In order to improve the data processing efficiency of light field images and realize real-time light field video presentation, embodiments of the present disclosure provide a light field image processing method, device, electronic equipment, video communication system, and storage medium.

In a first aspect, an embodiment of the present disclosure provides a light field image processing method, which is applied to a display device, and the method includes:

Obtain target viewpoint information and a light field image group sent by the acquisition device; the target viewpoint information represents the position information of the observer's eyes of the display device, and the light field image group includes each target light field in the acquisition device The light field image collected by the camera;

performing viewpoint fusion on foreground images in each light field image based on the target viewpoint information, to obtain foreground viewpoint images corresponding to the target viewpoint information;

Based on the target viewpoint information and the pre-generated background disparity map of each viewpoint position, perform viewpoint fusion on the background images in each light field image to obtain the background viewpoint image corresponding to the target viewpoint information; the viewpoint position represents A location corresponding to each target light field camera;

A target light field image is generated according to the foreground viewpoint image and the background viewpoint image.

In some implementation manners, performing viewpoint fusion on the foreground images in each light field image based on the target viewpoint information to obtain the foreground viewpoint image corresponding to the target viewpoint information includes:

performing image segmentation on each light field image in the light field image group to obtain a foreground image corresponding to each light field image;

For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, the first foreground image corresponding to the first light field image and the first foreground image corresponding to the second light field image Performing disparity estimation on the second foreground image to obtain a first foreground disparity map corresponding to the first foreground image and a second foreground disparity map corresponding to the second foreground image;

performing parallax mapping on the first foreground image based on the first foreground disparity map to obtain a first foreground map, and performing parallax mapping on the second foreground image based on the second foreground disparity map to obtain a second foreground map;

performing image fusion processing on the first foreground map and the second foreground map based on the target viewpoint information to obtain the foreground viewpoint image.

In some implementation manners, performing image segmentation on each light field image in the light field image group to obtain a foreground image corresponding to each light field image includes:

For each light field image in the light field image group, based on a pre-generated preset background image at the same viewpoint position as the light field image, image difference is performed on the light field image to obtain the light field image Corresponding foreground and background images.

In some embodiments, the parallax estimation is performed on the first foreground image corresponding to the first light field image and the second foreground image corresponding to the second light field image to obtain the first foreground image corresponding to the first foreground image A foreground disparity map and the second foreground disparity map corresponding to the second foreground image include:

performing downsampling on the first foreground image based on a preset downsampling coefficient to obtain a first downsampling image, and performing downsampling on the second foreground image to obtain a second downsampling image;

matching the positions of the same pixels on the first downsampled image and the second downsampled image to obtain a first disparity map corresponding to the first downsampled image and a second disparity map corresponding to the second downsampled image;

determining a first disparity search range according to the first disparity map and the preset downsampling coefficient, and determining a second disparity search range according to the second disparity map and the preset downsampling coefficient;

Performing disparity estimation on the first foreground image based on the first disparity search range to obtain the first foreground disparity map, and performing disparity estimation on the second foreground image based on the second disparity search range to obtain the second foreground image Foreground disparity map.

In some implementations, the first foreground map is obtained by performing disparity mapping on the first foreground image based on the first foreground disparity map, and the second foreground image is obtained by performing disparity mapping on the second foreground disparity map. Foreground maps, including:

matching each pixel on the first foreground image to a corresponding pixel position on the map according to the first foreground disparity map to obtain the first foreground map; according to the second foreground disparity map matching each pixel on the second foreground image to a corresponding pixel position on the map to obtain the second foreground map;

The performing image fusion processing on the first foreground map and the second foreground map based on the target viewpoint information to obtain the foreground viewpoint image includes:

determining a first weight and a second weight according to the target viewpoint information and the position information of any two adjacent target light field cameras;

Based on the first weight and the second weight, image fusion processing is performed on the first foreground map and the second foreground map to obtain the foreground viewpoint image.

In some embodiments, based on the target viewpoint information and the pre-generated background disparity map of each viewpoint position, viewpoint fusion is performed on the background images in each light field image to obtain the background corresponding to the target viewpoint information A viewpoint image; the viewpoint position represents a position corresponding to each target light field camera, including:

performing image segmentation on each light field image in the light field image group to obtain a background image corresponding to each light field image;

For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, based on the pre-generated first background disparity map at the same viewpoint position as the first light field image, the performing parallax mapping on the first background image of the first light field image to obtain a first background map, and based on the pre-generated second background parallax map at the same viewpoint position as the second light field image, performing parallax mapping on the second background image of the image to obtain a second background map;

performing image fusion processing on the first background map and the second background map based on the target viewpoint information to obtain the background viewpoint image.

In some implementation manners, the acquiring target viewpoint information includes:

collecting scene images through an image collection device arranged on the display device;

performing image detection according to the scene image to obtain position information of the observer's eyes in the scene image;

The target viewpoint information is generated based on the position information.

In some implementations, the process of acquiring the light field image group sent by the acquisition device includes:

sending the target viewpoint information to the acquisition device, so that the acquisition device determines one or more target light field cameras from a plurality of light field cameras according to the target viewpoint information;

The light field image group sent by the acquisition device is received.

In some embodiments, the light field image processing method described in the present disclosure further includes:

The background disparity map of each viewpoint position sent by the collection device is received and stored.

In a second aspect, the embodiment of the present disclosure provides a light field image processing method, which is applied to a collection device, and the method includes:

Collecting the current scene image respectively by a plurality of light field cameras arranged on the collection device, to obtain the scene image of the viewpoint position corresponding to each light field camera;

For any two adjacent light field cameras, according to the scene images collected by the two light field cameras respectively, a background disparity map of the viewpoint position of each light field camera is generated;

The background disparity map of each viewpoint position is sent to the display device, so that the display device stores the background disparity map of each viewpoint position.

In some embodiments, the light field image processing method described in the present disclosure further includes:

receiving target viewpoint information sent by the display device;

determining one or more target light field cameras from the plurality of light field cameras included in the acquisition device according to the target viewpoint information;

Collecting light field images by the target light field camera to obtain a light field image group, and sending the light field image group to the display device.

In a third aspect, an embodiment of the present disclosure provides a light field image processing device, which is applied to a display device, and the device includes:

An acquisition module configured to acquire target viewpoint information and a light field image group sent by the acquisition device; the target viewpoint information represents the position information of the observer's eyes of the display device, and the light field image group includes the acquisition device The light field image collected by each target light field camera in ;

The foreground fusion module is configured to perform viewpoint fusion on the foreground images in each light field image based on the target viewpoint information, to obtain the foreground viewpoint image corresponding to the target viewpoint information;

The background fusion module is configured to perform viewpoint fusion on the background images in each light field image based on the target viewpoint information and the pre-generated background disparity map of each viewpoint position, to obtain the background viewpoint corresponding to the target viewpoint information image; the viewpoint position represents a position corresponding to each target light field camera;

An image synthesis module configured to generate a target light field image according to the foreground viewpoint image and the background viewpoint image.

In some implementations, the foreground fusion module is configured to:

performing image segmentation on each light field image in the light field image group to obtain a foreground image corresponding to each light field image;

For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, the first foreground image corresponding to the first light field image and the first foreground image corresponding to the second light field image Performing disparity estimation on the second foreground image to obtain a first foreground disparity map corresponding to the first foreground image and a second foreground disparity map corresponding to the second foreground image;

performing parallax mapping on the first foreground image based on the first foreground disparity map to obtain a first foreground map, and performing parallax mapping on the second foreground image based on the second foreground disparity map to obtain a second foreground map;

performing image fusion processing on the first foreground map and the second foreground map based on the target viewpoint information to obtain the foreground viewpoint image.

In some implementations, the foreground fusion module is configured to:

For each light field image in the light field image group, based on a pre-generated preset background image at the same viewpoint position as the light field image, image difference is performed on the light field image to obtain the light field image Corresponding foreground and background images.

In some implementations, the foreground fusion module is configured to:

performing downsampling on the first foreground image based on a preset downsampling coefficient to obtain a first downsampling image, and performing downsampling on the second foreground image to obtain a second downsampling image;

matching the positions of the same pixels on the first downsampled image and the second downsampled image to obtain a first disparity map corresponding to the first downsampled image and a second disparity map corresponding to the second downsampled image;

determining a first disparity search range according to the first disparity map and the preset downsampling coefficient, and determining a second disparity search range according to the second disparity map and the preset downsampling coefficient;

Performing disparity estimation on the first foreground image based on the first disparity search range to obtain the first foreground disparity map, and performing disparity estimation on the second foreground image based on the second disparity search range to obtain the second foreground image Foreground disparity map.

In some implementations, the foreground fusion module is configured to:

matching each pixel on the first foreground image to a corresponding pixel position on the map according to the first foreground disparity map to obtain the first foreground map; according to the second foreground disparity map matching each pixel on the second foreground image to a corresponding pixel position on the map to obtain the second foreground map;

determining a first weight and a second weight according to the target viewpoint information and the position information of any two adjacent target light field cameras;

Based on the first weight and the second weight, image fusion processing is performed on the first foreground map and the second foreground map to obtain the foreground viewpoint image.

In some implementations, the background fusion module is configured to:

performing image segmentation on each light field image in the light field image group to obtain a background image corresponding to each light field image;

For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, based on the pre-generated first background disparity map at the same viewpoint position as the first light field image, the performing parallax mapping on the first background image of the first light field image to obtain a first background map, and based on the pre-generated second background parallax map at the same viewpoint position as the second light field image, performing parallax mapping on the second background image of the image to obtain a second background map;

performing image fusion processing on the first background map and the second background map based on the target viewpoint information to obtain the background viewpoint image.

In some implementations, the acquisition module is configured to:

collecting scene images through an image collection device arranged on the display device;

performing image detection according to the scene image to obtain position information of the observer's eyes in the scene image;

The target viewpoint information is generated based on the position information.

In some implementations, the acquisition module is configured to:

sending the target viewpoint information to the acquisition device, so that the acquisition device determines one or more target light field cameras from a plurality of light field cameras according to the target viewpoint information;

The light field image group sent by the acquisition device is received.

In some implementations, the acquisition module is configured to:

The background disparity map of each viewpoint position sent by the collection device is received and stored.

In a fourth aspect, an embodiment of the present disclosure provides a light field image processing device, which is applied to a collection device, and the device includes:

The image acquisition module is configured to respectively acquire the current scene image through a plurality of light field cameras arranged on the acquisition device, so as to obtain the scene image of the viewpoint position corresponding to each light field camera;

The parallax determination module is configured to generate a background parallax map of the viewpoint position of each light field camera for any two adjacent light field cameras according to the scene images respectively collected by the two light field cameras;

The sending module is configured to send the background disparity map of each viewpoint position to the display device, so that the display device stores the background disparity map of each viewpoint position.

In some implementations, the sending module is configured to:

receiving target viewpoint information sent by the display device;

determining one or more target light field cameras from the plurality of light field cameras included in the acquisition device according to the target viewpoint information;

Collecting light field images by the target light field camera to obtain a light field image group, and sending the light field image group to the display device.

In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:

processor; and

The memory stores computer instructions for causing the processor to execute the method according to any implementation manner of the first aspect, or execute the method according to any implementation manner of the second aspect.

In a sixth aspect, an embodiment of the present disclosure provides a video communication system, including:

A display device, including an image acquisition device and a first controller, the first controller is configured to execute the method according to any implementation manner of the first aspect;

The acquisition device includes a light field camera array and a second controller, the light field camera array includes a plurality of light field cameras, and the second controller is configured to execute the method according to any implementation manner of the second aspect.

In the seventh aspect, the embodiments of the present disclosure provide a storage medium, which stores computer instructions, and the computer instructions are used to make the computer execute the method according to any implementation manner of the first aspect, or execute the method according to any implementation manner of the second aspect. the method described.

The light field image processing method of the embodiment of the present disclosure includes acquiring the target pilot information and the light field image group sent by the acquisition device, performing viewpoint fusion on the foreground images in each light field image based on the target viewpoint information to obtain the foreground viewpoint image, and obtaining the foreground viewpoint image based on the target viewpoint information. Viewpoint information and the pre-generated background viewpoint image of each viewpoint position, perform viewpoint fusion on the background images in each light field image to obtain the background viewpoint image, and generate the target light field image according to the foreground viewpoint image and the background viewpoint image. In the embodiment of the present disclosure, the disparity map of the background image is generated in advance by means of foreground and background segmentation, so that there is no need to calculate the disparity map for the complex background in real time, greatly reducing the amount of data for viewpoint image synthesis, improving the speed and accuracy of image processing, and realizing Real-time light field video communication.

Description of drawings

In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description The drawings are some implementations of the present disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative work.

Fig. 1 is an architecture diagram of a video communication system according to some implementations of the present disclosure.

Fig. 2 is a schematic structural diagram of an electronic device according to some implementations of the present disclosure.

Fig. 3 is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

Fig. 4 is a flowchart of a light field image processing method according to some implementations of the present disclosure.

Fig. 5 is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

Fig. 6 is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

Fig. 7 is a schematic diagram of a circuit structure of a light field camera array according to some embodiments of the present disclosure.

Fig. 8 is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

Fig. 9 is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

Fig. 10 is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

Fig. 11 is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

Fig. 12 is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

Fig. 13 is a structural block diagram of a light field image processing device according to some embodiments of the present disclosure.

Fig. 14 is a structural block diagram of a light field image processing device according to some embodiments of the present disclosure.

Fig. 15 is a structural block diagram of an electronic device according to some embodiments of the present disclosure.

Detailed ways

The technical solutions of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described implementations are part of the implementations of the present disclosure, but not all of them. Based on the implementation manners in the present disclosure, all other implementation manners obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure. In addition, the technical features involved in different embodiments of the present disclosure described below may be combined with each other as long as they do not constitute a conflict with each other.

The definition of light field (Light Field) refers to the amount of light passing through each point in each direction. Light field images can record higher-dimensional light data than traditional two-dimensional images, thus presenting a more comprehensive image than traditional two-dimensional imaging and dual-dimensional imaging. Traditional 3D imaging represented by stereo vision provides higher-precision 3D information.

Light field video can accurately perceive the dynamic environment. Combined with eye tracking technology, when the user's viewing point of view changes, the video screen can follow the change of the point of view in real time, thus presenting the user with an immersive naked-eye 3D viewing experience.

The data collection of light field video requires the use of light field camera array, which includes several or even dozens of light field cameras at different positions, and each light field camera is responsible for collecting images of a viewpoint position, so that light field video The amount of data is huge, which leads to a large amount of computation for viewpoint synthesis of image data collected by different cameras in the later stage, and the data processing speed is very slow.

Therefore, the light field video in the related art is mainly used in an offline video scene, and it is difficult to realize a real-time video scene. For example, take the remote video chat scene as an example. User A watches the light field video of the scene where user B is located through the first device, and needs to process and send the light field video data of the scene where user B is located in real time. Composite rendering of field video images, the whole process involves a lot of data computation, and it is difficult to achieve real-time presentation.

Based on the defects in the above-mentioned related technologies, the embodiments of the present disclosure provide a light field image processing method, device, electronic equipment, video communication system and storage medium, aiming at improving the processing speed of light field data and realizing real-time light field video presented.

FIG. 1 shows an architecture diagram of a video communication system in some embodiments of the present disclosure. The application scenarios of the embodiments of the present disclosure will be described below with reference to FIG. 1 .

As shown in FIG. 1 , in some implementations, the video communication system includes a capture device 100 and a display device 200 , and the capture device 100 and the display device 200 establish a communicable connection through a wired or wireless network.

In an exemplary one-way video communication scenario, the collection device 100 may collect light field image data of the scene where user A is located, and send the light field image data to the display device 200 . The display device 200 obtains the viewpoint position currently viewed by the user B by tracking the eye position of the user B, combines the viewpoint position and the light field image data sent by the acquisition device 100 to perform viewpoint image synthesis, and renders the synthesis on the display device 200 The resulting light field image.

Of course, it can be understood that the above example only takes one-way video communication as an example, but the disclosed solution is not limited to the one-way video communication scene. For the two-way video communication scene, the display device 200 can also collect the light field image of the scene where user B is located. data, and send the light field image data to the acquisition device 100. At the same time, the acquisition device 100 can also track the eye position of user A to obtain the viewpoint position currently viewed by user A, combine the viewpoint position and the light field image data sent by the display device 200 to perform viewpoint image synthesis, and display it on the acquisition device 100 Rendering showing the composited light field image. Those skilled in the art can understand this, which is not repeated in the present disclosure.

Taking a two-way video communication scene as an example, FIG. 2 shows a schematic structural diagram of an electronic device in some implementations of the present disclosure. The electronic device can be either a collection device 100 or a display device 200 , which is not limited in the present disclosure.

As shown in FIG. 2 , the electronic device includes a display screen 110 , a light field camera array 120 and an image acquisition device 130 .

The display screen 110 is used to display light field images, and the display screen 110 may be any screen component suitable for implementation, such as an LCD (Liquid Crystal Display, liquid crystal display) screen, an OLED (Organic Light-Emitting Diode, organic light-emitting semiconductor) screen, etc., This disclosure does not limit this.

The light field camera array 120 includes a plurality of light field cameras 121, and these light field cameras 121 are deployed in an array form on the electronic device. Since each light field camera 121 is deployed in a different position on the electronic device, the light field camera array 120 can be Scene images from different viewpoint positions are collected.

For example, in the example shown in FIG. 2 , the light field camera array 120 includes 9 light field cameras 121 in total, and the 9 light field cameras 121 are evenly spaced on the upper edge of the display screen 110 . Of course, those skilled in the art can understand that the number and arrangement of cameras in the light field camera array 120 are not limited to those shown in FIG. 2 , and can also be in any other suitable implementation form, which is not limited in the present disclosure.

The image acquisition device 130 is a camera for realizing user eye tracking, which may be, for example, a high-precision RGB camera, that is, the image acquisition device 130 collects the current scene image and performs image detection on the scene image to determine the current user Viewpoint information, where the viewpoint information represents the location information of the user's eyes. In the example of FIG. 2 , the image capture device 130 is disposed under the display screen 110 of the electronic device, but it can be understood that the present disclosure does not limit the position of the image capture device 130 .

On the basis of the above video communication system shown in FIG. 1 and FIG. 2 , the light field image processing method according to the embodiment of the present disclosure will be described below.

It is worth noting that, for ease of understanding, the following embodiments of the present disclosure will take a one-way video communication scenario as an example, that is, the acquisition device 100 serves as a light field data acquisition end, and the display device 200 serves as a light field video display end. The principle of the two-way video communication scenario is exactly the same, which will not be repeated in this disclosure.

In some implementations, the present disclosure provides a light field image processing method, which can be applied to the display device 200, and the processor of the display device 200 executes the processing, which will be described below with reference to FIG. 3 .

As shown in FIG. 3 , in some implementations, the light field image processing method of the present disclosure includes:

S310. Acquire target viewpoint information and a light field image group sent by the acquisition device.

It can be understood that the light field image includes viewpoint images of many viewpoint positions, but the viewpoint positions that the user's eyes can receive are limited, that is, not all viewpoint images will enter the user's eyes, so the target light field rendered and displayed on the display device 200 The image is a viewpoint synthesis image generated in combination with the user's current viewpoint position. Therefore, in the embodiments of the present disclosure, it is necessary to combine the user's current target viewpoint information to perform viewpoint image synthesis to obtain the target light field image.

In combination with the scene shown in FIG. 1 , on the display device 200 side, through the image acquisition device 130 on the display device 200 combined with the eye tracking algorithm, the viewpoint tracking of user B can be realized, and the position information of user B's eyes can be obtained. The position information That is, the target viewpoint information described in this disclosure. For the process of obtaining target viewpoint information through eye tracking, the following embodiments of the present disclosure will be described, and details will not be described here.

At the same time, the display device 200 also needs to receive the light field image group from the collection device 100 , the light field image group refers to the image group collected by the light field camera array 120 on the collection device 100 . As shown in FIG. 2 , in this example, the light field camera array includes 9 light field cameras, and each camera can collect a light field image, and the light field image group is a collection of these light field images.

It is worth noting that, in some implementations, the light field image group may include light field images collected by each light field camera. For example, in the example in FIG. 2, 9 light field cameras collect a light field image respectively, Therefore, the light field image group includes 9 light field images in total, that is, each light field camera 121 in the light field camera array 120 is the target light field camera described in the present disclosure.

In some other implementations, it can be seen from the combination of the aforementioned principles that, due to the limited number of viewpoints that the human eye can receive, not all viewpoint images can enter the observer's eyes, that is, the light field images collected by the light field camera array 120 There are a large number of redundant images in the group, and this part of redundant image data will also slow down the data processing speed.

Therefore, one or more target light field cameras may be selected from multiple light field cameras based on the target viewpoint information to collect light field images, and the target light field camera is the light field camera at a position corresponding to the target viewpoint information. It can be understood that only using the target light field camera at the user's viewpoint to acquire the light field image group can filter out redundant image data, reduce the amount of computation while ensuring image accuracy, and improve data processing efficiency. This is for explanation.

S320. Perform viewpoint fusion on the foreground images in each light field image based on the target viewpoint information, to obtain foreground viewpoint images corresponding to the target viewpoint information.

S330. Based on the target viewpoint information and the pre-generated background disparity map for each viewpoint position, perform viewpoint fusion on the background images in each light field image to obtain a background viewpoint image corresponding to the target viewpoint information.

It can be understood that the light field images displayed on the display device 200 need to change with the position of the user's viewpoint. Therefore, when generating the target light field image, it is necessary to combine the user's target viewpoint information with the light field images in the light field image group. Image synthesis, and then obtain the target light field image under the user's new viewpoint, and realize the picture follow-up effect.

In the embodiments of the present disclosure, in order to speed up the synthesis speed of viewpoint images and meet the real-time requirements of light field video, the synthesis of viewpoint images is divided into synthesis of foreground viewpoint images and synthesis of background viewpoint images.

It is worth noting that, in combination with the video communication scene, it can be known that taking video conference as an example, the large-screen electronic equipment used to realize the video conference is usually in a relatively fixed position, so that during the video communication, the background in the scene image collected by the acquisition device 100 Parts hardly change, and generally only foreground people or objects move.

Therefore, in the embodiments of the present disclosure, it may be considered to divide the foreground and background in the light field image, and for the background image with little change, the background disparity map of each viewpoint position may be generated in advance through preprocessing.

Parallax (Diaparity) refers to the offset of image pixels due to different viewpoint positions. The Diaparity Map (Diaparity Map) is an image that reflects the pixel offset. The Disparity Map is a two-dimensional image whose size is equal to the original image. , each pixel on the disparity map represents the disparity value of the pixel at that position on the original image. For example, as shown in FIG. 2 , taking any two adjacent light field cameras 121 in the light field camera array 120 as an example, since the two are deployed at different positions, the pixels on the collected light field image also have deviations. The disparity map is used to record the pixel deviation on the two images.

For real-time light field video communication scenarios, since the light field camera array 120 includes a large number of cameras, and each camera position represents a viewpoint position, it is necessary to calculate the parallax between each light field image and adjacent images when combining viewpoint images Figure, the amount of calculation data is large, making it difficult to achieve real-time performance.

However, in the embodiments of the present disclosure, it is considered that in the scene where the user A is located, the background part hardly changes, so the disparity map can be generated in advance for the background part of the light field image. For example, when the acquisition device 100 is turned on, the light field camera array 120 on the acquisition device 100 can collect the current scene image, and pre-construct the background disparity map corresponding to each viewpoint position, so that in the subsequent acquisition device 100 and display device 200 When the video communication connection is established, the background disparity map is sent to the display device 200, and the display device 200 can directly synthesize the background viewpoint image of the background part in the current light field image based on the pre-obtained background disparity map, without recalculating the background Part of the disparity map to reduce the amount of data and speed up the synthesis of viewpoint images.

At the same time, for the foreground image in the light field image, combined with the video communication scene, it can be seen that the foreground figure generally only occupies a small area of the image. After the complex background image is segmented out, the speed of viewpoint image synthesis for the foreground image will also be greatly improved. Improve, so as to meet the real-time and low-latency requirements of video communication.

In some embodiments of the present disclosure, after receiving the light field image group sent by the acquisition device 100, the display device 200 may first perform image segmentation on each light field image in the light field image group to obtain the corresponding Foreground image and background image.

For the foreground image, since the foreground disparity map cannot be established in advance, real-time disparity map calculation can be performed on it to obtain the foreground disparity map of each viewpoint position, and then based on the foreground disparity map and target viewpoint information, the viewpoint fusion of each foreground image is performed. The foreground viewpoint image corresponding to the target viewpoint information is obtained.

For the background image, since the background disparity map of each viewpoint position has been established in advance, there is no need to calculate the disparity map, and the viewpoint fusion of each background image is directly based on the pre-established background disparity map and target viewpoint information, and the corresponding target viewpoint can be obtained Background viewpoint image under information.

For the viewpoint fusion process of the foreground viewpoint image and the background viewpoint image, this will be described in the following embodiments of the present disclosure, and will not be described in detail here.

S340. Generate a target light field image according to the foreground viewpoint image and the background viewpoint image.

It can be understood that the foreground viewpoint image represents the foreground image at the new viewpoint position (that is, the viewpoint position corresponding to the target viewpoint information), and the background viewpoint image represents the background image at the new viewpoint position (that is, the viewpoint position corresponding to the target viewpoint information).

Therefore, after the foreground viewpoint image and the background viewpoint image are obtained, the target light field image can be obtained by performing fusion processing on the two. For example, in an example, the target light field image can be obtained by pasting the foreground viewpoint image on the background viewpoint image.

After the target light field image is obtained, the target light field image can be rendered and displayed on the display screen of the display device 200 , and the user B can watch the target light field image through the display screen.

It can be understood that, in the above embodiment, the process of generating a frame of target light field image is described. For each frame of light field video image, the above process is repeated, and the light field video can be played on the display device 200 .

At the same time, it can be understood that in the process of generating the target light field image, it is necessary to combine the viewpoint position of the observer (that is, the viewpoint position corresponding to the target viewpoint information), so that when the observer's eye position moves, the target viewpoint information is also updated accordingly , so that the finally obtained target light field image is also changed to a light field image corresponding to the new viewpoint position, presenting a naked-eye 3D effect in which the video picture changes with the viewer's viewpoint. For example, in a video conferencing scenario, the user can have an immersive experience, eliminate the sense of distance generated by conventional two-dimensional video communication, and improve the user communication experience.

From the above, it can be seen that in the embodiments of the present disclosure, by dividing the light field image into the foreground and the background, and realizing the viewpoint image synthesis based on the pre-generated background disparity map, the data volume of the viewpoint image synthesis is reduced, and the calculation efficiency and accuracy are improved, and further realize Real-time low-latency communication of light field video improves user video communication experience.

In combination with the foregoing, it can be seen that in the embodiments of the present disclosure, it is necessary to pre-generate the background disparity map of each viewpoint position at the acquisition device 100 . The process of generating the background disparity map will be described below with reference to FIG. 4 .

As shown in FIG. 4 , in some implementations, the light field image processing method of the example of the present disclosure is applied to the acquisition device 100, and the process for the acquisition device 100 to generate the background disparity map includes:

S410. Collect the current scene image respectively through multiple light field cameras arranged on the collection device, and obtain the scene image of the viewpoint position corresponding to each light field camera.

S420. For any two adjacent light field cameras, generate a background disparity map of the viewpoint position of each light field camera according to the scene images respectively collected by the two light field cameras.

S430. Send the background disparity map of each viewpoint position to the display device, so that the display device stores the background disparity map of each viewpoint position.

In some implementation manners, the process of S410-S430 may be executed to generate the background disparity map in the current scene each time the acquisition device 100 is turned on.

For example, in the example in FIG. 1, when the collection device 100 is turned on, the collection device 100 has not established a video communication connection with the display device 200 at this time, so that there is no user A in the scene where the collection device 100 is located, and only the background is included. At this time, it can The background disparity map of each viewpoint position is generated by using the collection device 100 .

As shown in FIG. 2, the acquisition device 100 can respectively acquire a current scene image through each light field camera in the light field camera array 120 arranged above, that is, in the example of FIG. Each light field camera 121 captures a current scene image.

It can be understood that since the deployment positions of each light field camera 121 in the light field camera array 120 are different, the collected scene images correspond to different viewpoint positions, and the disparity map of a certain viewpoint position can be understood as, The pixel offset of the field image relative to the light field image at other viewpoint locations. In the embodiments of the present disclosure, the background disparity map of each viewpoint position is the disparity map of the light field image of the viewpoint position and the light field images of adjacent viewpoint positions.

In some embodiments of the present disclosure, the background disparity map of each disparity position relative to adjacent disparity positions can be determined based on the disparity estimation algorithm. For the specific process of the disparity estimation algorithm, you can refer to the following method for calculating the foreground disparity map in this paper. It will not be described in detail here.

In some other implementation manners, considering that the estimation process of the background disparity map is pre-built, there is no requirement on the operation speed, so a disparity estimation method with higher precision can be used. For example, in an example, a disparity estimation network based on a deep neural network (DNN, Deep Neural Network) can be used, and the disparity estimation network can be pre-trained by manual labeling, and then the scene images of any two adjacent viewpoint positions can be input into the disparity Estimate the network to obtain the background disparity map corresponding to each viewpoint position.

After the background disparity map of each viewpoint position is obtained through the above process, the acquisition device 100 may store the background disparity map in a cache.

After completing the preparatory work in the above-mentioned offline stage, when the display device 200 establishes a video communication connection with the collection device 100, it means that user B of the display device 200 needs to perform video communication with user A of the collection device 100. At this time, the collection device 100 can The background disparity map stored in the cache is sent to the display device 200. After receiving the background disparity map of each viewpoint position, the display device 200 may store the background disparity map to wait for the calling of the subsequent viewpoint image synthesis stage. The interaction process between the acquisition device 100 and the display device 200 during the video communication process will be described below.

It can be seen from the foregoing that in the video communication scenario, user B of the display device 200 can observe limited viewpoints, therefore, there are a large number of redundant images in the light field image group collected by the light field camera array 120 of the collection device 100 . In some embodiments of the present disclosure, the target light field camera may be determined from the multiple light field cameras of the collection device 100 according to the current eye position information of the user B, which will be described below with reference to FIG. 5 .

As shown in FIG. 5, in some implementations, the light field image processing method of the example of the present disclosure includes:

S510. The display device collects the scene image through the image collection device.

As shown in FIG. 1 and FIG. 2 , the display device 200 can capture a scene image including user B through the image capture device 130 provided on the device.

S520. The display device performs image detection according to the scene image, and obtains position information of the observer's eyes in the scene image.

Then, the display device 200 can perform image detection on the collected scene image based on the eye tracking algorithm, so as to determine the position information of the eyes of the observer (that is, user B), and the position information represents the position of the observer's eyes on the scene image. image location.

S530. The display device generates target viewpoint information based on the location information.

Afterwards, the display device 200 can map the position information to the viewpoint position of the light field camera array 120 of the display device 200 according to the position information of the observer's eyes to obtain the target viewpoint information, that is, the target viewpoint information indicates that the observer's eyes are displaying Location information in the light field camera array 120 of the device 200 .

For example, in an example, the position information of user B's eyes is mapped to the target viewpoint information in the light field camera array 120 of the display device 200 as shown in FIG. And the viewpoint position of 121-c.

S540. The display device sends the target viewpoint information to the collection device.

The display device 200 sends the target viewpoint information to the collection device 100, so that the collection device 100 determines one or more target light field cameras from multiple light field cameras according to the target viewpoint information.

S550. The acquisition device 100 determines one or more target light field cameras from multiple light field cameras according to the target viewpoint information.

In the embodiments of the present disclosure, as shown in FIG. 6 , the target viewpoint information represents the position information of the observer's eyes in the light field camera array 120, so that the acquisition device 100 can determine the predicted position from the light field camera array 120 according to the target viewpoint information. Sets the number of target light field cameras.

It can be understood that the target light field camera refers to the light field camera located near the target viewpoint information, the number of target light field cameras can be selected according to the specific scene, however, the number of target light field cameras should ensure that the viewpoint position of the target light field camera can Override the viewpoint position of the target viewpoint information. Therefore, for example, in the example shown in FIG. 6 , the number of target light field cameras can be set to three, and three target light field cameras can cover the viewpoints of both eyes of the user. Of course, those skilled in the art should understand that the number of target light field cameras is not limited to three, and may also be other numbers, which will not be repeated in this disclosure.

S560. The collection device collects light field images through the target light field camera to obtain a light field image group.

After the acquisition device 100 determines the target light field camera, it can use the target light field camera to capture the current scene image, and obtain the light field image collected by each target light field camera, and the light field image group is each target light field A collection of light field images captured by the camera.

In some implementations, after the target light field camera collects the light field images, preprocessing can also be performed on each light field image. The purpose of the preprocessing is to improve the accuracy of the light field image. The preprocessing can include, for example, image calibration, Distortion correction, etc.

In the embodiments of the present disclosure, in order to increase the preprocessing speed of the light field image and further ensure implementation, a circuit topology of a multi-processing chip may be used. For example, in the example shown in FIG. 7, Cam0-Cam8 are the light field cameras in the light field camera array 120 respectively. In the circuit topology, every three adjacent light-field cameras can be grouped into a group according to the numbers of Cam0-Cam8. , the circuit includes 3 processing chips, the input ends of the processing chip Tx_0 are respectively connected to Cam0, Cam3 and Cam6, the input ends of the processing chip Tx_1 are respectively connected to Cam1, Cam4 and Cam7, and the input ends of the processing chip Tx_2 are respectively connected to Cam2, Cam5 and Cam8 .

Therefore, through the circuit structure shown in Figure 7, for any three adjacent target light field cameras, one processing chip can be guaranteed to process one image processing alone, realizing parallel processing of light field image data and improving image preprocessing speed.

S570. The acquisition device sends the light field image group to the display device.

In the embodiments of the present disclosure, the collection device 100 obtains a light field image group after preprocessing the light field images collected by the target light field camera, and then sends the light field image group to the display device 200 .

From the above, it can be seen that in the embodiments of the present disclosure, based on the current target viewpoint information of the user, the target light field camera is determined from the light field camera array to collect light field images, so as to filter out the light field images of redundant viewpoint positions and reduce the synthesis of viewpoint images. The amount of data can improve the efficiency of light field image processing.

For the display device 200 , after receiving the light field image group sent by the acquisition device 100 , it can perform viewpoint image synthesis according to the target viewpoint information and the light field image group. It can be known from the foregoing that in the embodiments of the present disclosure, viewpoint image synthesis is divided into foreground viewpoint image synthesis and background viewpoint image synthesis. Therefore, after the display device 200 obtains the light field image group, it first needs to perform foreground and background image synthesis on the light field image. segmentation. In some implementations, the light field image processing method of the present disclosure includes:

Image segmentation is performed on each light field image in the light field image group to obtain a foreground image and a background image corresponding to each light field image.

It can be understood that the light field image group includes light field images collected by each target light field camera. For example, in the example of FIG. 7 , the light field image group received by the display device 200 includes 3 light field images.

Then, the display device 200 may perform image segmentation on each light field image, and the purpose of the image segmentation is to segment the foreground and background from the light field image. Taking a video communication scene as an example, the foreground is the person in the light field image, and the background is other image parts except the person.

In the embodiments of the present disclosure, there is no limitation on the specific algorithm for image segmentation, and those skilled in the art can use any image segmentation algorithm in the related art to realize the segmentation of foreground and background. However, in order to speed up image segmentation, in some embodiments of the present disclosure, image segmentation may be performed based on a pre-obtained background image.

It can be understood that, referring to the embodiment shown in FIG. 4 , in the process of pre-generating the background disparity map by the acquisition device 100 , the current scene image collected by the acquisition device 100 is the background image that does not include people. In other words, in the aforementioned S410, the acquisition device 100 has acquired the background image of each viewpoint position through each light field camera, and the background image is the preset background image described in the present disclosure.

Therefore, in some implementations, in S430, when the acquisition device 100 sends the background disparity map to the display device 200, at the same time, the scene images of each viewpoint position collected by each light field camera are sent to the display device 200, and the display device 200 These scene images are received and stored, and these scene images are used as the preset background images in the present disclosure.

Therefore, in the process of image segmentation, for a light field image at a certain viewpoint position, image difference can be performed on the light field image according to the pre-obtained preset background image corresponding to the viewpoint position, so that the same image position of the two is The background area and the different image positions of the two are the foreground area. Based on this principle, the foreground and background segmentation of each light field image can be quickly realized, and the image segmentation speed can be improved.

After image segmentation of each light field image through the above process, the foreground image and background image corresponding to each light field image can be obtained, and then based on the foreground image and background image, the foreground viewpoint image synthesis and background viewpoint image synthesis are realized, as follows Described separately.

As shown in FIG. 8 , in some implementations, the light field image processing method described in the present disclosure includes:

S810. For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, the first foreground image corresponding to the first light field image and the second light field image corresponding to the second light field image Perform disparity estimation on the foreground image to obtain a first foreground disparity map corresponding to the first foreground image and a second foreground disparity map corresponding to the second foreground image.

S820. Perform disparity mapping on the first foreground image based on the first foreground disparity map to obtain a first foreground map, and perform disparity mapping on the second foreground image based on the second foreground disparity map to obtain a second foreground map.

S830. Perform image fusion processing on the first foreground map and the second foreground map based on the target viewpoint information to obtain a foreground viewpoint image.

It is worth noting that in order to realize the synthesis of viewpoint images, it is first necessary to calculate the disparity maps of the images of two adjacent viewpoint positions. In the implementation of the present disclosure, two light field images of any adjacent viewpoint positions will be taken as an example , the process of synthesizing the two foreground viewpoint images and background viewpoint images will be described.

For example, in the example shown in Fig. 7, the light field image group includes three light field images in total, namely the first light field image I1, the second light field image I2 and the third light field image I3, wherein the first light field image I1 and The second light field image I2 is a light field image at an adjacent viewpoint position, and the second light field image I2 and the third light field image I3 are light field images at adjacent viewpoint positions. In the following embodiments of the present disclosure, the first light field image I1 and the second light field image I2 will be described as examples, and the second light field image I2 and the third light field image I3 can refer to the same method steps. This will not be repeated here.

In combination with the foregoing, it can be seen that by performing image segmentation on the first light field image I1, the first foreground image I _fore 1 and the first background image I _back 1 corresponding to the first light field image I1 can be obtained. Similarly, for the second light field image I2 Performing image segmentation can obtain a second foreground image I _fore 2 and a second background image I _back 2 corresponding to the second light field image I2.

For foreground viewpoint image synthesis, it is first necessary to determine a disparity map based on the first foreground image I _fore 1 and the second foreground image I _fore 2, which will be described below with reference to FIG. 9 .

As shown in FIG. 9 , in some implementations, the process of obtaining the first foreground disparity map and the second foreground disparity map in the light field image processing method of the example of the present disclosure includes:

S811. Perform down-sampling on the first foreground image based on a preset down-sampling coefficient to obtain a first down-sampled image, and perform down-sampling on the second foreground image to obtain a second down-sampled image.

It is worth noting that, in the embodiments of the present disclosure, considering that the scale of the original image of the light field image is large, if the disparity map is estimated based on the scale of the original image, the amount of data is large, so the foreground image of the scale of the original image can be down-sampled , so as to reduce the size of the image, reduce the amount of data, and improve the operation speed.

Referring to Fig. 10, firstly, the first foreground image I _fore 1 of the original image scale can be down-sampled to obtain the first down-sampled image I _fore 1', and the second foreground image I _fore 2 of the original image scale can also be down-sampled A second downsampled image I _fore 2' is obtained.

It is worth noting that the preset downsampling factor refers to the magnification of image reduction. This factor can be set according to specific scenarios. The larger the image magnification, the faster the calculation speed and the higher the accuracy. The smaller the image magnification, the slower the calculation speed. The higher the accuracy, based on this rule, in an example, the preset downsampling coefficient can be selected as 1/4, 1/8, 1/16, etc.

S812. Match the positions of the same pixels on the first downsampled image and the second downsampled image to obtain a first disparity map corresponding to the first downsampled image and a second disparity map corresponding to the second downsampled image.

In the embodiment of the present disclosure, after downsampling the first foreground image I _fore 1 and the second foreground image I _fore 2, the first downsampled image I _fore 1' and the second downsampled image after downsampling can be used I _fore 2' performs disparity estimation. The basic principle of disparity estimation is to match the position of the same pixel on the image to determine the pixel offset distance, that is, the pixel value on the disparity map represents the pixel offset distance.

Taking the first foreground image I _fore 1 as an example, the first disparity map I _forepara 1' corresponding to the first foreground image I _fore 1 means that a certain pixel on the first foreground image I _fore 1 is relative to the second foreground image I _fore The offset distance of the same pixel on 2.

Therefore, the parallax estimation can be realized through the above pixel matching process, and the first parallax map I _forepara 1' corresponding to the first foreground image I _fore 1, and the second parallax map I _forepara 2 corresponding to the second foreground image I _fore 2 are obtained. '. The first disparity map I _forepara 1' and the second disparity map I _forepara 2' may be shown in FIG. 10 .

S813. Determine a first disparity search range according to the first disparity map and a preset downsampling coefficient, and determine a second disparity search range according to the second disparity map and a preset downsampling coefficient.

It can be seen from FIG. 10 that the image scales of the first parallax map I _forepara 1' and the second parallax map I _forepara 2' are the same as the downsampled image scale, so it is necessary to combine the first parallax map I _forepara 1' and the second parallax map I _forepara 2'revert to the original scale.

Taking the first parallax map I _forepara 1' as an example, it can be understood that since the scale of the first parallax map I _forepara 1' is small, each pixel on the first parallax map I _forepara 1' corresponds to a range on the original image. In the embodiment of the present disclosure, the first disparity map I _forepara 1' can be divided into blocks, and the disparity search range (min', max') of the corresponding pixels of each image block is determined in units of image blocks. The disparity search range (min', max') indicates that the image block corresponds to the pixel range on the size of the original image, and the specific value of the disparity search range can be determined in combination with the maximum and minimum values of the pixel values of the image block.

Then, the disparity search range (min', max') is restored to the scale of the original image, that is, the minimum and maximum values of the disparity search range (min', max') are multiplied by the reciprocal of the preset sampling coefficient to obtain The first disparity search range (min, max), the first disparity search range (min, max) means the disparity search range on the scale of the original image. Similarly, the second parallax search range can be obtained by processing the second parallax map I _forepara 2' according to the above process.

S814. Perform disparity estimation on the first foreground image based on the first disparity search range to obtain a first foreground disparity map, and perform disparity estimation on the second foreground image based on the second disparity search range to obtain a second foreground disparity map.

In the embodiments of the present disclosure, taking the first foreground image I _fore 1 as an example, after the first disparity search range (min, max) is obtained through the foregoing, the first disparity search range represents the comparison between the first foreground image I _fore 1 and the first foreground image The search range for matching the same pixels on the two foreground images I _fore 2, so that the first foreground image I _fore 1 is subjected to parallax estimation based on the first disparity search range, and the first foreground corresponding to the first foreground image I _fore 1 can be obtained Disparity map I _forepara 1. Similarly, by performing disparity estimation on the second foreground image I _fore 2 based on the second disparity search range, the second foreground disparity map I _forepara 2 corresponding to the second foreground image I _fore 2 can be obtained.

From the above, it can be seen that in the embodiments of the present disclosure, by performing disparity estimation after down-sampling the foreground image, the amount of data can be effectively reduced, the image processing efficiency can be improved, and the real-time requirement of the video communication scene can be met.

In the embodiment of the present disclosure, after obtaining the first foreground parallax map I _forepara 1 and the second foreground parallax map I _forepara 2 corresponding to the first foreground image I _fore 1 and the second foreground image I _fore 2 respectively, the foreground parallax Figure 10 performs viewpoint synthesis on the foreground image to obtain a foreground viewpoint image at a new viewpoint position, which will be described below with reference to FIG. 10 .

As shown in FIG. 11 , in some implementations, the light field image processing method described in the present disclosure includes:

S1110. Match each pixel on the first foreground image to the corresponding pixel position on the map according to the first foreground disparity map to obtain the first foreground map; match each pixel on the second foreground image according to the second foreground disparity map Pixels are matched to corresponding pixel positions on the map to obtain a second foreground map.

In combination with the foregoing, it can be seen that the disparity map is an image that reflects the pixel position offset due to the difference in viewpoint position, and each pixel on the disparity map represents the disparity value of the pixel at the position on the original image. The first foreground disparity map I _forepara 1 and Take the second foreground disparity map I _forepara 2 as an example. In the case of ignoring the calculation error, the pixel coordinates on the first foreground image I _fore 1 plus the disparity value on the first foreground disparity map I _forepara 1 are the second foreground The pixel coordinates on the image I _fore 2, likewise, the pixel coordinates on the second foreground image I _fore 2 plus the parallax value on the second foreground disparity map I _forepara 2 is the pixel on the first foreground image I _fore 1 coordinate.

Therefore, after obtaining the first foreground parallax map I _forepara 1 and the second foreground parallax map I _forepara 2, the first foreground image I _fore can be processed according to the first foreground parallax map I _forepara 1 and the second foreground parallax map I _forepara 2 1 and the second foreground image I _fore 2 are subjected to pixel offset processing, and the pixel offset processing process is the pixel coordinate mapping process. After the first foreground image I _fore 1 is mapped, the first foreground map I _foremap 1 can be obtained, and after the second foreground image I _fore 2 is mapped, the first foreground map I _foremap 2 can be obtained.

Taking the first foreground map I _foremap 1 as an example, each pixel on the first foreground image I _fore 1 can be offset according to the disparity value on the first foreground disparity map I _forepara 1, so as to obtain A pixel position on the foreground map I _foremap 1, complete the pixel offset processing from the first foreground image I _fore 1 to the first foreground map I _foremap 1, and obtain the first foreground map I _foremap 1. Similarly, the second foreground image I fore 2 is mapped according to the second foreground disparity map I _forepara 2 in the same _process to obtain the corresponding second foreground map I _foremap 2 .

It is worth noting that, before performing the mapping process on the first foreground image I _fore 1 and the second foreground image I _fore 2, the first foreground parallax map I _forepara 1 and the second foreground parallax map I _forepara 2 can also be consistent Verification, the purpose of consistency verification is to correct the first foreground disparity map I _forepara 1 and the second foreground disparity map I _forepara 2, so as to improve the accuracy of the disparity map. For the consistency verification process, those skilled in the art may refer to related technologies, which will not be repeated in this disclosure.

S1120. Determine a first weight and a second weight according to the target viewpoint information and the position information of any two adjacent target light field cameras.

S1130. Based on the first weight and the second weight, perform image fusion processing on the first foreground map and the second foreground map to obtain a foreground viewpoint image.

In combination with the foregoing, it can be seen that the target viewpoint information refers to the user's current new viewpoint position, and it is necessary to combine the target viewpoint information to perform viewpoint image synthesis on the first foreground map I _foremap 1 and the second foreground map I _foremap 2, so as to obtain the user's new The foreground viewpoint image at the viewpoint position.

When performing viewpoint image synthesis on the first foreground map I _foremap 1 and the second foreground map I _foremap 2, it is first necessary to determine the weights of the first foreground map I _foremap 1 and the second foreground map I _foremap 2, the The weight represents the fusion coefficient of the first foreground map I _foremap 1 and the second foreground map I _foremap 2 .

It can be understood that the first foreground map I _foremap 1 corresponds to the viewpoint position of a target light field camera, the second foreground map I _foremap 2 corresponds to the viewpoint position of another target light field camera, and the viewpoint position corresponding to the target viewpoint information, It may appear at any position between the two. The new viewpoint position is different from the viewpoint position of the target light field camera, and the corresponding weights should also be different.

Based on this principle, in the embodiments of the present disclosure, the fusion coefficients for the left and right foreground maps, that is, the first weight and the second weight, can be determined according to the target viewpoint information and the position information of the left and right target light field cameras . For example, if the target viewpoint information is closer to the viewpoint position of the first foreground map I _foremap 1, the first weight w1 corresponding to the first foreground map I _foremap 1 is larger; otherwise, if the target viewpoint information is farther away from the second foreground map The viewpoint position of the graph I _foremap 2 is closer, and the second weight value w2 corresponding to the second foreground map graph I _foremap 2 is larger. Accordingly, those skilled in the art may determine specific values of the first weight w1 and the second weight w2 in combination with the target viewpoint information, which is not limited in the present disclosure.

After determining the first weight w1 and the second weight w2 according to the target viewpoint information, the first foreground map I _foremap 1 and the second foreground map I _foremap can be calculated according to the first weight w1 and the second weight w2 2 for fusion processing, the process of fusion processing can be expressed as:

P = w1* I _foremap 1+ w2* I _foremap 2 (1)

In formula (1), P represents the foreground viewpoint image obtained after fusion processing. Through the above process, the first foreground image I _fore 1 and the second foreground image I _fore 2 are combined with the target viewpoint information to perform viewpoint image synthesis to obtain the foreground viewpoint image P corresponding to the new viewpoint position.

As shown in FIG. 12 , in some implementations, the light field image processing method described in the present disclosure includes:

S1210. For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, based on the pre-generated first background disparity map at the same viewpoint position as the first light field image, perform the second light field image Perform parallax mapping on the first background image of a light field image to obtain the first background map, and based on the pre-generated second background parallax map at the same viewpoint position as the second light field image, the second background image of the second light field image Perform parallax mapping to obtain a second background map.

S1220. Perform image fusion processing on the first background map and the second background map based on the target viewpoint information to obtain a background viewpoint image.

For the synthesis of background viewpoint images, still take the first light field image I1 and the second light field image I2 of adjacent viewpoint positions in the light field image group as an example, the first light field image I1 obtained through image segmentation A background image I _back 1, and a second background image I _back 2 of the second light field image I2.

When performing background viewpoint image synthesis on the first background image I _back 1 and the second background image I _back 2, there is no need to calculate the foreground disparity map in the process of foreground viewpoint image synthesis, because through the implementation of the aforementioned Figure 4, the offline stage pre-generates the background disparity map for each viewpoint location, eliminating the need for online computation.

Therefore, for background viewpoint image synthesis, firstly, according to the viewpoint position of the first background image I _back 1, the first background disparity map I _backpara 1 with the same viewpoint position can be determined from the background disparity map. Similarly, according to the second background image I backpara 1 For the viewpoint position of the image I _back 2, a second background disparity map I _backpara 2 with the same viewpoint position is determined from the background disparity map.

After the background disparity map is determined, the first background image I _back 1 and the second background image I _back 2 can be synthesized by combining the new viewpoint position of the target viewpoint information, and the sum process is the same as the aforementioned foreground viewpoint image synthesis principle The same, only briefly described below.

First, each pixel on the first background image I _back 1 can be shifted according to the parallax value on the first background parallax map I _backpara 1, and the process of the shift processing is the mapping process for pixel coordinates , so as to obtain the pixel position on the first background map I _backmap 1, complete the pixel offset processing from the first background image I _back 1 to the first background map I _backmap 1, and obtain the first background map I _backmap 1 . Similarly, the second background image I back 2 is mapped according to the second background parallax map I _backpara 2 in the same _process to obtain the corresponding second background map I _backmap 2 .

It is worth noting that, before performing mapping processing on the first background image I _back 1 and the second background image I _back 2, the first background parallax map I _backpara 1 and the second background parallax map I _backpara 2 can also be consistent Verification, the purpose of the consistency verification is to correct the first background parallax map I _backpara 1 and the second background parallax map I _backpara 2, so as to improve the accuracy of the parallax map. For the consistency verification process, those skilled in the art may refer to related technologies, which will not be repeated in this disclosure.

Then, according to the target viewpoint information and the position information of the left and right target light field cameras, the fusion coefficients for the left and right background maps can be determined, and the fusion coefficients are also the aforementioned first weight w1 and second weight w2, I won't repeat it here. After that, the first background map I _backmap 1 and the second background map I _backmap 2 can be fused according to the first weight w1 and the second weight w2, and the fusion process can be expressed as:

Q = w1* I _backmap 1+ w2* I _backmap 2 (2)

In formula (2), Q represents the background viewpoint image obtained after fusion processing. Through the above process, the viewpoint image synthesis is performed on the first background image I _back 1 and the second background image I _back 2 combined with the target viewpoint information to obtain the background viewpoint image Q corresponding to the new viewpoint position.

Through the above process, the foreground viewpoint image P and the background viewpoint image Q are obtained, and then the foreground viewpoint image P and the background viewpoint image Q are fused, for example, the background viewpoint image Q is pasted on the foreground viewpoint image P to obtain the final corresponding new The target light field image at the viewpoint position. The above description is only based on the viewpoint image synthesis between the first light field image I1 and the second light field image I2. The viewpoint image synthesis process between any other two adjacent light field images is the same. The target light field image at each viewpoint position included in the user's target viewpoint information can then be obtained, which will not be described in detail in this disclosure.

After obtaining one or more target light field images, the target light field images can be rendered and displayed on the display screen 110 of the display device 200. For the process of image rendering and display, those skilled in the art can understand by referring to related technologies. Let me repeat.

From the above, it can be seen that in the embodiments of the present disclosure, the disparity map of the background image is generated in advance through the method of foreground and background segmentation, so that there is no need to calculate the disparity map for the complex background in real time, greatly reducing the amount of data for viewpoint image synthesis, and improving image processing speed and Accuracy, real-time light field video communication can be realized. Moreover, in the process of calculating the foreground disparity map, the disparity estimation is performed based on the downsampled small image, which further reduces the calculation amount and improves the data processing speed. In addition, the target light field camera is selected from the light field camera array through the target viewpoint information, and the operation of collecting, transmitting, and processing a large amount of redundant data is eliminated, the image processing speed is further improved, and low-latency video communication is realized.

In some implementations, the present disclosure provides a light field image processing device, which can be applied to a display device 200, as shown in FIG. 13 , the light field image processing device in the example of the present disclosure includes:

The acquisition module 10 is configured to acquire target viewpoint information and a light field image group sent by the acquisition device; the target viewpoint information represents the position information of the observer's eyes of the display device, and the light field image group includes each target light field camera in the acquisition device The collected light field image;

The foreground fusion module 20 is configured to perform viewpoint fusion on the foreground images in each light field image based on the target viewpoint information, to obtain the foreground viewpoint image corresponding to the target viewpoint information;

The background fusion module 30 is configured to perform viewpoint fusion on the background images in each light field image based on the target viewpoint information and the pre-generated background disparity map of each viewpoint position, to obtain the background viewpoint image corresponding to the target viewpoint information; viewpoint position represents the position corresponding to each target light field camera;

The image synthesis module 40 is configured to generate a target light field image according to the foreground viewpoint image and the background viewpoint image.

In some embodiments, the foreground fusion module 20 is configured to:

performing image segmentation on each light field image in the light field image group to obtain a foreground image corresponding to each light field image;

For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, the first foreground image corresponding to the first light field image and the second foreground image corresponding to the second light field image Perform parallax estimation to obtain a first foreground parallax map corresponding to the first foreground image and a second foreground parallax map corresponding to the second foreground image;

performing parallax mapping on the first foreground image based on the first foreground disparity map to obtain a first foreground map, and performing parallax mapping on the second foreground image based on the second foreground disparity map to obtain a second foreground map;

Image fusion processing is performed on the first foreground map and the second foreground map based on the target viewpoint information to obtain a foreground viewpoint image.

In some embodiments, the foreground fusion module 20 is configured to:

For each light field image in the light field image group, based on a pre-generated preset background image at the same viewpoint position as the light field image, image difference is performed on the light field image to obtain a foreground image and a background image corresponding to the light field image.

In some embodiments, the foreground fusion module 20 is configured to:

downsampling the first foreground image based on a preset downsampling coefficient to obtain a first downsampling image, and downsampling the second foreground image to obtain a second downsampling image;

Matching the positions of the same pixels on the first downsampled image and the second downsampled image to obtain a first disparity map corresponding to the first downsampled image and a second disparity map corresponding to the second downsampled image;

determining a first disparity search range according to the first disparity map and a preset downsampling coefficient, and determining a second disparity search range according to the second disparity map and a preset downsampling coefficient;

Performing disparity estimation on the first foreground image based on the first disparity search range to obtain a first foreground disparity map, and performing disparity estimation on the second foreground image based on the second disparity search range to obtain a second foreground disparity map.

In some embodiments, the foreground fusion module 20 is configured to:

Match each pixel on the first foreground image to the corresponding pixel position on the map according to the first foreground disparity map to obtain the first foreground map; match each pixel on the second foreground image according to the second foreground disparity map to the corresponding pixel position on the map to obtain the second foreground map;

determining the first weight and the second weight according to the target viewpoint information and the position information of any two adjacent target light field cameras;

Based on the first weight and the second weight, image fusion processing is performed on the first foreground map and the second foreground map to obtain a foreground viewpoint image.

In some embodiments, the background fusion module 30 is configured to:

Image segmentation is performed on each light field image in the light field image group to obtain a background image corresponding to each light field image;

For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, based on the pre-generated first background disparity map at the same viewpoint position as the first light field image, the first light field Perform parallax mapping on the first background image of the field image to obtain the first background map, and perform parallax on the second background image of the second light field image based on the pre-generated second background parallax map at the same viewpoint position as the second light field image Mapping to obtain a second background map;

Image fusion processing is performed on the first background map and the second background map based on the target viewpoint information to obtain a background viewpoint image.

In some embodiments, the acquisition module 10 is configured to:

Acquiring scene images through an image acquisition device arranged on the display device;

Perform image detection based on the scene image to obtain the position information of the observer's eyes in the scene image;

Target viewpoint information is generated based on the position information.

In some embodiments, the acquisition module 10 is configured to:

sending the target viewpoint information to the acquisition device, so that the acquisition device determines one or more target light field cameras from multiple light field cameras according to the target viewpoint information;

Receive the light field image group sent by the acquisition device.

In some embodiments, the acquisition module 10 is configured to:

The background disparity map of each viewpoint position sent by the acquisition device is received and stored.

From the above, it can be seen that in the embodiments of the present disclosure, the disparity map of the background image is generated in advance through the method of foreground and background segmentation, so that there is no need to calculate the disparity map for the complex background in real time, greatly reducing the amount of data for viewpoint image synthesis, and improving image processing speed and Accuracy, real-time light field video communication can be realized. Moreover, in the process of calculating the foreground disparity map, the disparity estimation is performed based on the downsampled small image, which further reduces the calculation amount and improves the data processing speed. In addition, the target light field camera is selected from the light field camera array through the target viewpoint information, and the operation of collecting, transmitting, and processing a large amount of redundant data is eliminated, the image processing speed is further improved, and low-latency video communication is realized.

In some implementations, the present disclosure provides a light field image processing device, which can be applied to the acquisition device 100, as shown in FIG. 14 , the light field image processing device in the example of the present disclosure includes:

The image acquisition module 50 is configured to respectively acquire the current scene image through a plurality of light field cameras arranged on the acquisition device, and obtain the scene image of the viewpoint position corresponding to each light field camera;

The parallax determination module 60 is configured to, for any two adjacent light field cameras, generate a background parallax map of the viewpoint position of each light field camera according to the scene images respectively collected by the two light field cameras;

The sending module 70 is configured to send the background disparity map of each viewpoint position to the display device, so that the display device stores the background disparity map of each viewpoint position.

In some implementations, the sending module 70 is configured to:

Receive the target viewpoint information sent by the display device;

Determining one or more target light field cameras from multiple light field cameras included in the acquisition device according to the target viewpoint information;

The light field image is collected by the target light field camera to obtain a light field image group, and the light field image group is sent to a display device.

From the above, it can be seen that in the embodiments of the present disclosure, the disparity map of the background image is generated in advance through the method of foreground and background segmentation, so that there is no need to calculate the disparity map for the complex background in real time, greatly reducing the amount of data for viewpoint image synthesis, and improving image processing speed and Accuracy, real-time light field video communication can be realized. Moreover, in the process of calculating the foreground disparity map, the disparity estimation is performed based on the downsampled small image, which further reduces the calculation amount and improves the data processing speed. In addition, the target light field camera is selected from the light field camera array through the target viewpoint information, and the operation of collecting, transmitting, and processing a large amount of redundant data is eliminated, the image processing speed is further improved, and low-latency video communication is realized.

In some implementations, the present disclosure provides a video communication system. The video communication system may be as shown in FIG. 1 , which includes:

The display device 200 includes an image acquisition device and a first controller, and the first controller is configured to execute the method in any of the foregoing implementation manners;

The acquisition device 100 includes a light field camera array and a second controller, the light field camera array includes a plurality of light field cameras, and the second controller is used to execute the method in any of the above-mentioned implementation manners;

In some implementations, the present disclosure provides a storage medium storing computer instructions for causing a computer to execute the method in any of the above implementations.

In some embodiments, the present disclosure provides an electronic device comprising:

processor; and

The memory stores computer instructions, and the computer instructions are used to cause the processor to execute the method in any of the above implementation manners.

In the embodiments of the present disclosure, the electronic device may be the above-mentioned collection device 100 or the display device 200 , which is not limited in the present disclosure. Specifically, FIG. 15 shows a schematic structural diagram of an electronic device 600 suitable for implementing the method of the present disclosure. Through the electronic device shown in FIG. 15 , corresponding functions of the aforementioned processor, controller, and storage medium can be realized.

As shown in FIG. 15 , the electronic device 600 includes a processor 601 that can perform various appropriate actions and processes according to programs stored in the memory 602 or loaded from the storage part 608 into the memory 602 . In the memory 602, various programs and data necessary for the operation of the electronic device 600 are also stored. The processor 601 and the memory 602 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to the bus 604 .

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 608 including a hard disk, etc. and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary so that a computer program read therefrom is installed into the storage section 608 as necessary.

In particular, according to the embodiments of the present disclosure, the above method process can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method described above. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 609 and/or installed from a removable medium 611 .

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code that contains one or more Executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

Apparently, the above-mentioned implementation manners are only examples for clear description, rather than limiting the implementation manners. For those of ordinary skill in the art, other changes or changes in different forms can be made on the basis of the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. And the obvious changes or changes derived therefrom are still within the scope of protection of the present disclosure.

Claims (16)

1. A light field image processing method, characterized in that being applied to a display device, said method comprising: Obtain target viewpoint information and a light field image group sent by the acquisition device; the target viewpoint information represents the position information of the observer's eyes of the display device, and the light field image group includes each target light field in the acquisition device The light field image collected by the camera; performing viewpoint fusion on foreground images in each light field image based on the target viewpoint information, to obtain foreground viewpoint images corresponding to the target viewpoint information; Based on the target viewpoint information and the pre-generated background disparity map of each viewpoint position, perform viewpoint fusion on the background images in each light field image to obtain the background viewpoint image corresponding to the target viewpoint information; the viewpoint position represents A location corresponding to each target light field camera; A target light field image is generated according to the foreground viewpoint image and the background viewpoint image. 2. The method according to claim 1, wherein, performing viewpoint fusion on the foreground images in each light field image based on the target viewpoint information, to obtain the corresponding foreground viewpoint image of the target viewpoint information, comprising : performing image segmentation on each light field image in the light field image group to obtain a foreground image corresponding to each light field image; For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, the first foreground image corresponding to the first light field image and the first foreground image corresponding to the second light field image Performing disparity estimation on the second foreground image to obtain a first foreground disparity map corresponding to the first foreground image and a second foreground disparity map corresponding to the second foreground image; performing parallax mapping on the first foreground image based on the first foreground disparity map to obtain a first foreground map, and performing parallax mapping on the second foreground image based on the second foreground disparity map to obtain a second foreground map; performing image fusion processing on the first foreground map and the second foreground map based on the target viewpoint information to obtain the foreground viewpoint image. 3. The method according to claim 2, wherein said performing image segmentation on each light field image in said light field image group to obtain a foreground image corresponding to each light field image comprises: For each light field image in the light field image group, based on a pre-generated preset background image at the same viewpoint position as the light field image, image difference is performed on the light field image to obtain the light field image Corresponding foreground and background images. 4. The method according to claim 2, wherein the parallax estimation is performed on the first foreground image corresponding to the first light field image and the second foreground image corresponding to the second light field image to obtain The first foreground disparity map corresponding to the first foreground image and the second foreground disparity map corresponding to the second foreground image include: performing downsampling on the first foreground image based on a preset downsampling coefficient to obtain a first downsampling image, and performing downsampling on the second foreground image to obtain a second downsampling image; matching the positions of the same pixels on the first downsampled image and the second downsampled image to obtain a first disparity map corresponding to the first downsampled image and a second disparity map corresponding to the second downsampled image; determining a first disparity search range according to the first disparity map and the preset downsampling coefficient, and determining a second disparity search range according to the second disparity map and the preset downsampling coefficient; Performing disparity estimation on the first foreground image based on the first disparity search range to obtain the first foreground disparity map, and performing disparity estimation on the second foreground image based on the second disparity search range to obtain the second foreground image Foreground disparity map. 5. The method according to claim 2, characterized in that, performing parallax mapping on the first foreground image based on the first foreground disparity map to obtain a first foreground map, and performing parallax mapping on the first foreground image based on the second foreground parallax map Two foreground images are subjected to parallax mapping to obtain a second foreground map, including: matching each pixel on the first foreground image to a corresponding pixel position on the map according to the first foreground disparity map to obtain the first foreground map; according to the second foreground disparity map matching each pixel on the second foreground image to a corresponding pixel position on the map to obtain the second foreground map; The performing image fusion processing on the first foreground map and the second foreground map based on the target viewpoint information to obtain the foreground viewpoint image includes: determining a first weight and a second weight according to the target viewpoint information and the position information of any two adjacent target light field cameras; Based on the first weight and the second weight, image fusion processing is performed on the first foreground map and the second foreground map to obtain the foreground viewpoint image. 6. The method according to claim 1, wherein the background disparity map of each viewpoint position based on the target viewpoint information and the pre-generated viewpoint fusion is carried out to the background image in each light field image to obtain The background viewpoint image corresponding to the target viewpoint information; the viewpoint position represents a position corresponding to each target light field camera, including: performing image segmentation on each light field image in the light field image group to obtain a background image corresponding to each light field image; For the first light field image and the second light field image corresponding to any two adjacent target light field cameras, based on the pre-generated first background disparity map at the same viewpoint position as the first light field image, the performing parallax mapping on the first background image of the first light field image to obtain a first background map, and based on the pre-generated second background parallax map at the same viewpoint position as the second light field image, performing parallax mapping on the second background image of the image to obtain a second background map; performing image fusion processing on the first background map and the second background map based on the target viewpoint information to obtain the background viewpoint image. 7. The method according to claim 1, wherein said acquiring target viewpoint information comprises: collecting scene images through an image collection device arranged on the display device; performing image detection according to the scene image to obtain position information of the observer's eyes in the scene image; The target viewpoint information is generated based on the position information. 8. The method according to claim 1, wherein the process of obtaining the light field image group sent by the acquisition device comprises: sending the target viewpoint information to the acquisition device, so that the acquisition device determines one or more target light field cameras from a plurality of light field cameras according to the target viewpoint information; The light field image group sent by the acquisition device is received. 9. The method of claim 1, further comprising: The background disparity map of each viewpoint position sent by the collection device is received and stored. 10. A light field image processing method, characterized in that it is applied to an acquisition device, the method comprising: Collecting the current scene image respectively by a plurality of light field cameras arranged on the collection device, to obtain the scene image of the viewpoint position corresponding to each light field camera; For any two adjacent light field cameras, according to the scene images collected by the two light field cameras respectively, a background disparity map of the viewpoint position of each light field camera is generated; The background disparity map of each viewpoint position is sent to the display device, so that the display device stores the background disparity map of each viewpoint position. 11. The method of claim 10, further comprising: receiving target viewpoint information sent by the display device; determining one or more target light field cameras from the plurality of light field cameras included in the acquisition device according to the target viewpoint information; Collecting light field images by the target light field camera to obtain a light field image group, and sending the light field image group to the display device. 12. A light field image processing device, characterized in that it is applied to a display device, the device comprising: An acquisition module configured to acquire target viewpoint information and a light field image group sent by the acquisition device; the target viewpoint information represents the position information of the observer's eyes of the display device, and the light field image group includes the acquisition device The light field image collected by each target light field camera in ; The foreground fusion module is configured to perform viewpoint fusion on the foreground images in each light field image based on the target viewpoint information, to obtain the foreground viewpoint image corresponding to the target viewpoint information; The background fusion module is configured to perform viewpoint fusion on the background images in each light field image based on the target viewpoint information and the pre-generated background disparity map of each viewpoint position, to obtain the background viewpoint corresponding to the target viewpoint information image; the viewpoint position represents a position corresponding to each target light field camera; An image synthesis module configured to generate a target light field image according to the foreground viewpoint image and the background viewpoint image. 13. A light field image processing device, characterized in that it is applied to acquisition equipment, and the device comprises: The image acquisition module is configured to respectively acquire the current scene image through a plurality of light field cameras arranged on the acquisition device, so as to obtain the scene image of the viewpoint position corresponding to each light field camera; The parallax determination module is configured to generate a background parallax map of the viewpoint position of each light field camera for any two adjacent light field cameras according to the scene images respectively collected by the two light field cameras; The sending module is configured to send the background disparity map of each viewpoint position to the display device, so that the display device stores the background disparity map of each viewpoint position. 14. An electronic device, characterized in that it comprises: processor; and A memory storing computer instructions for causing the processor to execute the method according to any one of claims 1 to 9, or to execute the method according to any one of claims 10 to 11. 15. A video communication system, comprising: A display device, comprising an image acquisition device and a first controller, the first controller is configured to execute the method according to any one of claims 1 to 9; The collection device includes a light field camera array and a second controller, the light field camera array includes a plurality of light field cameras, and the second controller is used to execute the method according to any one of claims 10 to 11. 16. A storage medium, characterized in that computer instructions are stored, and the computer instructions are used to make the computer execute the method according to any one of claims 1 to 9, or execute the method according to any one of claims 10 to 11. the method described.

Images