JP2019146010A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a display image with a little uncomfortable feeling for a display system that displays an image on a wide-field display unit. A display system including a display unit including a first acquisition unit that acquires an imaging angle of view when an input image is captured, and a display angle that is a prospective angle when the display unit is viewed from a viewpoint position. Rendering the projection plane based on a correspondence between the input image and the projection plane, second acquisition means for acquiring the display system information to be displayed, setting means for setting the projection plane in the virtual space, Generating means for generating a display image to be displayed on the display unit, the generating means sets a virtual viewpoint position in the virtual space based on the shooting angle of view, and the virtual viewpoint The projection plane is rendered from a position. [Selection] Figure 2

Description

  The present invention relates to a technique for generating an image for a display system that displays a wide-field image.

  2. Description of the Related Art Conventionally, as one display system for displaying an image, a system that gives a high sense of realism to a viewer by displaying the image on a display screen arranged so as to cover the viewer's field of view is known.

  Patent Document 1 describes a method of displaying an image on a spherical wide viewing angle screen with a concave surface facing the viewer. According to the method described in Patent Document 1, an image to be displayed on a screen is generated by performing a mapping process in which a planar image is pasted into a spherical shape.

JP 2007-318754 A

  In order to generate an image with a wide field of view, it is desirable that the image be captured with a wide shooting angle of view. However, there are cases where the angle of view of the image is not sufficient for the angle at which the viewer looks at the display unit (hereinafter referred to as the display angle). In such a case, when image processing for expanding the angle of view of the image so as to satisfy the display angle is performed on the image, the prior art does not consider the viewer's viewpoint position and the like. Will cause unnatural distortion and give viewers a sense of incongruity.

  SUMMARY OF THE INVENTION An object of the present invention is to generate a display image with less discomfort for a display system that displays an image on a wide-field display unit.

  In order to solve the above problems, the present invention provides a first acquisition unit that acquires a shooting angle of view when an input image is captured, and a display system that includes a display unit. A second acquisition unit configured to acquire display system information indicating a display angle, a setting unit configured to set a projection plane in a virtual space, and the projection plane based on a correspondence relationship between the input image and the projection plane. Generating means for generating a display image to be displayed on the display unit, and the generating means sets a virtual viewpoint position in the virtual space based on the shooting angle of view. The projection plane is rendered from the virtual viewpoint position.

  With respect to a display system that displays an image on a wide-field display unit, it is possible to generate a display image with less discomfort.

The figure which shows the hardware constitutions of an image processing apparatus Block diagram showing functional configuration of image processing apparatus A flowchart showing a flow of processing executed in the image processing apparatus A diagram showing a configuration example of a display system Flowchart showing the flow of display image generation processing The figure which shows the relationship between a display part and a projection surface The figure which shows the correspondence of a plane projection surface and an input image The figure which looked at the relation between the display section and the projection plane in the XZ plane Diagram showing how to set the virtual viewpoint position

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not necessarily limit the present invention, and not all combinations of features described in the present embodiment are essential for the solution means of the present invention.

<First Embodiment>
In the present embodiment, an example of an image processing apparatus for a display system that displays an image with a wide field of view by arranging three planar displays capable of displaying an image so as to cover the viewer's field of view will be described. To do. FIG. 4 shows an example of a display system assumed in this embodiment. In the display system according to the present embodiment, a display unit that displays an image is configured by three displays, the center display 401, the left display 402, and the right display 403. The center display 401, the left display 402, and the right display 403 are arranged so as to draw an isosceles trapezoid when viewed from above. Each display uses a self-luminous device such as a liquid crystal display. The center display 401 is disposed in front of the viewer. The left display 402 is disposed so as to be in contact with the left end as viewed from the viewer of the center display 401 and an angle formed with the center display 401 has an angle θsc. Similarly, the right display 403 is arranged so as to be in contact with the right end as viewed from the viewer of the center display 401 and to form an angle θsc with the center display 401. Therefore, the three displays are arranged so that the normal from each display surface has an intersection. And the display image corresponding to each is displayed on the intersection side of each display surface. By arranging three planar display screens (displays) in this way, the display unit covers the viewer's field of view. When an image is displayed on this display system, it is possible to give the viewer a sense of presence as if he / she was in a place where the displayed image was captured. In the present embodiment, an image processing apparatus that generates a display image to be displayed on the display system shown in FIG. 4 will be described.

  In the present embodiment, the angle at which the viewer looks at the image displayed on the display unit (three displays) in the display system is called a display angle. In addition, the display images displayed on each display are all generated from a common input image. In other words, in the present embodiment, three display images are generated based on an input image captured using one imaging device (for example, a digital camera).

  FIG. 1 shows a hardware configuration of an image processing apparatus according to this embodiment. The CPU 101 uses the RAM 102 as a work memory, executes programs stored in the ROM 103 and the hard disk drive (HDD) 105, and controls each component to be described later via the system bus 100. Thereby, various processes described later are executed. The HDD I / F 104 is an interface such as serial ATA (SATA), for example, and connects a secondary storage device such as the HDD 105 or an optical disk drive. The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD I / F 104. Further, the CPU 101 can expand the data stored in the HDD 105 in the RAM 102 and similarly store the data expanded in the RAM 102 in the HDD 105. The CPU 101 can execute the data expanded in the RAM 102 as a program. The input I / F 106 is a serial bus interface such as USB or IEEE1394, for example, and is connected to an input device 107 such as a keyboard and a mouse. The CPU 101 can read data from the input device 107 via the input I / F 106. The output I / F 108 is a video output interface such as DVI or HDMI (registered trademark), for example, and connects an output device 109 such as a liquid crystal display or a projector. The CPU 101 can send data to the output device 109 via the output I / F 108 to execute display. In the present embodiment, the output device 109 is a display system having the display unit shown in FIG.

  FIG. 2 is a block diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment. 2A, the image processing apparatus includes a projection plane setting unit 201, an image acquisition unit 202, a display device information acquisition unit 203, a viewpoint position information acquisition unit 204, an imaging parameter acquisition unit 205, a display image generation unit 206, and an image output. Part 207. The projection plane setting unit 201 sets a projection plane for generating a display image to be displayed on each display in the display system from the input image. In the present embodiment, three display images are generated from an input image using a planar virtual projection plane (hereinafter referred to as a plane projection plane) corresponding to the input image. The plane projection plane is set according to the aspect ratio of the input image and the imaging angle of view when the input image obtained by imaging is captured. Therefore, the projection plane setting unit 201 sets a plane projection plane based on the viewpoint position and the input image.

  The image acquisition unit 202 acquires an image obtained by imaging and outputs the acquired image to the display image generation unit 206 as an input image.

  The display system information acquisition unit 203 acquires information about a display unit (here, a display) in the display system. In the present embodiment, information indicating the number of displays, the shape and size of each display surface, and the positional relationship among a plurality of displays is acquired.

  The viewpoint information acquisition unit 204 acquires viewpoint information indicating the viewer's viewpoint position in the real space. The viewpoint information is information indicating the three-dimensional position of the viewer's viewpoint when viewing the image display unit in the display system. In this embodiment, it is assumed that a display image to be displayed on the display system is generated in advance before the viewer appreciates. However, in the display system shown in FIG. 4, when the display angle changes, the display image displayed on each display also changes. The display angle varies depending on how far the viewer views the display from a position away from the display. Therefore, in this embodiment, in order to generate a display image in advance, it is necessary to assume from which position the viewer will appreciate. In the present embodiment, it is assumed that the viewpoint position is specified by acquiring, as viewpoint information, a viewpoint position desirable for the viewer to appreciate the display.

  The imaging parameter acquisition unit 205 acquires imaging parameters of the imaging apparatus that were set when the input image was acquired by imaging. The imaging parameter acquisition unit 205 can acquire imaging parameters based on metadata attached to the input image. Or it is good also as a form which acquires an imaging parameter based on the information which the user input from the input device 107. FIG.

  The display image generation unit 206 generates a display image to be displayed on each display from one input image based on the positional relationship between the viewpoint position and each display. The display image generation unit 206 generates a wide-field display image by controlling the virtual viewpoint position based on the shooting angle of view when the input image is shot and the display angle on the display unit. Details of the display image generation unit 206 will be described later. The image output unit 207 outputs the generated three display images to each display.

  Hereinafter, a flow of processing executed by the image processing apparatus according to the present embodiment will be described.

  FIG. 3 is a flowchart showing the flow of processing performed by the image processing apparatus of this embodiment. The CPU 101 reads a program for realizing the flowchart shown in FIG. 3 stored in the ROM 103 or the HDD 104, and executes the RAM 102 as a work area. Thereby, the CU 101 plays a role as each functional configuration shown in FIG. In the following flowcharts, each step (step) is denoted as “S”.

  In step S <b> 301, the image acquisition unit 202 acquires captured image data representing a captured image stored in the HDD 105 as an input image, and stores the acquired image data in the RAM 102.

  In step S302, the imaging parameter acquisition unit 205 acquires imaging parameters from metadata attached to the captured image data. In the present embodiment, the imaging parameter acquisition unit 205 acquires, as imaging parameters, information for specifying a shooting angle of view and a lens projection method when an input image is captured. In the present embodiment, it is assumed that an input image is captured through a central projection type lens used in a general lens.

  In S303, the display system information acquisition unit 203 acquires display system information related to the display unit in the display system. In the present embodiment, the display system information acquisition unit 203 acquires information indicating the number of displays that display images, the shape and size of each display surface, and the arrangement of each display. In this embodiment, as shown in FIG. 4, the number of displays is three. The shape of each display is a plane and a rectangle with a width Wsc and a height Hsc. In the arrangement of the three displays, the opening angle between the center display and the right display, and the opening angle between the center display and the left display are all the angle θsc. Further, the viewing angle (display angle) when viewing a display unit composed of three displays from the viewpoint position is 2φ. The display system information is acquired from the input device 107 to the RAM 102 based on a user instruction. Alternatively, these pieces of information may be stored in advance in the HDD 105 as a display system information package, and may be selected from the HDD 105 as necessary.

  In step S304, the viewpoint information acquisition unit 204 acquires viewpoint information from the input device 107 based on a user instruction. In the present embodiment, the viewpoint information acquisition unit 204 acquires a distance Dview from the center position on the screen of the center display 401 as viewpoint information.

  In step S305, the projection plane setting unit 201 sets a plane projection plane used when generating a display image. Details of the projection plane setting process will be described later.

  In S306, the display image generation unit 206 generates display image data indicating a display image to be displayed on each display. Details of the display image generation processing will be described later.

  In S307, the image output unit 207 outputs a display image corresponding to each display generated from the RAM 102 to the output device 109 via the output I / F 108. Alternatively, the generated display image may be stored in the HDD 105.

<Projection plane setting process>
Next, the projection plane setting process will be described in detail. In the projection plane setting process in this embodiment, a planar projection plane (hereinafter referred to as a plane projection plane) corresponding to the input image is set. The plane projection plane is composed of a plane having the same aspect ratio as that of the input image. The projection plane setting processing unit 201 calculates the size and position so that the prospective angle of the plane projection plane when viewed from the viewpoint position in the virtual space matches the shooting angle of view, and arranges the plane projection plane in the virtual space. Is done. Here, let θ be the half field angle of the horizontal field of view. In the present embodiment, XYZ three-dimensional coordinates having the viewpoint position as the origin are defined in the virtual space. FIG. 8 is a diagram illustrating the XZ plane when the display unit and the plane projection plane are viewed from above in the virtual space. As shown in FIG. 8, the plane projection plane is arranged at a position parallel to the XY plane and the Z axis passing through the center of the plane projection plane. When the plane representing the display unit is arranged in the virtual space, each display of the display unit is arranged so that the distance between the origin (viewpoint position) and the center position of the center display becomes Dview. That is, the center coordinates of the center display are (0, 0, Dview). The three displays are arranged so as to be symmetric about the Z axis.

<Display image generation processing>
Here, the display image generation processing will be described in detail. The display image generation unit 206 generates a display image corresponding to each display by rendering the planar projection plane with a virtual camera arranged at the viewpoint position in the virtual space.

  FIG. 2B is a block diagram illustrating a detailed configuration of the display image generation unit 206. The first calculation unit 2061 calculates the correspondence between the input image and the planar projection plane. The rendering processing unit 2062 renders the planar projection plane using the correspondence relationship between the input image and the planar projection plane.

  FIG. 5 is a flowchart showing details of the display image generation processing in this embodiment. Hereinafter, each step of the flowchart will be described.

  In step S501, the first calculation unit 2061 performs association between the three-dimensional coordinates of each vertex of the planar virtual projection plane and the two-dimensional coordinates indicating the pixel position of the captured image. This is the same processing as UV mapping in general CG rendering. Association is performed between the three-dimensional coordinates of each vertex of the plane projection plane in the virtual space and the two-dimensional coordinates indicating the pixel position of the input image. This is the same processing as UV mapping in general CG rendering. When an input image captured through a central projection lens is used, as shown in FIG. 7, the coordinates of the four corners of the planar projection surface are associated with coordinates indicating the pixel positions of the four corners of the input image. Here, in the input image, the pixel position of each pixel is indicated by a UV coordinate system in which the upper left pixel is the origin (0, 0). The first calculation unit 2061 acquires the UV coordinates of each vertex in the input image and associates it with the three-dimensional coordinates of each vertex of the planar projection plane. The UV coordinates of each pixel other than the four corners are calculated by linear interpolation. If the lens projection method is equidistant projection or equisolid angle projection, such as a fisheye lens, the same processing may be performed after re-projecting with central projection.

  In step S502, the rendering processing unit 2062 compares the display angle with the shooting angle of view in the horizontal direction. If the display angle is larger than the shooting angle of view, the process proceeds to S503, and if not, the process proceeds to S504.

  In step S503, the rendering processing unit 2062 sets a virtual viewpoint position (hereinafter referred to as a virtual viewpoint position) serving as a reference for rendering processing. Specifically, the virtual viewpoint position is set so that the virtual display angle is a virtual display angle when the display unit is viewed from the virtual viewpoint position, and the virtual display angle coincides with the shooting angle of view. FIG. 8 is a diagram illustrating the relationship between the planar projection plane and the display unit when the virtual viewpoint position matches the viewpoint position indicated by the viewpoint information. If the display angle of the display unit is larger than the shooting angle of view as shown in FIG. 8, if the plane projection plane is projected onto each display surface constituting the display unit as it is, images are not displayed on the left and right sides of the display unit. An area will be generated. Therefore, in this embodiment, the rendering processing unit 2062 sets the virtual viewpoint position for rendering to a position different from the viewpoint position according to the shooting angle of view and the display angle. FIG. 9 is a diagram for explaining the virtual viewpoint position. The rendering processing unit 2062 moves the virtual viewpoint position in the negative Z-axis direction to a position where the virtual display angle φ ′ matches the shooting angle of view θ.

In step S504, the rendering processing unit 2062 calculates the X coordinate x _{m ′} of the point M ′ on the plane projection plane corresponding to the point M (x, y, z) on the display unit using Expression (1).

Here, φ _m is an angle formed by the vector OM _xz and the Z axis when a perpendicular line extending from the point M to the XZ plane is a point M _xz . W _flat is the width of the plane projection plane.

Rendering unit 2062 in S505 is calculated using equation (1) used in S504 the 'Y coordinate y _m of _the' point M at the point M (x, y, z) planar projection surface corresponding to the display unit . At this time, a vertical shooting field angle is used instead of the horizontal shooting field angle θ. Further, instead of the angle φ _m , an angle formed by the vector OM _yz and the Z axis when a perpendicular line extending from the point M to the YZ plane is the point M _yz is used. Further, the height of the plane projection plane is used instead of W _flat for calculation. As for the Z coordinate, since the plane projection plane is a plane, the Z coordinate does not change at any point on the plane projection plane. Therefore, the Z coordinate of the plane projection plane set in the projection plane setting process is always matched.

  In step S <b> 506, the rendering processing unit 2062 uses the correspondence relationship between the three-dimensional coordinates indicating the planar projection plane and the two-dimensional coordinates indicating the input image to determine the position I in the input image from the three-dimensional coordinates of the point M ′ on the plane projection plane. (U, v) is calculated and sampled. Specifically, the pixel values of four pixels in the vicinity of the position I (u, v) are acquired, and an interpolation operation is performed according to the distance between each of the four pixels and I (u, v) to display at the point M. A pixel value to be determined is determined.

  In step S507, the rendering processing unit 2062 determines whether or not the pixel values of the pixels in the display image to be displayed for all the points on the display unit have been calculated. If the pixel values have been calculated for all the pixels, the process ends. If not completed, the point M is updated to an unprocessed pixel that has not yet been processed, and the process returns to S504 to continue the process.

  With the above processing, when the display angle when viewing the display unit is larger than the shooting angle of view, the virtual viewpoint position is set so that the shooting angle of view matches the display angle at the virtual viewpoint position, and the display image Is generated. Thereby, it is possible to generate a display image with less discomfort that satisfies the display angle.

  In the present embodiment, the case where the configuration shown in FIG. 2 is realized by causing the CPU 101 to execute a program has been described as an example. However, some or all of the components shown in FIG. 2 may be realized by a dedicated processing circuit.

<Other embodiments>
In the above-described embodiment, the case where the display unit includes three displays in the display system that displays a display image has been described as an example. In addition to the self-luminous display, the display unit may use a screen that projects an image using a projector. In this case, as a display system, a plurality of projectors corresponding to each of the plurality of screens are installed so that a display image can be projected on each screen. Alternatively, the above embodiment can also be applied to a display system in which a plurality of projectors project onto a white wall. When the wall on which the image is projected is viewed from above, if the shape is the same as the display shown in FIG. 4, the virtual viewpoint position is set so that the shooting angle of view matches the display angle at the virtual viewpoint position. The same effect can be obtained by generating a display image. In this case, the display system information is acquired by regarding the area where the image is projected on the white wall as the display unit.

  A curved display screen in which a plane is curved in the horizontal direction can also be used as the display unit.

  Further, in the above-described embodiment, the display unit using a large display as shown in FIG. 4 has been described as an example. However, a similar display system can be configured even if the display unit is large enough to cover only the head using, for example, display screens on a plurality of planes. For example, a head mounted display. Also in this case, the display screen constituting the display unit is a plane, and the image display side is arranged so that the normal of each display screen has an intersection on the image display side, thereby displaying a wide-field image with a sense of reality. be able to. For such a display system, a wide-field display image without a sense of incongruity can be generated by rendering a common input image, as in the above-described embodiment.

  In the above-described embodiment, the viewpoint information is specified by designating a desired viewer position. However, for example, the position of the viewer may be actually detected, and the viewpoint information may be set according to the viewer who is actually viewing the display system.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 201 Projection surface setting part 202 Image acquisition part 203 Display system information acquisition part 204 Viewpoint information acquisition part 205 Imaging parameter acquisition part 206 Display image generation part 207 Image output part

Claims (8)

First acquisition means for acquiring a shooting angle of view when an input image is captured;
In a display system including a display unit, a second acquisition unit that acquires display system information indicating a display angle that is an expected angle when the display unit is viewed from a viewpoint position;
Setting means for setting a projection plane in a virtual space;
Generation means for generating a display image to be displayed on the display unit by rendering the projection plane based on a correspondence relationship between the input image and the projection plane;
The image processing apparatus, wherein the generation unit sets a virtual viewpoint position in the virtual space based on the shooting angle of view, and renders the projection plane from the virtual viewpoint position. And third acquisition means for acquiring viewpoint information indicating a viewpoint position for viewing the display unit;
Comparing means for comparing a display angle that is an expected angle when the display unit is viewed from the viewpoint position indicated by the viewpoint information and the shooting angle of view;
The generation unit sets the virtual viewpoint position at a position different from the viewpoint position indicated by the viewpoint information when the comparison unit determines that the display angle is larger than the shooting angle of view. The image processing apparatus according to claim 1.   The generating means sets the virtual viewpoint position to the viewpoint position indicated by the viewpoint information when the comparing means determines that the display angle is smaller than the shooting angle of view. The image processing apparatus according to claim 2.   The generating unit is configured to move the generating unit in a direction away from the projection plane from the viewpoint position indicated by the viewpoint information when the comparing unit determines that the display angle is larger than the shooting angle of view. The image processing apparatus according to claim 2, wherein the virtual viewpoint position is moved.   The image processing apparatus according to claim 1, wherein the setting unit sets the projection plane based on the input image.   The image processing apparatus according to claim 1, wherein the display unit includes a plurality of display screens capable of displaying images.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 6. Obtain the shooting angle of view when the input image was shot,
In a display system including a display unit, display system information indicating a display angle that is an expected angle when viewing the display unit from a viewpoint position is acquired;
Set the projection plane in the virtual space,
A virtual viewpoint position is set in the virtual space based on the shooting angle of view, and the projection plane is rendered from the virtual viewpoint position based on a correspondence relationship between the input image and the projection plane. An image processing method for generating a display image to be displayed on the display unit.

Images