JP2018113683A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

In setting a path of a virtual camera, a user can grasp in advance the resolution of an object in a generated virtual viewpoint video.
An image processing apparatus that generates a virtual viewpoint video that is a video viewed from a virtual camera using a plurality of viewpoint videos, and each camera group including a plurality of real cameras having the same angle of view. In addition, a derivation unit that derives an imaging region common to the plurality of real cameras, and a setting unit that sets a parameter related to the virtual camera based on a user input via a GUI, the GUI includes the The imaging area for each camera group derived by the deriving means is displayed in a recognizable manner.
[Selection] Figure 4

Description

  The present invention relates to a technique for setting parameters relating to a virtual viewpoint.

Virtual viewpoint video technology is a technology for reproducing video from a camera (virtual camera) that does not actually exist and is virtually arranged in a three-dimensional space using videos taken by a plurality of real cameras. In the generation of the virtual viewpoint video, there is an area where the video from the virtual camera cannot be reproduced due to the blind spot between the videos taken by the real camera. Further, when the virtual camera is closer to a person or the like (object) in the shooting scene than the real camera, the resolution of the object is reduced, resulting in a blurred image. If the path of the virtual camera (positional movement of the virtual camera along the time axis) is set without taking these into consideration, the obtained virtual viewpoint video has low image quality. Therefore, there is a possibility that the completed virtual viewpoint video is confirmed on the preview screen or the like, and the virtual camera path setting is repeated (reset) several times.

  In this regard, for example, a technique has been proposed in which information on blind spots between videos taken with a real camera is presented to the user (Patent Document 1). In the technique of this patent document 1, it is possible to confirm the blind spot area in advance without actually generating the virtual viewpoint video by visualizing the blind spot area or the area that the observer does not want to show on the 2D map. I was doing.

JP 2011-172169 A

  However, with the method of Patent Document 1, it is impossible to confirm the correspondence between the position of the virtual viewpoint and the quality of the virtual viewpoint video before setting the virtual viewpoint. Therefore, according to the present invention, the correspondence between the position of the virtual viewpoint and the quality of the virtual viewpoint video can be confirmed before setting the virtual viewpoint. Accordingly, an object of the present invention is to suppress the reworking of the virtual camera path setting.

  An image processing apparatus according to the present invention is an image processing apparatus that generates a virtual viewpoint video, which is a video viewed from a virtual camera, using a plurality of viewpoint videos, and includes a plurality of real cameras having the same angle of view. For each camera group, a derivation unit for deriving an imaging region common to the plurality of real cameras, and a setting unit for setting a parameter related to the virtual camera based on a user input via a GUI, The GUI is characterized in that an imaging region for each camera group derived by the deriving means is displayed in a recognizable manner.

  According to the present invention, in setting a route of a virtual camera, a user can grasp in advance the resolution of an object in a completed virtual viewpoint video. As a result, the reworking of the virtual camera path setting is suppressed.

It is a figure which shows an example of a structure of a virtual viewpoint video system. (A) is a figure which shows an example of camera arrangement | positioning, (b) is a figure which shows the imaging region of the camera which belongs to each camera group. It is the flowchart which showed the whole flow until a virtual viewpoint image | video is produced | generated. It is a flowchart which shows the process in which the GUI screen used for the parameter setting of a virtual camera is produced | generated. It is a figure explaining the derivation | leading-out method of the common imaging | photography area | region for every camera group. It is a figure which shows an example of the result as which the common imaging | photography area | region for every camera group was projected on the field map. It is a flowchart which shows the flow of the parameter setting process of a virtual camera. It is a figure which shows an example of the GUI screen for parameter setting of a virtual camera. 10 is a flowchart showing details of a virtual camera height adjustment process according to the second embodiment. It is a figure which shows an example of the sample image of a virtual viewpoint image | video. (A)-(c) is a figure explaining the production method of a sample image. 10 is a flowchart illustrating details of a virtual camera and gaze point designation process according to the third embodiment. It is a figure explaining intersection determination.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

Embodiment 1

  FIG. 1 is a diagram illustrating an example of a configuration of a virtual viewpoint video system in the present embodiment. The virtual viewpoint video system shown in FIG. 1 includes an image processing apparatus 100 and two types of camera groups 109 and 110. The image processing apparatus 100 includes a CPU 101, a main memory 102, a storage unit 103, an input unit 104, a display unit 105, and an external I / F 106, and each unit is connected via a bus 107. First, the CPU 101 is an arithmetic processing device that controls the image processing apparatus 100 in an integrated manner, and executes various programs by executing various programs stored in the storage unit 103 or the like. The main memory 102 temporarily stores data and parameters used in various processes, and provides a work area to the CPU 101. The storage unit 103 is a large-capacity storage device that stores various programs and various data necessary for GUI (graphical user interface) display. For example, a nonvolatile memory such as a hard disk or a silicon disk is used. The input unit 104 is a device such as a keyboard, mouse, electronic pen, or touch panel, and accepts various user inputs. The display unit 105 is configured by a liquid crystal panel or the like, and displays a GUI for setting a virtual camera path when generating a virtual viewpoint video. The external I / F unit 106 is connected to each camera constituting the camera groups 109 and 110 via a network (here, the LAN 108), and transmits and receives video data and control signal data. A bus 107 connects the above-described units and performs data transfer.

  The two types of camera groups are a zoom camera group 109 and a wide-angle camera group 110, respectively. The zoom camera group 109 includes a plurality of cameras equipped with lenses having a narrow angle of view (for example, 10 degrees). The wide-angle camera group 110 includes a plurality of cameras equipped with lenses having a wide angle of view (for example, 45 degrees). Each camera constituting the zoom camera group 109 and the wide-angle camera group 110 is connected to the image processing apparatus 100 via the LAN 108. The zoom camera group 109 and the wide-angle camera group 110 also start and stop shooting, change camera settings (shutter speed, aperture, etc.), and transfer shot video data based on control signals from the image processing apparatus 100. Do.

  In addition to the above, there are various components of the system configuration, but the description is omitted because it is not the main point of the present invention.

  FIG. 2A is a diagram illustrating an example of camera arrangement in an imaging system including two types of camera groups, a zoom camera group 109 and a wide-angle camera group 110, in a stadium where, for example, soccer is performed. A player as an object 202 exists on the field 201 where the competition is performed. Then, twelve zoom cameras 203 constituting the zoom camera group 109 and twelve wide angle cameras 204 constituting the wide angle camera group 110 are arranged so as to surround the field 201. An area 213 surrounded by a dotted line in FIG. 2B indicates a shooting area of the zoom camera 203. Although the zoom camera 203 has a narrow angle of view so that the shooting area is narrow, the zoom camera 203 has a high resolution when shooting the object 202. In addition, an area 214 surrounded by a dotted line in FIG. 2C indicates an imaging area of the wide-angle camera 204. The wide-angle camera 204 has a wide angle of view and thus a wide shooting area, but has a characteristic that the resolution when shooting the object 202 is low. In FIG. 2B, the shape of the shooting area is indicated by an ellipse for the sake of convenience. However, as will be described later, the actual shape of the shooting area of each camera is generally rectangular or trapezoidal.

  FIG. 3 is a flowchart showing the overall flow until the virtual viewpoint video is generated in the image processing apparatus 100. This series of processing is realized by the CPU 101 reading a predetermined program from the storage unit 103 and developing it in the main memory 102, which is executed by the CPU 101.

  In step 301, shooting parameters such as exposure conditions during shooting and a shooting start signal are transmitted to the zoom camera group 109 and the wide-angle camera group 110. Each camera belonging to each camera group starts shooting according to the received shooting parameters, and holds the obtained video data in a memory in each camera.

  In step 302, multi-view video data captured by each zoom camera 203 belonging to the zoom camera group 109 and multi-view video data captured by each wide-angle camera 204 belonging to the wide-angle camera group 110 are acquired. The acquired video data of a plurality of viewpoints (here, each of 12 viewpoints) is developed in the main memory 102.

  In step 303, the process for estimating the three-dimensional shape of the object is executed using the multi-viewpoint video data acquired in step 302. As the estimation method, a known method such as a Visual-hull method using the contour information of an object or a Multi-view stereo method using triangulation may be applied.

  In step 304, various parameters such as a moving path of the virtual camera necessary for generating the free start video are set based on the estimated object shape data. Details of processing related to the virtual camera parameter setting will be described later.

  In step 305, a virtual viewpoint video is generated according to the set virtual camera parameters. The virtual viewpoint video can be generated by using a computer graphics technique for a video viewed from a set virtual camera with respect to the estimated object shape.

  In step 306, it is determined whether to change the parameter setting of the virtual camera to generate a new virtual viewpoint video. This process is performed based on an instruction from a user who has viewed the virtual viewpoint video displayed on a preview screen (not shown) and confirmed the image quality. If the user wants to regenerate the virtual viewpoint video, the parameter setting for the virtual camera is performed again (return to step 304). When the parameter is changed, a virtual viewpoint video is newly generated with the changed content. On the other hand, if there is no problem in the generated virtual viewpoint video, this process is finished. The above is a rough flow until the virtual viewpoint video is generated according to the present embodiment.

  Next, the preparation process of the GUI screen used in the virtual camera parameter setting process in the above-described step 304 will be described. FIG. 4 is a flowchart showing a process of generating a GUI screen used for parameter setting of the virtual camera. Note that the processing in FIG. 4 may be executed prior to each processing shown in FIG. 3 or may be executed at the timing at which step 304 is executed.

  In step 401, camera information of all camera groups (here, two types of zoom camera group 109 and wide-angle camera group 110) is acquired. The camera information includes information such as the installation location of the camera belonging to each camera group, the point of interest position, and the angle of view. In this embodiment, the gazing point position is a position that is the center of photographing of all 24 cameras. Such information may be obtained by reading out information previously stored in the storage unit 103, or obtained by accessing each camera (or one camera representing each camera group) via the LAN 108. May be.

  In step 402, based on the acquired camera information of each camera group, an area (common shooting area) where shooting areas of a plurality of cameras having substantially the same angle of view overlap is derived for each camera group. Here, the expression “substantially the same” is because a fine change in the angle of view occurs between the cameras due to a difference in distance from the camera to the gazing point. Note that the angles of view of the cameras belonging to the same camera group may not be completely the same. FIG. 5 is a diagram for explaining a method for deriving a common imaging region for each camera group. Now, two cameras 501 and 502 belonging to the same camera group with respect to the field 201 are shooting at the same point of sight. At this time, the imaging region of the camera group to which the camera 501 and the camera 502 belong includes an intersection plane 511 between the square pyramid and the field 201 from the camera 501 toward the gazing point, and a quadrangular pyramid and the field 201 from the camera 502 toward the gazing point. It is calculated | required as the area | region 513 with which the intersection surface 512 overlaps. Here, for convenience of explanation, explanation has been given with two cameras 501 and 502, but the concept is the same with three or more cameras. Thus, a common shooting area for each camera group is derived.

  In step 403, each of the derived common shooting areas for each camera group is visualized and projected on an overhead view (field map) of the entire shooting space including the field 201. FIG. 6 is a diagram illustrating an example of a result of recognizing and projecting a common shooting area for each camera group on the field map. In FIG. 6, a rectangular frame 600 indicates a field map. A dashed ellipse 601 indicates a common shooting area of the zoom camera group 109 (hereinafter, zoom camera group shooting area). Within the zoom camera group shooting area 601, it is possible to generate a virtual viewpoint video using a plurality of viewpoint videos shot by the zoom camera group 109. In the zoom camera group shooting area 601, the resolution of the object is relatively high, so that the image quality of the virtual viewpoint video is maintained even when the virtual camera is brought close to the object (even if the height of the virtual camera is lowered). Is possible. In the present embodiment, the zoom camera group shooting area 601 is an area where the shooting areas of all twelve zoom cameras 203 belonging to the zoom camera group 109 overlap. A dashed-dotted ellipse 602 indicates a common shooting area of the wide-angle camera group 110 (hereinafter, a wide-angle camera group shooting area). Within the wide-angle camera group shooting area 602, it is possible to generate a virtual viewpoint video using a plurality of viewpoint videos shot by the wide-angle camera group 110. Since the resolution of the object is relatively low in the wide-angle camera group shooting region 602, the image quality of the virtual viewpoint video deteriorates when the virtual camera is brought closer to the object than a certain level (the height of the virtual camera is lowered below a certain level). Will do. In the present embodiment, the wide-angle camera group shooting area 602 is an area where the shooting areas of all 12 wide-angle cameras 204 belonging to the wide-angle camera group 110 overlap. In the field map 600, an area 603 indicated by hatching does not overlap with the shooting areas of the wide-angle cameras 204 belonging to the wide-angle camera group 110 having a wide angle of view in the present embodiment, and it is difficult to generate a virtual viewpoint video of a certain quality. A possible area (hereinafter, a virtual viewpoint video generation impossible area) is shown. In the example of FIG. 6, the outer edges of the zoom camera group shooting area and the wide-angle camera group shooting area are indicated by broken lines and alternate long and short dash lines, respectively, and the virtual viewpoint video non-generation area is indicated by oblique lines, but this is not limitative. It suffices if the common shooting area for each camera group capable of generating a virtual viewpoint video is displayed so that it can be recognized by the user. For example, each area may be indicated by color coding or the like. In FIG. 6, for convenience of explanation, the shape of the common shooting area of each camera group is indicated by an ellipse, but actually, it is a polygon. In this way, the GUI screen used for the virtual camera parameter setting process is ready.

  Next, the virtual camera parameter setting process according to the present embodiment will be described with reference to the flowchart of FIG. The process in FIG. 7 corresponds to the process in step 304 in FIG. First, in step 701, a virtual camera parameter setting GUI screen is displayed on the display unit 105. FIG. 8 is a diagram illustrating an example of a GUI screen. In the GUI screen 800 shown in FIG. 8A, the field map 600 obtained by the above-described preparation process is displayed on the left side of the screen, and the object shape 810 estimated at the above-described step 303 is mapped thereon. .

  In step 702, designation of a virtual camera movement path (camera path) and a gaze point movement path (gaze point path) is accepted via the displayed GUI screen. After the user presses the button 801 or 803 and operates the mouse or the like on the field map 600 to move the cursor, the movement locus is designated as a camera path or a gazing point path. In FIG. 8A, a thick arrow 811 indicates a designated camera path, and a dotted arrow 812 indicates a designated gaze point path. The camera path and the gazing point path specified in this way are associated with each other so as to match in the setting frame. That is, when the virtual viewpoint video is composed of, for example, 600 frames, each division point obtained by dividing each path by the number of frames is sequentially associated with the start point of the arrow. When the user moves the cursor to an arbitrary position (coordinates) on the camera path 811 and performs a click operation with a mouse or the like, the division point P0 closest to the cursor is selected and displayed, and the gazing point position Q0 corresponding to P0 is also displayed. indicate. At this time, an angle of view area 813 viewed from the virtual camera is also displayed. Since the user can grasp the positional relationship between the gazing point Q0, the object 810, and the view angle area 813, the gazing point Q0 and the object 810 cannot generate the zoom camera group shooting area 601, the wide angle camera group shooting area 602, and the virtual viewpoint video. The designation work can be performed while confirming which area of the area 603 fits, that is, what resolution the virtual viewpoint video has. Therefore, the camera path and the like can be appropriately set, and the number of times of performing the designated work can be suppressed. It is also possible to correct the path by moving P0 and Q0 with a mouse or the like. The height from the field 201 of the virtual camera path is a default value (for example, 15 m), and the default value (for example, 1 m) is set as the height of the gazing point path that is lower than the virtual camera path.

  In step 703, the process is divided depending on whether or not the height of the virtual camera and the point of gaze is adjusted. If the user presses the camera path height edit button 802 or the gaze point path edit button 804, the process proceeds to step 704.

  In step 704, processing for adjusting the heights of the virtual camera and the point of interest (height adjustment processing) is executed. Here, a process for adjusting the height of the virtual camera will be described as an example. The user moves the cursor to an arbitrary position (coordinates) on the camera path 811 and performs a click operation with a mouse or the like to specify the position (height edit point) of the virtual camera whose height is to be changed. Here again, as in step 703, the division point closest to the cursor is selected and displayed as the height edit point. In FIG. 8B, a portion indicated by a cross on the camera path 811 indicates a height edit point designated by the user. A plurality of height editing points can be set. In the example of FIG. 8B, two height editing points P1 and P2 are set, and points Q1 and Q2 on the gazing point path 812 corresponding to P1 and P2 are displayed accordingly. Also, the distance connecting P1 and Q1 and P2 and Q2 is displayed on the GUI. This is because the size of the object in the virtual viewpoint video changes depending on the distance between the virtual camera and the gazing point. The closer the distance, the larger the size of the object and the more likely it will be blurred. In the case of FIG. 8B, the distance between P1 and Q1 is 24 m, and the distance between P2 and Q2 is 20 m, and P2 to Q2 are more easily blurred. When the height edit point is set, a height setting window 820 is displayed in the GUI screen 800, and the user can enter an arbitrary value (unit: m) in the input field 821 corresponding to each edit point in the height setting window 820. ) Can be changed to change the height of the virtual camera at that position. At this time, the user can specify any value while confirming the distance between the zoom camera group shooting area 601, the wide-angle camera group shooting area 602, and the virtual camera and the gazing point. Can be set. It should be noted that the heights other than the portion where the altitude is changed by the height editing point are adjusted so that the height does not change abruptly by interpolating from the height editing point at the adjacent position or the default value.

  The above is the content of the parameter setting process of the virtual camera according to the present embodiment. In the GUI screen of this embodiment, each camera group shooting area is displayed in two dimensions, but may be displayed in three dimensions.

  As described above, according to the present embodiment, the user can set parameters such as the moving path and height of the virtual camera while grasping the imaging region of each camera group. As a result, it is possible to suppress redo of the parameter setting work when generating the virtual viewpoint video.

Embodiment 2

  In the first embodiment, a common imaging area for each camera group is visualized on the GUI screen for parameter setting of the virtual camera and projected onto the field map, so that the camera path and the like can be set appropriately. Next, a mode in which the user can directly check the resolution of the object in the height adjustment of the virtual camera will be described as a second embodiment.

  Since the basic system configuration and the general flow of the self-viewpoint video generation process are the same as those in the first embodiment, the description thereof will be omitted, and the following description will focus on the differences.

  FIG. 9 is a flowchart showing details of the virtual camera height adjustment process (step 704 described above) according to the present embodiment.

  In step 901, designation of a height edit point is accepted. Here, it is assumed that two points P1 and P2 are designated as shown in FIG. In the subsequent step 902, the lower limit value of the height at the designated height editing point is acquired. The lower limit of the height indicates the height of the virtual camera corresponding to the minimum level acceptable as the resolution of the object, the distance between the virtual camera and the point of interest, the angle of view of the virtual camera, the note It is determined in advance by the combination of the camera group shooting area where the viewpoint is located. That is, when the gazing point is located in the high-resolution camera group photographing region, the distance between the virtual camera and the gazing point can be shortened, and thus the lower limit value of the height is lowered. On the other hand, when the gazing point is located in the low-resolution camera group photographing region, the lower limit value of the height is increased. When the angle of view is wide, the lower limit value of the height is lowered, and when the angle of view is narrow, the lower limit value of the height is increased. Whether or not the resolution is acceptable is determined based on a criterion such as whether or not the resolution is below the resolution of the object in the actual camera. When acquiring the lower height limit, first, obtain the gaze point coordinates of the specified height edit point, the classification information of each camera group shooting area corresponding to the coordinates, the distance between the virtual camera and the gaze point And information on the angle of view of the virtual camera. The classification information means information such as a flag that is given for each coordinate position and that indicates a zoom camera group photographing region 601 or a wide-angle camera group photographing region 602. And based on these acquired information, the height lower limit of the virtual camera in the designated height edit point is acquired.

  In step 903, a sample image of the virtual viewpoint video corresponding to the acquired height lower limit value is displayed. The sample image is an object cut out from the video of the real camera. The clipping method will be described with reference to FIGS. 11 (a) and 11 (b). FIG. 11A shows an image 1101 captured by a certain real camera. The image 1101 includes objects A to E corresponding to five players. By identifying the faces and spine numbers of the objects A to E with respect to the image 1101 using an existing recognition technique, a sample image obtained by cutting out the area for each object can be obtained. The obtained sample image data is held in the storage unit 103 together with information on the number of pixels (width and height). FIG. 11B shows an image 1102 taken with another real camera. A similar process is performed on the image 1102 to obtain a sample image. If the number of pixels of the newly created sample image exceeds the number of pixels of the previously created sample image, the process of overwriting (updating) with the newly created sample image is repeated for each object. By executing this processing with all real cameras, a sample image with the highest resolution can be prepared for each object. When displaying the sample image, the sample image is reduced according to the distance between the virtual camera and the object and the angle of view of the virtual camera. This reduction method will be described with reference to FIG. First, an object closest to the gazing point is detected, and sample image data of the object is read from the storage unit 103 into the main memory 102. Next, the number of pixels in the object area in the virtual viewpoint video is calculated from the distance between the virtual camera and the object and the angle of view of the virtual camera. Specifically, it can be calculated by placing a three-dimensional model 1122 having a standard height size of the player at the position of the detected object and drawing a preview image 1123 when viewed from the virtual camera 1121. Then, the sample image is reduced so that the object area in the preview image 1123 and the number of pixels (width and height) of the sample image match or approximate. After the reduction, in order to display on the GUI screen 800, it is enlarged by simple interpolation to an appropriate size. The sample image is displayed on the GUI screen 800 in a format such as a sub window shown in FIG. As a result, the user can confirm the resolution of the object when the height is lowered to the lower limit with an actual image. At this time, as shown in FIG. 10A, the corresponding height lower limit value may be displayed together with the sample image.

  In step 904, a change in height input via the height setting window 820 is accepted for the designated height editing point. In the following step 905, it is determined whether or not the changed height is lower than the height lower limit value. When the changed height is below the lower limit value, the process proceeds to step 906. On the other hand, if the height after the change is equal to or greater than the lower limit value, the process is exited.

  In step 906, the sample image is updated to the content corresponding to the height after the change, that is, the sample image with a low resolution exceeding the allowable limit. FIG. 10B shows an example of the updated sample image, and it can be seen that the sample image corresponding to the height adjustment point P2 is blurred. This updated sample image is a reduced / enlarged display of the sample image unique to the object according to the distance between the virtual camera and the point of gaze after setting and the angle of view of the virtual camera, as in step 903 described above. It is. At this time, a message (warning display) indicating that the height is below the lower limit value, for example, “below the lower limit value” may be displayed. In the subsequent step 907, the processing is switched depending on whether or not the height is changed again. If a new height is designated, the process returns to step 904. If no new height is specified, this process is exited.

  In this embodiment, the video of the actual camera that shot the actual game was used as the sample image. However, for example, a video that is being practiced, a portrait shot in the studio, or the like may be used. In addition, an image obtained by rendering a person may be used. Further, since it is sufficient if the resolution can be confirmed, an image using an object different from the actual object may be used as the sample image.

  The above is the content of the height adjustment process of the virtual camera according to the present embodiment. In the case of this embodiment, when editing the height of the virtual camera, the sample image corresponding to the lower limit value of the height at the specified height editing point is visualized and displayed. It becomes easy to judge.

Embodiment 3

  In the first embodiment, a common imaging area for each camera group is visualized on the GUI screen for parameter setting of the virtual camera and projected onto the field map, so that the camera path and the like can be set appropriately. Next, a mode in which the user can check a region where the resolution of the object rapidly changes will be described as a third embodiment.

  Since the basic system configuration and the general flow of the virtual viewpoint video generation processing are the same as those in the first embodiment, the description thereof will be omitted, and the following description will focus on the differences.

  FIG. 12 is a flowchart showing details of the gaze point path setting process (corresponding to step 702 described above) according to the present embodiment.

  In step 1201, after the designation of the gazing point is accepted, the gazing point path is searched. This is a process for investigating which camera group the photographing area of which the coordinate value of the gazing point moves. Next, in step 1202, it is determined whether or not the gazing point path determined by the search intersects the imaging areas of a plurality of camera groups. FIG. 13 is a diagram for explaining this intersection determination. In the example of FIG. 13, the gazing point path 812 starting from the wide-angle camera group shooting area 602 passes through the zoom camera group shooting area 601 and returns to the wide-angle camera group shooting area 602 again. Therefore, in this case, it is determined that they intersect. In the case of the determination result that the vehicle intersects in this way, the process proceeds to step 1203. On the other hand, in the case of a determination result that there is no intersection, the present process is terminated. In step 1203, as shown in FIG. 13, a resolution step mark 1301 is displayed at the coordinate position where the imaging region and the gazing point path intersect. This is because the designated gaze point moves so as to straddle the imaging areas of a plurality of camera groups, and the resolution may suddenly fluctuate near the step mark 1301 where the focal length of the camera changes. This is to warn the user of this. In such a case, the user needs to take measures such as moving the virtual camera away from the point of sight near the step mark 1301.

  The above is the content of the resolution change warning process according to the present embodiment. In the case of this embodiment, when editing a virtual camera or a gazing point path, a change in resolution of the virtual viewpoint video is visualized, so that the user can easily determine a more appropriate virtual camera or gazing point path setting. Become.

  In the first to third embodiments, the image processing apparatus 100 generates a virtual camera (virtual viewpoint) setting GUI, displays the GUI, accepts a user operation on the GUI, and generates a virtual viewpoint video. An example of doing all of this was explained. However, it is not limited to this example. For example, a device that generates and displays a GUI and accepts a user operation and a device that generates a virtual viewpoint video based on the user operation may be different devices. In addition, a virtual viewpoint video system including a device that generates a GUI (typically one device), a plurality of devices that accept GUI display and user operations, and one or a plurality of devices that generate a virtual viewpoint video. The present invention can also be applied to.

  In Embodiments 1 to 3 described above, a common shooting area where the shooting areas of all the cameras belonging to the zoom camera group 109 overlap, and a common shooting area where the shooting areas of all the cameras belonging to the wide-angle camera group 110 overlap. The description has been made centering on an example of displaying such that it can be identified. However, it is not limited to this example. For example, out of 12 cameras belonging to the zoom camera group 109, an area in which the imaging areas of 10 or more cameras overlap may be set as the common imaging area.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (14)

An image processing device that generates a virtual viewpoint video that is a video viewed from a virtual camera using a plurality of viewpoint videos,
Deriving means for deriving an imaging region common to the plurality of real cameras for each camera group composed of a plurality of real cameras having the same angle of view;
Setting means for setting parameters relating to the virtual camera based on user input via a GUI;
With
The image processing apparatus characterized in that the GUI displays a recognizable shooting area for each camera group derived by the deriving unit.   The image processing apparatus according to claim 1, wherein an area where a virtual viewpoint video cannot be generated is displayed recognizable on the GUI.   3. The image processing apparatus according to claim 1, wherein in the GUI, an imaging region for each camera group is projected and displayed on a field map corresponding to the entire imaging space of the multi-view video. . Further comprising an acquisition means for acquiring camera information including at least information of an angle of view of each camera belonging to each camera group,
The image processing apparatus according to claim 1, wherein the deriving unit derives a shooting area for each camera group based on the acquired camera information. It further comprises storage means for previously holding the camera information,
The image processing apparatus according to claim 4, wherein the acquisition unit acquires the camera information from the storage unit. The multiple real cameras that make up each camera group are connected via a network,
The image processing apparatus according to claim 4, wherein the acquisition unit acquires the camera information by accessing the real camera via the network.   4. When adjusting the height of the virtual camera as the parameter, a sample image of a virtual viewpoint video corresponding to a lower limit value of the height is further displayed on the GUI. The image processing apparatus according to claim 1.   5. The image processing apparatus according to claim 4, wherein, when a height lower than the lower limit value is designated in the GUI, a warning display indicating that the height is lower than the lower limit value is performed.   The image processing apparatus according to claim 5, wherein the sample image is updated to an image corresponding to a height lower than the lower limit value together with the warning display. The sample image is
Generate from the frame with the highest resolution of the object area in the real camera video,
The image processing apparatus according to claim 7, wherein at the time of the display, the image processing apparatus is reduced according to at least one of a distance between the virtual camera and the object and an angle of view of the virtual camera.   The image processing apparatus according to claim 10, wherein the sample image is generated for each object.   2. When setting the gazing point path of the virtual camera, if the gazing point path intersects a shooting area for each camera group, a warning display to the effect that the resolution changes is performed. The image processing apparatus described. A GUI control method in an image processing apparatus that generates a virtual viewpoint video that is a video viewed from a virtual camera using a plurality of viewpoint videos,
A derivation step for deriving an imaging region common to the plurality of real cameras for each camera group composed of a plurality of real cameras having the same angle of view;
A setting step for setting a parameter related to the virtual camera based on a user input via a GUI;
Including
The control method, wherein the GUI displays a recognizable shooting area for each camera group derived in the deriving step.   The program for functioning a computer as an image processing apparatus of any one of Claims 1 thru | or 6.

Images