JP2023001850A - Information processing device, information processing method, and program - Google Patents

Abstract

An object of the present invention is to suppress the influence caused by the presence of a normal imaging camera in the imaging range of a virtual viewpoint image generating camera. Kind Code: A1 An information processing device includes first viewpoint information for specifying a virtual viewpoint corresponding to a virtual viewpoint image, and a first viewpoint information existing in an imaging range of a first imaging device used for generating the virtual viewpoint image. and second viewpoint information representing the viewpoint of the second imaging device to be captured, and the second imaging device specified by the second viewpoint information is placed in the field of view of the virtual viewpoint specified by the first viewpoint information. When the position of is included, control is performed so that a captured image obtained by capturing an image by the second imaging device is output. [Selection drawing] Fig. 1

Description

The present disclosure relates to processing based on captured images.

There is a method of installing a plurality of imaging devices (cameras for generating virtual viewpoint images) at different positions and generating a virtual viewpoint image using captured images obtained by capturing images of an object from a plurality of viewpoints. Also, an imaging device (ordinary imaging camera) different from the virtual viewpoint image generation camera may perform imaging. Then, there is a method of appropriately switching and displaying a virtual viewpoint image and a captured image obtained by capturing an image by a normal imaging camera.

Patent Document 1 describes a method of switching between an image captured by a normal imaging camera and a virtual viewpoint image so as to reduce the viewer's sense of discomfort.

JP 2020-42665 A

However, when the normal imaging camera exists in the imaging range of the virtual viewpoint image generating camera, the normal imaging camera may be reflected in the virtual viewpoint image. When the normal imaging camera is reflected in the virtual viewpoint image, the quality of the virtual viewpoint image may deteriorate.

An object of the technique of the present disclosure is to suppress the influence caused by the presence of the normal imaging camera in the imaging range of the virtual viewpoint image generating camera.

An information processing apparatus according to the technology of the present disclosure includes first viewpoint information for specifying a virtual viewpoint corresponding to a virtual viewpoint image, and imaging by a first imaging device used to generate the virtual viewpoint image. second viewpoint information representing a viewpoint of a second imaging device existing within a range; obtaining means for obtaining second viewpoint information; and outputting the virtual viewpoint image or the captured image obtained by imaging by the second imaging device. and, if the field of view of the virtual viewpoint specified by the first viewpoint information includes the position of the second imaging device specified by the second viewpoint information, the output means outputs the and control means for controlling such that the captured image obtained by imaging by the imaging device of No. 2 is output.

According to the technology of the present disclosure, it is possible to suppress the influence of the presence of the normal imaging camera in the imaging range of the virtual viewpoint image generating camera.

2 is a block diagram showing the functional configuration of the image processing device; FIG. 2 is a block diagram showing a hardware configuration example of an image processing apparatus; FIG. FIG. 10 is a diagram for explaining an imaging range of a camera group for generating a virtual viewpoint image; 4 is a flowchart showing an example of image output control processing; FIG. 11 is a diagram for explaining a viewpoint comparison table; FIG. FIG. 10 is a diagram for explaining a display screen for a user who performs an output image switching operation; 4 is a flowchart of image output control processing; FIG. 4 is a diagram for explaining the relative velocity of an object with respect to a virtual camera; FIG. 10 is a diagram for explaining the ratio of objects in a virtual viewpoint image; 4 is a flowchart showing an example of image output control processing;

Hereinafter, details of the technology of the present disclosure will be described based on embodiments with reference to the accompanying drawings.

<First Embodiment>
[Configuration of image processing system]
FIG. 1 is a configuration diagram showing the entire image processing system of this embodiment. The image processing system includes a virtual viewpoint image generation camera group 110 , a normal imaging camera 120 , an image processing device 100 , and an image switching device 130 .

The image processing device 100 of this embodiment is an information processing device capable of generating a virtual viewpoint image. A virtual viewpoint image is an image representing a view from a different viewpoint (called a virtual viewpoint) from the viewpoint of an actually installed camera, and is also called a free viewpoint image or an arbitrary viewpoint image. A virtual viewpoint image may be a moving image or a still image. In this embodiment, the virtual viewpoint image will be described as a moving image.

In addition, in this embodiment, the virtual viewpoint may be replaced with a virtual camera (virtual camera) for explanation. At this time, the position of the virtual viewpoint corresponds to the position of the virtual camera, and the line-of-sight direction from the virtual viewpoint corresponds to the orientation direction of the virtual camera. A virtual viewpoint image corresponds to a captured image obtained by virtually capturing an image with a virtual camera. The position and orientation of the virtual camera can be specified by the operator of the virtual camera. Therefore, it is possible to generate an image from an arbitrary viewpoint.

The virtual viewpoint image in this embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user. A corresponding image is also included in the virtual viewpoint image. Also, in the present embodiment, the case where the designation of the virtual viewpoint is performed by the user's operation will be mainly described, but the designation of the virtual viewpoint may be automatically performed based on the result of image analysis or the like.

The image processing apparatus 100 includes a virtual viewpoint captured image acquisition unit 101, a virtual viewpoint designation unit 102, a virtual viewpoint image generation unit 103, a physical viewpoint captured image acquisition unit 104, a camera information acquisition unit 106, a similarity calculation unit 107, and It has an output control unit 108 .

The virtual viewpoint captured image acquisition unit 101 captures each of the virtual viewpoint image generation cameras 110, which are a plurality of imaging devices arranged so as to surround an imaging range such as a studio, in synchronization with each other. A captured image corresponding to the angle of view of the camera is acquired. The number and arrangement of the cameras forming the virtual viewpoint image generation camera group 110 are not limited.

The virtual viewpoint designation unit 102 generates virtual camera viewpoint information that defines at least the position and orientation (orientation) of the virtual camera designated by the operator of the virtual camera.

The operator of the virtual camera can specify the desired position and direction of the virtual camera through an operation unit (not shown) connected to the image processing apparatus 100 . The operation unit (not shown) is, for example, a device such as a joystick, but the operation unit (not shown) is not limited to a joystick. In addition, a device such as a mouse or a keyboard used for operating a personal computer may be used.

The viewpoint information of the virtual camera includes the three-dimensional position (position of the virtual camera) on world coordinates, orientation (orientation of the virtual camera), focal length, and principal point (center on the virtual camera image).
The position, orientation, and the like of the virtual camera are defined by generating the viewpoint information of the virtual camera.

The virtual viewpoint image generation unit 103 uses the positional relationship between the plurality of captured images acquired by the virtual viewpoint captured image acquisition unit 101 and the virtual viewpoint image generation camera group 110 to generate images from the virtual camera specified by the virtual viewpoint specification unit 102 . Generate a virtual viewpoint image that represents the appearance of Then, the virtual viewpoint image generation unit 103 outputs an image within the angle of view of the virtual camera as a virtual viewpoint image.

Here, as an example of a method of generating a virtual viewpoint image, a three-dimensional model representing the three-dimensional shape of an object is generated, and a two-dimensional image when the three-dimensional model is viewed from a virtual camera is represented by a projection operation. A method of generating a virtual viewpoint image will be described. A three-dimensional model representing the three-dimensional shape of an object is also called three-dimensional shape data.

First, a three-dimensional model of an object within an imaging range is generated based on images captured by the virtual viewpoint image generation camera group 110 and location information. As a method for constructing a three-dimensional model, there is a method called a visual volume intersection method or VisualHull (hereinafter referred to as VisualHull). In VisualHull, the silhouette of the object on the captured image of each virtual viewpoint image generation camera is virtually back-projected from the optical principal point position of the virtual viewpoint image generation camera in the direction of the object. As a result, a cone region is formed whose vertex is the position of the optical principal point and whose cross section is the silhouette of the object. Then, a three-dimensional model of the object is generated by using an overlap region (logical product) of the cone regions formed for each virtual viewpoint generation camera as a three-dimensional model.

Next, the camera that captured the captured image used for coloring the 3D model is determined from among the cameras constituting the virtual viewpoint image generation camera group 110, and rendering processing is performed to appropriately color the 3D model. As a processing method for determining the cameras to be used for coloring, for example, there is a method of generating a distance image representing the distance from each virtual viewpoint image generation camera to each point constituting a 3D model and determining based on the distance image. be. Coloring is performed by using the distance image to select the color of the captured image of which virtual viewpoint image generation camera is to be used.

Note that the method of generating the virtual viewpoint image is not limited to the method described above.
As a method of generating a virtual viewpoint image, an image-based image processing method such as morphing or billboarding may be used instead of a method of generating a three-dimensional model.

The virtual viewpoint image generation processing is described assuming that the image data sent from the virtual viewpoint image generation camera group 110 is aggregated in the image processing apparatus 100, which is a computer device connected to a network. The network connection is assumed to be Ethernet, which is most commonly used in computer networks, but is not limited to Ethernet. The image processing apparatus 100 is implemented by a device such as a personal computer, workstation, or server. However, the form of the image processing apparatus 100 is not limited to the form described above because the computing power required for the computer device differs depending on the virtual viewpoint image to be generated. In addition, for example, the image processing apparatus 100 may be configured by a plurality of devices, and the necessary image generation processing may be shared among the plurality of devices. When the image processing apparatus 100 is composed of a plurality of devices, the plurality of devices are connected so that data can be exchanged through the aforementioned network connection.

The physical viewpoint captured image acquisition unit 104 acquires the actual captured image captured by the normal imaging camera 120 . The normal imaging camera 120 is an actual imaging device arranged within the imaging range of the virtual viewpoint image generating camera group 110 .

A virtual viewpoint image may not be suitable for expressing a person's facial expression by zooming in. Therefore, the normal imaging camera 120 is used to obtain an image with a higher percentage of objects within the angle of view than the virtual viewpoint image generating camera group 110 . For example, when the object is a person, the normal imaging camera 120 is arranged for the purpose of imaging the facial expression of the person.

The normal imaging camera 120 is, for example, a camera held by a cameraman for imaging, or a camera installed on a tripod or a crane for imaging. The image processing device 100 and the normal imaging camera 120 are connected by, for example, an SDI (Serial Digital Interface) cable. SDI is an interface standard mainly used for professional video equipment. In this embodiment, the normal imaging camera 120 will be described as an imaging device that is handheld by a cameraman.

The camera information acquisition unit 106 acquires viewpoint information defining a virtual camera from the virtual viewpoint image generation unit 103 . The viewpoint information representing the position and orientation of the virtual camera is output from the virtual viewpoint designation unit 102 to the virtual viewpoint image generation unit 103 and used to generate a virtual viewpoint image. Therefore, the camera information acquisition unit 106 can acquire viewpoint information of the virtual camera 304 from the virtual viewpoint image generation unit 103 .

In addition, the camera information acquisition unit 106 acquires viewpoint information indicating the position and orientation of the normal imaging camera 120 as the information of the normal imaging camera 120 from the physical viewpoint captured image acquisition unit 104 .

The acquisition method of each viewpoint information of the camera information acquisition unit 106 described above is an example, and the camera information acquisition unit 106 may directly acquire the viewpoint information of the virtual camera from the virtual viewpoint designation unit 102 . Further, the camera information acquisition unit 106 may directly acquire the viewpoint information of the normal imaging camera 120 from the normal imaging camera 120 . In this case, the camera information acquisition unit 106 acquires viewpoint information from the normal imaging camera 120 through the above-described Ethernet connection.

The viewpoint information of the normal imaging camera 120 can be acquired by mounting a sensor device capable of detecting the position and orientation in the normal imaging camera 120 . Alternatively, the position information of the normal imaging camera 120 may be acquired based on the reflection from the marker obtained by setting a marker in advance in the space in which the normal imaging camera 120 can move and projecting infrared light. Alternatively, the orientation information of the normal imaging camera 120 may be obtained by using an acceleration sensor, a gyro sensor, or the like.

The position and orientation of the normal imaging camera 120 are adjusted in advance so that the positional relationship matches the world coordinates used by the virtual viewpoint image generation unit 103 . By performing this adjustment, the viewpoint information of the normal imaging camera 120 is obtained as information representing the position and orientation in the world coordinate system common to the viewpoint information of the virtual camera. As a method of adjusting the position and orientation in the world coordinate system, for example, the virtual camera and the normal imaging camera 120 are set to have the same angle of view. Then, based on the relationship between the world coordinates and the coordinates of the normal imaging camera 120 at that time, it is possible to adjust the viewpoint information of the normal imaging camera 120 so as to match the world coordinates. Alternatively, it is possible to arrange the normal imaging camera 120 in advance at a location corresponding to the position on the world coordinates, and adjust from the relationship between the coordinates of the normal imaging camera 120 and the world coordinates.

Based on the viewpoint information acquired by the camera information acquisition unit 106, the similarity calculation unit 107 calculates the similarity for determining whether the viewpoint of the virtual camera and the viewpoint of the normal imaging camera 120 are similar. Details will be described later.

The output control unit 108 instructs the image switching device 130 to perform output control according to the degree of similarity calculated by the degree of similarity calculation unit 107 . For example, when it is determined that the viewpoints are similar based on the degree of similarity, the image switching device 130 is instructed to output images by output control according to the similarity of the viewpoints. Details will be described later.

The image switching device 130 acquires virtual viewpoint images from the image processing device 100 . Also, an image captured by the normal imaging camera 120 corresponding to the virtual viewpoint image is acquired from the normal imaging camera 120 . The image switching device 130 acquires the virtual viewpoint image and the captured image via the SDI cable. The image switching device 130 is an output device (output unit) that outputs any of the acquired images. The image output from the image switching device 130 is displayed on the display section viewed by the viewer. Specifically, the image switching device 130 outputs either the acquired virtual viewpoint image or the captured image to broadcasting equipment, a distribution server, or the like (not shown).

In this embodiment, the image switching device 130 appropriately switches between a captured image obtained by capturing an object at a high zoom magnification with the normal imaging camera 120 and a virtual viewpoint image from a desired virtual viewpoint. Control is performed so that it is displayed on the display unit. By switching between the image captured by the normal image capturing camera 120 and the virtual viewpoint image, it is possible to provide the viewer with an image with a higher sense of reality. That is, it is possible to appropriately display an image that captures facial expressions while making use of the features of a virtual viewpoint image with a high degree of freedom.

The image switching device 130 is implemented by, for example, a device called a switcher. Normally, a user switches images to be output by operating a switch provided in the switcher. In this embodiment, the switching of the output image of the image switching device 130 is controlled based on the instruction of the output control unit 108 of the image processing apparatus 100 in addition to the switching instruction by the user.

Each functional unit realized in the image processing apparatus 100 shown in FIG. 1 will be described as being realized by executing a predetermined program by the CPU 201 (see FIG. 2) of the image processing apparatus, which will be described later. It is not limited. For example, hardware such as GPUs (Graphics Processing Units) and FPGAs (Field Programmable Gate Arrays) for speeding up calculations may be used. That is, each functional unit of the image processing apparatus 100 may be implemented by cooperation of software and hardware such as a dedicated IC, or some or all of the functions may be implemented by hardware alone. Also, a configuration may be used in which a plurality of image processing apparatuses 100 are used to distribute and execute the processing of each functional unit.

[Hardware Configuration of Image Processing Apparatus]
FIG. 2 is a block diagram showing a hardware configuration example of the image processing apparatus 100. As shown in FIG. The image processing apparatus 100 has a CPU 201 , a RAM 202 , a ROM 203 , an external storage device 204 and an I/F 205 .

The CPU 201 controls the entire computer using computer programs and data stored in the RAM 202 and ROM 203 .

The RAM 202 has an area for temporarily storing computer programs and data loaded from an external storage device 204, data externally acquired via an I/F (interface) 205, and the like. Furthermore, the RAM 202 has a work area used when the CPU 201 executes various processes. That is, the RAM 202 can be allocated, for example, as frame memory, or can provide other various areas as appropriate. The ROM 203 stores setting data of the computer, a boot program, and the like.

The external storage device 204 is a large-capacity information storage device typified by a hard disk drive. An external storage device 204 stores an OS (operating system) and a computer program for causing the CPU 201 to implement each function of the image processing apparatus 100 shown in FIG. Furthermore, image data to be processed may be stored in the external storage device 204 . Computer programs and data stored in the external storage device 204 are appropriately loaded into the RAM 202 under the control of the CPU 201 and are processed by the CPU 201 .

An I/F 205 is an interface for connecting to a network such as a LAN or the Internet, and the image processing apparatus 100 can acquire or transmit various information via the I/F 205 . Also, a display unit, an operation unit, and other devices (not shown) can be connected via the I/F 205 . A bus 206 is a bus for connecting the units described above.

The display unit (not shown) is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) for the user to operate the image processing apparatus 100, and the like. An operation unit (not shown) is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 201 in response to user operations. The CPU 201 also operates as a display control unit that controls the display unit and an operation control unit that controls the operation unit.

Note that the hardware configuration of the image switching device 130 is also the same as that of FIG. 2, so the description is omitted.

[About camera placement]
FIG. 3 is a diagram showing a state in which an imaging range 300 of the virtual viewpoint image generation camera group 110 is viewed from above. For example, the imaging range 300 is a studio where objects 303 such as singers, dancers, etc. perform. As shown in FIG. 3, the virtual viewpoint image generating camera group 110 is arranged around the studio, and captures images of the studio from various angles in synchronization with each other. As a result, captured images of a plurality of viewpoints are obtained.

Further, in the imaging range 300, there are a normal imaging camera 120 and a cameraman 308 who uses the normal imaging camera 120 to perform imaging. That is, in the studio, imaging by the normal imaging camera 120 and imaging by the virtual viewpoint image generation camera group 110 are performed simultaneously.

The normal imaging camera 120 exists within the imaging range of at least one of the cameras that constitute the virtual viewpoint image generating camera group 110 . Therefore, depending on the position of the normal imaging camera 120 , the subject may be included as a subject within the angle of view of one or more cameras among the cameras forming the virtual viewpoint image generating camera group 110 .

Virtual camera 304 in FIG. 3 represents the position and orientation of the virtual camera specified by the operator of the virtual camera. A direction 305 two-dimensionally represents the orientation of the virtual camera 304 . In this case, if a virtual viewpoint image is generated from the virtual camera 304, a virtual viewpoint image containing not only the original object 303 but also the normal imaging camera 120 and the cameraman 308 is generated. In this case, for example, since the normal imaging camera 120 and the cameraman 308 may become an eyesore, processing may be required to prevent the normal imaging camera 120 and the cameraman 308 from being represented in the virtual viewpoint image. .

For example, of the cameras forming the virtual viewpoint image generation camera group 110, the normal imaging camera 120 and the cameraman 308 generate virtual viewpoint images without using the camera data included in the angle of view. By processing in this way, a method of generating a virtual viewpoint image that does not include the normal imaging camera 120 and the cameraman 308 can be considered. FIG. 3 shows that the angle of view 307 of the camera 110b that constitutes the virtual viewpoint image generating camera group 110 includes the normal imaging camera 120 and the cameraman 308. FIG. Therefore, in the example of FIG. 3, it is conceivable to generate a virtual viewpoint image by excluding the data of the camera 110b. However, the object 303 is also included in the angle of view 307 of the camera 110b. If the virtual viewpoint image is generated without using the data of the camera 110b that has captured the object 303, the quality of the virtual viewpoint image may deteriorate.

In addition, there is a case where the cameraman 308 takes images while moving frequently. In this case, even if the object 303 is stationary, the camera used to generate the virtual viewpoint image changes every several frames. Therefore, if the data of the camera 110b that was able to capture the object 303 becomes unusable, the virtual viewpoint image will change for each frame, and if the virtual viewpoint image is displayed as a moving image, it will appear unnatural to the viewer. It can make you feel.

Therefore, in the present embodiment, based on viewpoint information representing the position and orientation of the virtual camera and viewpoint information representing the position and orientation of the normal imaging camera 120, it is determined whether the respective viewpoints are similar. Then, when the images are similar, a method of controlling so that the image captured by the normal image capturing camera 120 is output will be described. By performing such control, it is possible to prevent a person other than the original object, such as a cameraman, from being captured while providing an image from the virtual camera.

In order to simplify the explanation, FIG. 3 shows an example of a two-dimensional view of an object to be imaged, but it is also possible to deal with three-dimensional parameters.

[About similarity]
Based on the viewpoint information of the virtual camera 304 and the viewpoint information of the normal imaging camera 120, the similarity calculation unit 107 calculates the degree of similarity between the virtual viewpoint, which is the viewpoint of the virtual camera 304, and the viewpoint of the normal imaging camera 120. A degree of similarity, which is a value representing the degree, is calculated. An example of similarity calculation will be described with reference to FIG.

In FIG. 3, a direction 309 represents the orientation (orientation) of the normal imaging camera 120 two-dimensionally. A direction 305 represents the orientation (orientation) of the virtual camera 304 two-dimensionally. An angle θ 310 indicates the difference between the orientation of the virtual camera 304 and the orientation of the normal imaging camera 120 (the angle formed by the directions 305 and 309). A distance 311 indicates the distance from the position of the virtual camera 304 to the position of the normal imaging camera 120 . In this embodiment, the distance 311 and the angle 310 are calculated as the similarity of viewpoints. A distance 311 and an angle 310 are calculated based on each viewpoint information.

By comparing the calculated degree of similarity with a predetermined threshold value, it is possible to determine whether the viewpoint of the virtual camera 304 and the viewpoint of the normal imaging camera 120 are similar. In this embodiment, when the distance 311 is smaller than a predetermined first threshold and the angle 310 is smaller than a predetermined second threshold, the viewpoint of the virtual camera 304 and the viewpoint of the normal imaging camera 120 are are determined to be similar.

[About image output control]
FIG. 4 is a flowchart for explaining the flow of image output control processing. A series of processes shown in the flowchart of FIG. 4 is performed by the CPU 201 of the image processing apparatus 100 developing the program code stored in the ROM 203 in the RAM 202 and executing the program code. Also, some or all of the functions of the steps in FIG. 4 may be realized by hardware such as ASIC or electronic circuits. Note that the symbol "S" in the description of each process means a step in the flowchart, and the same applies to subsequent flowcharts.

It should be noted that the description will be made on the assumption that virtual viewpoint images are generated in parallel with the processing of the flowchart of FIG. good.

In S<b>401 , the camera information acquisition unit 106 acquires viewpoint information (first viewpoint information) representing the position and orientation of the virtual camera from the virtual viewpoint image generation unit 103 . In this step, viewpoint information of the virtual camera corresponding to the virtual viewpoint image output when the image switching device 130 switches the output to the virtual camera is acquired. Since the image output control process is performed repeatedly, when the process of this step is performed next time, the viewpoint information of the virtual camera corresponding to the virtual viewpoint image to be output next is acquired.

In S<b>402 , the camera information acquisition unit 106 acquires viewpoint information (second viewpoint information) representing the position and orientation of the normal imaging camera 120 . The viewpoint information acquired in this step is the viewpoint information of the normal imaging camera 120 when obtaining the captured image corresponding to the virtual viewpoint image in S401. For example, at the same time as the time of the virtual viewpoint image based on the virtual viewpoint acquired in S401, the viewpoint information of the normal imaging camera 120 when the normal imaging camera 120 captured the image is acquired.

In S403, the similarity calculation unit 107 calculates the similarity between the two viewpoints based on the obtained viewpoint information of the normal imaging camera 120 and the viewpoint information of the virtual camera. As described above, in this embodiment, as similarity, the angle indicating the difference between the orientation of the virtual camera and the orientation of the normal imaging camera 120, and the distance between the position of the virtual camera and the position of the normal imaging camera 120, Calculated.

In S404, the output control unit 108 determines whether the viewpoint of the normal imaging camera 120 and the virtual camera are similar based on the degree of similarity derived in S403. If there are a plurality of normal imaging cameras 120, it is determined whether there is a normal imaging camera 120 with a similar viewpoint to that of the virtual camera. Then, the processing is switched according to the result of the determination.

If it is determined that there is a normal imaging camera 120 similar to the virtual viewpoint (YES in S404), the process proceeds to S405, and the output control unit 108 turns on the output control based on the degree of similarity for the image switching device 130. give instructions.

Output control based on similarity is control for automatically switching the image to be output to the captured image of the normal imaging camera 120 . In addition, when the image captured by the normal imaging camera 120 has already been output, even if the user issues an instruction to switch the image to be output to the virtual viewpoint image, the user's instruction is not accepted and the normal imaging camera 120 is used to capture the image. This is control for outputting an image. Therefore, it is possible to output and display an image that has an angle of view similar to that of the virtual viewpoint image and that does not include the normal imaging camera 120 and the cameraman 308 .

On the other hand, if it is determined that there is no normal imaging camera 120 similar to the virtual viewpoint (NO in S404), the process proceeds to S406, and the output control unit 108 turns off the output control based on the degree of similarity for the image switching device 130. give instructions to Therefore, the image switching device 130 outputs an image according to the user's switching instruction.

The image processing apparatus 100 continuously performs the above-described processes of S401 to S406 at predetermined cycles (constant time intervals) for a predetermined period. That is, when the predetermined number of frames are output, the processing of S401 to S406 is performed again, and for the output of the next predetermined number of frames, an instruction to turn ON or OFF the output control based on the degree of similarity is given. A decision is made as to whether to issue an instruction to The shorter the time interval at which the next steps S401 to S406 are performed, that is, the higher the frequency of the decision processing to turn ON or OFF the output control based on the similarity, the more the motion of the virtual camera and the normal imaging camera 120 is affected. , the tracking performance becomes higher, but the computational load increases.

A plurality of virtual cameras may be designated, and a plurality of normal imaging cameras may exist. Even in this case, it is possible to perform the control of this embodiment.

FIG. 5 is a view showing a viewpoint comparison table 500 representing the result of comparing the respective viewpoints when there are a plurality of virtual cameras and a plurality of normal imaging cameras. The viewpoint comparison table in FIG. 5 shows a viewpoint comparison table when there are four virtual cameras 1 to 4 as virtual cameras and two normal imaging cameras 1 to 2 as normal imaging cameras. When there are a plurality of virtual cameras, the similarity calculation unit 107 calculates the similarity of the viewpoint with the normal imaging camera for each virtual camera. Then, the output control unit 108 compares the degree of similarity with the threshold value, determines whether the viewpoint of the virtual camera and the viewpoint of the normal imaging camera are similar, and stores the result in a viewpoint comparison table.

The output control unit 108 instructs the image switching device 130 to perform output control according to the viewpoint comparison table 500 as output control based on the degree of similarity. For example, in the flowchart of FIG. 4, after calculating the degree of similarity in S403, in S404 the output control unit 108 generates the viewpoint comparison table 500 and determines whether there is a normal imaging camera similar to the virtual camera. If it is determined that there is a normal imaging camera similar to the virtual camera, in S405 the image switching device 130 is instructed to perform output control based on the degree of similarity in accordance with the viewpoint comparison table 500 . Note that instead of the processing of S404 to S406, the output control unit 108 generates the viewpoint comparison table 500, and performs output control according to the viewpoint comparison table 500 as output control based on the degree of similarity. 130 may be instructed.

An example of output control according to the viewpoint comparison table 500 executed in the image switching device 130 will be described with reference to FIG. As shown in FIG. 5, it is determined that the viewpoint of the normal imaging camera 1 is similar to the viewpoint of the virtual camera 1, and that the viewpoint of the normal imaging camera 2 is not similar. In this situation, when the user gives an instruction to output the virtual viewpoint image corresponding to the virtual camera 1, the image switching device 130 automatically outputs the captured image of the normal imaging camera 1 whose viewpoint is similar to that of the virtual camera 1. Control the output to switch. In addition, output control is performed so that switching from the normal imaging camera 1 to the normal imaging camera 2 can be performed at any timing of the user.

When the user gives an instruction to output the virtual viewpoint image corresponding to the virtual camera 2, the image switching device 130 automatically switches the output to the captured image of the normal imaging camera 2 whose viewpoint is similar to that of the virtual camera 2. conduct. In addition, output control is performed so that the output can be switched from the captured image of the normal imaging camera 2 to the captured image of the normal imaging camera 1 at arbitrary timing of the user.

It has been determined that the viewpoint of virtual camera 3 is not similar to the viewpoint of either normal imaging camera. In this situation, when the user gives an instruction to output the virtual viewpoint image corresponding to the virtual camera 3, the image switching device 130 performs output control so as to switch to the virtual viewpoint image of the virtual camera 3 at the user's arbitrary timing.

The viewpoint of the virtual camera 4 is similar to that of either normal imaging camera. In this situation, when the user gives an instruction to output the virtual viewpoint image corresponding to the virtual camera 4, the image switching device 130 performs output control to automatically switch the output to the captured image of one of the normal imaging cameras. As to whether to switch to the captured image of the normal imaging camera 1 or the normal imaging camera 2, the priority is set in advance to the normal imaging camera, and the output is switched to the captured image of the camera with the higher priority. . Alternatively, instead of determining whether or not the viewpoints are similar, a normal imaging camera whose viewpoint is more similar to the virtual camera is determined based on the degree of similarity, and the output is switched to the image captured by the determined camera. You can do it. For example, the image is switched to the captured image of the normal imaging camera that is closer to the virtual camera.

As described above, according to the present embodiment, when the virtual viewpoint image and the captured image of the normal imaging camera are switched and output, the image can be displayed even when the cameraman or the like is reflected in the angle of view of the virtual camera. It becomes possible to suppress deterioration in image quality.

In the above description, the method of calculating the degree of similarity used for determining whether the viewpoint of the virtual camera and the viewpoint of the normal imaging camera are similar from the viewpoint information representing the position and orientation of the camera has been described. . Alternatively, it may be determined whether the viewpoint of the virtual camera and the viewpoint of the normal imaging camera are similar based on the similarity between the virtual viewpoint image and the image captured by the normal imaging camera.

For example, the camera information acquisition unit 106 acquires the virtual viewpoint image from the virtual viewpoint image generation unit 103 and the captured image of the normal imaging camera from the physical viewpoint captured image acquisition unit 104 . The camera information acquisition unit 106 outputs the acquired virtual viewpoint image and captured image to the similarity calculation unit 107 . A similarity calculation unit 107 calculates the similarity of images from the image data of the virtual viewpoint image and the captured image of the normal imaging camera. The output control unit 108 determines that the viewpoints are similar when the degree of similarity between the images exceeds a predetermined threshold. The method of output control when it is determined that the viewpoints are similar is the same as the method described above.

As a method of calculating the degree of similarity between images, for example, there is a method of extracting feature points on an image and calculating the degree of matching of the feature points as the degree of similarity. Alternatively, objects can be identified in advance using a machine learning method, and the similarity can be calculated by identifying the objects from the virtual viewpoint image and the image captured by the normal imaging camera, and comparing the positional relationship of the objects. can be calculated. These similarity calculation methods are examples, and the method for calculating the image similarity is not limited in this embodiment.

In addition, the similarity between the viewpoint based on the viewpoint information of the virtual camera and the viewpoint information of the normal imaging camera and the similarity between the virtual viewpoint image and the image captured by the normal imaging camera are combined to determine the viewpoint. Similarity may be determined. For example, calculation of image similarity requires a higher computational load on the image processing apparatus 100 than calculation of viewpoint similarity. For this reason, for example, only when it is determined that the distance 311 is smaller than a predetermined first threshold value and the angle 310 is smaller than a predetermined second threshold value based on the viewpoint information, the image similarity is further evaluated. calculate. Then, it may be determined that the viewpoints are similar when the similarity of the images is higher than a predetermined threshold. By determining whether the viewpoints are similar in two stages in this way, it is possible to increase the accuracy of determining whether the viewpoints are similar while suppressing the load.

<Second embodiment>
In the first embodiment, as the output control executed by the output control unit 108, when there is a normal imaging camera having a viewpoint similar to that of the virtual camera, a method of executing control so that an image captured by the normal imaging camera is output. explained. In the present embodiment, when there is a normal imaging camera having a viewpoint similar to that of the virtual camera, a method of notifying the user who operates the switch of the image switching device 130 to switch the output image will be described. This embodiment will be described with a focus on differences from the first embodiment. Parts that are not particularly specified have the same configuration and processing as in the first embodiment.

In this embodiment, two virtual cameras, virtual camera 1 and virtual camera 2, are designated, and a case where two cameras, normal imaging camera 1 and normal imaging camera 2, exist will be described.

FIG. 6 is a diagram showing a screen 601 displayed on a display unit (not shown) that can be viewed by the user of the image switching device 130. As shown in FIG. It is assumed that a display unit (not shown) on which screen 601 is displayed is connected to image switching device 130 .

The screen 601 includes screens 602 to 605 on which the virtual viewpoint image and the captured image at the same time are displayed. For example, when the object 303 is being imaged simultaneously by four cameras, two virtual cameras and two normal imaging cameras, the screen 601 is displayed on the display unit (not shown) of the image switching device 130. to be displayed. A user who switches images to be output by the image switching device 130 can select an image while viewing the screen 601 and switch the images so that the selected image is output.

A screen 602 is a screen for displaying an imaged image obtained by imaging by the normal imaging camera 1 . A screen 603 is a screen for displaying an imaged image obtained by imaging by the normal imaging camera 2 . A screen 604 is a screen for displaying a virtual viewpoint image seen from the virtual camera 1 . A screen 605 is a screen for displaying a virtual viewpoint image representing the view from the virtual camera 2 .

A screen 604 displays an image output from the image switching device 130 . That is, the image displayed on the screen 604 is the image output according to the user's instruction. The screen 604 is display-controlled such that the thickness of the frame of the screen is emphasized compared to the other screens 602 , 603 , and 605 . Although not shown in FIG. 6, the screen 604 is colored differently from the other screens. For example, the screen 604 is surrounded by a thick red frame. Therefore, the user can easily confirm that the image currently being output by the image switching device 130 is the image displayed on the screen 604 .

Output control according to the present embodiment, which is executed by the image switching device 130 based on instructions from the output control unit 108, will be described with reference to FIG. As described above, it is assumed that the image whose output is instructed by the user who operates the image switching device 130 is the virtual viewpoint image of the virtual camera 1 . Then, of the normal imaging camera 1 and the normal imaging camera 2, it is determined that the normal imaging camera 1 has a similar viewpoint to the virtual camera 1. FIG. In this case, on the screen 604 of the currently selected virtual camera 1, a notification 607 indicating that there is a normal imaging camera with a similar viewpoint is displayed superimposed on the virtual viewpoint image.

Note that the notification 607 to the user who operates the image switching device 130 may include information for specifying a similar normal imaging camera. That is, the notification 607 may include information that the viewpoint of the normal imaging camera 1 is similar to that of the virtual camera 1 .

As a method of notifying the user, instead of the notification 607 or in addition to the notification 607, notification by sound or light may be performed using a mechanism for emitting sound or light so as to call the attention of the user. good.

In this embodiment, the case where there are multiple cameras has been described. Therefore, as described in the first embodiment, the output control unit 108 may generate the viewpoint comparison table 500 and instruct the image switching device 130 to perform output control based on the viewpoint comparison table 500 .

Alternatively, one virtual camera and one normal imaging camera may be provided. In this case, in the flowchart of FIG. 4, if an instruction to turn on output control based on similarity is given in S405, the image switching device 130 notifies the user. If an instruction to turn off the output control based on the degree of similarity is given in S406, control is performed so as not to notify the user.

Note that the display unit that displays the screen 601 may be connected to the image processing apparatus 100 . In this case, the CPU 201 of the image processing apparatus 100 performs display control so that the screen 601 is displayed based on instructions from the output control unit 108 .

As described above, according to the present embodiment, the output image is switched according to the user's instruction. Therefore, even if there are normal imaging cameras with similar viewpoints, it is possible for the user to adjust the switching timing, such as not switching the output.

<Third Embodiment>
In this embodiment, a method of instructing the image switching device 130 to skip the similarity calculation and turn off the output control based on the similarity when a predetermined condition is satisfied will be described. This embodiment will be described with a focus on differences from the first embodiment. Parts that are not particularly specified have the same configuration and processing as in the first embodiment. The functional configuration of the image processing apparatus 100 differs from that of the first embodiment in that the similarity calculation unit 107 performs different processing.

FIG. 7 is a flowchart for explaining image output control processing in this embodiment. A series of processes shown in the flowchart of FIG. 7 are performed by the CPU 201 of the image processing apparatus 100 developing the program code stored in the ROM 203 in the RAM 202 and executing the program code. S702 to S707 are the same processes as S401 to S406 shown in FIG. In this embodiment, before performing S702 to S707, in S701, the similarity calculation unit 107 determines whether or not a condition (exclusion condition) for not executing viewpoint similarity calculation is satisfied.

A case where the exclusion condition is satisfied is, for example, a case where the relative velocity of the object with respect to the virtual camera exceeds a predetermined threshold. Alternatively, it may be determined that the exclusion condition is satisfied when the ratio of the area representing the object to the entire virtual viewpoint image is below a predetermined threshold value.

Even if the normal imaging camera 120 or the cameraman 308 is included in the angle of view of the virtual camera, if the object 303 moves quickly or the proportion of the object 303 occupied by the object 303 is small, the image quality of the virtual viewpoint image may deteriorate. hard to notice. In such a case, it is preferable to turn off the output control based on the degree of similarity, since it is only necessary to switch and display the images as instructed by the user. Therefore, in this embodiment, when the exclusion condition is satisfied (YES in S701), the process proceeds to S707. S707 is the same processing as S406. As with the processing of the flowchart of FIG. 4 of the first embodiment, the processing of the flowchart of FIG. 7 is repeatedly executed at regular time intervals.

FIG. 8 is a diagram showing a state in which the imaging range 300 of the virtual viewpoint image generation camera group 110 is viewed from above. As an example of the processing of S701, an example of determining that the exclusion condition is satisfied when the relative velocity of the object with respect to the virtual camera exceeds a predetermined threshold will be described with reference to FIG.

In FIG. 8, the normal imaging camera 120 and the virtual camera 304 represent a state in which the object 303 is imaged at the same time, as in FIG. As described above, S701 to S707 are repeatedly executed at regular time intervals. A position 801 is the position of the virtual camera 304 obtained in the previous processing of S701, and a position 802 indicates the position of the virtual camera 304 obtained in the current processing of S701. Thus, the virtual camera 304 in FIG. 8 indicates that it has been specified to image the object 303 while moving. FIG. 8 shows that the object 303 existed at a position 803 during the previous processing of S701, and was located at a position 804 during the current processing of S701.

In this example, the similarity calculation unit 107 acquires and holds the position of the virtual camera 304 in S701. Therefore, if it is determined in S701 that the exclusion condition is not satisfied, the position of the virtual camera 304 may not be determined in S704 in this embodiment. Further, in S701, information on the orientation of the virtual camera may be acquired in addition to the position of the virtual camera. In this case, S703 may be skipped.

Then, in S701, the similarity calculation unit 107 calculates the moving speed of the virtual camera. Since a certain time interval for executing S701 to S707 in the flowchart of FIG. 7 is set in advance, the moving speed of the virtual camera is calculated by calculating the moving distance of the virtual camera from the change in the position of the virtual camera. be able to.

Next, in S<b>701 , the similarity calculation unit 107 acquires the position of the object 303 and calculates the moving speed of the object 303 . As with the virtual camera, the movement speed can be calculated by calculating the movement distance from the change in the position of the object 303 .

In the process of generating the three-dimensional model of the object in order for the virtual viewpoint image generating unit 103 to generate the virtual viewpoint image, the position of the three-dimensional model of the object on the world coordinates is obtained. Therefore, the similarity calculation unit 107 can obtain the outline of the position of the object from the virtual viewpoint image generation unit 103 .

When the three-dimensional model of the object is not generated, the object is identified from the images of a plurality of cameras in the virtual viewpoint image generation camera group 110 . Then, it is possible to calculate the position of the object from the positional relationship of each virtual viewpoint image generation camera that is grasped in advance. The technology for identifying objects from images is not limited. For example, a technique of separating moving objects from the background, or a method of learning and identifying objects in advance by machine learning may be used.

Then, in S701, the similarity calculation unit 107 calculates the relative speed of the object with respect to the virtual camera based on the moving speed of the virtual camera and the moving speed of the object. Then, the similarity calculation unit 107 determines whether the relative velocity of the object with respect to the virtual camera exceeds a predetermined threshold. If the threshold is exceeded, it is determined that the exclusion condition is satisfied.

When the relative velocity of the object with respect to the virtual camera exceeds a predetermined threshold, the viewer of the image is less likely to perceive the deterioration of the image quality. Therefore, it is not necessary to perform the output control based on the degree of similarity described in the above embodiment. Therefore, when it is determined in S701 that the exclusion condition is satisfied, the output control unit 108 instructs the image switching device 130 to turn off the output control based on the degree of similarity.

If there are two objects (two people) or more, for example, the relative velocities of all the objects are calculated, and if the relative velocities of all the objects exceed the threshold, it is determined in S701 that the exclusion condition is satisfied. . Alternatively, the relative velocities of objects within a predetermined range from the position of the virtual camera may be calculated, and if all the calculated relative velocities exceed the threshold, it may be determined in S701 that the exclusion condition is satisfied.

FIG. 9 is a diagram showing a state in which the imaging range 300 of the virtual viewpoint image generation camera group 110 is viewed from above, and a virtual viewpoint image 901 corresponding to the virtual camera 304 . Next, as an example of the processing of S701, an example of determining that the exclusion condition is satisfied when the ratio of the area representing the object in the virtual viewpoint image is smaller than the threshold will be described with reference to FIG.

FIG. 9 shows a state in which the normal imaging camera 120 and the virtual camera 304 are imaging the object 303 at the same time. The virtual camera 304 in FIG. 9 is designated to be positioned outside the imaging range 300 of the virtual viewpoint image generation camera group 110 . In this way, the virtual camera 304 can virtually move anywhere within the area where the world coordinates are defined. Therefore, the position of the virtual camera 304 is a general state. It is also possible to move the virtual camera 304 to a position overlapping the object 303 .

In FIG. 9, the virtual camera 304 is capturing an image of the object 303, and the virtual viewpoint image 901 is an image showing the angle of view of the virtual camera 304 in FIG. Since the virtual camera 304 in FIG. 9 captures the object 303 in a drawn state, the proportion of the entire image occupied by the object 303 on the virtual viewpoint image 901 is small. When an object occupies a small proportion of the entire image, as in the virtual viewpoint image 901, the viewer is less likely to notice the deterioration of the image quality. Therefore, if it is determined that the proportion of the area representing the object included in the virtual viewpoint image to the entire virtual viewpoint image is smaller than the threshold, it is determined that the exclusion condition is satisfied, and the process proceeds to S707.

The calculation of the ratio of the area representing the object to the entire image of the virtual viewpoint image (angle of view of the virtual camera) can be performed, for example, in the process of generating the virtual viewpoint image. Specifically, after generating the three-dimensional model of the object, the virtual viewpoint image generating unit 103 performs perspective projection transformation on the vertices of the circumscribing rectangular parallelepiped of the three-dimensional model to the virtual camera coordinate side of the virtual viewpoint image. Thus, it is possible to obtain the ratio of the area representing the object to the entire virtual viewpoint image. The ratio of the area representing the object obtained by this method to the entire image can be output from the virtual viewpoint image generation unit 103 and obtained by the similarity calculation unit 107 .

Alternatively, the similarity calculation unit 107 obtains the virtual viewpoint image from the virtual viewpoint image generation unit 103 and performs processing to identify the object from the virtual viewpoint image, thereby calculating the proportion of the area representing the object to the entire virtual viewpoint image. may be determined. In this case, the method of extracting the object is not limited. For example, as described above, a technique for separating moving objects from the background, or a method for identifying objects by learning them in advance by machine learning may be used.

Alternatively, the similarity calculation unit 107 may determine similarity when the angle of view (field of view) of the virtual camera 304, that is, the virtual viewpoint image 901 includes the normal imaging camera 120. .

When such determination is made, it is assumed that the image processing system is provided with a mechanism capable of acquiring the position of the normal imaging camera 120, and that acquisition is possible by this mechanism. As a mechanism for acquiring the position of the camera, there is a means for calculating the position information of the camera itself by setting a plurality of reflective markers in advance in the photographing area and photographing the reflective markers. For the acquired position of the normal imaging camera 120, the camera coordinates can be associated with the coordinates in the virtual space by calculating the correspondence from the relationship between the origin in the virtual space and the coordinates of the camera installed at the origin, for example. . Further, by calculating the position of the normal imaging camera 120 in the virtual space and subjecting the position to the virtual viewpoint image 901 by projective transformation, it becomes possible to determine whether or not it is included in the angle of view of the virtual viewpoint image 901 .

Also, the normal imaging camera 120, which is a three-dimensional model as one of the objects in the virtual space, is projected onto the angle of view of the virtual camera 304, and a part of the normal imaging camera 120 is included in the virtual viewpoint image 901. It may be determined whether or not In addition, object recognition processing is performed using the virtual viewpoint image 901 itself as image data, and when the normal imaging camera 120 is identified, it is determined that the virtual viewpoint image 901 includes the normal imaging camera 120 . may Alternatively, it may be determined whether the above-described normal imaging camera 120 is included in a predetermined range within the angle of view instead of the entire angle of view of the virtual viewpoint image 901 .

In this way, when the normal imaging camera 120 is included in the angle of view of the virtual camera 304, the output control unit 108 performs image switching so that the image of the normal imaging camera 120 is output as output control based on the degree of similarity. The device 130 is instructed. Although it has been described that the similarity is determined to be high when the normal imaging camera 120 is included in the angle of view of the virtual camera 304, the present invention is not limited to this. That is, independently of the processing based on similarity determination, when the normal imaging camera 120 is included in the angle of view of the virtual camera 304, the processing of switching the output image to the image captured by the normal imaging camera 120 is performed. may

Processing for switching the output image to the image captured by the normal imaging camera 120 when the angle of view of the virtual camera 304 includes the normal imaging camera 120 will be described below with reference to FIG. 10 . Note that processing steps similar to those in FIG. 4 are denoted by the same reference numerals, and descriptions thereof are omitted. In the following description, it is assumed that the similarity calculation unit 107 determines whether the normal imaging camera 120 is included in the angle of view of the virtual camera 304. Any configuration may be used.

In S<b>1001 , the similarity calculation unit 107 identifies the angle of view (field of view) of the virtual camera 304 based on viewpoint information representing the position and orientation of the virtual camera 304 . The similarity calculation unit 107 also calculates the relationship between the position of the normal imaging camera 120 represented by the viewpoint information of the normal imaging camera 120 and the angle of view of the virtual camera 304 .

In S<b>1002 , the similarity calculation unit 107 determines whether the position of the normal imaging camera 120 is included in the angle of view of the virtual camera 304 based on the relationship calculated in S<b>1002 . As a determination method at this time, for example, it is specified whether the position coordinates of the normal imaging camera 120 are included in the angle of view of the virtual camera 304 , or if the virtual viewpoint image corresponding to the virtual camera 304 is displayed by the normal imaging camera 120 . A method of specifying whether is included is used.

If included, the output control unit 108 turns on output control for the image switching device 130 in S1003. If not included, the output control unit 108 turns OFF the output control for the image switching device 130 in S1004. Note that the processes in S1003 and S1004 are the same as those in S405 and S406, respectively.

By the processing described above, when the normal imaging camera 120 is included in the angle of view of the virtual camera 304, the output image is switched to the image captured by the normal imaging camera 120, so that the normal imaging camera 120 is reflected in the virtual viewpoint image. can be suppressed. It should be noted that determination based on the degree of similarity may be further combined with the processing described with reference to FIG. 10 .

Further, the output control unit 108 instructs the virtual viewpoint image generation unit 103 to move the virtual camera 304 to the position of the normal imaging camera 120 when switching to output the image of the normal imaging camera 120 . may be performed. At this time, when the virtual camera 304 reaches the position of the normal imaging camera 120 , the image is switched to that of the normal imaging camera 120 .

As described above, according to the present embodiment, it is possible to control the output of an image specified by the user while suppressing deterioration in the quality of the output virtual viewpoint image.

<Other embodiments>
In the above embodiment, the image processing device 100 and the image switching device 130 are described as being separate devices, but the functions of the image switching device 130 may be included in the image processing device 100 .

The present disclosure provides a program that implements one or more functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in a computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 image processing device 106 camera information acquisition unit 107 similarity calculation unit 108 output control unit

Claims (16)

First viewpoint information for specifying a virtual viewpoint corresponding to a virtual viewpoint image, and a viewpoint of a second imaging device existing within an imaging range of the first imaging device used to generate the virtual viewpoint image an acquisition means for acquiring second viewpoint information representing
output means for outputting the virtual viewpoint image or the captured image obtained by capturing by the second imaging device;
When the field of view of the virtual viewpoint specified by the first viewpoint information includes the position of the second imaging device specified by the second viewpoint information, the second imaging device is selected by the output means. and control means for controlling such that the captured image obtained by imaging is output. calculating means for calculating a similarity between viewpoints based on the first viewpoint information and the second viewpoint information;
When the similarity of the viewpoints is higher than a threshold, the control means performs control so that the captured image obtained by the second imaging device is output by the output means. The information processing device according to claim 1 . image acquisition means for acquiring the virtual viewpoint image corresponding to the virtual viewpoint and the captured image obtained by capturing by the second imaging device;
a calculating means for calculating a degree of similarity between the virtual viewpoint image and the captured image captured by the second imaging device;
has
When the degree of similarity of the images is higher than a threshold, the control means controls the output means to output the captured image obtained by imaging by the second imaging device. The information processing device according to claim 1 . The acquisition means is
A first position that is the position of the virtual viewpoint and a first direction that is a line-of-sight direction from the virtual viewpoint are obtained as the first viewpoint information, and the second imaging is performed as the second viewpoint information. obtaining a second position, which is the position of the device, and a second direction, which is the orientation direction of the second imaging device;
The calculation means is
4. The information processing apparatus according to claim 2, wherein an angle representing a distance from said first position to said second position and a difference between said first direction and said second direction is calculated. . The control means is
Control according to the degree of similarity to the output means capable of switching and outputting either the captured image obtained by imaging by the second imaging device or the virtual viewpoint image according to a user's instruction. 5. The information processing apparatus according to any one of claims 2 to 4, wherein an instruction is given so that the The control means is
6. The method according to any one of claims 2 to 5, wherein when there are a plurality of said second imaging devices, it is determined whether the degree of similarity between each viewpoint of said plurality of second imaging devices and said virtual viewpoint is higher than a threshold. The information processing apparatus according to any one of items 1 and 2. The control means is
When there are a plurality of the second imaging devices with the degree of similarity higher than a threshold, a captured image obtained by imaging by one of the plurality of second imaging devices is output. 7. The information processing apparatus according to claim 6, wherein control is performed so as to: a receiving means for receiving an instruction from a user to switch an image output by the output means;
The control means is
When the similarity between the virtual viewpoint and the viewpoint of the second imaging device is higher than a threshold, and the virtual viewpoint image is output, the captured image obtained by imaging by the second imaging device. and even if the user instructs to switch the output image from the captured image obtained by imaging by the second imaging device to the virtual viewpoint image, the second imaging is performed without following the instruction. 8. The information processing apparatus according to any one of claims 2 to 7, wherein the output means is instructed to perform control for outputting the captured image obtained by imaging by the apparatus. The control means is
The information processing apparatus according to any one of claims 2 to 8, wherein control not based on the degree of similarity is performed when a predetermined condition is satisfied. the imaging range includes an object;
10. The information processing apparatus according to claim 9, wherein when the predetermined condition is satisfied, the relative speed of the object with respect to the virtual viewpoint is greater than a predetermined value. the imaging range includes an object;
11. The information processing apparatus according to claim 9, wherein the predetermined condition is satisfied when the proportion of the object in the virtual viewpoint image is smaller than a predetermined value. The output means is
outputting the virtual viewpoint image or the captured image obtained by imaging by the second imaging device based on a user's instruction;
The control means is
When the field of view of the virtual viewpoint specified by the first viewpoint information includes the position of the second imaging device specified by the second viewpoint information, a predetermined notification is sent to the user. 12. The information processing apparatus according to any one of claims 1 to 11, wherein control is performed to The predetermined notification includes information indicating that the field of view of the virtual viewpoint specified by the first viewpoint information includes the position of the second imaging device specified by the second viewpoint information. 13. The information processing apparatus according to claim 12, which is a notification. When the virtual viewpoint image includes an image representing the second imaging device, the control means causes the output means to output the captured image obtained by imaging by the second imaging device. The information processing apparatus according to any one of claims 1 to 13, characterized by controlling. First viewpoint information for specifying a virtual viewpoint corresponding to a virtual viewpoint image, and a viewpoint of a second imaging device existing within an imaging range of the first imaging device used to generate the virtual viewpoint image an obtaining step of obtaining second viewpoint information representing
an output step of outputting the virtual viewpoint image or the captured image obtained by capturing by the second imaging device;
When the field of view of the virtual viewpoint specified by the first viewpoint information includes the position of the second imaging device specified by the second viewpoint information, the second imaging device is selected in the output step. and a control step of controlling so that the captured image obtained by imaging is output. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 14.

Images