JP2019080183A - Image transmission device, image transmission method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce tightness of a transmission band of a network for transmitting a captured image necessary for generating a virtual viewpoint image. In an image processing system that generates a virtual viewpoint image using a plurality of captured images captured by a plurality of imaging devices, at least a part of the plurality of captured images is transmitted to generate the virtual viewpoint image. The image transmission device that determines whether or not a captured image captured by at least one of the plurality of imaging devices is suitable for generating the virtual viewpoint image, and is not suitable for generating the virtual viewpoint image. Limit transmission of the determined captured image. [Selection] Figure 2

Description

  The present invention relates to a technology for transmitting a plurality of captured images captured by a plurality of imaging devices.

Multiple imaging devices can be installed at different positions, synchronized shooting with multiple viewpoints, and using multiple images obtained by the shooting, not only images taken from the installation position of the imaging device, but also viewpoints can be arbitrarily changed There is a technology to generate a simple virtual viewpoint image.
A virtual viewpoint image is generated by an image processing unit such as a server collecting images taken by a plurality of cameras, generating a three-dimensional model, performing processing such as rendering, and transmitting it to the user terminal for viewing Be done.
In Patent Document 1, there is a technology in which a plurality of imaging devices are arranged so as to surround the same range, and a virtual viewpoint image corresponding to any designation is generated and displayed using an image obtained by capturing the same range. It is disclosed.

JP, 2014-215828, A

  In a system that transmits captured images captured by a plurality of imaging devices to a server that generates a virtual viewpoint image via a network, when the transmission bandwidth of the network becomes tight, at least a part of the captured images of multiple cameras is transmitted to the server In some cases, this may not happen. If the transmission band of the network is too tight to transmit the captured image required for generation to the virtual viewpoint image to the server, part of the generated virtual viewpoint image may be dropped, the image quality may be lowered, or the virtual viewpoint It may take time to generate an image. Furthermore, the same problem may occur in the case of generating an image using a plurality of photographed images such as a panoramic image as well as the generation of a virtual viewpoint image.

  The present invention has been made in view of the above problems, and has as its object to reduce the strain on the transmission band of a network that transmits a photographed image necessary for generating a virtual viewpoint image.

The present invention is an image processing system for generating a virtual viewpoint image using a plurality of captured images captured by a plurality of imaging devices, wherein at least a part of the plurality of captured images is generated to generate the virtual viewpoint image. A transmission device,
An acquisition unit configured to acquire a captured image captured by at least one of the plurality of imaging devices;
Transmission means for transmitting the photographed image acquired by the acquisition means;
A determination unit that determines whether the captured image acquired by the acquisition unit is suitable for generating the virtual viewpoint image;
A control unit that restricts transmission of the captured image by the transmission unit when it is determined by the determination unit that the captured image is not suitable for generating the virtual viewpoint image;
It is characterized by having.

  According to the present invention, it is possible to reduce the tightness of the transmission band of the network transmitting the photographed image necessary for generating the virtual viewpoint image.

FIG. 1 is a block diagram for explaining a configuration of an image processing system 100. FIG. 2 is a block diagram for explaining a functional configuration of a camera adapter 120. FIG. 7 is a sequence diagram for describing an imaging start process. It is a figure which shows the data format which the camera adapter 120 outputs. 5 is a flowchart for explaining the operation of the camera adapter 120. It is a figure which shows the flow of the data which the camera adapter 120 outputs. 5 is a flowchart for explaining the operation of the camera adapter 120. It is a figure which shows the flow of the data which the camera adapter 120 outputs. FIG. 2 is a block diagram showing a hardware configuration of a camera adapter 120.

  Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 is a diagram showing an example of an outline of an image processing system according to the present embodiment. The image processing system shown in FIG. 1, for example, generates a virtual viewpoint image using images taken by a plurality of cameras installed in a field having a competition and the like and a stadium or a concert hall having an audience seat. It is a system. The image processing system 100 includes camera systems 110 a to 110 z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

  A controller 300 is an information processing apparatus that performs management of operation states, control of parameter settings, and the like with respect to respective components of the image processing system 100 through networks 300 a, 180 a, 180 b, and 170 a to 170 y. Here, the network may be Ethernet (registered trademark) IEEE standard compliant GbE (Gigabit Ethernet) or 10 GbE, or may be configured by combining an interconnect Infiniband, industrial Ethernet, or the like. Moreover, it is not limited to these, It may be networks of other types, such as a wireless network. Further, the form of connection between the respective devices is not limited to that shown in FIG. 1. For example, the respective devices may be connected via a network such as the Internet.

  The camera systems 110a to 110z are systems for transmitting an image captured by a camera that is an imaging device. The camera systems 110a-110z describe the operation of transmitting 26 sets of images from the camera system 110z to the image computing server 200.

  In the present embodiment, 26 sets of systems from the camera system 110 a to the camera system 110 z will be described as the camera system 110 without distinction, unless otherwise specified. Similarly, devices in each camera system 110 are described as camera 112, sensor 114, and camera adapter 120 without distinction unless otherwise described. Although 26 sets are described as the number of camera systems, this is merely an example, and the number is not limited to this. In the present embodiment, unless otherwise noted, the term “image” is described as including the concept of a moving image and a still image. That is, the image processing system 100 of the present embodiment can process both still images and moving images.

  The camera systems 110a to 110z have one camera 112a to 112z, respectively. That is, the image processing system 100 has a plurality of cameras for shooting a shooting target from a plurality of directions. The plurality of camera systems 110 may be connected by a daisy chain. With this connection form, it is possible to reduce the number of connection cables and save labor of wiring work in increasing the resolution of the captured image to 4K, 8K, etc. and increasing the capacity of image data accompanying the increase in frame rate.

  However, the present invention is not limited to this. As a connection form, each camera system 110a to 110z may be connected to the switching hub 180, and may have a star network configuration for transmitting and receiving data between the camera systems 110 via the switching hub 180. .

  Further, FIG. 1 shows a configuration in which all of the camera systems 110a to 110z are cascaded to form a daisy chain, but the present invention is not limited to this. For example, the plurality of camera systems 110 may be divided into several groups, and the camera systems 110 may be daisy-chained among the divided group units. Then, the camera adapter 120 serving as the end of the division unit may be connected to the switching hub to input an image to the image computing server 200. Such an arrangement is particularly effective in stadiums. For example, it may be considered that a stadium is composed of a plurality of floors and the camera system 110 is deployed on each floor. In this case, the input to the image computing server 200 can be performed every floor or every half cycle of the stadium, and installation is simplified even in a place where it is difficult to connect all the camera systems 110 by one daisy chain and System flexibility can be achieved.

  Further, control of image processing in the image computing server 200 can be switched depending on whether there is one or two or more camera adapters 120 connected in a daisy chain and performing image input to the image computing server 200. . That is, the control is switched depending on whether the camera system 110 is divided into a plurality of groups. When there is one camera adapter 120 that performs image input, images are generated all around the stadium while performing image transmission by daisy chain connection, so the image computing server 200 is synchronized when the image data of all the circumferences are aligned It is taken. That is, if the camera system 110 is not divided into groups, synchronization can be achieved.

  However, when there are a plurality of camera adapters 120 that perform image input (the camera system 110 is divided into groups), it is conceivable that the delays may differ depending on the lanes (paths) of the respective daisy chains. Therefore, it may be necessary to perform image processing of the latter stage while checking the concentration of the image data by synchronous control in which the image computing server 200 waits and synchronizes until the image data of all the circumferences are aligned.

  The camera system 110a includes a camera 112a, a sensor 114a, and a camera adapter 120a. In addition, it does not limit to this structure, You may have another structure. Also, the camera system 110a may be configured to have at least one camera adapter 120a and one camera 112a. Also, for example, the camera system 110a may be configured by one camera adapter 120a and a plurality of cameras 112a, or may be configured by a single camera 112a and a plurality of camera adapters 120a. Also, the camera system 110a may operate the camera adapter 120a, which the camera 112a will be described later, with only one camera 112a.

  The plurality of cameras 112 and the plurality of camera adapters 120 in the image processing system 100 correspond to each other by N to M (N and M are both integers of 1 or more). The camera system 110 may also include devices other than the camera 112 and the camera adapter 120. Also, the camera 112 and the camera adapter 120 may be integrally configured. Furthermore, the image computing server 200 may have at least a part of the functions of the camera adapter 120. The camera systems 110a to 110z may have the same configuration as the camera system 110a. The camera systems 110 are not always limited to the same configuration, and each camera system 110 may have a different configuration.

  The camera adapter 120 is an image transmission device that transmits an image. For example, an image captured by the camera 112a is subjected to image processing described later in the camera adapter 120a, and then transmitted to the camera adapter 120b of the camera system 110b through the daisy chain network 170. Similarly, the camera system 110b transmits an image captured by the camera 112b to the camera system 110c in combination with the image acquired from the camera system 110a.

  By continuing the operation described above, the photographed images acquired by the camera systems 110a to 110z are transmitted from the camera system 110z to the switching hub 180 using the network 180b, and then transmitted to the image computing server 200. As described above, in the system according to the present embodiment, an image captured by the camera 112 of a certain camera system 110 (hereinafter referred to as a self-camera image) and an image transmitted from the upstream camera system 110 (hereinafter referred to as another camera image) ) To the downstream camera system 110. The images taken by the camera systems 110 a to 110 z arrive at the image computing server 200 while repeating such routine processing in each camera system 110. In such a distributed system based on daisy chain connection, the number of captured images to be transmitted increases as going downstream, and the amount of transmission data increases, so there is a possibility that the transmission band may become tight particularly in the downstream. In the present embodiment, data is appropriately reduced before the transmission band is exceeded.

  The sensor 114 is a sensor that senses information related to the imaging condition of the camera system 110. The sensor 114 may be, for example, an infrared sensor, a gyro sensor, an acceleration sensor, a millimeter wave sensor, a GPS or an electronic compass. The sensor 114 may be one of these sensors or a combination of a plurality of sensors.

  The time server 290 has a function of distributing time and synchronization signals, and distributes the time and synchronization signals to the camera systems 110a to 110z through the switching hub 180. The camera adapters 120a to 120z having received the time and the synchronization signal perform Genlock on the cameras 112a to 112z based on the time and the synchronization signal to perform time synchronization of the captured image. That is, the time server 290 synchronizes the imaging timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, so that it is possible to suppress the degradation of the quality of the virtual viewpoint image due to the shift of the capturing timing. In the present embodiment, the time server 290 manages time synchronization of a plurality of cameras 112. However, the present invention is not limited to this. Each camera 112 or each camera adapter 120 independently performs processing for time synchronization. May be

  The image computing server 200 processes data acquired from the camera system 110z. The image computing server 200 generates a three-dimensional model based on a plurality of photographed images acquired from the camera systems 110a to 110z, and generates a virtual viewpoint image by performing processing such as rendering. Model-based rendering (MBR) may be used as a method of generating the virtual viewpoint image. The MBR is a method of generating a virtual viewpoint image using a three-dimensional model generated based on a plurality of captured images obtained by capturing an object from a plurality of directions. Specifically, the MBR utilizes the three-dimensional shape (model) of the target scene obtained by a three-dimensional shape restoration method such as the visual volume intersection method or MULTI-VIEW-STEREO (MVS) to make the scene visible from a virtual viewpoint. Is a technology to generate as an image. Further, the method of generating the virtual viewpoint image is not limited to the MBR, and, for example, a method using image-based rendering may be mentioned. Image-based rendering is a rendering method that generates a virtual viewpoint image from captured images of multiple viewpoints without modeling (process of creating a shape of an object using geometric figures). The image computing server 200 receives a designation of a viewpoint from the controller 300 or the end user terminal 190 for an image acquired from the camera system 110 z, and performs a rendering process based on the accepted viewpoint to generate a virtual viewpoint image. Do.

  The image computing server 200 transmits the rendered image to the end user terminal 190. The user operating the end user terminal 190 can view an image according to the designation of the viewpoint. That is, the image computing server 200 generates a virtual viewpoint image based on the captured images (plural viewpoint images) captured by the plurality of cameras 112 and the viewpoint information. More specifically, the image computing server 200 is virtual based on, for example, image data of a predetermined area extracted from images captured by the plurality of cameras 112 by the plurality of camera adapters 120 and a viewpoint designated by the user operation. Generate a viewpoint image. Then, the image computing server 200 provides the end-user terminal 190 with the generated virtual viewpoint image.

  The image computing server 200 generates a virtual viewpoint image as an image obtained when the subject is photographed from a virtual viewpoint. The virtual viewpoint image can also be said to be an image representing the appearance at a designated viewpoint. The virtual viewpoint (virtual viewpoint) may be designated by the user, or may be automatically designated based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily specified by the user. Further, an image corresponding to a viewpoint specified by the user from a plurality of candidates and an image corresponding to a viewpoint automatically specified by the apparatus are also included in the virtual viewpoint image.

  In addition, the image computing server 200 also transmits the virtual viewpoint image to the H.264 system. Alternatively, the data may be compressed and encoded by a standard technology represented by H.264 and HEVC, and then transmitted to the end user terminal 190 using the MPEG-DASH protocol. Also, the virtual viewpoint image may be transmitted to the end user terminal 190 uncompressed. In particular, the former that performs compression coding assumes a smartphone or a tablet as the end user terminal 190, and the latter assumes a display capable of displaying uncompressed images. That is, it is specified that the image format can be switched according to the type of the end user terminal 190. Further, the transmission protocol of the image is not limited to the MPEG-DASH, and for example, HLS (HTTP Live Streaming) or another transmission method may be used.

  The image computing server 200 may convert the image data generated by the camera systems 110a to 110z and the meta information of the data into a common schema and a data type. As a result, even if the cameras 112 of the camera systems 110a to 110z change to cameras of other models, the possibility that the controller 300 does not operate properly can be reduced. The controller 300 may obtain images directly from the camera systems 110a to 110z without passing through the image computing server 200.

  The configuration of the image computing server 200 is not limited to this. The end user terminal 190 or the controller 300 may have at least a part of the functions of the image computing server 200.

  The end user terminal 190 is an information processing apparatus that acquires a rendering-processed virtual viewpoint image from the image computing server 200 and displays the acquired virtual viewpoint image.

  As described above, in the image processing system 100, a virtual viewpoint image is generated by the image computing server 200 based on image data based on shooting by a plurality of cameras 112 for shooting a subject from a plurality of directions. . The image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be logically configured.

  FIG. 9 is a diagram illustrating an example of a hardware configuration of each device of the image processing system 100. The apparatus 1200 includes a CPU 1201, a ROM 1202, a RAM 1203, an auxiliary storage device 1204, a display unit 1205, an operation unit 1206, a communication unit 1207, and a bus 1208.

  The CPU 1201 controls the entire apparatus 1200 using computer programs and data stored in the ROM 1202 and the RAM 1203. The ROM 1202 stores programs and parameters that do not need to be changed. The RAM 1203 temporarily stores programs and data supplied from the auxiliary storage device 1204, data supplied from the outside via the communication unit 1207, and the like. The auxiliary storage device 1204 is configured by, for example, a hard disk drive and stores content data such as still images and moving images.

  The display unit 1205 is configured of, for example, a liquid crystal display, and displays a graphical user interface (GUI) or the like for the user to operate the device 1200. The operation unit 1206 includes, for example, a keyboard and a mouse, and receives various operations from the user and inputs various instructions to the CPU 1201. A communication unit 1207 communicates with an external device. For example, when the device 1200 is connected to an external device by wire, a LAN cable or the like is connected to the communication unit 1207. Note that in the case where the device 1200 has a function of performing wireless communication with an external device, the communication unit 1207 includes an antenna. A bus 1208 connects the units of the device 1200 to transmit information.

  In the present embodiment, the display unit 1205 and the operation unit 1206 exist inside the device 1200, but the device 1200 may not include at least one of the display unit 1205 and the operation unit 1206. In addition, at least one of the display unit 1205 and the operation unit 1206 exists as another device outside the apparatus 1200, and the CPU 1201 controls the display unit 1205, and the operation control unit controls the operation unit 1206. It may operate as

  The CPU 1201 may be configured of a single or a plurality of CPUs, or may be a multi-core CPU. In addition, instead of the CPU 1201 or together with the CPU 1201, hardware such as an ASIC, an FPGA, or a GPU may be included. In this case, hardware such as an ASIC, an FPGA, or a GPU may perform some or all of the processing that the CPU 1201 should perform. In addition, a part of the processing of the apparatus 1200 may be performed by hardware, and another part of the processing may be realized by software processing using a CPU. Note that not all devices of the image processing system 100 need to have the same configuration, and depending on the devices, some components may not be included, or other components may be included.

  Next, functional blocks of the camera adapter 120 according to the present embodiment will be described with reference to FIG. The functional block of the camera adapter 120 shown in FIG. 2 is realized by the CPU 1201 of the camera adapter 120 executing a computer program stored in the ROM 1202 or the RAM 1203 to calculate information and control each hardware. Note that part or all of the components shown in FIG. 2 may be realized by dedicated hardware. The dedicated hardware is, for example, an ASIC, an FPGA, or a GPU.

  The camera control unit 12011 is connected to the camera 112 and has functions of controlling the camera 112, acquiring a captured image, providing a synchronization signal, setting a time, and the like, and acquiring an image captured by the camera 112. The control of the camera 112 may include, for example, setting and reference of shooting parameters (such as aperture, shutter speed, sensitivity, number of pixels, color depth, frame rate, and setting of white balance). Further, control of the camera 112 may include acquisition of the state of the camera 112 (during photographing, stopping, synchronizing, error, etc.), start and stop of photographing, focus adjustment, and the like. In the present embodiment, focus adjustment is performed via the camera 112. However, when a removable lens is attached to the camera 112, the camera adapter 120 is connected to the lens to directly adjust the lens. It is also good. In addition, the camera adapter 120 may perform lens adjustment such as zooming via the camera 112. The synchronization signal is provided by the time synchronization control unit 12038 providing the photographing timing (control clock) to the camera 112 using the time synchronized with the time server 290. The time setting may be performed by providing the time synchronized with the time server 290 by the time synchronization control unit 12038, for example, with a time code conforming to the format of SMPTE 12M. As a result, the time code provided to the image data received from the camera 112 is given. The format of the time code is not limited to SMPTE 12M, and may be another format. In addition, the camera control unit 12011 may not provide the time code to the camera 112, and may assign the time code to the image data received from the camera 112.

  The sensor control unit 12012 is connected to the sensor 114, and has a function of acquiring sensor information indicating the result of the sensing by the sensor 114. For example, when a gyro sensor is used as the sensor 114, information representing vibration can be obtained. The sensor information acquired by the sensor control unit 12012 is input to the image correction unit 12013. The sensor of the camera system 110 is not limited to the sensor 114, and may be a sensor built in the camera adapter 120 or the camera 112.

  The image correction unit 12013 corrects the image data acquired from the camera control unit 12011 based on the sensor information acquired by the sensor control unit 12012, and outputs the corrected image data to the image processing unit 12021. If the sensor 114 is a gyro sensor or an acceleration sensor, etc., and the sensor information is information representing vibration or shake, the image correction unit 12013 performs shake correction of the image data and outputs the shake correction amount to the meta information generation unit 12022 Do. When the sensor 114 is an infrared sensor or a millimeter wave sensor, and the sensor information is information indicating the presence or absence of a shielded object in the imaging region, the image correction unit 12013 performs region detection of the shielded object in the image. . Then, the image correction unit 12013 outputs the size of the area of the shielded object to the meta information generation unit 12022. If the sensor 114 is an electronic compass or the like and the sensor information is information representing a change in the imaging area, the image correction unit 12013 detects the degree of change in the imaging area, and the degree is used as the meta information generation unit 12022. Output.

  An image processing unit 12021 performs processing such as edge processing, noise removal, and color conversion on the image data captured by the camera 112. The image processing unit 12021 may perform foreground / background separation processing. In the foreground / background separation process, a region of a predetermined object which is a specific person such as a ball or a player may be extracted from the image as a foreground region, and the remaining image obtained by extracting the foreground region from the image may be used as a background region. The camera adapter 120 may separately communicate with the foreground image data indicating the foreground area and the background image data indicating the background area.

  The meta information generation unit 12022 generates information based on sensor information, such as a time code or sequence number at the time of shooting an image, a data type, an identifier indicating an individual of the camera 112, a blur correction amount acquired from the image correction unit 12013, and the like. Generate as meta information of data. Note that the information generated by the meta information generation unit 12022 may be part of these or other information. For example, the meta information generation unit 12022 may generate information necessary for determination processing by an inappropriate image determination unit 12034, which will be described later, of the other camera adapters 120. Also, for example, the meta information generation unit 12022 may generate the determination result of the inappropriate image determination unit 12034 as meta information of the image data. Also, for example, the meta information generation unit 12022 may generate the sensor information acquired by the sensor control unit 12012 as meta information of the image data.

  The data amount acquisition unit 12031 is data to be output, and acquires the data amount of the image processed by the image processing unit 12021. The output data amount calculation unit 12032 adds the data amount of the self-camera image acquired by the data amount acquisition unit 12031 and the data amount of the other camera image acquired by the data amount measurement unit 12036 so that the camera adapter 120 Calculate the amount of data to be output. That is, the camera adapter 120 acquires the data amount of the photographed image acquired by the own system and the photographed image acquired from another camera system. The output data amount determination unit 12033 compares the data amount calculated by the output data amount calculation unit 12032 with a predetermined transmission bandwidth restriction amount to confirm the transmission possibility. Specifically, it is determined whether the amount of data to be output to data transmission unit 12042 exceeds a threshold of the amount of data that can be output specified in advance. In addition, the output data amount determination unit 12033 may dynamically change the threshold of the output data amount according to the use status of the transmission band of the network 170.

  The inappropriate image determination unit 12034 determines an image that is inappropriate for generation of a virtual fantasy image, from among the captured image acquired by the own system and the captured images acquired from another camera system. The inappropriate image determining unit 12034 determines, for example, an image that is inappropriate for generation of a virtual fantasy image, for a captured image with a large amount of blur. When a photographed image with a large amount of blurring is used to generate a virtual viewpoint image, blurring occurs in the virtual viewpoint image as well, and the image quality is degraded. In addition, the inappropriate image determination unit 12034 determines, for example, an image that is inappropriate in the generation of a virtual fantasy image, in which a captured image in which a shielded object is captured is generated. When a photographed image in which an occluded object is captured is used to generate a virtual viewpoint image, when the area in which the occluded object is imaged is mapped in the virtual viewpoint image, the occluded object that can not be seen from the virtual viewpoint is reflected in the virtual viewpoint image It is for. Also, for example, when detecting that the imaging area of the camera 112 has been unintentionally changed due to mischief or the like, the inappropriate image determination unit 12034 determines an inappropriate image in generation of the virtual fantasy viewpoint image. When a photographed image whose photographing region is unintentionally changed due to mischief or the like is mapped in the virtual viewpoint image, the photographing region is shifted, so that mapping can not be performed at an appropriate position, and the image quality is degraded. The inappropriate image determination unit 12034 may determine only a part of these, or may determine an inappropriate image using other criteria in generating the virtual viewpoint image.

  The inappropriate image determination unit 12034 may determine whether the image is inappropriate for generation of the virtual viewpoint image based on the acquired meta information of the captured image. The inappropriate image determination unit 12034 acquires the meta information of the self-camera image generated by the meta information generation unit 12022 and the meta information of the other camera image extracted by the meta information extraction unit 12037, and determines an inappropriate image. May be Also, for example, the inappropriate image determination unit 12034 may compare the camera shake correction amount included in the meta information with a threshold and determine image data having a large shake correction amount as an inappropriate image. In addition, the inappropriate image determination unit 12034 may determine that one of the own camera image and the other camera image having a larger shake amount is inappropriate in generation of the virtual viewpoint image. In addition, the inappropriate image determination unit 12034 may determine whether the image is inappropriate for generating a virtual viewpoint image based on the image processing result of the captured image. For example, the inappropriate image determination unit 12034 may detect the presence or absence of the shield in the captured image by image processing, and determine that the captured image in which the shield is detected is inappropriate in generation of the virtual viewpoint image. Also, for example, the inappropriate image determination unit 12034 compares a plurality of captured images captured at different times by the same camera 112, and detects that the imaging range or the imaging area is changing by image processing. Good. In this case, the inappropriate image determination unit 12034 may determine that the captured image after the change of the imaging range is inappropriate in the generation of the virtual viewpoint image.

  The inappropriate image determination unit 12034 may determine an inappropriate image in generation of the virtual viewpoint image based on the sensing result of the sensor 114. An example of the determination of the inappropriate image determination unit 12034 when the sensor 114 is a gyro sensor or an acceleration sensor will be described. When the sensing result indicates that the camera 112 vibrates when shooting, the inappropriate image determining unit 12034 improperly generates an image captured by the camera 112 in generating a virtual viewpoint image. An image may be determined. An example of the determination by the inappropriate image determination unit 12034 in the case where the sensor 114 is an infrared sensor or a millimeter wave sensor will be described. In the case where an object whose imaging result in a predetermined range (for example, 5 m from the camera 112) intrudes these sensing results intruding, the inappropriate image determination unit 12034 virtualizes the image captured by the camera 112. An inappropriate image may be determined in generation of a viewpoint image. An example of the determination by the inappropriate image determination unit 12034 when the sensor 114 is an electronic compass will be described. If the sensing result of the electronic compass indicates that the shooting direction of the camera 112 has changed, the inappropriate image determination unit 12034 determines an image captured by the camera 112 as an inappropriate image in generation of a virtual viewpoint image. May be An example of the determination by the inappropriate image determination unit 12034 when the sensor 114 is the GPS will be described. When the sensing result of GPS indicates that the position of the camera 112 has changed, the inappropriate image determination unit 12034 determines an image captured by the camera 112 as an inappropriate image in generation of a virtual viewpoint image. It is also good.

  The image transmission processing unit 12035 performs control to transmit image data to another camera adapter 120 (another camera system 110) or the image computing server 200 via the data transmission unit 12042. The image transmission processing unit 12035 creates a message including the image data and the meta information of each data, and transmits the created message. The data amount measuring unit 12036 measures the data amount of the other camera image received by the data receiving unit 12041. The meta information extraction unit 12037 extracts meta information from the other camera image message received by the data reception unit 12041.

  The time synchronization control unit 12038 performs processing relating to time synchronization with the time server 290. The data receiving unit 12041 and the data transmitting unit 12042 perform data communication with the other camera adapters 120, the image computing server 200, the time server 290, and the controller 300 via the networks 170, 180a, 180b, 291 and 300a. For example, the data transmission unit 12042 outputs the captured image captured by the camera 112, which has been processed by the image processing unit 12021, to the next camera adapter 120. As each camera adapter 120 outputs an image, a virtual viewpoint image is generated based on the images captured from a plurality of viewpoints. The time control unit 12043 stores time stamps of data transmitted to and received from the time server 290, and performs time synchronization with the time server 290.

  Subsequently, the operation of the image processing system 100 having the above-described configuration will be described. First, photographing processing in the camera system 110 will be described with reference to FIG. FIG. 3 is a diagram showing a sequence of photographing processing.

  First, the time server 290 performs time synchronization with, for example, the GPS 2201 and the like, and sets the time managed in the time server (06801). The time setting of the time server 290 is not limited to the method using GPS, and the time may be set by another method such as NTP (Network Time Protocol).

  Next, the camera adapter 120 communicates with the time server 290, corrects the time managed in the camera adapter 120, and performs time synchronization with the time server 290 (06802). The camera adapter 120 provides a synchronized photographing signal such as a Genlock signal or a ternary synchronization signal and a time code signal to the camera 112 based on the time synchronized with the time server 290. The camera adapter 120 starts provision of the synchronized shooting signal and the time code signal to the camera 112 in synchronization with the shooting frame rate (for example, 60 fps) of the camera 112 (06803). The information to be provided is not limited to the time code, and may be other information as long as it is an identifier that can identify the shooting frame.

  Next, the camera adapter 120 instructs the camera 112 to start photographing (06804). When the camera 112 receives the imaging start instruction, the imaging is performed in synchronization with the Genlock signal (06805). Next, the camera 112 transmits the photographed image to the camera adapter 120 (06806). Shooting synchronized with the Genlock signal is performed until the camera 112 stops shooting.

  Further, the camera adapter 120 acquires sensor information (06808) sensed by the sensor 114 (06807). The camera adapter 120 may transmit to the sensor 114 in synchronization with the imaging frame rate of the camera 112 a synchronous imaging signal such as a Genlock signal or a ternary synchronization signal and a time code signal. Then, the sensor 114 may perform sensing based on the synchronous imaging signal or the time code signal. Also, the camera adapter 120 may be configured to associate sensing information from the sensor 114 sensed at a time when the shooting time and the sensing time are close to the image data.

  Subsequently, an example of a message for transmitting image data transmitted and received between the camera adapters 120 is shown in FIG. In FIG. 4, reference numeral 401 denotes a camera number for identifying the camera 112 that has captured the image included in this message, and the identification number of each of the cameras 112 a to 112 z is input. In FIG. 4, the camera number 401 is set with a number 112a indicating the camera 112a, and it is shown that it is the camera 112a that photographed the image data 404 described later.

  Reference numeral 402 denotes a time code of the image data 404, which indicates the time when the image data 404 was captured. 403 is information based on sense information acquired by the sensor 114 by sensing. FIG. 4 shows an example in which the correction amount of the data corrected by the image correction unit 12013 is stored in the information 403 based on the sense information. The shake correction amount may be expressed in 10 steps of 0-9. In this case, the shake correction amount 0 may indicate that the camera 112 has no vibration and shake correction is not performed, and the values 1 to 9 may indicate that the shake correction amount increases as the number increases. The information included in the information 403 based on the sense information is not limited to the blur correction amount. For example, the sensor information itself acquired from the sensor 114 may be input to the information 403 based on the sense information. 404 is image data.

  Among the data 401 to 404, the data 401 to 403 other than the image data are called meta information. The message shown in FIG. 4 is an example, and other information may be included, or some of the data 401 to 404 may not be present. For example, the determination result of the inappropriate image determination unit 12034 may be added to the message as meta information of the image data. Also, meta information such as 401 to 403 and the image data 404 may be communicated as separate messages.

  Next, the flow of the output process of the camera adapter 120 will be described using the flowchart of FIG. The flowchart in FIG. 5 may be started, for example, at each data output period of the camera adapter 120. The camera adapter 120 may output data for each shooting frame rate of the camera 112, or may start output processing shown in the flowchart of FIG. For example, when the camera 112 shoots at 60 fps, the process shown by the flowchart of FIG. 5 may be started every 1/60 seconds. Also, the process shown by the flowchart of FIG. 5 may be started each time image data indicating an image captured by the camera 112 of the own system is acquired. The process shown in the flowchart to be described later is realized by the CPU 1201 of the camera adapter 120 executing a program to calculate information and control each hardware. Note that at least a part of the steps of the flowchart to be described later may be executed by dedicated hardware. The dedicated hardware is, for example, an ASIC, an FPGA or a GPU.

  The camera adapter 120 acquires an image captured by the camera 112 of the own system, and generates meta information of the acquired image by the meta information generation unit 12022 (S501). The meta information generation unit 12022 uses, as meta information of the image data, a time code or sequence number when the image was captured, a data type, an identifier indicating an individual of the camera 112, a blur correction amount acquired from the image correction unit 12013, and the like. Generate

  The camera adapter 120 acquires the data amount of the image data indicating the image captured by the camera 112 of the own system (S502). In the process of S502, the data amount of the image data which is the processing result of the image processing unit 12021 may be acquired, or the data amount of the image data after the compression process may be acquired. Also, in the process of S502, the camera adapter 120 performs image processing on the foreground image as a result of subjecting the image captured by the camera 112 of the own system to foreground / background separation processing and / or data on image data representing the background image. You may get an amount.

  The camera adapter 120 receives a message including image data indicating an image captured by the camera 112 of another camera system 110 from the adjacent camera adapter 120 upstream. Then, the camera adapter 120 measures the data amount of the received message (S503). That is, the camera adapter 120 measures the data amount of the message acquired from the upstream camera adapter 120 via the data receiving unit 12041 by the data amount measuring unit 12036.

  Next, the camera adapter 120 outputs an amount of data to be output to the downstream camera adapter 120d based on the amount of data acquired by the data amount acquisition unit 12031 and the amount of data measured by the data amount measurement unit 12036 as an output data amount calculation unit 12032 It calculates by (S504).

  Next, the camera adapter 120 determines whether the amount of output data calculated in S504 exceeds the amount of data that can be output (S505). In step S505, the output data amount determination unit 12033 of the camera adapter 120 compares the output data amount with the transmission band restriction amount, and determines whether the output data amount exceeds the transmission band restriction amount. Specifically, the camera adapter 120 determines whether the amount of data to be output to the data transmission unit 12042 exceeds a predetermined threshold of the amount of output data. For example, it is assumed that the camera system 110 z has an effective network speed of 100 Gbps for transmitting data to the image computing server 200. In addition, it is assumed that a shooting frame rate at which the cameras 112a to 112z shoot is 100 fps, and each of the camera systems 110a to 110z performs output processing every 1/100 second. In the transmission of the camera system 110z, the camera system 110z needs to reduce the amount of data to less than 1 Gbit for each output process so as not to exceed the amount of data that can be transmitted by the network. Therefore, when the amount of data of own camera data is 1/26 Gbit, each of the 26 camera adapters 120a to 120z does not exceed the amount of data that can be transmitted by the network in transmission of the camera system 110z. That is, in this example, the camera adapter 120 may determine in S505 whether or not the output data amount exceeds (1 + n) / 26 Gbit (n is the number of upstream camera systems).

  If it is determined in S505 that the amount of output data does not exceed the amount of data that can be output, which is a threshold (NO in S505), the camera adapter 120 performs normal transmission (S506). In S506, the camera adapter 120 transmits all of the self-camera image data acquired in S501 and the other camera image data acquired in S503 to the other downstream camera adapter.

  If it is determined in S505 that the amount of output data exceeds the amount of data that can be output that is a threshold (YES in S505), the camera adapter 120 reduces the amount of data and transmits image data (S507).

  The data reduction transmission process (S507) will be described in detail with reference to the flowchart of FIG. The camera adapter 120 acquires meta information of the self-camera image data generated in S501 (S701). The camera adapter 120 extracts meta information of the other camera image data acquired in S503 (S702).

  The inappropriate image determination unit 12034 of the camera adapter 120 determines an image unsuitable for use in generating a virtual viewpoint image, out of the self-camera image acquired in S701 and the other camera image acquired in S702. The camera adapter 120 may determine the shooting conditions of the acquired self-camera image and the other camera image, and may determine an image shot in a situation where the image quality of the virtual viewpoint image may deteriorate. In addition, the inappropriate image determination unit 12034 of the camera adapter 120 determines an image unsuitable for use in generating a virtual viewpoint image based on sensor information associated with each of the own camera image and the other camera image. Good. In addition, the inappropriate image determination unit 12034 of the camera adapter 120 is used to generate a virtual viewpoint image based on the meta information of the own camera image data acquired in S701 and the meta information of the other camera image data acquired in S702. It is possible to determine an image inappropriate for Then, the camera adapter 120 removes the image data indicating the most inappropriate image for generating the virtual viewpoint image from the output target, and deletes the image data to limit the transmission of the captured image to the network. (S703).

  The camera adapter 120 calculates the output data amount after reduction at S703 again (S704), and determines whether the output data amount calculated at S704 exceeds the data amount that can be output (S705).

  If it is determined in S 705 that the output data amount does not exceed the threshold that can be output, which is the threshold (NO in S 705), the camera adapter 120 transmits the image data to be output to the other downstream camera adapter (S706). On the other hand, when it is determined in S705 that the amount of output data exceeds the amount of data that can be output, which is a threshold (YES in S505), the camera adapter 120 performs the processing from S703 again.

  A flow of data between camera adapters 120 at the time of normal transmission (S506) will be described using FIG.

  In FIG. 6, the camera adapter 120a transmits image data indicating a photographed image acquired from the camera 112a, which is self-camera image data, to the downstream camera adapter 120b. 600 is data transmitted from the camera adapter 120a. Data 600 includes self-camera image data generated by the camera system 110 a. Data 600 includes a camera number 601 of the camera 112a, a time code 602 indicating a photographing time, a blur correction amount 603 corrected by the image correction unit 12013 based on sensor information of the sensor 114a, and an image processed by the image processing unit 12021 604 is stored.

  The camera adapter 120b acquires image data indicating a photographed image acquired from the camera 112b, which is self-camera image data. Also, the camera adapter 120b acquires, from the camera adapter 120a, image data indicating another image captured by the camera 112a, which is other camera image data. The camera adapter 120b determines that the output data (the own camera image data indicating the photographed image acquired from the camera 112b and the other camera image data acquired from the camera adapter 120a) does not exceed the outputable data amount, and normally transmits the output data Do.

  Data 605 and 610 are data transmitted from the camera adapter 120 b. Reference numeral 605 denotes data transferred from the data received from the upstream camera adapter 120a, and reference numeral 610 denotes self-camera image data generated by the camera system 110b.

  The camera adapter 120c acquires self-camera image data indicating a photographed image acquired from the camera 112c. Also, the camera adapter 120c acquires, from the camera adapter 120b, other camera image data indicating a captured image captured by the camera 112a and the camera 112b. The camera adapter 120c determines that the output data (the own camera image data acquired from the camera 112c and the other camera image data acquired from the camera adapter 120b) does not exceed the amount of data that can be output, and normally transmits the output data.

  606, 615, and 620 are data transmitted from the camera adapter 120c. Reference numerals 606 and 615 denote transfer data of data received from the upstream camera adapter 120b, and reference numeral 620 denotes own camera image data generated by the camera system 110c.

  As described above, at the time of normal transmission, each camera adapter 120 transfers the data received from the upstream camera adapter 120 together with the own camera image data to the downstream camera adapter without leakage.

  Subsequently, the flow of data between the camera adapters 120 when the data reduction transmission process of S507 is performed in the camera adapter 120c will be described with reference to FIG. In FIG. 8, data transmitted by the camera adapter 120 a and the camera adapter 120 b are the same as in FIG. 6.

  In FIG. 8, the camera adapter 120 c acquires self-camera image data indicating a photographed image acquired from the camera 112 c. Also, the camera adapter 120c acquires, from the camera adapter 120b, other camera image data indicating a captured image captured by the camera 112a and the camera 112b. When it is determined that the output data (the own-camera image data acquired from the camera 112c and the other-camera image data acquired from the camera adapter 120b) exceeds the outputable data amount, the camera adapter 120c performs the data reduction transmission process of S507. .

  The camera adapter 120c determines an image unsuitable for generating a virtual viewpoint image among the acquired self-camera image and the other camera image. Here, the camera adapter 120c determines the shooting conditions of the acquired self-camera image and the other camera image, and determines an image shot in a situation in which the image quality may deteriorate. For example, the camera adapter 120c acquires meta information of each image, and determines an image captured in a situation where blurring occurs from the acquired meta information. From the meta information of each image, the camera adapter 120c determines that the image data 615 captured by the camera 112b, which is an image with the largest blur correction amount, is an image not suitable for generating a virtual viewpoint image. The camera adapter 120 c removes the image data 615 determined to be an image unsuitable for generating the virtual viewpoint image from the transmission target, and deletes the image data 615. The camera adapter 120c determines that the data amount of the output data after deleting the image data 615 does not exceed the possible output data amount, and transmits the output data to the downstream camera adapter 120d.

  606 and 620 are data transmitted from the camera adapter 120c in FIG. Compared with FIG. 6, since the image data 615 is deleted, it is possible to reduce the tightness of the transmission band of the network that transmits the photographed image necessary for generating the virtual viewpoint image. Further, since the image to be deleted is an image unsuitable for generating a virtual viewpoint image, even if the image to be transmitted is reduced, it is possible to suppress the image quality deterioration of the virtual viewpoint image.

  As described above, in this embodiment, regardless of the self-camera image and the other-camera image, the image data determined to be unsuitable for creating the virtual viewpoint image is appropriately deleted from the transmission target to prevent the transmission band from being over. be able to. In addition, unnecessary data reduction can be suppressed by deleting the image data determined to be unsuitable for creating the virtual viewpoint image only when the transmission band is tight.

  The sensor information is not limited to the information indicating the vibration of the camera, and the sensor 114 may sense the direction or position information of the camera.

  Moreover, the above-mentioned embodiment was explained focusing on the example in case image processing system 100 is installed in facilities, such as a stadium and a concert hall. Other examples of the facility include, for example, an amusement park, a park, a racetrack, a bicycle race track, a casino, a pool, a skating rink, a ski resort, a live house and the like. In addition, events performed in various facilities may be performed indoors or outdoor. Further, the facilities in the present embodiment also include facilities temporarily (limitedly) built.

  Further, in S703 of FIG. 7, as described above, the inappropriate image determination unit 12034 determines not only the blur amount but also the determination result of the imaging condition such as the presence or absence of the shielding object in the imaging region and the change in the imaging region. The image may be identified as an inappropriate image in generation of a virtual viewpoint image. Further, in S703, as described above, the inappropriate image determination unit 12034 may determine whether the image is inappropriate for generation of a virtual viewpoint image based on the image processing result of the captured image as described above. Also, the inappropriate image determination unit 12034 generates an image that is inappropriate for generating a virtual viewpoint image based on the shooting parameters (aperture, shutter speed, white balance, sensitivity, etc.) set in the camera 112 that has captured the image. You may judge. The inappropriate image determination unit 12034 may specify an image captured using a shooting parameter different from the prescribed value as an image unsuitable for generation of a virtual viewpoint image. When the virtual viewpoint image is generated, if the imaging parameters are different from those of the other images, the color tone and the like are different, and the image quality may be degraded. In this case, imaging parameters may be attached as meta information of the image data.

  Also, in S703, the inappropriate image determination unit 12034 is an image that is inappropriate for generating a virtual viewpoint image based on the position of the virtual viewpoint related to the virtual viewpoint image generated by the image computing server 200, The image to reduce may be determined. In addition, the inappropriate image determination unit 12034 reduces data, which is an image unsuitable for generating a virtual viewpoint image, based on the position of the virtual viewpoint specified by the controller 300 or the end user terminal 190 and the gaze direction. An image may be determined. In this case, the inappropriate image determination unit 12034 sets the data of the position of the virtual viewpoint related to the generated virtual viewpoint image, the gaze direction, the position of the designated virtual viewpoint, the image of the shooting area not captured as viewed from the gaze direction. It may be specified as an object to be reduced. The position of the virtual viewpoint related to the virtual viewpoint image to be generated, the direction of the gaze direction, the position of the designated virtual viewpoint, and the image of the imaging region which does not show the area viewed from the gaze direction contributes less to generation of the virtual viewpoint image. Therefore, even if this image is not transmitted, the influence on the image quality deterioration of the generated virtual viewpoint image is low.

  Further, in S703 of FIG. 7, the inappropriate image determination unit 12034 determines an image captured by the malfunctioning camera 112 as an image that is inappropriate for generation of a virtual viewpoint image and that reduces data. You may In the case where breakage of the camera 112 or lens breakage occurs, the inappropriate image determination unit 12034 reduces the data of the image captured by the camera 112 as an image unsuitable for generating a virtual viewpoint image. It may be determined as an image. In this case, as the meta information of the image data, an operation state or a damage state of the camera 112 may be added with a photographing parameter.

  In addition, the camera adapter 120 performs the data reduction transmission process of S507 according to the remaining amount of the transmission band of the network and the usage status in the determination of S505, but the data reduction of S507 regardless of the remaining amount of the transmission band of the network. Transmission processing may be performed. That is, the camera adapter 120 may omit steps S503 to S504 and step S506 and perform the process of step S507.

  Also, the example in which the camera adapter 120 reduces the output data amount by deleting the image determined to be inappropriate in S703 has been described. However, for example, the number of pixels and resolution of the image determined to be inappropriate are reduced or compressed to reduce the amount of data and transmit the image determined to be inappropriate, thereby the network of the captured image It may be configured to limit the transmission to.

  As mentioned above, although the embodiment of the present invention was explained in full detail, the present invention is not limited to the above-mentioned embodiment, and various modification and change within the range of the gist of the present invention described in the claim are included. Is possible.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

100 image processing system 110 camera system 112 camera 114 sensor 120 camera adapter 180 switching hub 190 end user terminal 200 image computing server 290 time server 300 controller

Claims (23)

In an image processing system for generating a virtual viewpoint image using a plurality of captured images captured by a plurality of imaging devices, an image transmission device transmitting at least a part of the plurality of captured images to generate the virtual viewpoint image And
An acquisition unit configured to acquire a captured image captured by at least one of the plurality of imaging devices;
Transmission means for transmitting the photographed image acquired by the acquisition means;
A determination unit that determines whether the captured image acquired by the acquisition unit is suitable for generating the virtual viewpoint image;
A control unit that restricts transmission of the captured image by the transmission unit when it is determined by the determination unit that the captured image is not suitable for generating the virtual viewpoint image;
An image transmission apparatus comprising:   2. The apparatus according to claim 1, wherein the determination unit determines whether the captured image is suitable for generating the virtual viewpoint image based on a capturing condition of the captured image acquired by the acquisition unit. Image transmission apparatus as described.   The determination unit is characterized by determining that the captured image is not suitable for generation of the virtual viewpoint image when it is determined that blurring occurs in the captured image acquired by the acquisition unit. Item 3. The image transmission device according to item 1 or 2.   The determination unit determines that the captured image is not suitable for generation of the virtual viewpoint image when it is determined that an object blocking the imaging target of the captured image acquired by the acquisition unit is included in the captured image. The image transmission apparatus according to any one of claims 1 to 3, wherein it is determined that   The determination unit determines that the captured image is not suitable for generation of the virtual viewpoint image when it is determined that the imaging device for capturing the captured image acquired by the acquisition unit is broken or damaged. The image transmission device according to any one of claims 1 to 4, characterized in that:   The determination unit determines that the captured image is not suitable for generation of the virtual viewpoint image when it is determined that the imaging area of the imaging device for capturing the captured image acquired by the acquisition unit has changed. The image transmission apparatus according to any one of claims 1 to 5, characterized in that   The determination means is characterized by determining whether or not the photographed image is suitable for generation of the virtual viewpoint image based on a photographing parameter when the photographed image acquired by the acquisition means is photographed. The image transmission apparatus according to any one of claims 1 to 6.   The determination means determines whether or not the captured image is suitable for generation of the virtual viewpoint image based on sensor information obtained by sensing a capturing condition at the time of capturing the captured image acquired by the acquisition device. The image transmission device according to any one of claims 1 to 7, characterized in that:   The determination means is based on sensor information which is a result of sensing by an infrared sensor, a gyro sensor, an acceleration sensor, a millimeter wave sensor, GPS or an electronic compass when the photographed image acquired by the acquisition means is photographed. The image transmission apparatus according to any one of claims 1 to 8, wherein it is determined whether or not the photographed image is suitable for generation of the virtual viewpoint image.   The determination means determines whether or not the photographed image is suitable for generation of the virtual viewpoint image based on the position of the viewpoint related to the virtual viewpoint image and the direction of the line of sight. 9. The image transmission device according to any one of up to 9.   The said determination means determines whether it is suitable for the production | generation of the said virtual viewpoint image based on the image processing result of the said picked-up image, The any one of Claim 1 to 10 characterized by the above-mentioned. Image transmission device.   When the determination unit determines that the captured image is not suitable for generating the virtual viewpoint image, the control unit reduces the data amount of the captured image to transmit the captured image by the transmission unit. The image transmission device according to any one of claims 1 to 11, characterized in that it is restricted.   13. The apparatus according to claim 1, wherein the control means does not transmit the photographed image to the transmission means when the judging means judges that the photographed image is not suitable for generation of the virtual viewpoint image. The image transmission device according to any one of the above. The information processing apparatus further includes identification means for identifying the data amount of data to be transmitted by the transmission means.
The said control means limits transmission by the said transmission means of the said picked-up image, when the data amount specified by the said identification means exceeds a threshold value, It is any one of Claim 1 to 13 characterized by the above-mentioned. Image transmission device. The acquisition unit acquires a first captured image captured by a first imaging device of the plurality of imaging devices and a second captured image captured by a second imaging device of the plurality of imaging devices.
2. The image processing apparatus according to claim 1, wherein the determination unit determines a captured image not suitable for generating the virtual viewpoint image from the first captured image and the second captured image acquired by the acquisition unit. 14. The image transmission device according to any one of items 1 to 14.   The transmission unit transmits the photographed image acquired by the acquisition unit when the judgment unit judges that the photographed image is suitable for generating the virtual viewpoint image. 15. The image transmission apparatus according to any one of items 1 to 15.   The transmission unit transmits the photographed image acquired by the acquisition unit to another image transmission apparatus that transmits at least a part of the plurality of photographed images or a server that generates the virtual viewpoint image. The image transmission apparatus according to any one of claims 1 to 16.   The determination means is suitable for generating the virtual viewpoint image when the sensor information obtained as a result of sensing by the gyro sensor or the acceleration sensor indicates that the imaging device for capturing the captured image is vibrating. The image transmission apparatus according to claim 9, wherein it is determined that there is no.   In the case where the determination unit indicates that there is an object that shields the imaging region of the imaging device that captures the captured image acquired by the acquisition unit, sensor information that is a result of sensing by an infrared sensor or a millimeter wave sensor. The image transmission apparatus according to claim 9, wherein the image transmission apparatus determines that the captured image is not suitable for generating the virtual viewpoint image.   When the determination unit indicates that sensor information which is a result of sensing by the electronic compass indicates that the imaging direction of the imaging device for capturing the captured image acquired by the acquisition unit has changed, the captured image corresponds to the virtual viewpoint image The image transmission apparatus according to claim 9, wherein the image transmission apparatus is determined not to be suitable for the generation of.   When the determination unit indicates that the position of the imaging device for capturing the captured image acquired by the acquisition unit has changed as the sensor information which is the result of the sensing of the GPS, the captured image is a generation of the virtual viewpoint image The image transmission apparatus according to claim 9, wherein the image transmission apparatus is determined not to be suitable for In an image processing system including a plurality of imaging devices, a plurality of image transmission devices transmitting at least a part of a plurality of photographed images photographed by the plurality of imaging devices, and a server generating a virtual viewpoint image An image transmission method,
And transmitting the photographed image not suitable for generating the virtual viewpoint image among the plurality of photographed images to the server.   A program for operating a computer as an image transmission device according to any one of claims 1 to 16.

Images