JP7660357B2 - Imaging device, control method, and program - Google Patents

Description

The present invention relates to an imaging apparatus, a control method , and a program.

There is a multi-lens imaging device that can capture a wide range by arranging multiple imaging units in a circular shape on the imaging device. The multiple imaging units are fixedly arranged on the multi-lens imaging device. However, if there is an obstacle such as a wall or a pillar near the installation location of the multi-lens imaging device, the shooting range of the multiple imaging units may be blocked. Therefore, a mounting mechanism for a multi-lens imaging device that can freely change the position and attitude of the imaging units so that the shooting range of the imaging units can be changed has been disclosed (Patent Document 1). A multi-lens imaging device with a function to uniquely set the arrangement of the multiple imaging units (multi-lens preset function) has also been proposed.

JP 2015-119476 A

However, there are cases where the shooting range captured from the arrangement of multiple imaging units previously set using the multi-lens preset function differs from the shooting range the user wants to capture, which poses an issue in that the user must manually change the arrangement.

The present invention aims to improve the work efficiency involved in adjusting the shooting range of an imaging device having multiple imaging units.

In order to achieve the object of the present invention, an imaging device according to an embodiment of the present invention comprises the following configuration: a first imaging mode in which the multiple imaging units are arranged on the same circle and rotatable around a central axis, a second imaging mode in which at least one of the multiple imaging units is arranged around the central axis and another imaging unit of the multiple imaging units that is different from the imaging unit arranged around the central axis is arranged near the central axis, and a third imaging mode in which the multiple imaging units are arranged in a straight line, a setting means for setting a relative arrangement between the multiple imaging units based on a result acquired by the acquisition means, and a control means for rotating the multiple imaging units around the central axis while maintaining the relative arrangement between the multiple imaging units set by the setting means.

The present invention can improve the work efficiency of adjusting the shooting range using an imaging device having multiple imaging units.

FIG. 1 is an overall diagram of a device configuration of an imaging system. FIG. FIG. 2 is a functional block diagram of an imaging apparatus and an information processing apparatus. 5A and 5B are schematic diagrams illustrating the arrangement of imaging units according to a first setting operation. 5A and 5B are schematic diagrams illustrating user interfaces for a first setting operation and a second setting operation. FIG. 11 is a schematic diagram illustrating a second setting operation. 11A and 11B are schematic diagrams illustrating the operation of the imaging unit according to the second setting operation. 6 is a flowchart illustrating a process flow of a first setting operation and a second setting operation. FIG. 4 is a schematic diagram illustrating a changing means by a mechanical mechanism operation. FIG. 11 is a schematic diagram for explaining a modification means by a cutout process. FIG. 4 is a schematic diagram illustrating a cutout process. 6 is a flowchart for explaining automatic adjustment of the positions and orientations of multiple imaging units based on target recognition. 11 is a flowchart for explaining automatic adjustment of the positions and orientations of a plurality of imaging units based on a region selected by a user.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

In the embodiment described below, a network camera is used as the imaging device 101, but this is not limited thereto, and the use of other imaging devices (e.g., video cameras, still cameras, mobile phones, and personal digital assistants) is not excluded. In addition, an imaging system 10 including the imaging device 101 and other devices is described below, and here the information processing device 102 is used as an imaging control device that controls the imaging device 101.

Another device may have one or more functions of the information processing device 102 according to the embodiments described below. For example, the imaging device 101 may have one or more functions of the information processing device 102 according to each embodiment. An imaging device 101 such as a network camera may have the function of the information processing device 102 that performs imaging control. In addition, an imaging control device according to an embodiment of the present invention may be composed of multiple information processing devices 102 connected via a network, for example. The imaging control device may be mounted on the imaging device 101, or on a server that communicates with the imaging device 101.

Below, an example of an embodiment is shown in Figures 1 to 13, and the imaging control device will be described with reference to the drawings.

Fig. 1 is a diagram showing an example of the configuration of an imaging system 10 in this embodiment. The imaging system 10 is configured to include an imaging device 101, an information processing device 102, a display device 103, and a network 105. The imaging device 101 and the information processing device 102 are connected to the network 105 via a cable 104, and the information processing device 102 and the display device 103 are connected by the cable 104.

The imaging device 101 is installed outdoors, for example, in residential areas, parks, commercial facilities, etc., and is a device for photographing various objects, such as a surveillance camera or a network camera. The imaging device 101 has a function for photographing objects, and is equipped with multiple imaging units including an imaging unit 202 and a lighting unit (not shown) on the lower side of the device body. The multiple imaging units capture moving images at a predetermined angle of view, and transmit various data including the moving images, the imaging time, and identification information of the multiple imaging units to the information processing device 102 via the network 105. The lighting unit is, for example, a visible light and infrared light used to ensure illuminance in low-light environments such as at night. The various data includes imaging setting information such as the pan angle, tilt angle, rotation angle, zoom magnification, focus position, exposure, and white balance of the imaging unit 202.

The information processing device 102 is a device for controlling the imaging device 101, and is, for example, a PC or the like equipped with a program for executing a control function. The information processing device 102 also transmits the following control commands to the imaging device 101. For example, these are control commands for pan angle, tilt angle, rotation angle, zoom magnification, focus position, exposure, white balance, and the like. When performing a first setting operation and a second setting operation described later, the information processing device 102 can transmit these control commands so as to control the position and attitude of each imaging unit according to the settings. On the other hand, the information processing device 102 may transmit setting information such as a shooting mode set in the first setting operation and the second setting operation as a control command to the imaging device 101. In this case, the imaging device 101 may control the rotation angle of each imaging unit according to the setting information. The information processing device 102 can also control the operation of multiple imaging units of the imaging device 101, and may control the imaging device 101 by connecting multiple information processing devices 102 to the imaging device 101.

The display device 103 is a device that displays the image processing results of the information processing device 102 and moving images captured by the imaging device 101, and is, for example, a liquid crystal display (LCD). The display device 103 is connected to the information processing device 102 via a cable 104 that complies with a communication standard such as HDMI (registered trademark). Note that the information processing device 102 and the display device 103 may be configured as a single unit, or the imaging device 101 and the information processing device 102 may be configured as a single unit.

The image processing results by the information processing device 102 and the images captured by the imaging device 101 may not only be displayed on the display device 103, but may also be displayed on a display device of another external device (e.g., a server) that is not included in the imaging system 10. For example, the image processing results and moving images may be displayed on a display device of a personal digital assistant (PDA) such as a mobile phone, smartphone, or tablet terminal connected via the network 105.

The cable 104 is a LAN cable, such as a twisted pair cable or an optical fiber cable, that conforms to a communication standard such as Gigabit Ethernet (registered trademark). Note that the cable 104 is not limited to a wired LAN, and may be configured as a wireless LAN that conforms to the Wi-Fi (registered trademark) standard.

The network 105 is, for example, a LAN or WAN that is composed of multiple routers, switches, cables, etc. that comply with a communication standard such as Ethernet (registered trademark).

Fig. 2(a) is a side view showing an example of the configuration of the imaging device 101 of this embodiment. The imaging device 101 is fixed to, for example, a support structure or the like via a common axis (not shown) passing through the center of the imaging device 101. The lower part of the imaging device 101 stores multiple imaging units, and has a protective cover (not shown) for protecting the multiple imaging units from, for example, wind, rain, etc.

2(b) shows the arrangement of multiple imaging units when the imaging device 101 is viewed from the arrow A in FIG. 2(a). The imaging device 101 in this embodiment has four imaging units, but may have less than four imaging units, or may have four or more imaging units. The imaging units 202, 204, 206, and 208 are arranged on the circumference of the same circle. The arrangement of the four imaging units may be such that all imaging units are arranged in different circles, or some imaging units are arranged in the same circle and other imaging units are arranged in the same circle different from the above. In addition, the multiple imaging units are not limited to the arrangement shown in the figure, and can be freely arranged on the circumference (direction of arrow B) and diameter (direction of arrow C) of the imaging device 101.

Figure 3(a) shows the configuration of the imaging device 101, information processing device 102, and imaging unit 202 for realizing the functions of this embodiment. The imaging device 101 is configured to include imaging unit 202, imaging unit 204, imaging unit 206, and imaging unit 208. The imaging unit 202 is configured to include a lens 307, imaging section 308, image processing section 309, focus driving section 310, zoom driving section 311, and control section 312. Note that imaging units 204 to 208 have the same configuration as imaging unit 202, so illustration and description of the components will be omitted. Below, the configuration of imaging unit 202 will be described as an example of multiple imaging units.

The imaging unit 308 is a two-dimensional image sensor that converts a two-dimensional optical image into an electrical physical quantity by photoelectric conversion, such as a CCD sensor or a CMOS sensor. The imaging unit 308 outputs the captured RAW data to the image processing unit 309. The image processing unit 309 performs processing such as black correction, demosaicing, and filtering on the RAW data received from the imaging unit 308, and stores the developed data in the RAM 304 via the network 105.

The control unit 312 controls the zoom driving unit 311, which moves a zoom lens (not shown) in the lens 307 along the optical axis to achieve zooming in or zooming out. Similarly, the control unit 312 controls the focus driving unit 310, which moves a focus lens (not shown) in the lens 307 along the optical axis to achieve focusing. The control unit 312 can also control the position and attitude of the imaging unit 202 based on control commands received from the CPU 301.

The position and orientation recognition unit 313 (position recognition means) is a sensor that recognizes the position and orientation of the imaging unit 202, and includes, for example, an encoder, a gyro, a GPS, and an acceleration sensor. The information on the position and orientation of the imaging unit 202 acquired by the position and orientation recognition unit 313 is processed by the CPU 301 and stored in the RAM 304 as information on the shooting direction and shooting angle of view of the imaging unit 202.

The information processing device 102 is configured, for example, as a general-purpose computer or a workstation, and includes a CPU 301, an image compression unit 302, an I/F 303, a RAM 304, a ROM 305, and a recording device 306.

The CPU 301 is a device for controlling the overall imaging system 10. The CPU 301 transmits the development data transmitted by each of the multiple imaging units to the image compression unit 302, and controls the transmission of each compressed development data to the network 105. The CPU 301 can also calculate the imaging direction and imaging angle of view of each of the multiple imaging units based on the position and orientation of each of the multiple imaging units acquired from the position and orientation recognition unit 313, and stores information on the imaging direction and imaging angle of view in the RAM 304.

The image compression unit 302 performs compression processing on the developed data transmitted from the multiple imaging units under the control of the CPU 301 to generate compressed data. The compressed data is output to the display device 103 or to another external device via the network 105. For example, the compression processing of the developed data applies a compression method conforming to the JPEG standard to still images, and applies a compression method conforming to standards such as MOTION-JPEG, MPEG2, AVC/H.264, and AVC/H.265 to moving images.

I/F 303 is a device for communicating with external devices based on a network protocol. I/F 303 connects to input devices such as operation keys including a relay switch and a power switch, a cross key, a joystick, a touch panel, a keyboard, and a pointing device (e.g., a mouse). I/F 303 has a user interface for receiving instruction information from a user, etc., and notifies CPU 301 of the received information.

RAM 304 is a memory for temporarily storing the programs executed by CPU 301, and is, for example, a volatile memory such as SRAM or DRAM. RAM provides CPU 301 with a work area for executing and processing the programs.

The ROM 305 stores programs and data for the CPU 301 to control the imaging system 10, and is, for example, a non-volatile memory such as an EEPROM or a flash memory. The CPU 301 executes the programs to realize the functions of each part according to this embodiment, which will be described later.

The recording device 306 is a device for recording image data captured by multiple imaging units and image processing related data, and is, for example, an HDD, SSD, eMMC, etc.

Figure 3(b) shows the functions of the information processing device 102 of this embodiment. Note that each function shown in the figure is realized by the CPU 301 executing a computer program stored in the ROM 305.

The communication unit 400 can communicate with the imaging device 101, the recording device 306, etc., via the I/F 303 and the network 105. The communication unit 400 can acquire captured images captured by multiple imaging units. For example, the communication unit 400 has a function of receiving image data captured by multiple imaging units and transmitting control commands for controlling the multiple imaging units to each of the multiple imaging units. The control commands are, for example, commands from the information processing device 102 to cause the multiple imaging units to capture images and commands to change the positions and orientations of the multiple imaging units.

The storage unit 401 has information on the first setting operation and the second setting operation of the setting unit 404 described below. The storage unit 401 has a function of storing image processing results and image processing-related data in the RAM 304 and the recording device 306. Image processing-related data includes, for example, a large amount of image data showing the characteristics of a target.

The acquisition unit 402 can acquire an overall image by integrating images captured by multiple imaging units. An overall image is an image that captures a larger range than the imaging range of a single imaging unit, and is obtained by integrating images captured by multiple imaging units in, for example, a 360° imaging mode, which will be described later. The imaging mode used when acquiring captured images is not limited to the 360° imaging mode, and multiple images may be integrated. The overall image is used, for example, for recognizing an object from the overall image and for a user to select an area. The acquisition unit 402 can also acquire the absolute position of an object or a selected area based on the positions and orientations of the multiple imaging units acquired by the position and orientation recognition unit 313.

The recognition unit 403 can perform object recognition processing on images captured by multiple imaging units. The recognition unit 403 can read image data indicating the characteristics of an object stored in the storage unit 401 and recognize the object from the captured image, for example, by a pattern matching method. When recognizing, for example, a road from the captured image, the recognition unit 403 can probabilistically recognize the road by comparing the captured image with multiple matching patterns indicating the characteristics of the road. The matching patterns of the road may be multiple matching patterns when viewed from a perspective such as an oblique direction or an upward direction. In addition, the matching pattern does not have to indicate the overall characteristics of the road, but may rather be a matching pattern indicating part of the road's characteristics (for example, road forks and road signs, etc.).

The setting unit 404 can set the arrangement of the multiple imaging units. The setting unit 404 can perform a first setting operation to determine the arrangement of the multiple imaging units according to a setting selected from a plurality of settings that respectively define the relative arrangement between the multiple imaging units, for example. The setting unit 404 can also perform a second setting operation to determine the arrangement of the multiple imaging units by changing the arrangement of the multiple imaging units while maintaining the relative arrangement between the multiple imaging units. The setting unit 404 can automatically set the arrangement of the multiple imaging units so that the target recognized in the object recognition process of the recognition unit 403 is included in the shooting range.

The processing unit 405 can perform image processing, such as cropping of each of the images captured by the multiple imaging units. When performing the above processing, it is sufficient that the captured images are sufficiently large compared to the cropped image. The control unit 406 can control the arrangement of the multiple imaging units according to the settings made by the setting unit 404. The control unit 406 can also manage the transfer of control between the imaging device 101 and the multiple information processing devices 102.

4(a) to 4(c) show the arrangement of multiple imaging units arranged in the same circle after the first setting operation. Note that the multiple imaging units do not have to be arranged in the same circle as long as the imaging modes described below can be realized. Examples of the arrangement include a 360° shooting mode in which multiple imaging units are arranged to be able to shoot the entire circumference of the imaging device 101, and a 270° shooting mode in which only a part of the entire circumference of the imaging device 101 and an area directly below the imaging device 101 can be shot. There is also a line shooting mode in which the vectors of the imaging directions of the multiple imaging units starting from the imaging device 101 are arranged on the same plane. The user can remotely set the arrangement of the multiple imaging units by inputting any of the first setting operations to the setting unit 404 via the I/F 303. Each of the illustrated shooting modes will be described below.

Figure 4 (a) shows the arrangement of multiple imaging units in 360° shooting mode, which is used, for example, when shooting a vast space without partitions. By arranging the multiple imaging units with a 90° offset from each other, the multiple imaging units can capture the entire 360° circumference of the imaging device 101.

Figure 4 (b) shows the arrangement of multiple imaging units in 270° shooting mode, which is used, for example, when shooting an intersection with a road corner. In 270° shooting mode, the imaging unit 206 shoots the area directly below the imaging device 101, and the other imaging units can each shoot within a 270° range around the imaging device 101. This makes it possible for the multiple imaging units to shoot only a portion of the entire circumference of the imaging device 101 and the area directly below the imaging device 101.

Figure 4 (c) shows the arrangement of multiple imaging units in line shooting mode, which is used, for example, when shooting objects lined up in a straight line. Each of the multiple imaging units is arranged in a straight line, and the imaging directions of the multiple imaging units are set to different directions (or imaging direction vectors). The imaging directions (or imaging direction vectors) of the multiple imaging units are expressed as tilt angles that indicate the inclination of the posture of the imaging units. For example, the imaging direction directly below the imaging device 101 can be set to a 0° tilt, with imaging unit 202 at a +90° tilt, imaging unit 204 at a +30° tilt, imaging unit 206 at a -30° tilt, and imaging unit 208 at a -90° tilt. This allows the multiple imaging units to shoot objects lined up in a straight line from various angles. Note that the shooting modes in this embodiment are not limited to those described above.

Next, the I/F 303 has a user interface including a first setting unit that accepts a selection from multiple settings for a first setting operation, and a second setting unit that accepts an instruction to change the arrangement of multiple imaging units for a second setting operation. FIG. 5(a) is a schematic diagram illustrating the user interface of the first setting unit. In the user interface, rectangular buttons for selecting the first setting operation including the 360° shooting mode, the 270° shooting mode, and the line shooting mode are arranged on the screen. When information indicating that the user has pressed any of the above buttons is transmitted to the setting unit 404, the first setting operation becomes effective. Depending on the effective first setting operation, the setting unit 404 can set the arrangement of multiple imaging units.

FIG. 5B is a schematic diagram illustrating a user interface for the second setting operation. In the second setting operation, multiple imaging units can be rotated by the same angle with respect to a common axis or the center of a circle provided in the imaging device 101. In the user interface in FIG. 5B, buttons for the second setting operation including +1°, +5°, +10°, -1°, -5°, and -10° are arranged on the screen. Note that the above + indicates setting in the clockwise direction and - indicates setting in the counterclockwise direction, respectively, with respect to the Z axis in FIG. 2B as the reference. In the second setting operation, the arrangement of multiple imaging units can be determined by changing the arrangement of multiple imaging units, for example, by one of the angles shown above, while maintaining the relative arrangement between the multiple imaging units set in the first setting operation. For example, when the relative arrangement between multiple imaging units is rotated by +10° clockwise, the user presses the +10° button once, and the relative arrangement of the multiple imaging units is changed to an arrangement rotated by +10°. Note that the user interface in the figure does not show a button for changing the tilt angles of multiple imaging units, but a button for changing the tilt angles may be provided, or a separate user interface for setting the tilt angles may be provided.

As a specific example of imaging, the imaging device 101 is installed, for example, on the inside of a corner of a road, and the relative arrangement between the multiple imaging units is set so that a part of the imaging range of the entire circumference of the imaging device 101 can be captured. FIG. 6(a) is a schematic diagram for explaining the first and second setting operations when the imaging device 101 is installed on the inside of the corner of a three-way intersection (upper right part of the figure). The three-way intersection has a road extending upward (first direction) in the figure and a road extending left and right (second direction) from the corner based on the installation position of the imaging device 101. The imaging device 101 sets the 270° imaging mode in the first setting operation in order to capture images in the first and second directions.

Here, FIG. 7(a) shows the arrangement and attachment mechanism of multiple imaging units in the 270° shooting mode. The exterior 210 is an external component of the imaging device 101, and functions as a fixing member that allows the imaging device 101 to hold multiple imaging units. The fixing parts 212 are, for example, screws for fixing the imaging device 101 and the exterior 210, and are provided in four places on the circumference of the exterior 210. Note that the positions of the exterior 210 and the fixing parts 212 on the imaging device 101 do not change due to the first setting operation and the second setting operation. Meanwhile, the positions of the multiple imaging units are each determined by the fixed position of the exterior 210 relative to the imaging device 101. The determined positions of the multiple imaging units determine the shooting direction of each of the multiple imaging units.

Returning to the explanation, FIG. 6(b) is an image of a road in a second direction captured by the imaging unit 202 in FIG. 6(a). FIG. 6(b) also shows that the imaging range of the captured image is shifted in the counterclockwise direction from the imaging range that the user wants to capture. In such a case, the user would conventionally manually change the arrangement of the multiple imaging units so that the arrangement matches the imaging range that the user wants to capture. Manual rearrangement refers to, for example, the user changing the mounting position of the exterior 210 relative to the imaging device 101. In this embodiment, in order to improve the efficiency of the work involved in rearrangement of the multiple imaging units, the arrangement of the multiple imaging units can be changed by remote control via the user's mobile terminal and the network 105, etc.

To change the shooting range of the multiple imaging units, the user can change the arrangement of the multiple imaging units in a clockwise direction relative to the center of the circle by the second setting operation. The user provides an input to the user interface of the second setting operation to change the relative arrangement of the multiple imaging units by the same angle, for example, in a clockwise direction, while maintaining the relative arrangement of the multiple imaging units. Here, FIG. 7(b) shows an arrangement in which the arrangement of the multiple imaging units has been changed in a clockwise direction by the second setting operation. The second setting operation can change the arrangement of the multiple imaging units in a clockwise direction (the direction of the arrow) without changing the positions of the exterior 210 and the fixing part 212.

Figure 6(c) shows the arrangement of the multiple imaging units after the second setting operation. The road shape and the installation location of the imaging device 101 in Figure 6(c) are the same as those in Figure 6(a). The arrangement of the multiple imaging units has moved generally in a clockwise direction compared to before the second setting operation. Figure 6(d) is an image of the road in the second direction captured by the imaging unit 202 as in Figure 6(c), and the image has moved generally in a clockwise direction compared to Figure 6(c). This allows the imaging unit 202 to capture the road in the second direction in the shooting range that the user wants to capture. According to this embodiment, the arrangement of the multiple imaging units can be efficiently changed in the circumferential direction of the imaging device 101 in the second setting operation according to the shooting range that the user wants to capture after setting the first setting operation.

Figure 8 (a) is a flowchart showing the processing flow of the first setting operation. This function is realized by the CPU 301 executing a program stored in the ROM 305. In S701, the setting unit 404 determines whether or not any of the 360° shooting mode, 270° shooting mode, and line shooting mode has been set as the first setting operation. If the user sets the first setting operation via the user interface (YES in S701), the process proceeds to S702. If none of the first setting operations has been set (NO in S701), the process ends. In S702, the setting unit 404 sets the arrangement between the multiple imaging units based on the first setting operation input in S701. Through the above processing, the first setting operation of the imaging device 101 is executed.

Figure 8 (b) is a flowchart showing the processing flow of the second setting operation. This function is realized by the CPU 301 executing a program stored in the ROM 305. In S703, the setting unit 404 judges whether the second setting operation has been set. If the user has set the second setting operation via the user interface (YES in S703), proceed to S704. If none of the second setting operations has been set (NO in S703), the process ends. In S704, based on the second setting operation input in S703, the multiple imaging units are rotated by the same angle around a common axis or a center within a circle of the imaging device 101 while maintaining the relative arrangement between the multiple imaging units. This makes it possible to change the arrangement of the multiple imaging units. The means for changing the shooting range of the multiple imaging units include a change by a mechanical mechanism and a change by image processing (e.g., a cutout process), and the details of these will be described later. Through the above processing, the second setting operation of the imaging device 101 is executed.

Figure 9 is a schematic diagram explaining the movement direction of each of the multiple imaging units by the mechanical mechanism. The mechanical mechanism is one of the means for changing the arrangement between the multiple imaging units described in S704 of Figure 8. As described in Figure 2 (b), the multiple imaging units are arranged in the same circle, and each can move in the circumferential direction (the direction of arrow B) and the radial direction (the direction of arrow C). In addition, the multiple imaging units can realize the above movement by themselves at their respective fixed positions. In this way, the multiple imaging units can move freely within the circle of the imaging device 101, so that the relative arrangement of the first setting operation can be realized. In addition, the mechanical mechanism makes it possible to change the arrangement of the multiple imaging units in the second setting operation. By providing this mechanical mechanism, for example, the arrangement of the multiple imaging units in the 270° shooting mode and line shooting mode of the first setting operation can be realized.

On the other hand, instead of changing the arrangement of the multiple imaging units, the shooting range of the image corresponding to each imaging unit may be changed by changing the cutout range from the image captured by each imaging unit. FIG. 10 is a schematic diagram for explaining a method of changing the shooting range by image processing. The image processing in this embodiment is a cutout process for each of the images captured by the multiple imaging units, and is one of the means for changing the arrangement of the multiple imaging units described in S704 of FIG. 8. In this way, if the shooting range that the user wants to shoot can be realized by the cutout process for the captured image, the processing unit 405 does not need to change the arrangement of the multiple imaging units. This configuration can be used, in particular, when adjusting the shooting range in the second setting operation after setting the relative arrangement of the imaging units using the first setting operation. In S1001, the setting unit 404 determines whether any of the second setting operations has been set. If the user has set the second setting operation via the user interface (YES in S1001), proceed to S1002. If none of the second setting operations has been set (NO in S1001), the processing ends.

In step S1002, the processing unit 405 determines whether or not the cutout process can be performed on each image captured by the multiple imaging units. If image data larger than the images captured by the imaging units 308 of the multiple imaging units is acquired (YES in S1002), the processing unit 405 can perform the cutout process. If the processing unit 405 needs to cut out an area other than the captured image (NO in S1002), the processing unit 405 changes the arrangement of the multiple imaging units using a mechanical mechanism. In S1003, the processing unit 405 performs the cutout process on each of the captured images captured by the multiple imaging units. The size of the cutout area in the cutout process may be a preset size, or may change depending on the shooting conditions such as the focal length at the time of shooting recorded in the image. According to this embodiment, the shooting range desired by the user can be realized by performing the cutout process on each of the captured images of the multiple imaging units.

FIG. 11 is a schematic diagram for explaining a specific example of the cutout process. The whole image 1100 and the image 1120 are images captured by one of the multiple imaging units. The image 1140 is an image after the processing unit 405 has performed cutout processing from the whole image 1100. The processing of the processing unit 405 will be explained when the acquisition unit 402 has acquired the whole image 1100 that is larger than the image captured by the imaging unit 308 of one of the multiple imaging units. The processing unit 405 can perform cutout processing from the whole image 1100 and obtain the image 1140 based on, for example, a user's instruction to change the shooting direction. Note that the image 1120 is the same as the captured image in FIG. 6B, and the image 1140 after the cutout processing is the same as FIG. 6D. In this way, when multiple imaging units capture a whole image larger than the captured image and there is an instruction to change the shooting direction, the processing unit 405 can perform the cutout processing.

Figure 12 is a flowchart explaining how the second setting operation of multiple imaging units is automatically performed based on object recognition. As an example of an embodiment, for example, when the imaging device 101 is installed inside a corner of a three-way intersection, the recognition unit 403 can recognize a first direction and a second direction from the corner on a road extending from the corner.

In S1201, the setting unit 404 executes the 360° shooting mode in the first setting operation to acquire an overall image. The multiple imaging units are set to an arrangement corresponding to the 360° shooting mode. In S1202, the acquisition unit 402 acquires an overall image by integrating the images captured by the multiple imaging units. Note that the captured image used when acquiring the overall image may be a part of the above. In S1203, the recognition unit 403 can automatically recognize a three-way intersection as an object from the overall image, for example, by a pattern matching method. In S1204, the setting unit 404 executes the 270° shooting mode in the first setting operation to capture roads in the first and second directions of the three-way intersection. Note that the setting unit 404 can set the optimal shooting mode in the first setting operation according to, for example, the number of objects and the direction of the objects recognized by the recognition unit 403.

In S1205, the setting unit 404 changes the arrangement of the multiple imaging units. At this time, the setting unit 404 can change the arrangement so that the direction from the imaging device 101 toward the center of the imaging range using the multiple imaging units matches the direction from the imaging device 101 toward the center of the imaging target around the imaging device 101. In this example, the imaging target is a road. In particular, in the example of FIG. 6(a), the center direction of the imaging target is the center direction 601 between the two roads extending from the position of the imaging device 101. Therefore, as shown in FIG. 6(c), the arrangement of the multiple imaging units is changed while maintaining the relative arrangement so that the direction toward the center of the imaging range using the multiple imaging units 202 (the direction of the imaging units 202 in this example) faces the center direction 601.

To perform this operation, the acquisition unit 402 refers to the position and orientation of each of the multiple imaging units acquired by the position and orientation recognition unit 313 in S1203. Based on this reference, the acquisition unit 402 can acquire the absolute position (position in space) of the road from the position on the image of the road recognized by the recognition unit 403 in S1203. The acquisition unit 402 can also acquire the positions and orientations of the multiple imaging units set in S1204 via the position and orientation recognition unit 313. This allows the setting unit 404 to automatically perform a second setting operation based on the absolute position of the road and the positions and orientations of the multiple imaging units so that the road is included in the shooting range of the multiple imaging units.

As another example, suppose that the position and orientation of the imaging unit 202 are associated when acquiring the absolute position of the road in the second direction shown in FIG. 6(a). In this case, the setting unit 404 can change the arrangement of the multiple imaging units until the shooting range of the imaging unit 202 includes the absolute position of the road in the second direction. Note that multiple imaging units may be associated with the absolute position of the road, and the user may decide which range of the shooting range of the associated imaging unit includes the absolute position of the road. The user may set this information in the setting unit 404 in advance, thereby changing the arrangement of the multiple imaging units. According to this embodiment, it is possible to change the arrangement of the multiple imaging units so that a specific shooting range (including corners) of the entire circumference of the imaging device 101 is captured according to the recognized target.

Figure 13 is a flowchart explaining how the arrangement of multiple imaging units is automatically changed based on a user-selected area on a captured image. A selected area refers to an area selected by a user from an image captured by multiple imaging units, or from an overall image obtained by integrating these captured images. This embodiment is used when the imaging device 101 is installed in a large area such as a parking lot.

In S1301, the setting unit 404 executes the 360° shooting mode of the first setting operation to acquire an overall image. The multiple imaging units are set to an arrangement corresponding to the 360° shooting mode. In S1302, the acquisition unit 402 acquires an overall image that integrates the images captured by the multiple imaging units. In S1303, the user selects, for example, multiple vehicles lined up in a straight line from the overall image as a selection area. At this time, the user may select the object with a symbol such as a circle or a square that can cover the object.

In S1304, the setting unit 404 executes the multi-lens preset function. Here, the setting unit 404 can select a multi-lens preset that covers the attention area selected by the user in S1303. For example, if the attention area is linear, the setting unit 404 can select the line shooting mode as the first setting operation. As a specific example, if the setting unit 404 determines that the multiple selection areas selected by the user and the imaging device 101 are arranged in a straight line, the setting unit 404 can select the line shooting mode. On the other hand, if the attention area is radial, the setting unit 404 can select the 270° shooting mode or the 360° shooting mode as the first setting operation. As a specific example, if the setting unit 404 determines that the multiple selection areas selected by the user are arranged radially with the imaging device 101 as the center, the setting unit 404 sets the following shooting mode. For example, if the selection area spreads over a narrower range, the setting unit 404 can select the 270° shooting mode, and if the selection area spreads over a wider range, the setting unit 404 can select the 360° shooting mode.

In S1305, the acquisition unit 402 can acquire the absolute position of the selected area selected in S1303 based on the respective positions and orientations of the multiple imaging units acquired by the position and orientation recognition unit 313 at the time of S1303. The acquisition unit 402 can also acquire the positions and orientations of the multiple imaging units set in S1304 via the position and orientation recognition unit 313. The setting unit 404 can set the arrangement of the multiple imaging units by a second setting operation based on the absolute position of the selected area and the positions and orientations of the multiple imaging units so that the selected area is included in the shooting range of the corresponding multiple imaging units. According to this embodiment, it is possible to automatically set the arrangement of the multiple imaging units according to the target present in the user's selected area included in the entire image.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

10: Imaging system, 101: Imaging device, 102: Information processing device, 103: Display device, 104: Cable, 105: Network

Claims (9)

A plurality of imaging units arranged on the same circle and rotatable around a central axis;
an acquisition means for acquiring an instruction to select one of a first shooting mode in a state in which the plurality of imaging units are arranged at substantially equal intervals around the central axis, a second shooting mode in a state in which at least one of the plurality of imaging units is arranged around the central axis and another imaging unit of the plurality of imaging units different from the imaging unit arranged around the central axis is arranged near the central axis, and a third shooting mode in which the plurality of imaging units are arranged in a straight line;
a setting means for setting a relative arrangement between the plurality of imaging units based on the result of acquisition by the acquisition means ;
a control unit that rotates the plurality of imaging units around the central axis while maintaining the relative arrangement between the plurality of imaging units set by the setting unit.
Imaging device. the control means rotates the plurality of imaging units around the central axis such that the plurality of imaging units rotate at the same angle with respect to the central axis of the plurality of imaging units;
The imaging device according to claim 1 . the acquiring means receives an instruction regarding a rotation angle of the plurality of imaging units for rotating the plurality of imaging units about the central axis while maintaining the relative arrangement between the plurality of imaging units set by the setting means ;
The imaging device according to claim 1 . The imaging device further includes a position recognition unit that recognizes the position and orientation of each of the plurality of imaging units.
The imaging device according to claim 1 . The object recognition unit further includes an object recognition unit that performs object recognition processing on the images captured by the plurality of imaging units.
the control means rotates the plurality of imaging units around the central axis at rotation angles of the plurality of imaging units determined based on a result of the object recognition processing.
The imaging device according to claim 1 . a processing unit that performs a cutout process on each of the images captured by the plurality of imaging units,
the processing means changes a cut-out portion in the cut-out process in accordance with an instruction to change a shooting direction of each of the images corresponding to the plurality of imaging units.
The imaging device according to claim 1 . the control means acquires an entire image by synthesizing the images captured by the plurality of imaging units.
The imaging device according to claim 1 . A control method for controlling an imaging device having a plurality of imaging units arranged on the same circle and rotatable around a central axis, comprising :
an acquisition step of acquiring an instruction to select one of a first shooting mode in a state where the plurality of imaging units are arranged at substantially equal intervals around the central axis, a second shooting mode in a state where at least one of the plurality of imaging units is arranged around the central axis and another imaging unit of the plurality of imaging units different from the imaging unit arranged around the central axis is arranged near the central axis, and a third shooting mode in which the plurality of imaging units are arranged in a straight line;
a setting step of setting a relative arrangement between the plurality of imaging units based on the result of the acquisition step ;
a control step of rotating the plurality of imaging units around the central axis while maintaining the relative arrangement between the plurality of imaging units set by the setting step.
Control methods. A program for causing a computer to function as the imaging device according to any one of claims 1 to 7 .

Images