JP2010056953A - Image capturing apparatus, image processing device and imaging system - Google Patents

Abstract

An imaging apparatus, an image processing apparatus, and an imaging system capable of generating a high-resolution image at high speed are provided.
A light splitting unit 2 splits incident light into a plurality of light beams, which are incident on a plurality of image sensors 3. Of the plurality of imaging elements, one captures a full pixel image (high resolution image), and the other captures an image (low resolution image) thinned out according to the number of lights divided by the light dividing unit. The timing control unit 4 controls the shooting timing and data transfer timing of each image sensor. The image processing unit 7 generates a high resolution image by calculating a movement vector between the low resolution images and adding it to the high resolution image.
[Selection] Figure 1

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, and an imaging system that can perform high-speed imaging.

There is a high-speed imaging device capable of continuous shooting at high speed.
When performing high-speed imaging with a high-speed imaging device, the processing time required to output data from the pixels of the solid-state imaging device, that is, the processing rate becomes a problem.

One technique for performing high-speed imaging is to perform imaging at short intervals using a plurality of image sensors.
Each image sensor photoelectrically converts incident light to generate an image signal.
Here, in order for each image sensor to perform the next shooting at a short interval, it is necessary to read the image signal from each image sensor.
For example, when shooting is performed at a shooting speed of 10,000 frames / second by a solid-state imaging device having 100,000 pixels, the image signal must be read out at a speed of 1 GHz. However, from the viewpoint of memory reading / writing speed, there is a limit to speeding up the reading process from the image sensor.

Methods for solving such problems are disclosed in, for example, Patent Documents 1 to 3 and the like.
In Patent Document 1, as a pixel peripheral recording type imaging device, a pixel peripheral recording type imaging device including a charge signal accumulation unit including a charge coupled device obliquely extending linearly from an individual photodiode having a relatively large area ( An oblique CCD type image pickup device) is disclosed.
In the oblique CCD type image sensor disclosed in Patent Document 1, charge signals are not continuously read out from the element during photographing, but are continuously overwritten on an image signal storage unit provided around each pixel. Then, all the electrical signals stored in the image signal storage unit after the photographing is completed are read out.

As another method, Patent Document 2 discloses a technique that improves the data transfer speed instead of accumulating a large number of image data for each pixel in the element.
In Patent Document 2, a data detection circuit and an A / D conversion circuit are provided for each column of an image sensor by a solid-state image sensor called a column parallel CMOS sensor, so that data output is parallelized and an image from the image sensor is obtained. The signal reading speed is improved.

As an approach different from the techniques disclosed in Patent Documents 1 and 2 described above, for example, there is a technique disclosed in Patent Document 3.
Patent Document 3 discloses a technology that includes a light splitting prism that splits incident light into three parts, and photoelectrically converts the split incident light with a plurality of imaging elements.
That is, in order to break the limit of shortening the readout time from one image sensor, the first image sensor generates an image signal of the first image data. Then, while this is read out, the second image sensor generates an image signal of the second image data so that high-speed shooting is possible by shifting the shooting timing of each image sensor. Incident light is divided and made incident on the plurality of image pickup devices, for example, by a prism or the like.
JP 2001-345441 A JP 2005-323331 A JP-A-4-68876

The technique disclosed in Patent Document 1 has an advantage that the shooting speed is not affected by the output rate, but has a disadvantage that the number of images that can be shot, that is, the shooting time is limited by the memory capacity.
For example, in a high-speed imaging apparatus that can store up to 100 images, that is, up to 100 image signals in a memory, it is naturally impossible to save the 101st and subsequent images.
This problem can be solved to some extent by increasing the memory capacity, but if the memory capacity is to be increased, the physical size itself must be increased, and in recent imaging devices that are required to be downsized. It is difficult to take such a solution.
In addition, the physical size of the memory presses on other components, and there is a problem that the number of pixels of the solid-state imaging device has to be reduced or the pitch between the pixels has to be increased.
Furthermore, when it is desired to perform shooting at a high resolution, it is necessary to increase the number of pixels and reduce the arrangement pitch between the pixels, but the size of the memory must be reduced.

In addition, the technique disclosed in Patent Document 2 is improved for high-speed shooting at high resolution, but has a disadvantage that it is not yet fully compatible with ultra-high-speed shooting at high resolution.
Furthermore, the technique disclosed in Patent Document 3 also has a disadvantage that it is difficult to capture a high-resolution image at high speed as recently requested.

  The present invention provides an imaging apparatus, an image processing apparatus, and an imaging system that can generate a high-resolution image at high speed.

  In order to eliminate the disadvantages described above, the imaging apparatus according to the first aspect of the present invention includes a light dividing unit that divides the light amount of incident light into a plurality of light beams and the light beams divided into the plurality of light beams. In response, a plurality of image sensors that generate an image signal and transfer the generated image signal to a subsequent circuit, timing control that controls image signal generation timing of the image sensor, and transfer timing that transfers the generated image signal And at least one of the plurality of image sensors generates a higher resolution image than the other image sensors, and the timing control unit transfers data of the image sensor that generates a high resolution image. During this, the image pickup operation of the other image pickup device is executed a plurality of times to generate an image signal, and data transfer of the generated image signal is executed.

  An image processing apparatus according to a second aspect of the present invention provides a plurality of low resolution images in which at least one high resolution image signal and the same subject as the high resolution image generated at equal intervals following the high resolution image signal are photographed. An image signal is acquired, and based on the plurality of low resolution image signals, the high resolution image signal is complemented to generate a plurality of high resolution image signals corresponding to the plurality of low resolution image signals.

  An imaging system according to a third aspect of the present invention includes a light dividing unit that divides incident light into a plurality of pieces, a plurality of image pickup elements that respectively enter the plurality of divided lights and generate image signals according to the incident light. , An image processing unit that performs predetermined image processing on the image signals transferred from the plurality of image sensors, an image signal generation timing of the image sensor, and a timing for controlling a transfer timing of transferring the generated image signals A control unit, wherein at least one of the plurality of image sensors generates a higher resolution image than the other image sensors, and the timing control unit generates data of the image sensor that generates a high resolution image. During the transfer, the image pickup operation of the other image pickup device is executed a plurality of times, the data transfer is also executed following the image pickup operation, and the image processing unit performs the plurality of low resolution image signals as predetermined image processing. Based complements the high-resolution image signal, performs processing for generating a plurality of high resolution image signals corresponding to the plurality of low-resolution image signal.

  The present invention can generate a high-resolution image at high speed.

Hereinafter, the imaging system 100 of the present invention will be described.
<First Embodiment>
Hereinafter, a first embodiment of the imaging system 100 will be described.
FIG. 1 is a block diagram illustrating an example of the configuration of the imaging system 100.
As shown in FIG. 1, an imaging system 100 includes an optical system 1, a light dividing unit 2, a plurality of imaging elements 3, a timing control unit 4, a plurality of primary data storage units 5, a secondary data storage unit 6, and image processing. Unit 7, storage device 8, monitor image processing unit 9, and monitor 10.

Note that the imaging system 100 includes an imaging unit 101 and an image processing system 102.
The imaging unit 101 includes an optical system 1, a light dividing unit 2, an imaging element 3, a timing control unit 4, a primary data storage unit 5, and a secondary data storage unit 6.
The image processing system 102 includes an image processing unit 7, a storage device 8, a monitor image processing unit 9, and a monitor 10.
The imaging unit 101 generates an image signal based on the incident light, and the image processing system performs image processing on the image signal and outputs it as image data, and performs various processes such as storage and display.

The optical system 1 includes a lens or the like, and causes reflected light from a subject to enter the light splitting unit 2.
The light dividing unit 2 divides the light incident through the optical system 1 into four.
For example, as shown in FIG. 2, the light splitting unit 2 includes a plurality of prisms.

FIG. 2 is a diagram illustrating an example of the configuration of the light splitting unit 2 according to the first embodiment.
As shown in FIG. 2, the light splitting unit 2 includes three prisms 21 to 23.
Each prism 21-23 is a triangular prism, for example, and divides incident light from one direction into two directions.
As shown in FIG. 2, in the first embodiment, the incident light is divided into four by the prisms 21 to 23 and is incident on different image pickup devices 3. Here, the number of lights divided by the light dividing unit 2 corresponds to the number of the image sensors 3.

  In the present invention, the number of prisms in the light splitting unit and the number of splitting incident light are not particularly limited. However, even if the number of divisions is increased, the amount of light is insufficient or the cost of the image sensor is increased. Therefore, the number of divisions of incident light is preferably about 3 to 6.

The image sensor 3 is an image sensor that converts incident light into an image signal.
The image sensor 3 includes a plurality of image sensors corresponding to the light divided by the light dividing unit 2.
In the first embodiment, since the light dividing unit 2 divides the light into four, it has four image pickup devices 3A to 3D.
The image pickup devices 3A to 3D may be configured by a semiconductor image pickup device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).
In the present embodiment, these imaging elements 3A to 3D have the following imaging capabilities, for example.
That is, for example, a shutter time is 50 μs, a full pixel image (for example, 1900 × 1280 pixels, hereinafter, a high resolution image) can be taken, and a data transfer time of the high resolution image is 1 ms. As the shutter, an electronic global shutter capable of simultaneously shuttering the entire image is used. The data transfer time is the time from when the image signals generated in the image pickup devices 3A to 3D are A / D (analog / digital) converted until the transfer to the primary data storage units 5A to 5D is completed. is there.
In the present embodiment, the image pickup device 3A generates a high-resolution image, and the image pickup devices 3B to 3D are image signals (hereinafter referred to as ¼) whose data has been thinned to ¼ according to the incident light divided by the light dividing unit 2. Low resolution image).

  In addition, the incident light divided | segmented as shown in FIG. 2 injects into image pick-up element 3A-3D. Since the image signals generated by the four image pickup devices 3A to 3D will be continuous image data later, the pixel positions of the image pickup devices 3A to 3D need to be exactly overlapped with each other. This is realized by performing correction and confirmation in advance (for example, when the imaging system 100 is constructed).

The timing control unit 4 is a control block for controlling the shutter timing of the image sensors 3A to 3D and the image signal readout timing from the image sensors 3A to 3D. The timing control unit 4 performs timing control by outputting a clock signal, for example. The timing control of the timing control unit 4 will be described later in detail.
The primary data storage unit 5 is a buffer memory that temporarily stores an image signal generated by the image sensor 3.
The plurality of primary data storage units 5 are prepared in the same number as the image pickup devices 3 and store image signals transferred from the corresponding image pickup devices 3, respectively. Therefore, in the first embodiment, four primary data storage units 5A to 5D are prepared.
The primary data storage units 5A to 5D are composed of, for example, a DRAM (Dynamic Random Access Memory).
The primary data storage unit 5A is the image sensor 3A, the primary data storage unit 5B is the image sensor 3B, the primary data storage unit 5C is the image sensor 3C, and the primary data storage unit 5D is the image signal of the image sensor 3D. Remember each one.

  The secondary data storage unit 6 stores the image signals transferred from the primary data storage units 5A to 5D by a technique such as parallel writing. The secondary data storage unit 6 is a mass storage device such as an HDD (Hard Disk Drive).

The image processing unit 7 acquires an image signal from the secondary data storage unit 6 (imaging unit 101), performs image processing such as inter-pixel interpolation, time series arrangement, data compression / encoding, and generates digital image data. . Then, for example, the data is converted into a data format that can be streamed.
Details of the image processing performed by the image processing unit 7 will be described later.
The storage device 8 is a storage medium that stores digital image data in a format capable of streaming distribution generated by the image processing unit 7. The storage device 8 is, for example, a large-capacity storage device such as an HDD, or an optical disk drive such as a DVD or a BD (Blu-ray Disc).

The monitor image processing unit 9 acquires an image signal from any one of the imaging elements 3 </ b> A to 3 </ b> D of the imaging unit 101, performs monitor image processing so that monitoring can be performed in real time, and outputs the monitor signal to the monitor 10.
The monitor 10 is a monitor device for monitoring an image captured by the imaging unit 101 in real time.

  With the configuration described above, the image processing system 102 can acquire the image signal generated by the imaging unit 101, perform image processing, and store or output the image signal.

Next, the imaging timing and transfer timing control processing of the imaging devices 3A to 3D by the timing control unit 4 in the first embodiment will be described.
FIG. 3 is a time chart for explaining the imaging timing and transfer timing control processing of the image sensors 3A to 3D by the timing control unit 4 in the first embodiment.

In FIG. 3, the black thick line indicates the photographing timing (shutter timing), and the hatched portion subsequent thereto indicates the image signal transfer timing (time required for data output).
As shown in FIG. 3, the timing control unit 4 first performs imaging of the image sensor 3A (t1 in FIG. 3). In FIG. 3, the time when the image pickup of the image sensor 3A is first performed is set to 0 for convenience in the following.
Then, control is performed so that the image sensor 3B is imaged at the next timing when the image sensor 3A is imaged (t2).
Similarly, the imaging elements 3C and 3D are also photographed at the same interval (t3 and t4).
When the timing control unit 4 causes the imaging device 3D to perform imaging, the timing controller 4 next performs imaging in the order of the imaging devices 3B, 3C, and 3D (t5, t6, and t7).
Next, the image pickup by the image pickup device 3A is performed again. In the present image sensor system, one image pickup unit is from the first image pickup by the image pickup element A to the next image pickup by the image pickup element A, which is hereinafter referred to as one set. One set of imaging is performed, for example, in 1 ms (milliseconds). In this case, the shooting timing in FIG. 3 is 1/7 ms.

Now, in this image sensor system, incident light is divided into four by the light splitting unit 2 and made incident on the four image sensors 3A to 3D. Then, only the image pickup device 3A generates a high-resolution image signal, and the other three image pickup devices 3B to 3D generate low-resolution images in which the data amount is thinned to ¼.
Therefore, the data amount of the low-resolution image generated by the image sensors 3B to 3D is 1/4 of the data amount of the image generated by the image sensor 3A.
As described above, the data transfer time of the high resolution image is 1 ms. Therefore, in FIG. 3, the data transfer time of the image pickup device 3A is 1 ms, but the data transfer time of the image pickup devices 3B to 3D is 1/4 ms, which is 1/4 ms. Since one scale shown in FIG. 3 is 1/7 ms, reading of the image sensors 3B to 3D takes two scales.
That is, as shown in FIG. 3, while the image pickup device 3A takes 1 ms from t1 and outputs data, the image pickup devices 3B to 3D take images twice while shifting by 1/7 ms and also output data. It will be. The data output from the image sensor 3A ends at 1 ms, but the second output from the image sensor 3D has not ended yet, and this ends at the time 8/7 ms.

As described above, the timing control unit 4 performs shooting by shifting the shooting timing in the order of the image sensors 3A, 3B, 3C, 3D, 3B, 3C, and 3D as illustrated in FIG. Then, this combination is taken as one set and repeated thereafter.
An image signal is output from each image sensor under the control of the timing control unit 4 as described above.
The image signal output at each timing is referred to as follows.
The image output from the image sensor 3A is An (n is a positive integer and indicates the number of times the above-described one set of imaging is repeated).
The images output from the image sensor 3B are bn1 (first image in one set described above) and bn2 (second image in one set).
The images output from the image sensor 3C are cn1 (first image in one set described above) and cn2 (second image in one set).
The images output from the image sensor 3D are dn1 (first image in one set described above) and dn2 (second image in one set).

Next, image processing in the image processing unit 7 of the image processing system 102 will be described.
In the image processing of the image processing unit 7 to be described below, one set of image signals A1, b11, c11, d11, b12, c12, d12 generated by the image pickup devices 3A to 3D according to the control of the timing control unit 4 described above. Suppose you use

The image processing unit 7 first generates a low resolution image a11 obtained by thinning out the high resolution image signal A1 to ¼.
Since the image A1 generated by the image sensor 3A is a high-resolution image as described above, and the images b11 to d12 are low-resolution images thinned by ¼, the image A1 is processed in order to process them in the same row. Generates a low-resolution image.
Next, a movement vector between images taken at adjacent timings is calculated. That is, the movement vectors between the images a11 and b11, b11 and c11, c11 and d11, d11 and b12, b12 and c12, and c12 and d12 are calculated.

The movement vector calculation method of the image processing unit 7 is not particularly limited in the present invention. For example, a rotation vector image processing method or a spatio-temporal movement vector method is assumed on the assumption that an image obtained by photographing an object moving at high speed is processed. Can be used.
The rotation vector image processing method is a method for extracting a moving body in an image by a spatio-temporal movement vector method using a rotation vector.
The spatio-temporal movement vector method is a method of detecting a moving body between two images by block matching processing.

  Next, inter-image movement vectors are sequentially added to the high resolution image signal A1. That is, a high resolution image signal B11 can be obtained by adding a movement vector between a11 and b11 to the image signal A1. Similarly, a high-resolution image signal C11 can be obtained by adding a movement vector between b11 and c11 to the image signal B11. Similarly, high-resolution image signals D11, B12, C12, and D12 are generated.

The image processing unit 7 performs inter-image complementation as described above, and obtains one set of high-resolution image signals A1, B11, C11, D11, B12, C12, and D12.
However, in the case where sufficient complementation cannot be performed only by complementation using a movement vector between low-resolution images, for example, a technique for generating a high-resolution image from a low-resolution image disclosed in Japanese Patent Application Laid-Open No. 2004-208339 is used. Thus, a high resolution image may be generated.
Then, the image processing unit 7 arranges a plurality of high-resolution images obtained by the above-described processing in order of the generated time series, and performs additional processing such as noise removal, contrast adjustment, compression / encoding, and the like. Generate data.

According to the imaging system 100 as described above, a high-resolution image can be captured at high speed. In the example described above, a high-resolution image can be taken every 1/7 ms, for example. Therefore, 7000 high-resolution images can be taken per second.
However, in the present invention, the interval at which the imaging devices 3A to 3B perform shooting is not limited to 1/7 ms. As described above, the shooting interval varies depending on how many seconds are taken for one set. In the present embodiment, a shooting speed of 7000 images per second was obtained by setting one set to 1 ms. However, by changing the time of one set, more images or fewer images can be captured per second. It becomes possible to do.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described.
The second embodiment has substantially the same configuration as the first embodiment, but the configuration of the light splitting unit 2 is slightly different and splits the light into three. Accordingly, the number of image pickup devices 3 is also three (3A to 3C), and the primary data storage unit 5 is also composed of three (3A to 3C). Further, the shooting timing and transfer timing control processing of the image sensor 3 by the timing control unit 4 is slightly different from the first embodiment in accordance with the change in configuration.
Below, a different point from 1st Embodiment is demonstrated.

First, in the second embodiment, the light splitting unit 2 has a configuration as shown in FIG.
FIG. 4 is a diagram illustrating an example of the configuration of the light splitting unit 2 of the second embodiment.
As shown in FIG. 4, in the second embodiment, the light splitting unit 2 includes two prisms 24 and 25.
With such a configuration, the light dividing unit 2 divides the light into one light having a light amount that is ½ of the incident light and two light beams having a light amount that is ¼ of the incident light.
Here, it is assumed that light having a light amount that is ½ of incident light is incident on the image sensor 3A, and light having a light amount that is ¼ of incident light is incident on the image sensors 3B and 3C.

Next, imaging timing and transfer timing control processing of the imaging devices 3A to 3C by the timing control unit 4 of the second embodiment will be described.
FIG. 5 is a time chart for explaining the imaging timing and transfer timing control processing of the imaging devices 3A to 3C by the timing control unit 4 in the second embodiment.

In FIG. 5, the black thick line indicates the photographing timing (shutter timing), and the hatched portion subsequent thereto indicates the transfer timing of the image signal (time required for data output).
As shown in FIG. 5, the timing control unit 4 first performs imaging with the image sensor 3 </ b> A (t <b> 11 in FIG. 5). In FIG. 5, the time point when the imaging device 3 </ b> A is first photographed is hereinafter set to 0 for convenience.
Then, control is performed so that the image sensor 3B is imaged at the next timing when the image sensor 3A is imaged (t12).
Similarly, the image sensor 3C is also photographed at the same interval (t13).
Next, the timing control unit 4 causes the imaging devices 3B and 3C to perform imaging again in the order (t14 and t15).
Next, the image pickup by the image pickup device 3A is performed again. In the present image sensor system, one image pickup unit is from the first image pickup by the image pickup element A to the next image pickup by the image pickup element A, which is hereinafter referred to as one set. One set of imaging is performed, for example, in 1 ms (milliseconds). In this case, the shooting timing in FIG. 5 is 1/5 ms.

As described above, the data transfer time of the high resolution image is 1 ms. Therefore, in FIG. 5, the data transfer time of the image sensor 3A is 1 ms, but the data transfer time of the image sensors 3B and 3C is 1/3 of 1/3 of 1 ms. Since one scale shown in FIG. 5 is 1/5 ms, data transfer of the image sensors 3B and 3C takes two scales.
That is, as shown in FIG. 5, while the image sensor 3A takes 1 ms from t11 to output data, the image sensors 3B and 3C take images twice while shifting by 1/5 ms, and also output data. It will be. The data output of the image sensor 3A ends at 1 ms, but the second output of the image sensor 3C has not ended yet, and this ends at the time of 6/5 ms.
Furthermore, in the second embodiment, the timing control unit 4 is configured to read out signal amounts (charge amounts) for two pixels at a time for the image sensors 3B and 3C. That is, the information for two pixels is added and transferred. In FIG. 5, at the time of photographing with the image sensors 3B and 3C, photographing for two times and data transfer are continuously performed.
This is equivalent to doubling the light receiving area in the image sensors 3B and 3C, and means that the sensitivity of the image sensors 3B and 3C is doubled.

As described above, the timing control unit 4 causes the imaging to be performed while shifting the imaging timing in the order of the imaging elements 3A, 3B, 3C, 3B, and 3C as illustrated in FIG. Then, this combination is taken as one set and repeated thereafter.
In the second embodiment, since photographing is performed five times within 1 ms, 5000 images can be generated per second.

As described above, according to the second embodiment of the imaging system 100, the amount of light incident on the imaging device 3A is ½ of the entire incident light that is twice that of the first embodiment. In the second embodiment, the image sensors 3B and 3C are configured to read out signal amounts (charge amounts) for two pixels at a time.
For this reason, the sensitivity of the imaging device 3 as a whole has doubled compared to the first embodiment, and high-sensitivity imaging can be performed.
This makes it possible to cope with, for example, a dark environment or a situation where high-speed shooting is performed with the shutter speed halved, and image blurring can be suppressed.

  In the second embodiment described above, the image sensors 3B and 3C transfer the signal amount for two pixels at a time, but the present invention is not limited to this. That is, for example, the image pickup devices 3B and 3C may transfer the signal amount for four pixels at a time. In this case, it means that the light receiving area of the image pickup devices 3B and 3C is quadrupled, so that higher sensitivity shooting can be performed.

<Third Embodiment>
Hereinafter, a third embodiment of the imaging system 100 will be described.
In the third embodiment, the configuration is almost the same as that of the second embodiment, but the timing control processing of the image sensor 3 by the timing control unit 4 is slightly different.
Hereinafter, imaging timing and transfer timing control processing of the imaging devices 3A to 3C by the timing control unit 4 of the third embodiment will be described.
In the third embodiment, as in the second embodiment, the light dividing unit 2 divides incident light into three.

  FIG. 6 is a time chart for explaining the imaging timing and transfer timing control processing of the imaging devices 3A to 3C by the timing control unit 4 in the third embodiment.

In FIG. 6, the black thick line indicates the photographing timing (shutter timing), and the hatched portion subsequent thereto indicates the image signal transfer timing (time required for data output).
As shown in FIG. 6, the timing control unit 4 first performs imaging of the image sensor 3A (t21 in FIG. 6). In FIG. 6, the time when the image pickup by the image pickup device 3 </ b> A is first performed is hereinafter set to 0 for convenience.
Then, control is performed so that the image sensor 3B is imaged at the next timing when the image sensor 3A is imaged (t22).
Similarly, the image sensor 3C is also photographed at the same interval (t23).
Next, the timing control unit 4 causes the imaging devices 3B, 3C, and 3B to capture images again in the order (t24, t25, and t26).
Here, one imaging unit is from the first imaging by the imaging device A to the next imaging by the imaging device A, which is hereinafter referred to as one set.
Then, as the second set, the next is photographing by the image sensor 3A again (t27).
Then, photographing is sequentially performed in the order of the imaging elements 3C, 3B, 3C, 3B, and 3C (t28, t29, t30, t31, and t32).

  Therefore, one set of photographing is performed, for example, in 2 ms (for two data transfer times of the image sensor 3A). In this case, the imaging timing in FIG. 6 is 1/6 ms.

In FIG. 6, the data transfer time of the image sensors 3B, 3C is 1/3 of 1/3 of 1 ms. Since one scale shown in FIG. 6 is 1/6 ms, data transfer of the image sensors 3B and 3C takes two scales.
In the third embodiment, as shown in FIG. 6, when the image sensor 3B is photographed three times and the image sensor 3C is photographed twice (t22 to t26) in one set, the image sensor 3B is photographed in one set. There are cases where the image sensor 3C is imaged twice (t28 to t32).
For example, when the image sensor 3B is imaged three times and the image sensor 3C is imaged twice in one set, and the image sensor 3B is imaged twice and the image sensor 3C is imaged three times in one set, for example, alternately. .

  That is, in the third embodiment, the shooting timing of the low-resolution image pickup devices (image pickup devices 3B and 3C) in one set is not necessarily fixed, and can be changed according to the situation. The timing control unit 4 controls each image sensor 3.

  In this way, by configuring so that the shooting timing of the low-resolution image sensor in one set can be changed according to the situation, the time required for data transfer of the high-resolution image and the data transfer of the low-resolution image Even if the time required for the change is changed, it becomes possible to cope.

This will be specifically described below.
For example, Tf is an interval between photographing timings, Th is a data transfer time for a high resolution image, Tl is a data transfer time for a high resolution image, and n is an imaging element for a low resolution image (n is a positive integer).
In this case, the number of shootings that should be performed by the low-resolution imaging device in one set is Th / Tf = k (+ remainder) (k is a positive integer).
That is, the timing control unit 4 may determine the shooting timings of the image sensors 3B and 3C so as to satisfy the number of shootings to be performed by the low resolution image sensor in one set calculated in this way.

For example, the shooting timing of the image sensor is set to 0, and the shooting timing of the low-resolution image sensor is sequentially set every time Tf elapses. What is necessary is just to set to an image pick-up element. In this way, the timing control unit 4 can automatically set the shooting timing based on the given conditions.
It should be noted that Tf × n> Tl is a necessary condition for setting the photographing timing as described above.

Although the case where there is one high-resolution image sensor has been described here, the present invention is not limited to this. That is, even when there are two or more high-resolution image sensors, the timing controller 4 can set the shooting timing of each image sensor in the same manner as described above.
In the third embodiment, the number of image sensors is assumed to be three. However, even when the number of image sensors changes, the timing control unit 4 uses the above-described method to capture the shooting timing of each image sensor. Can be set.

  As described above, in the first to third embodiments of the imaging system 100 of the present invention, the light splitting unit 2 divides the incident light into a plurality of parts, and each enters the plurality of imaging elements 3. Of the plurality of imaging elements, one captures a full pixel image (high resolution image), and the other captures an image (low resolution image) thinned out according to the number of lights divided by the light dividing unit. The timing control unit 4 controls the shooting timing and data transfer timing of each image sensor. The image processing unit 7 generates a high resolution image by calculating a movement vector between the low resolution images and adding it to the high resolution image. Therefore, high-speed and high-resolution image capturing can be performed.

In particular, since the low-resolution image is captured and transferred even during the transfer of the high-resolution image data, the data transfer time of the high-resolution image can be secured even with a small number of image sensors.
Further, by partially using the low-resolution image, it is possible to suppress the total required memory capacity, particularly the capacity of the memory (primary data storage unit 5 in the above-described embodiment) that requires high-speed transfer.

  Furthermore, in the second and third embodiments, light with a large amount of light is incident on the high-resolution image sensor and light with a small amount of light is incident on the low-resolution image sensor, and the data of a plurality of pixels is added to the low-resolution image sensor. Therefore, high-sensitivity shooting can be performed.

The present invention is not limited to the embodiment described above.
That is, when implementing the present invention, various modifications and alternatives may be made to the components of the above-described embodiments within the technical scope of the present invention or an equivalent scope thereof.
In the above-described embodiment, in the shooting timing control processing of the image sensor 3 by the timing controller 4, the high resolution image sensor (image sensor 3A) is first shot within one set. It is not limited. Imaging may be started from any imaging element.
In the above-described embodiment, a plurality of imaging elements having the same performance are used. However, the present invention is not limited to this. Even when the performance differs for each image sensor, the timing control unit 4 can control the photographing timing and the data transfer timing according to the performance.

Further, in the above-described embodiment, color photography is not mentioned, but the present invention can also be applied to color photography. For example, high-resolution imaging may be performed through any one of RGB filters, low-resolution imaging may be performed through each of the RGB filters, and a full-color image may be synthesized at the time of high-resolution image synthesis.
In the above-described embodiment, the movement vector is used to generate a high-resolution image based on the low-resolution image. However, the present invention is not limited to this. Any known method may be used.
In the above-described embodiment, the image signal generated by the image sensor 3 is temporarily stored in the primary data storage unit 5 and then transferred to the secondary data storage unit 6. It is not limited. That is, the data may be transferred directly to the secondary data storage unit 6 without buffering the temporary data in the primary data storage unit 5 by reducing the frame rate when generating the image signal.

  In the above-described embodiment, the imaging system 100 has the imaging unit 101 and the image processing system 102 independent, but the present invention is not limited to this. That is, an imaging apparatus that includes the image processing unit 7 and performs up to image processing may be used.

FIG. 1 is a block diagram illustrating an example of the configuration of an imaging system. FIG. 2 is a diagram illustrating an example of the configuration of the light splitting unit of the first embodiment. FIG. 3 is a time chart for explaining the imaging timing and transfer timing control processing of the image sensor by the timing control unit in the first embodiment. FIG. 4 is a diagram illustrating an example of the configuration of the light splitting unit according to the second embodiment. FIG. 5 is a time chart for explaining the imaging timing and transfer timing control processing of the image sensor by the timing control unit in the second embodiment. FIG. 6 is a time chart for explaining the imaging timing and transfer timing control processing of the image sensor by the timing control unit in the third embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Imaging system, 101 ... Imaging part, 102 ... Image processing system, 1 ... Optical system, 2 ... Light splitting part, 21-25 ... Prism, 3 ... Imaging element, 4 ... Timing control part, 5 ... Primary data storage , 6 ... Secondary data storage unit, 7 ... Image processing unit, 8 ... Storage device, 9 ... Monitor image processing unit, 10 ... Monitor

Claims (8)

A light splitting section for splitting the amount of incident light into a plurality of parts;
A plurality of image sensors that respectively enter the divided light, generate an image signal according to the amount of incident light, and transfer the generated image signal to a subsequent circuit;
A timing control unit for controlling the image signal generation timing of the image sensor and the transfer timing for transferring the generated image signal;
Have
At least one of the plurality of image sensors generates a higher resolution image than the other image sensors,
The timing control unit causes the image pickup operation of the other image pickup device to be executed a plurality of times during data transfer of the image pickup device that generates a high-resolution image, generates an image signal, and executes data transfer of the generated image signal. Imaging device. The imaging device according to claim 1, wherein the number of the plurality of imaging elements corresponds to the number of incident light divided by the light dividing unit. The timing control unit is configured to perform the plurality of imaging operations of the other imaging device during the data transfer of the imaging device that generates the high-resolution image, and when the data transfer following the imaging operation is completed, The imaging device according to claim 2, wherein an imaging device that generates a high-resolution image executes generation of a next high-resolution image. The imaging device according to claim 3, wherein the light dividing unit divides the incident light so that at least one light amount is different from the other light amount. The timing control unit includes: an imaging element that generates the high-resolution image; an imaging element that generates a low-resolution image generated by the other imaging element; and a time required for image signal transfer and an imaging operation between the imaging elements The imaging apparatus according to claim 4, wherein an image signal generation timing and a transfer timing for transferring the generated image signal are set according to the interval. Obtaining at least one high-resolution image signal, and a plurality of low-resolution image signals generated at equal intervals following the high-resolution image signal, in which the same subject as the high-resolution image signal is captured,
An image processing apparatus that complements the high resolution image signal based on the plurality of low resolution image signals and generates a plurality of high resolution image signals corresponding to the plurality of low resolution image signals. The image processing apparatus according to claim 6, wherein the interpolation is performed by calculating a movement vector between low resolution images based on the plurality of low resolution image signals and adding the calculated movement vector to the high resolution image signal. . A light splitting section for splitting incident light into a plurality of parts;
A plurality of imaging elements that respectively enter the divided light and generate an image signal according to the incident light;
An image processing unit that performs predetermined image processing on image signals transferred from the plurality of image sensors;
A timing control unit for controlling the image signal generation timing of the image sensor and the transfer timing for transferring the generated image signal;
Have
At least one of the plurality of image sensors generates a higher resolution image than the other image sensors,
The timing control unit causes the imaging operation of the other imaging device to be executed a plurality of times during data transfer of the imaging device that generates a high-resolution image, and causes the data transfer to be executed following the imaging operation,
The image processing unit complements the high resolution image signal based on the plurality of low resolution image signals as predetermined image processing, and generates a plurality of high resolution image signals corresponding to the plurality of low resolution image signals. An imaging system that performs processing.

Images