JP2018175871A - Imaging device and control program for imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To secure a sufficient frame rate for observation. An image pickup apparatus according to an embodiment includes an image sensor, a control unit, and an image generation unit. The image sensor includes a plurality of pixels arranged in a matrix that generate an electric signal by receiving light, and starts exposure sequentially at least one row from the first row to the last row of the plurality of pixels. This is a rolling shutter type image sensor that repeats the process of outputting the electric signal in order from the row at which the exposure is completed for each frame. The control unit sets the period based on the blanking period corresponding to the period from the end of the output of the electric signal from the last row to the start of the output of the electric signal from the first row. Only the plurality of pixels receive light. The image generation unit generates an image based on the electric signal. [Selection diagram] Fig. 1

Description

  Embodiments of the present invention relate to an imaging device and a control program of the imaging device.

  In imaging devices, for example, endoscopic devices, CCD (Charge Coupled Device) image sensors have been the mainstream in the past, but in recent years CMOS (with low cost, single power supply, low power consumption, etc.) has advantages Complementary Metal Oxide Semiconductor) image sensors have become mainstream. In a CMOS image sensor, a rolling shutter system is generally adopted in many cases.

International Publication No. 2014/125724 Patent No. 6019167 gazette

  The problem to be solved by the present invention is to provide an imaging device and a control program of the imaging device capable of securing a frame rate sufficient for observation.

  An imaging device according to an embodiment includes an image sensor, a control unit, and an image generation unit. The image sensor includes a plurality of pixels arranged in a matrix to generate an electric signal by light reception, and starts exposure sequentially at least every other row from the first row to the last row of the plurality of pixels. The rolling shutter type image sensor is a rolling shutter type image sensor that repeats the process of outputting the electric signal in order from the row in which the exposure is completed for each frame. The control unit performs a period based on a blanking period corresponding to a period from the end of the output of the electrical signal from the last row to the start of the output of the electrical signal from the first row. Only the plurality of pixels are received. The image generation unit generates an image based on the electrical signal.

FIG. 1 is a diagram illustrating a configuration example of an imaging system including an imaging device according to the first embodiment. FIG. 2 is a diagram for explaining an example of the operation of the imaging device according to the comparative example. FIG. 3 is a diagram for explaining an example of the imaging operation of the imaging device according to the first embodiment. FIG. 4A is a diagram for explaining an example of image processing performed by the image processing circuit. FIG. 4B is a diagram showing an example of a composite image. FIG. 5 is a flowchart showing the flow of control processing according to the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of the imaging device according to the third embodiment. FIG. 7 is a view for explaining an example of an image according to the fourth embodiment. FIG. 8 is a diagram for explaining an example of mode switching performed by the imaging device according to the fifth embodiment. FIG. 9 is a diagram for explaining an example of the operation of the imaging device according to the sixth embodiment.

  Hereinafter, an imaging device and a control program of the imaging device according to each embodiment will be described with reference to the drawings. The embodiment is not limited to the following contents. In addition, the contents described in one embodiment or modification can be applied to other embodiments or modifications in the same manner in principle.

First Embodiment
FIG. 1 is a diagram showing a configuration example of an imaging system 1 provided with an imaging device 10 according to the first embodiment. As shown in FIG. 1, the imaging system 1 according to the first embodiment includes an imaging device 10, a light source device 30, and an optical fiber 31.

  The imaging device 10 is, for example, a device that is used as a medical rigid endoscope and images the inside of the subject 100. The imaging device 10 includes a scope 11, a camera head 12, a camera cable 13, and a CCU (Camera Control Unit) 14.

  The scope 11 is inserted into the body of the subject 100 when imaging is performed. An objective lens 11 a is provided at the tip of the scope 11. The scope 11 has a hardness that does not bend.

  The camera head 12 includes an excitation light cut filter 12a, a spectral prism 12b, three image sensors 12c to 12e (12c, 12d, 12e), and an image sensor control circuit 12f.

  The excitation light cut filter 12 a is an optical element that is disposed to face the imaging region of the plurality of pixels, and causes the plurality of pixels to receive light other than the excitation light emitted from the light source device 30. The excitation light cut filter 12a is disposed between the scope 11 and the incident surface 12b_1 of the spectral prism 12b, and is a filter that cuts excitation light out of light incident from the objective lens 11a and transmits light other than the excitation light. It is. Therefore, light other than the excitation light among reflected light (return light) of the light irradiated to the body tissue of the subject 100 by the light source device 30 is incident on the incident surface 12b_1 of the spectral prism 12b. In the following description, the reflected light of white light that is incident on the spectral prism 12 b and that is incident on the image sensor 12 c may be simply described as white light.

  The splitting prism 12b splits the incident light into red (R) light, green (G) light and blue (B) light. Then, the spectral prism 12 b focuses blue light on the imaging surface of the image sensor 12 c. Further, the spectral prism 12b focuses green light on the imaging surface of the image sensor 12d. The spectral prism 12b focuses red light on the imaging surface of the image sensor 12e.

  Each of the image sensors 12c to 12e is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensors 12c to 12e are arranged such that the imaging surfaces of the image sensors 12c to 12e substantially coincide with the image forming surface of the spectral prism 12b. A plurality of pixels of each of the three (plurality) image sensors 12c to 12e output an image signal by receiving light of the corresponding type.

  Each of the image sensors 12c to 12e includes a plurality of pixels (imaging elements). The plurality of pixels are arranged in a matrix on the imaging plane. Each pixel generates a video signal (electrical signal) by receiving light and driving the image sensor control circuit 12f under drive control, and outputs the generated video signal. For example, each pixel of the image sensor 12c outputs a B signal (image signal of B) by receiving blue light. In addition, each pixel of the image sensor 12d outputs a G signal (G video signal) by receiving green light. In addition, each pixel of the image sensor 12e outputs an R signal (R video signal) by receiving red light. For example, the camera head 12 including the image sensors 12 c to 12 e outputs RGB signals to the CCU 14 via the camera cable 13. The image sensors 12c to 12e output video signals of analog format.

  Here, the imaging apparatus 10 according to the first embodiment is used, for example, when performing a surgical operation by an ICG (indocyanine green) fluorescence imaging method on the subject 100. In the first embodiment, the subject 100 is administered ICG. The ICG is excited by excitation light emitted from the IR laser 30d, and emits near infrared fluorescence (hereinafter referred to as fluorescence) of about 800 to 850 nm. This fluorescence passes through the excitation light cut filter 12a and is imaged on the imaging surface of the image sensor 12e by the spectral prism 12b. That is, the image sensor 12e outputs an R signal by receiving fluorescence based on excitation light. Hereinafter, a case where the image sensor 12e receives fluorescence and the imaging device 10 performs various processes will be described. However, the imaging device 10 does not provide the excitation light cut filter 12a, and the image sensor reflects the reflected light of the excitation light The same process may be performed by receiving light at 12e.

  Each of the image sensors 12c to 12e sequentially starts exposure for at least one row from the first row to the last row of the plurality of pixels, and outputs a video signal in order from the row where the exposure is completed, It is a rolling shutter type image sensor that repeats every one frame (image). Here, exposure means, for example, that the pixel stores charge.

  Here, when the camera head 12 includes the plurality of image sensors 12 c to 12 e as described above, the resolution of the image may be increased by using a method of shifting the pixels by half a pixel.

  The camera head 12 may be provided with a Bayer filter and one image sensor instead of the spectral prism 12 b and the three image sensors 12 c to 12 e. In this case, a Bayer filter is disposed on the imaging surface side of the image sensor such that any one of the red filter, the green filter, and the blue filter faces each pixel. Thus, one color filter corresponds to one pixel. The image sensor outputs, for each pixel, an image signal of the color of the filter corresponding to the pixel. The image processing circuit 14c described later executes, for each pixel, an estimation process of estimating each of the remaining two color video signals that can not be obtained directly based on the video signals output from pixels around the pixel. Ru. As a result, the imaging device 10 can obtain RGB signals as video signals for each pixel.

  The image sensor control circuit 12 f drives and controls the image sensors 12 c to 12 e based on control signals output from the control circuit 14 a described later and various synchronization signals output from the timing signal generation circuit 14 f described later. For example, based on the control signal and various synchronization signals, the image sensor control circuit 12f appropriately applies gain (analog gain) to the video signal in analog format output from the image sensors 12c to 12e (to amplify the video signal). Control the image sensors 12 c to 12 e so as to output the gain-applied video signal to the CCU 14. Alternatively, when the image sensors 12c to 12e incorporate an AD converter (not shown), the image sensor control circuit 12f appropriately applies a gain (digital gain) to the digital image signal output from the image sensors 12c to 12e to obtain a gain. The image sensors 12 c to 12 e are controlled so as to output the video signal to which the image is applied to the CCU 14.

  The camera cable 13 is a cable that accommodates signal lines for transmitting and receiving video signals, control signals, and synchronization signals between the camera head 12 and the CCU 14.

  The CCU 14 performs various types of image processing on the video signal output from the camera head 12 to generate image data indicating an image to be displayed on the display 101, and outputs the image data to the display 101 connected to the CCU 14. The video signal subjected to various image processing is image data indicating an image to be displayed on the display 101.

  The CCU 14 includes a control circuit 14a, a storage control circuit 14b, an image processing circuit 14c, an image combining circuit 14d, an output circuit 14e, a timing signal generation circuit 14f, and a storage circuit 14g. When the image sensors 12c to 12e do not include an AD converter, the CCU 14 also includes an AD (Analog to Digital) converter (not shown). The AD converter converts, for example, an analog video signal output from the camera head 12 into a digital video signal.

  The control circuit 14 a controls various components of the imaging device 10. For example, the control circuit 14a outputs control signals to the image sensor control circuit 12f, the storage control circuit 14b, the image processing circuit 14c, the image synthesis circuit 14d, the output circuit 14e, and the timing signal generation circuit 14f. Control each circuit. The control circuit 14 a reads the control program of the imaging device 10 stored in the storage circuit 14 g and executes the read control program to execute control processing for controlling various components of the imaging device 10. Alternatively, the control circuit 14a internally includes a storage circuit (not shown) and executes a control program stored in the storage circuit. The control circuit 14a is realized by, for example, a processor such as an MPU (Micro-Processing Unit).

  The storage control circuit 14b stores the video signal output from the camera head 12 in the storage circuit 14g based on the control signal output from the control circuit 14a and various synchronization signals output from the timing signal generation circuit 14f. Control. The storage control circuit 14b also reads the video signal stored in the storage circuit 14g row by row based on the control signal and the synchronization signal. Then, the storage control circuit 14 b outputs the read video signal for one line to the image processing circuit 14 c.

  The image processing circuit 14c performs various processing on the video signal output from the storage control circuit 14b based on the control signal output from the control circuit 14a and various synchronization signals output from the timing signal generation circuit 14f. Perform image processing. Thereby, the image processing circuit 14 c generates image data indicating an image to be displayed on the display 101. That is, the image processing circuit 14c generates an image based on the video signal. For example, the image processing circuit 14 c adjusts the brightness of the image by applying a gain (digital gain) to the video signal output from the storage control circuit 14 b. In addition, the image processing circuit 14 c may perform noise reduction processing for reducing noise, contour emphasis processing for emphasizing a contour, or the like on the video signal output from the storage control circuit 14 b. Then, the image processing circuit 14c outputs a video signal (image data representing an image to be displayed on the display 101) subjected to various types of image processing to the image combining circuit 14d.

  The image combining circuit 14d combines the video signal output from the image processing circuit 14c on the basis of the control signal output from the control circuit 14a and various synchronization signals output from the timing signal generation circuit 14f. Generate image data. Then, the image combining circuit 14 d outputs the combined image data to the display 101.

  For example, the storage control circuit 14 b, the image processing circuit 14 c, and the image combining circuit 14 d are realized by one processor such as a DSP (Digital Signal Processor). Further, for example, the storage control circuit 14b, the image processing circuit 14c, the image synthesis circuit 14d, and the timing signal generation circuit 14f are realized by one FPGA (Field Programmable Gate Array). The control circuit 14a, the storage control circuit 14b, the image processing circuit 14c, and the image combining circuit 14d may be realized by one processing circuit. The processing circuit is realized by, for example, a processor.

  The output circuit 14 e outputs the combined image data output from the image combining circuit 14 d to the display 101. Thereby, the display 101 displays the composite image indicated by the composite image data. The composite image is an example of an image. The output circuit 14e is realized by, for example, an HDMI (registered trademark) (High-Definition Multimedia Interface) driver IC (Integrated Circuit), an SDI (Serial Digital Interface) driver IC, or the like.

  The timing signal generation circuit 14f unifies various timings such as emission timing of light from the light source device 30, exposure timing of the image sensors 12c to 12e, output timing of video signals, and control timing of the storage circuit 14g by the storage control circuit 14b. to manage. The control circuit 14a and the timing signal generation circuit 14f are examples of a control unit.

  The timing signal generation circuit 14 f receives various synchronization signals such as a horizontal synchronization signal and a vertical synchronization signal based on a clock signal generated by an oscillation circuit (not shown) and other synchronization signals for synchronizing the entire imaging device 10. Generate Then, the timing signal generation circuit 14f outputs the generated various synchronization signals to the image sensor control circuit 12f, the control circuit 14a, the storage control circuit 14b, the image processing circuit 14c, the image synthesis circuit 14d, and the output circuit 14e. .

  The timing signal generation circuit 14f generates a light source control signal based on the clock signal and the control signal output from the control circuit 14a. The light source control signal is a control signal for controlling the light emitted from the light source device 30 and synchronizing the entire imaging system 1. Then, the timing signal generation circuit 14 f outputs the generated light source control signal to the light source device 30.

  For example, the waveform of the light source control signal is a rectangular wave, and the light source control signal has two levels (states) of high level and low level. For example, while the light source control signal is high level, white light is emitted from the white LED 30b or excitation light is emitted from the IR laser 30d, and the white LED 30b is turned off during low level; Control signal for stopping the emission of the excitation light.

  The storage circuit 14 g is realized by, for example, a random access memory (RAM), a semiconductor memory device such as a flash memory, a hard disk, an optical disk, or the like. The storage circuit 14g stores various programs. For example, the storage circuit 14g stores a control program executed by the control circuit 14a. Further, a video signal is temporarily stored in the storage circuit 14 g by the storage control circuit 14 b.

  The light source device 30 emits white light or excitation light based on the light source control signal. The light source device 30 includes a drive circuit 30a, a white LED (Light Emitting Diode) 30b, a drive circuit 30c, and an IR laser 30d.

  The drive circuit 30a performs drive control to drive and light the white LED 30b based on the light source control signal output from the timing signal generation circuit 14f. The white LED 30b emits white light by drive control by the drive circuit 30a. White light is, for example, visible light. White light is an example of light. White light is also an example of second light.

  The drive circuit 30c drives the IR laser 30d based on the light source control signal output from the timing signal generation circuit 14f, and performs drive control to emit excitation light from the IR laser 30d. The IR laser 30d emits excitation light by drive control by the drive circuit 30c. As described above, the excitation light is cut by the excitation light cut filter 12a. In addition, the ICG is excited by the excitation light and the fluorescence (fluorescence based on the excitation light) emitted from the ICG passes through the excitation light cut filter 12a and is received by the image sensor 12e. Excitation light is an example of light. The excitation light is an example of the first light.

  The optical fiber 31 guides the white light and the excitation light from the light source device 30 to the tip of the scope 11 and causes the tip of the scope 11 to emit the light.

  The configuration example of the imaging device 10 of the imaging system 1 according to the first embodiment has been described above. Here, an imaging device according to a comparative example will be described. An imaging device according to the comparative example includes a rolling shutter type image sensor.

  FIG. 2 is a diagram for explaining an example of the operation of the imaging device according to the comparative example. FIG. 2 shows an example of the relationship between the exposure timing of each row of a plurality of pixels included in the image sensor of the imaging device according to the comparative example and the output timing of the video signal output from the image sensor. As shown in FIG. 2, in the imaging device according to the comparative example, in imaging of the first frame, exposure is sequentially started for each row from the first row of the plurality of pixels to the last row. In the imaging device according to the comparative example, the exposure period is 1/60 [s]. Here, the exposure period means, for example, a period from when the pixel starts accumulation of charge until it ends.

  Then, as shown in FIG. 2, in the imaging device according to the comparative example, video signals are output in order from the row in which the exposure is completed. That is, video signals are sequentially output from each row from the first row to the last row. Here, in the imaging device according to the comparative example, a period (readout period) in which a video signal of one frame is output from the image sensor is 1/60 [s] which is the same as the exposure period. Then, as shown in FIG. 2, the same processing is performed for the second and subsequent frames.

  In the imaging device according to the comparative example, in the image sensor, the light reception period (light reception period) is the same as the exposure period in all the rows of the plurality of pixels. Therefore, from the first row to the last row, the light reception periods of the respective rows are sequentially shifted in the time axis direction. As described above, in the imaging device according to the comparative example, since the light reception period is different for each row, distortion may occur in the image. In this case, for a user such as a doctor who observes an image, there is a problem that the image quality is not sufficient for observation.

  Therefore, the imaging apparatus 10 according to the first embodiment performs the following operation so as to ensure the image quality and the frame rate sufficient for the user to observe with the above-described configuration.

  In the present embodiment, the exposure period of the image sensors 12 c to 12 e is half of the period in which the image signal of one frame is output from the imaging device 10 to the display 101. Then, the control circuit 14a performs control to receive the plurality of pixels of each of the image sensors 12c to 12e only during a blanking period or a period centered on the blanking period. For example, the control circuit 14a outputs a control signal for causing the image sensor control circuit 12f to perform such control to the image sensor control circuit 12f.

  Here, the control circuit 14a performs control to receive light of a plurality of pixels in a period equal to or less than the reading period. The readout period indicates, for example, a period in which a video signal of one frame is output from the image sensors 12 c to 12 e. Further, the reading period refers to, for example, a period from the start of the output of the video signal from the first row in one frame to the end of the output of the video signal from the last row.

  The blanking period is, for example, after the output of the video signal from the last row of the image sensors 12c to 12e is finished in the imaging of the n (n is a positive integer) frame in the present embodiment, for example. , And in the imaging of the (n + 1) th frame, the period corresponding to the period until the output of the video signal from the first row of the image sensors 12c to 12e is started.

  In the present embodiment, the frame rate of the video signal (image) output from the imaging device 10 to the display 101 is A [fps (frame per second)]. In this case, an image sensor capable of setting the readout period to 1 / (M · A) [s] is used as the image sensors 12 c to 12 e of the imaging device 10. That is, an image sensor capable of outputting a video signal from each row is used as the image sensors 12 c to 12 e for each 1 / (M · k · A) [s]. However, “M” is a number larger than 1 and “k” is the number of rows of pixels of each of the image sensors 12 c to 12 e. Hereinafter, the case of M = 2 will be described as an example, but M may be a number different from 2 and a number larger than 1.

  Then, the control circuit 14a controls the image sensor 12c to 12e to output a control signal for outputting a video signal of one frame in the reading period 1 / (2A) [s] shorter than the exposure period 1 / A [s]. It outputs to the control circuit 12f.

  In the following, the case of A = 60 will be described as an example. That is, in the case where the exposure period and the period in which the imaging device 10 outputs a video signal of one frame to the display 101 are the same 1/60 [s] and the readout period is 1/120 [s], This will be described below.

  FIG. 3 is a diagram for explaining an example of the imaging operation of the imaging device 10 according to the first embodiment. In FIG. 3, emission timings of white light and excitation light emitted from the light source device 30 according to the first embodiment, and exposure timings of each row of a plurality of pixels included in the image sensors 12c to 12e of the imaging device 10; An example of the relationship between the output timing of the video signal output from the image sensors 12c to 12e and the output timing of the video signal output from the output circuit 14e is shown. In FIG. 3, the horizontal axis indicates time.

  In FIG. 3, since the illustration is simplified, the time (timing) when the output of the video signal from the last row of the image sensors 12c to 12e is finished in the imaging of the nth frame, and the imaging of the (n + 1) th frame Two different times of the time to start the output of the video signal from the first row of the image sensors 12c to 12e are indicated by the same "Tp (p = n + 1)". Actually, as described above, in the imaging of the n th frame, after the output of the video signal from the last row of the image sensors 12 c to 12 e is finished, in the imaging of the (n + 1) th frame, the image sensor 12 c A blanking period exists as a period until the output of the video signal from the first row of 12e is started.

  In FIG. 3, (n) shows the exposure timing in the imaging of the nth frame. However, n is a positive integer as described above. In the first embodiment, the light amount of the fluorescence generated by the irradiation of the excitation light is smaller than the light amount of the reflection of the white light emitted from the white LED 30b. For example, the amount of fluorescence is so small as to be negligible compared to the amount of reflection of white light. Therefore, it is considered that the influence of the fluorescence image obtained from the fluorescence based on the excitation light on the color image obtained from white light is extremely small. For this reason, in the first embodiment, excitation light is emitted constantly (continuously) from the IR laser 30d. Therefore, the light amount of the fluorescence light received by the image sensor 12e is larger than that in the case where the excitation light is not always emitted from the IR laser 30d. For this reason, the light reception sensitivity of fluorescence can be made high, and the fall of the brightness of the image by lack of light quantity can be suppressed.

  In FIG. 3, when n is an odd number, IRn indicates an R signal output from the image sensor 12e in the imaging of the nth frame. That is, when n is an odd number, IRn indicates an R signal output from the image sensor 12e that has received fluorescence based on excitation light.

  In FIG. 3, when n is an even number, Gn indicates a G signal output from the image sensor 12 d in the imaging of the nth frame. In addition, when n is an even number, Bn indicates a B signal output from the image sensor 12c in the imaging of the nth frame. Further, when n is an even number, “Rn + IRn” indicates an R signal output from the image sensor 12e in the imaging of the nth frame. That is, "Rn + IRn" indicates an R signal output from the image sensor 12e that simultaneously receives white light and fluorescence based on excitation light in imaging of the nth frame. “Wn + IRn” indicates RGB signals output from the image sensors 12 c to 12 e in the imaging of the nth frame. That is, “Wn + IRn” is a composite signal of Gn, Bn and Rn.

  In the example of FIG. 3, the control circuit 14a sends a control signal for outputting the first light source control signal to the timing signal generation circuit 14f before starting imaging of the first frame or starting imaging of the first frame. Output. The first light source control signal is a control signal for continuously emitting excitation light from the IR laser 30d.

  The timing signal generation circuit 14f outputs a first light source control signal for continuously emitting excitation light to the drive circuit 30c based on the above-described control signal output from the control circuit 14a. The drive circuit 30c causes the IR laser 30d to continuously emit excitation light based on the first light source control signal output from the timing signal generation circuit 14f.

  As shown in FIG. 3, in the imaging of the first frame, exposure is sequentially started for each row from the first row of the plurality of pixels to the last row. In imaging of the first frame, fluorescence based on excitation light is received by the image sensor 12e, and the image sensor 12e outputs an R signal (IR1) in a readout period 1/120 [s] from time T1 to time T2 . That is, the image sensor 12e outputs a video signal from each row every 1 / (120 k) [s]. The time T1 is a time at which the output of the video signal from the first row starts in the first frame. The time T2 is a time when the output of the video signal from the last row is finished in the first frame.

  A specific control method will be described. The control circuit 14a causes the image sensor control circuit 12f to output a control signal for causing the image sensor 12e to output a video signal of the first frame in a reading period 1/120 [s] from time T1. Output. That is, the control circuit 14a causes the image sensor control circuit 12f to start controlling the image sensor 12e to start outputting the video signal from each row sequentially every 1 / (120 k) [s] from time T1. Output.

  The image sensor control circuit 12 f drives and controls the image sensor 12 e based on the control signal. As a result, the image sensor 12e outputs video signals from all the rows (k rows) in a reading period 1/120 [s] from time T1 to time T2.

  Then, the storage control circuit 14 b temporarily stores the video signal output from each row in the storage circuit 14 g each time the video signal output from each row of the image sensor 12 e is input in imaging of the first frame. . Here, from time T1 to time T2, the storage control circuit 14b causes the storage circuit 14g to store the video signal output from each row from the first row to the last row in order. Thus, the video signal stored in the imaging of the first frame is stored in the storage circuit 14g at least until time T5.

  Next, as shown in FIG. 3, in the imaging of the second frame, exposure is sequentially started for each row from the first row to the last row of the plurality of pixels of each of the image sensors 12c to 12e.

  Here, in the imaging of the first frame, the period from the time T2 when the output of the video signal from the last row ends to the time when the output of the video signal from the first row starts in the imaging of the second frame is , Corresponds to the blanking period 40, as shown in FIG. The blanking period 40 is shown as a single dotted line in FIG. 3 because it is a relatively short period as compared to the exposure period etc. However, actually, a period having a fixed width in the time axis direction It is.

  In the present embodiment, as shown in FIG. 3, white light is emitted from the white LED 30 b only in the blanking period 40 or the period 41 centered on the blanking period 40 in imaging of the second frame. For example, when white light is emitted only in the period 41 centered on the blanking period 40, in the imaging of the second frame, the first row among all the plurality of pixels of the image sensors 12c to 12e. White light is emitted to the at least one row 42 and the at least one row 43 on the last row side only for a time shorter than the period 41.

  In the blanking period 40, white light may be emitted from the white LED 30b. In this case, within the blanking period 40, white light may be emitted at any timing without being limited to a period centered on the center of the blanking period 40. Also, the period 41 is, for example, a period longer than the blanking period 40, includes the blanking period 40, and includes a period before and a period after the blanking period 40. The blanking period, the period within the blanking period, and the period centered on the blanking period are examples of periods based on the blanking period.

  On the other hand, as shown in FIG. 3, in one or more rows 44 excluding the rows 42 and 43 among all the rows, the light receiving periods for receiving white light substantially coincide. That is, the light reception period is different between the row 44, the row 42 and the row 43. Therefore, the imaging device 10 masks the image obtained based on the video signals output from the rows 42 and 43, and sets the range of the image obtained based on the video signal output from the row 44 as the effective image range. Set As a result, the display 101 displays an image in which a portion in which unevenness in brightness may occur is masked. Therefore, the occurrence of image distortion can be suppressed. Therefore, according to the imaging device 10 according to the first embodiment, it is possible for the user such as a doctor who observes an image to ensure sufficient image quality for observation.

  Further, as shown in FIG. 3, in the imaging of the third frame, at least one row on the first row side among all the rows of the plurality of pixels receives only the fluorescence based on the excitation light during the exposure period. Instead, they also receive white light. In addition, in the first frame imaging, at least one row on the last row side among all the rows of the plurality of pixels receives white light instead of receiving only fluorescence based on excitation light during the exposure period It also receives light. Therefore, since the image signal output from these rows that also receive white light in addition to fluorescence also includes the white light component, a fluorescent image having good image quality consisting only of the fluorescent component is generated. It is difficult.

  Therefore, the imaging device 10 according to the first embodiment similarly masks the image obtained based on the video signal output from the row 42 and the row 43 also in the imaging of the first frame and the imaging of the third frame. The range of the image obtained based on the video signal output from the line 44 is set as the effective image range. As a result, the display 101 displays an image in which a portion that may not have good image quality is masked. Therefore, according to the imaging device 10 according to the first embodiment, it is possible to secure the image quality sufficient for the user such as the doctor who observes the image from this point as well.

  Here, an example of the reason why the period 41 is a period centered on the blanking period will be described. If the period 41 is a period not centered on the blanking period, the size of the masked area at one end and the masked area at the other end in the image masked at both ends. The size is different, and there is a possibility that the user who observes the image may feel discomfort.

  However, since the period 41 is a period centered on the blanking period, the sizes of the masked area on one end side and the masked area on the other end substantially match, which makes the user uncomfortable. It is possible to suppress giving.

  Then, the image sensors 12c to 12e output video signals (RGB signals) from all the rows (k rows) in a reading period 1/120 [s] from time T2 to time T3.

  A specific control method will be described. The control circuit 14a outputs a control signal for causing the image sensors 12c to 12e to output an image signal to the image sensor control circuit 12f in a reading period 1/120 [s] from time T2. That is, the control circuit 14a outputs, to the image sensor control circuit 12f, a control signal for causing the image sensors 12c to 12e to start outputting the video signal from each row every time 1 / (120 k) [s] from time T2. Do.

  The image sensor control circuit 12 f drives and controls the image sensors 12 c to 12 e based on the control signal. As a result, the image sensor 12c outputs the video signal (B signal) from all the rows (k rows) in the reading period 1/120 [s] from time T2 to time T3. Further, the image sensor 12d outputs video signals (G signals) from all the rows (k rows) in a reading period 1/120 [s] from time T2 to time T3. Further, the image sensor 12e outputs the video signal (R signal (R2 + IR2)) from all the rows (k rows) in the second frame imaging in the readout period 1/120 [s] from time T2 to time T3. Do.

  Then, from time T2 to time T3, the storage control circuit 14b temporarily stores the video signal output from each row in the storage circuit 14g every time the video signal output from each row of the image sensor 12c is input. . Also, from time T2 to time T3, the storage control circuit 14b temporarily stores the video signal output from each row in the storage circuit 14g every time the video signal output from each row of the image sensor 12d is input. . Also, from time T2 to time T3, the storage control circuit 14b temporarily stores the video signal output from each row in the storage circuit 14g every time the video signal output from each row of the image sensor 12e is input. . Here, in the imaging of the second frame, from time T2 to time T3, the video signal output from each row is stored in the storage circuit 14g sequentially from the first row to the last row. Thus, the video signal stored in the imaging of the second frame is stored in the storage circuit 14g at least until time T5.

  Then, as shown in FIG. 3, in the imaging of the third frame, exposure is sequentially started for each row from the first row of the plurality of pixels to the last row. In imaging of the third frame, fluorescence based on excitation light is received by the image sensor 12e, and the image sensor 12e outputs an R signal (IR3) in a readout period 1/120 [s] from time T3 to time T4. . That is, each row of the image sensor 12e outputs a video signal every 1 / (120 k) [s]. The time T3 is a time at which the output of the video signal from the first row starts in the imaging of the third frame. A time T4 is a time when the output of the video signal from the last row ends in imaging of the third frame.

  A specific control method will be described. The control circuit 14a outputs, to the image sensor control circuit 12f, a control signal for causing the image sensor 12e to output a video signal in a reading period 1/120 [s] from time T3. That is, the control circuit 14a outputs, to the image sensor control circuit 12f, a control signal for causing the image sensor 12e to start outputting a video signal from each row every 1 / (120 k) [s] from time T3.

  The image sensor control circuit 12 f drives and controls the image sensor 12 e based on the control signal. As a result, the image sensor 12e outputs video signals from all the rows (k rows) in a readout period 1/120 [s] from time T3 to time T4.

  Then, the storage control circuit 14 b temporarily stores the video signal output from each row in the storage circuit 14 g each time the video signal output from each row of the image sensor 12 e is input in imaging of the third frame. . Here, in the imaging of the third frame, from time T3 to time T4, the video signal output from each row is stored in the storage circuit 14g sequentially from the first row to the last row. Thus, the video signal stored in the imaging of the third frame is stored in the storage circuit 14g at least until time T7.

  Here, the process which "(W2 + IR2) + (IR1 + IR3) / 2" of FIG. 3 shows is demonstrated. At the stage of time T3, the memory circuit 14g stores the video signal (IR1) of the first frame and the video signal (W2 + IR2) of the second frame. Then, from time T3, as described above, the video signal (IR3) of the third frame is started to be stored in the storage circuit 14g.

  Then, in the present embodiment, the imaging device 10 combines the video signal (IR1) of the first frame, the video signal (W2 + IR2) of the second frame, and the video signal (IR3) of the third frame, and displays it. Output to

  Here, simply, each time the imaging device 10 outputs the video signal (IR3) from each row of the image sensor 12e in the imaging of the third frame, the video signal (IR3) output from each row and from each row A case will be described where the output video signal (IR1) and the video signal (W2 + IR2) output from each row of the image sensors 12c to 12e are combined and the combined image is output to the display 101. In this case, a composite image of the video signals output from all the rows is output to the display 101 at 1/120 [s].

  Then, after the video signal of the last row is output to the display 101 in the imaging of a certain frame, 1/120 [s] until the video signal of the first row is output to the display 101 in the imaging of the next frame. In the meantime, there is a period in which the video signal is not output to the display 101. The period in which the video signal is not output is also a period in which the image displayed on the display 101 is not updated. Therefore, a period of 1/120 [s] in which the image is updated and a period of 1/120 [s] in which the image is not updated are alternately arranged. In such a case, the display 101 may not be able to handle such an input because the image update frequency is not constant.

  Therefore, in the present embodiment, the frame rate of the video signal of one frame output from the output circuit 14e to the display 101 is set to 60 [fps] so that the image update frequency becomes constant. That is, the period from the output of the video signal of the last row in the imaging of a certain frame to the output of the video signal of the first row in the imaging of the next frame is shortened.

  Therefore, the storage control circuit 14b outputs the video signal (IR1) output from each row in the imaging of the first frame every 1 / (60 k) [s] from time T3 and outputs from each row in the imaging of the second frame The read video signal (W2 + IR2) and the video signal (IR3) output from each row in the imaging of the third frame are sequentially read out from the storage circuit 14g. For example, the storage control circuit 14b outputs the video signal (IR1) and the video signal (IR3) output from the mth row of the image sensor 12e and the video signal (image signal (IR) output from the mth row Read W2 + IR2). Here, m is a positive integer. The storage control circuit 14b then outputs the video signal (IR1) and the video signal (IR3) output from the mth row of the image sensor 12e and the video signal (i.e., the video signal output from the mth row It outputs W2 + IR2) to the image processing circuit 14c.

  The image processing circuit 14c then outputs the video signal (IR1) and the video signal (IR3) output from the mth row of the image sensor 12e, and the video signal output from the mth row of the image sensors 12c to 12e. Various image processing is performed on (W2 + IR2), and the result is output to the image combining circuit 14d.

  Hereinafter, an example of various image processing performed by the image processing circuit 14c will be described. For example, the image processing circuit 14c applies digital gain to the video signal (IR1) output from the m-th row of the image sensor 12e in the imaging of the first frame. The image processing circuit 14c amplifies the video signal (IR1) output from the mth row of the image sensor 12e by, for example, 30 times by the digital gain and the above-described analog gain.

  Similarly, the image processing circuit 14c applies digital gain to the video signal (IR3) output from the m-th row of the image sensor 12e in the imaging of the third frame. The image processing circuit 14c amplifies the video signal (IR3) output from the m-th row of the image sensor 12e to, for example, 30 times in combination with the digital gain and the above-described analog gain.

  Note that the image processing circuit 14c does not apply digital gain and analog gain to the m-th video signal (W2 + IR2) output from the image sensors 12c to 12e in the imaging of the second frame.

  In addition, the image processing circuit 14 c according to the present embodiment masks an image obtained based on the video signal output from the above-described line 42 and line 43. Further, the image processing circuit 14 c sets the range of the image obtained based on the video signal output from the row 44 described above as the effective image range.

  FIG. 4A is a diagram for explaining an example of image processing performed by the image processing circuit 14 c. As shown in the example of FIG. 4A, for example, the image processing circuit 14c replaces the image obtained based on the video signal (W2 + IR2) output from the above-described row 42 and row 43 with the black image 51, and A color image 50 is generated in which the range of the image obtained based on the video signal (W2 + IR2) output from the row 44 is set as the effective image range 52.

  The color image 50 illustrated in the example of FIG. 4A is a color when light is received from a plurality of pixels of the image sensors 12c to 12e only in a period in which 1/1200 [s] is added to the blanking period with the blanking period as the center. An example of an image is shown. In the color image 50, an area of 10% of the entire color image 50 is masked.

  Similarly, the image processing circuit 14c replaces the image obtained based on the video signal (IR1) output from the above-described rows 42 and 43 with a black image, and the video signal output from the above-described row 44 A fluorescence image is generated in which the range of the image obtained based on (IR1) is set as the effective image range. Further, the image processing circuit 14c replaces the image obtained based on the video signal (IR3) output from the above-described lines 42 and 43 with a black image, and outputs the video signal (from the above-described line 44) A fluorescence image is generated in which the range of the image obtained based on IR3) is set as the effective image range.

  The image synthesis circuit 14d combines the video signal (IR1) output from each row of the image sensor 12e in the imaging of the first frame and the video signal (IR3) output from each row of the image sensor 12e in the imaging of the third frame. Synthesize to generate a composite image. For example, the image combining circuit 14d combines the video signal (IR1) and the video signal (IR3) to generate a combined image ((IR1 + IR3) / 2).

  Then, the image synthesis circuit 14d extracts, for example, a portion where the luminance is equal to or higher than the threshold value in the synthesized image ((IR1 + IR3) / 2). Then, the image combining circuit 14d generates a marker having the same position and range as the extracted part and to which a predetermined color (for example, high saturation green color) is added. Then, in the imaging of the second frame, the image combining circuit 14 d superimposes the generated marker on the video signal (W2 + IR2) output from each row to generate a combined image (W2 + IR2 + (IR1 + IR3) / 2). Then, the image combining circuit 14d outputs the generated combined image (W2 + IR2 + (IR1 + IR3) / 2) to the output circuit 14e. The image combining circuit 14d generates a combined image using the color image and a plurality of (two) fluorescence images generated before and after the color image.

  For example, the color tone of a color image obtained by imaging the inside of the subject 100 is a reddish tint as a whole. Therefore, for example, even if the marker is given a red color, the marker is not noticeable. Therefore, the marker is made to stand out by providing the marker with green which is complementary to red.

  The image combining circuit 14 d may extract a contour component from a fluorescence image obtained by fluorescence instead of a marker, and may superimpose the extracted contour component on a color image to generate a composite image.

  The image synthesis circuit 14d performs such processing on the video signals output from all the rows. That is, the image combining circuit 14d outputs the combined image of the video signals output from all the rows to the output circuit 14e in a period 1/60 [s] from time T3 to time T5. Thus, as indicated by “W2 + IR2 + (IR1 + IR3) / 2” in FIG. 3, the output circuit 14e generates a composite image of video signals of all the rows in a period 1/60 [s] from time T3 to time T5. Are output to the display 101. For this reason, the composite image displayed on the display 101 is updated every 1/60 [s].

  The imaging operation similar to the imaging operation described in the imaging of the first frame, the imaging of the second frame, and the imaging of the third frame is repeatedly performed for the fourth and subsequent frames. Therefore, the control circuit 14 a continuously emits excitation light from the light source device 30 and generates two frames from the light source device 30 so as to synchronize white light different in type from the excitation light with the blanking period 40 or the period 41. To emit at a rate of once. The image processing circuit 14c generates a fluorescence image based on video signals generated in a plurality of pixels of the image sensor 12e by receiving fluorescence based on excitation light. In addition, the video signal generated at the plurality of pixels of the image sensor 12e by the light reception of the fluorescence is an example of the first electric signal. In addition, the fluorescence image is an example of the first image.

  Further, the image processing circuit 14 c generates a color image based on video signals generated in a plurality of pixels of the image sensors 12 c to 12 e by simultaneous reception of fluorescence and white light. In addition, the video signal which generate | occur | produced in the several pixel of image sensor 12c-12e by simultaneous light reception of fluorescence and white light is an example of a 2nd electrical signal. Also, a color image is an example of a second image.

  The image combining circuit 14d generates a combined image using the fluorescence image and the color image. For example, the image processing circuit 14c and the image synthesis circuit 14d are an example of an image generation unit.

  FIG. 4B is a diagram showing an example of a composite image. In the example of FIG. 4B, an example of a composite image in which the marker 53 is superimposed on a color image 50 in which a portion where unevenness in brightness may occur is masked is shown. Therefore, the user can easily observe, for example, the marker 53 which is the place to be noted. In addition, the user can obtain information of the marker 53 while observing the color image 50 which is a color image of content that can be actually recognized by a human.

  Further, in the color image 50, the exposure timings coincide. For this reason, according to the imaging device 10 according to the first embodiment, it is possible to suppress the occurrence of image distortion. Therefore, according to the imaging device 10 according to the first embodiment, it is possible for the user such as a doctor who observes an image to ensure sufficient image quality for observation.

  As shown in FIG. 4B, the image processing circuit 14c sets the effective image range 52 excluding at least the end of the composite image corresponding to the first row and the last row as the composite image. Further, the image processing circuit 14 c and the image combining circuit 14 d generate a combined image in which the range other than the effective image range 52 is replaced with the black image (another image) 51.

  Here, the case where the imaging target is moving in a predetermined direction in the composite image will be described. Further, the case where a composite image (W2 + IR2 + IR1) is generated by superimposing a marker obtained from the fluorescence image (IR1) on a color image (W2 + IR2) will be described. In this case, since the imaging target is moving, the position of the center of gravity of the marker is temporally delayed with respect to the position of the center of gravity of the imaging target depicted in the color image (W2 + IR2).

  Next, a case where a composite image (W2 + IR2 + IR3) is generated by superimposing a marker obtained from the fluorescence image (IR3) on a color image (W2 + IR2) will be described. In this case, since the imaging target is moving, the position of the center of gravity of the marker temporally advances with respect to the position of the center of gravity of the imaging target depicted in the color image (W2 + IR2).

  Here, in the present embodiment, a marker obtained from a composite image ((IR1 + IR3) / 2) of the fluorescence image (IR1) and the fluorescence image (IR3) is superimposed on the color image (W2 + IR2) to form a composite image (the image). W2 + IR2 + (IR1 + IR3) / 2) is generated. According to the present embodiment, even if the imaging target is moving, the fluorescence image (IR1) and the fluorescence image (IR3) captured at the imaging timing which is approximately the same time before and after the imaging timing of the color image (W2 + IR2) The marker obtained from the composite image ((IR1 + IR3) / 2) of is superimposed on the color image (W2 + IR2). For this reason, the difference between the gravity center position of the color image (W2 + IR2) and the gravity center position of the composite image ((IR1 + IR3) / 2) is relatively small. Therefore, it is possible to suppress the deviation of the position of the center of gravity of the marker with respect to the position of the center of gravity of the imaging target depicted in the composite image (W2 + IR2 + (IR1 + IR3) / 2). Therefore, according to the first embodiment, it is possible to suppress an uncomfortable feeling given to the user from the composite image (W2 + IR2 + (IR1 + IR3) / 2) with respect to the movement.

  Next, control processing executed by the control circuit 14a according to the first embodiment will be described. FIG. 5 is a flowchart showing the flow of control processing according to the first embodiment. The control process is executed by the control circuit 14a when an instruction to start imaging in the body of the subject 100 is input to the CCU 14 by an input device that receives an instruction from the user such as a mouse or a keyboard (not shown).

  As shown in FIG. 5, the control circuit 14a sets "1" to the variable N (step S101). Then, the control circuit 14a outputs, to the timing signal generation circuit 14f, a control signal for outputting a light source control signal for continuously emitting excitation light from the IR laser 30d (step S102). Then, the control circuit 14a outputs a control signal for starting exposure of the Nth frame by the rolling shutter method to the image sensor control circuit 12f in order to start imaging of the Nth frame (step S103).

  Then, the control circuit 14a starts the image sensor 12e in the readout period 1/120 [s] from the time when the output of the video signal of the first row of the plurality of pixels of the image sensor 12e starts in imaging the Nth frame. Control signal for causing the image sensor control circuit 12f to output the control signal (step S104). In step S104, the storage control circuit 14b temporarily stores the video signal output from each row in the storage circuit 14g each time the video signal output from each row of the image sensor 12e is input.

  Then, the control circuit 14a increments the value of the variable N by one (step S105). Then, the control circuit 14a outputs a control signal for starting exposure of the Nth frame by the rolling shutter method to the image sensor control circuit 12f in order to start imaging of the Nth frame (step S106).

  The control circuit 14a determines whether the current time is a blanking period or a period centered on the blanking period (step S107). If the current time is not a blanking period or a period centered on the blanking period (step S107; No), the control circuit 14a performs the determination of step S107 again.

  On the other hand, when the current time is a period centered on the blanking period or the blanking period (step S107; Yes), the control circuit 14a performs the following processing. That is, the control circuit 14a outputs a control signal for outputting a light source control signal for causing the white LED 30b to emit white light only to the blanking period or a period centered on the blanking period to the timing signal generating circuit 14f ( Step S108).

  Then, the control circuit 14a controls the image sensor to control the control signal for outputting the video signal to the image sensors 12c to 12e in the reading period 1/120 [s] from the time when the output of the video signal from the first row starts. It outputs to the circuit 12f (step S109). In step S109, the storage control circuit 14b temporarily stores the video signal output from each row in the storage circuit 14g each time the video signal output from each row of the image sensors 12c to 12e is input.

  Then, the control circuit 14a increments the value of the variable N by one (step S110). Then, the control circuit 14a outputs a control signal for starting exposure of the Nth frame by the rolling shutter method to the image sensor control circuit 12f in order to start imaging of the Nth frame (step S111). In step S111, the storage control circuit 14b temporarily stores the video signal output from each row in the storage circuit 14g each time the video signal output from each row of the image sensor 12e is input.

  Then, in step S112, the image processing circuit 14c outputs the video signal (IR1) output from each row in the first frame imaging, the video signal (W2 + IR2) output from each row in the second frame imaging, and three frames Various image processing is performed on the video signal (IR3) output from each row in imaging of the eye. Then, in step S112, the image combining circuit 14d outputs the combined image (W2 + IR2 + (IR1 + IR3) / 2) of the video signal (IR1), the video signal (W2 + IR2) and the video signal (IR3) to the output circuit 14e.

  Then, the control circuit 14a determines whether an instruction to end the control processing has been received through the input device (step S113). When the instruction to end the control process is not received (step S113; No), the control circuit 14a increments the value of the variable N by one (step S114), and returns to step S102. On the other hand, when an instruction to end the control processing is received (step S113; Yes), the control circuit 14a ends the control processing.

  The imaging device 10 according to the first embodiment has been described above. According to the imaging device 10 according to the first embodiment, as described above, it is possible to secure an image quality sufficient for the user to observe.

  Also, for example, in the technology described in Patent Document 1 above, two light beams (white light and IR excitation light) having different wavelengths are alternately emitted, and the imaging device outputs a video signal at 120 fps. The case where the exposure period is set to 1/60 [s] will be described. In this case, the white light image is output to the monitor at 30 fps.

  On the other hand, in the first embodiment, for example, when the image sensor 12c outputs a video signal at 120 fps and the exposure period is 1/60 [s], the color image based on white light is 60 [60]. is output to the display 101 at fps]. Therefore, according to the imaging device according to the first embodiment, it is possible to suppress a decrease in frame rate. Therefore, a frame rate sufficient for the user to observe can be secured.

(Modification of the first embodiment)
Next, a modification of the first embodiment will be described. In the first embodiment, the color image (W2 + IR2) is an image based on video signals output from the image sensors 12c to 12e by simultaneous reception of white light and fluorescence. For this reason, the color image (W2 + IR2) is a composite image of a color image based on white light and a fluorescence image based on fluorescence. Thus, the color image (W2 + IR2) contains a fluorescence image.

  In the first embodiment, it is premised that the light amount of fluorescence is negligibly small compared to the light amount of reflection of white light, but in the modification of the first embodiment, the light amount of reflection of white light is compared The case where the light quantity of the fluorescence is so large that it can not be ignored will be described. In this case, since the fluorescence image is a reddish tint, the color image (W2 + IR2) also has a reddish tint.

  Therefore, in the modification of the first embodiment, the image processing circuit 14c subtracts the composite image ((IR1 + IR3) / 2) of the fluorescence image (IR1) and the fluorescence image (IR3) from the color image (W2 + IR2). Thus, a color image based on substantially white light is generated. That is, the image processing circuit 14c uses the plurality of fluorescence images (IR1) and fluorescence images (IR3) generated before and after the color image (W2 + IR2) to reduce the color component of fluorescence (W2 + IR2). Correct the Therefore, according to the modification of the first embodiment, it is possible to perform correction to suppress the fluorescence component from the color image (W2 + IR2). In addition, the component of fluorescence is an example of the component by 1st light.

Second Embodiment
Next, an imaging device according to the second embodiment will be described. In the second embodiment, the same components as those in the first embodiment may be assigned the same reference numerals and descriptions thereof may be omitted. The imaging device according to the second embodiment operates the imaging device 10 according to the first embodiment except that the white light emitted by the white LED 30b is controlled such that the brightness of the image becomes the target brightness. Do the same as The configurations of the imaging device and the imaging system according to the second embodiment are similar to those of the first embodiment.

  For example, in the rolling shutter type image sensors 12c to 12e, during imaging of a certain frame, all the charges accumulated up to that point are discharged (reset) in all the rows of a plurality of pixels, and the charges are discharged. There is a shutter function that performs exposure again to re-accumulate the charge. When such a shutter function operates, the light reception periods may not be the same in the row 44 described above. In this case, unevenness in brightness may occur in the image.

  Therefore, the imaging device according to the second embodiment controls the white light emitted by the white LED 30b so that the brightness of the image becomes the target brightness less than the threshold value so that the shutter function does not operate.

  Here, an example of a method of calculating the brightness of the image will be described. In the second embodiment, the control circuit 14a executes the following processing for each frame. For example, the control circuit 14a acquires, from the image processing circuit 14c, image data of a color image obtained as a result of being subjected to various types of image processing. Then, the control circuit 14a selects, for each pixel, the highest luminance from among the luminance of R, the luminance of G, and the luminance of B which constitute the color image indicated by the image data. For example, each pixel which constitutes a picture corresponds to each pixel of image sensors 12c-12e. Then, the control circuit 14a calculates the sum of the luminances selected for each pixel. Then, the control circuit 14a divides the total of the calculated luminances by the number of pixels (number of pixels) forming the image to calculate an average value of luminance per pixel. Then, the control circuit 14a treats the calculated average value of the brightness as the brightness of the image.

  Then, the control circuit 14a compares the brightness of the image with the brightness of the target. The target brightness is, for example, a value of brightness designated by the user. Further, the target brightness is an example of a predetermined brightness. If the brightness of the image is larger than the brightness of the target, the control circuit 14a responds to the difference between the brightness of the image and the brightness of the target in order to reduce the amount of white light emitted to the subject 100. Then, the light source control signal whose length of the high level period is shortened is output to the timing signal generation circuit 14 f. For example, as the difference between the brightness of the image and the target brightness increases, the control circuit 14a causes the timing signal generation circuit 14f to output a light source control signal in which the length of the high level period is shortened. Describing the specific example, the control circuit 14a calculates the length of the high level period according to the difference between the brightness of the image and the brightness of the target, and the high level period obtained as a result of the calculation. The length is output to the timing signal generation circuit 14f. Thus, the timing signal generation circuit 14f outputs a light source control signal whose length of the high level period is shortened to the drive circuit 30a.

  If the target brightness is larger than the image brightness, the control circuit 14a sets the length of the high level period within the readout period or less according to the difference between the image brightness and the target brightness. The light source control signal thus lengthened is output to the timing signal generation circuit 14 f.

  That is, the control circuit 14a and the timing signal generation circuit 14f control the white light received by the plurality of pixels by changing the length of the high level period so that the brightness of the image becomes the target brightness. Do. Specifically, the control circuit 14a and the timing signal generation circuit 14f control the light reception period so that the brightness of the image becomes the target brightness. The control circuit 14a and the timing signal generation circuit 14f control the light reception period to control the amount of white light emitted to the subject 100. The control circuit 14a and the timing signal generation circuit 14f control the light reception period to control the brightness of the color image.

Third Embodiment
Next, an imaging device according to a third embodiment will be described. In the description of the third embodiment, the same components as those in the above-described embodiments and the above-described modification may be given the same reference numerals and descriptions thereof may be omitted.

  The imaging device according to the third embodiment is used when the light amounts of white light and excitation light (fluorescence) are sufficient or when mixing of white light and excitation light is suppressed.

  FIG. 6 is a diagram for explaining an example of the operation of the imaging device according to the third embodiment. In FIG. 6, emission timings of white light and excitation light emitted from the light source device 30 according to the third embodiment, and exposure timings of each row of a plurality of pixels included in the image sensors 12c to 12e of the imaging device 10; An example of the relationship between the output timing of the video signal output from the image sensors 12c to 12e and the output timing of the video signal output from the output circuit 14e is shown. In FIG. 6, the horizontal axis indicates time.

  The operation of the imaging apparatus according to the third embodiment shown in FIG. 6 is different from the operation of the imaging apparatus 10 according to the first embodiment shown in FIG. In the first embodiment, the case where the control circuit 14a continuously emits the excitation light has been described. However, in the third embodiment, the control circuit 14a switches the type of light received by the plurality of pixels for each frame. Specifically, the control circuit 14 a switches the light emitted from the light source device 30 to white light or excitation light alternately for each frame. Thereby, the control circuit 14 a switches the light to be received by the plurality of pixels alternately to white light or fluorescence for each frame.

  In the third embodiment, the control circuit 14 a causes the light source device 30 to alternately emit excitation light and white light of different types in synchronization with a blanking period or a period centered on the blanking period. The image processing circuit 14c generates a fluorescence image based on video signals generated in a plurality of pixels by receiving fluorescence based on excitation light, and a color image based on video signals generated in a plurality of pixels by receiving white light. Generate The image combining circuit 14d generates a combined image using the fluorescence image and the color image. In addition, the video signal generated at the plurality of pixels of the image sensor 12e by the reception of the fluorescence based on the excitation light is an example of the first electric signal. The video signals generated at the plurality of pixels of the image sensors 12c to 12e by the reception of the white light are an example of the second electrical signal.

  The imaging apparatus according to the third embodiment can ensure sufficient image quality for observation, as in the first embodiment.

  Also, for example, in the technology described in Patent Document 1 above, two light beams (white light and IR excitation light) having different wavelengths are alternately emitted, and the imaging device outputs a video signal at 120 fps. The case where the exposure period is set to 1/60 [s] will be described. In this case, each of the white light image and the fluorescence observation image is output to the monitor at 30 [fps].

  On the other hand, in the third embodiment, for example, two light beams (white light and excitation light) having different wavelengths are alternately emitted, and the image sensors 12c to 12e output video signals at 120 fps to perform exposure. When the period is 1/60 [s], a composite image of a white light-based color image and a fluorescence-based fluorescence image is output to the display 101 at 60 [fps]. Therefore, according to the imaging device of the third embodiment, it is possible to suppress a decrease in frame rate.

Fourth Embodiment
Next, an imaging device according to a fourth embodiment will be described. In the description of the fourth embodiment, the same components as those of the above-described embodiments and the above-described modification may be denoted by the same reference numerals and description thereof may be omitted.

  In the first embodiment, for example, a case has been described in which a plurality of pixels of the image sensors 12c to 12e receive light only in a period in which 1/1200 [s] is added to the blanking period with the blanking period as the center. In this case, the area to be masked is approximately 10% of the area of the entire screen.

  FIG. 7 is a view for explaining an example of an image according to the fourth embodiment. However, in order to secure the light intensity, the light reception periods of the plurality of pixels of the image sensors 12c to 12e are extended, for example, about 50% of the entire screen area is masked as shown in the example of FIG. It is also good. In this case, the image processing circuit 14c may generate an image (enlarged image) 50a in which the effective image range 52 is expanded. As shown in the example of FIG. 7, the user can observe the marker 53 a more easily by enlarging.

Fifth Embodiment
Next, an imaging device according to the fifth embodiment will be described. In the description of the fifth embodiment, the same components as those in the above-described embodiments and the above-described modification may be denoted by the same reference numerals and description thereof may be omitted.

  The control circuit 14a of the imaging device according to the fifth embodiment has a mode (first mode) for executing control processing according to one of the first to fourth embodiments, and white. A mode (second mode) in which only the LED 30 b or the IR laser 30 d is driven to perform imaging of a color image based on white light or a fluorescence image based on excitation light is switched according to a user's instruction.

  For example, in the second mode, the exposure period is 1/60, which is a double period of the exposure period 1/120 [s] according to the first to fourth embodiments. In the second mode, the control circuit 14a causes the plurality of pixels of the image sensors 12c to 12e to receive fluorescence or white light, and displays a fluorescence image or a color image at a predetermined frame rate 1/60 [s]. Make it output to 101.

  FIG. 8 is a diagram for explaining an example of mode switching performed by the imaging device according to the fifth embodiment. As shown in the example of FIG. 8, for example, when a user such as a doctor or the like observes the color image 60 in the second mode, if there is a portion to be scrutinized, an input device (not shown) Input an instruction to switch to the first mode. The control circuit 14 a switches to the first mode in accordance with the input user's instruction, and causes the display 101 to display a composite image in which the marker 53 is superimposed on the color image 50.

  In the fifth embodiment, the frame rate output from the imaging apparatus to the display 101 does not change even when the mode is switched. Therefore, according to the imaging device according to the fifth embodiment, it is possible to suppress a sense of discomfort given to the user when switching the mode.

Sixth Embodiment
Next, an imaging device according to a sixth embodiment will be described. In the first to fourth embodiments and the modification of the first embodiment described above, the control circuit 14a sets a blanking period, a period within the blanking period, as a period based on the blanking period, Alternatively, the case has been described in which the plurality of pixels are received only during a period centered on the blanking period. However, as a period based on the blanking period, the control circuit 14a may cause a plurality of pixels to receive light only during a part of the period centered on the blanking period. Thus, such an embodiment will be described as a sixth embodiment.

  FIG. 9 is a diagram for explaining an example of the operation of the imaging device according to the sixth embodiment. The horizontal axis in FIG. 9 indicates time. For example, first, similarly to the first embodiment, the control circuit 14a causes excitation light to be continuously emitted from the light source device 30, and synchronizes white light with a period based on the blanking period 40. The case where light is emitted from the light source device 30 once in two frames will be described.

  In this case, as shown in FIG. 9, the control circuit 14a emits white light from the light source device 30 only during a part of the period 41a of the period 41 centered on the blanking period 40 at a rate of once every two frames. Let Thereby, the plurality of pixels are received only during a part of the period 41 a of the period 41. Here, the period 41 a is a period before the blanking period 40 in time series.

  Next, as in the third embodiment, the control circuit 14 a causes the light source device 30 to alternately emit excitation light and white light of different types in synchronization with a period based on the blanking period 40. The case will be described.

  Also in this case, as shown in FIG. 9, the control circuit 14a alternately emits the excitation light and the white light from the light source device 30 only during the period 41a. Also in this case, the plurality of pixels are received only during part of the period 41 a.

  According to the sixth embodiment, although the size of the masked area on one end side of the composite image displayed on the display 101 is different from the size of the masked area on the other end side, the other embodiments are different. Similar to the form, for a user such as a doctor who observes an image, a frame rate and image quality sufficient for observation can be secured.

  Each embodiment has been described above. In each embodiment, the case where the imaging device 10 receives the reflected light of white light and receives the image sensors 12 c to 12 e and performs various processes has been described. However, the imaging device 10 may perform similar processing by causing the image sensor 12 e to receive fluorescence (fluorescence based on white light) emitted from the ICG after the ICG is excited by white light.

  According to the imaging device according to at least one embodiment or modification described above, it is possible to secure a frame rate sufficient for the user to observe.

  While certain embodiments of the present invention have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. These embodiments can be implemented in other various forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the invention described in the claims and the equivalents thereof as well as included in the scope and the gist of the invention.

1 imaging system 10 imaging device 12c, 12d, 12e image sensor 14a control circuit 14c image processing circuit 14d image combining circuit

Claims (16)

It includes a plurality of pixels arranged in a matrix to generate an electric signal by light reception, and exposure starts sequentially at least every other row from the first row to the last row of the plurality of pixels, and the exposure ends A rolling shutter type image sensor that repeats the process of outputting the electric signal in order from the left row, for each frame;
After the output of the electrical signal from the last row is ended, the plurality of the plurality of periods are only based on a blanking period corresponding to a period from when the output of the electrical signal from the first row is started. A control unit that receives the pixels of
An image generation unit that generates an image based on the electrical signal;
An imaging device provided with   The period based on the blanking period is the blanking period, a period within the blanking period, a period centered on the blanking period, or a partial period of a period centered on the blanking period. The imaging device according to claim 1, wherein   The control unit performs the plurality of pixels in a period equal to or less than a period from when output of the electrical signal from the first row in one frame starts to when output of the electrical signal from the last row ends. The imaging device according to claim 1, wherein the light is received.   The imaging according to any one of claims 1 to 3, wherein the image generation unit sets an effective image range excluding at least an end of the image corresponding to the first row and the last row to the image. apparatus. The image generation unit generates the image in which a range other than the effective image range is replaced with another image.
The imaging device according to claim 4. The image generation unit generates the image in which the effective image range is expanded,
The imaging device according to claim 4 or 5. The control unit causes the light source device to continuously emit the first light and synchronizes the second light whose type is different from the first light with a period based on the blanking period. The light source device emits light at a rate of once every two frames,
The image generation unit generates a first image based on first electric signals generated in the plurality of pixels by receiving the reflected light of the first light or fluorescence generated by the first light, Based on second electric signals generated in the plurality of pixels by simultaneous reception of the reflected light of the first light or the fluorescence and the reflected light of the second light or the fluorescence generated by the second light Generating a second image, and using the first image and the second image to generate the image.
The imaging device according to any one of claims 1 to 6. The control unit causes the light source device to alternately emit first light and second light of different types so as to be synchronized with a period based on the blanking period.
The image generation unit generates a first image based on first electric signals generated in the plurality of pixels by receiving the reflected light of the first light or fluorescence generated by the first light, A second image is generated based on second electric signals generated in the plurality of pixels by reception of the reflected light of the second light or fluorescence generated by the second light; Generating the image using the second image;
The imaging device according to any one of claims 1 to 6. The image generation unit generates the image using the second image and a plurality of the first images generated before and after the second image.
The imaging device according to claim 7. The image generation unit corrects the second image such that a component of the first light is reduced by using a plurality of the first images generated before and after the second image.
The imaging device according to claim 7. The optical device is further provided with an optical element disposed to face the imaging region of the plurality of pixels, and causing the plurality of pixels to receive light other than the excitation light emitted from the light source device as the first light.
The imaging device according to any one of claims 7 to 10. The control unit controls the brightness of the image by controlling a light reception period of the pixel.
The imaging device according to any one of claims 1 to 11. The control unit causes the plurality of pixels to receive light only during a predetermined exposure period of the plurality of pixels in response to an instruction, and the image at a predetermined frame rate. Reflected light of the first light, fluorescence generated by the first light, in the plurality of pixels during a first mode of outputting and an exposure period twice as long as the predetermined exposure period; And one of the fluorescence generated by the second light is received, and switching is made between a second mode in which an image based on the one light is output at the predetermined frame rate.
The imaging device according to any one of claims 7 to 11. Comprising a plurality of said image sensors,
Each of the plurality of image sensors outputs an electrical signal by receiving light of a corresponding type,
The imaging device according to any one of claims 1 to 13. The image generation unit generates the image obtained by combining the first image and the second image.
The imaging device according to any one of claims 7 to 11. It includes a plurality of pixels arranged in a matrix to generate an electric signal by light reception, and exposure starts sequentially at least every other row from the first row to the last row of the plurality of pixels, and the exposure ends It is a control program for controlling an imaging device provided with a rolling shutter type image sensor, which repeats the process of outputting the electric signal in order from a row to a row, for each frame,
After the output of the electrical signal from the last row is ended, the plurality of the plurality of periods are only based on a blanking period corresponding to a period from when the output of the electrical signal from the first row is started. Receive the pixels of
Generating an image based on the electrical signal,
Control program.

Images