TWI636558B - Electronic device and image sensor - Google Patents

Abstract

The present disclosure relates to an electronic device and an image sensor that can achieve power saving and measurement accuracy when measuring illuminance using an output of an image sensor.

An electronic device according to a first aspect of the present disclosure includes: an image sensor unit including a plurality of pixels; and a control unit that reduces driving of the plurality of pixels constituting the image sensor unit, and cycles The illuminance calculation unit is configured to calculate the illuminance by using the output of the image sensor unit when the plurality of pixels are reduced in driving. For the present disclosure, it is applicable to devices based on illuminance control actions.

Description

Electronic device and image sensor

The present disclosure relates to an electronic device and an image sensor, and more particularly to an electronic device and image sensing for measuring illumination based on an output of an image sensor mounted for realizing an imaging function. Device.

In recent years, an illuminometer has been incorporated into home electric appliances such as television receivers, air conditioners, and refrigerators, and the home electric appliance is formed to operate in accordance with the measured illuminance. Specifically, for example, the brightness of the screen is adjusted according to the illuminance, or the touch sensor for preventing malfunction is disabled, or the specific function is stopped to suppress power consumption.

Further, in an electronic device such as a personal computer or a smart phone having an imaging function, an illuminance is measured using an output of an image sensor for realizing an imaging function (see, for example, Patent Document 1).

[Previous Technical Literature] [Patent Literature]

[Patent Document 1] Japanese Patent Laid-Open Publication No. 2008-306254

When the image sensor output is used for illuminance measurement in the electronic device, there is a method of using all pixel outputs of the image sensor (driving all pixels), and using a part of the pixel output of the image sensor (minus) The method of driving pixels).

In the case of the method of using all pixel outputs, since the brightness of the entire viewing angle is averaged, the value close to the actual illuminance can be measured. However, there is a problem in terms of power consumption in order to drive all the pixels.

On the other hand, in the case of using a method of partial pixel output of an image sensor, there may be a high-intensity subject, a blinking subject, a moving subject, or the like due to a viewing angle. The detector itself sways and causes a value to deviate from the actual illuminance.

The present disclosure has been made in view of such a situation, and when the illuminance is measured using the output of the image sensor, the power saving and the accuracy of the measured value can be improved.

An electronic device according to a first aspect of the present disclosure includes: an image sensor unit including a plurality of pixels; and a control unit that reduces driving of the plurality of pixels constituting the image sensor unit, and cycles The illuminance calculation unit is configured to calculate the illuminance by using the output of the image sensor unit when the plurality of pixels are reduced in driving.

The illuminance calculation unit may calculate the illuminance each time the plurality of pixels are driven to be reduced, and further output a value obtained by moving and averaging the calculated illuminance.

The electronic device according to the first aspect of the present disclosure may further include a drive unit that operates based on the control based on the illuminance output from the illuminance calculation unit.

The control unit may divide the plurality of pixels constituting the image sensor unit into regions of a specific size, and periodically change the region to be driven for each frame or each time it is activated.

The image sensor unit may include a short-time storage pixel and a long-term storage pixel, and the illuminance calculation unit may calculate the illuminance using an output of the short-time storage pixel that is simultaneously driven and an output of the long-term storage pixel.

The image sensor unit may include pixels for detecting an image plane phase difference, and the illuminance calculation unit may not use the pixel having the image surface phase difference detection function that is driven. The output is calculated to calculate the illuminance.

In the first aspect of the present disclosure, the plurality of pixels constituting the image sensor section are reduced in driving, and the pixels to be driven are periodically changed, and the image in which the plurality of pixels are used is reduced in driving. The illuminance is calculated by the output of the sensor section.

The image sensor according to the second aspect of the present disclosure includes a plurality of pixels, and simultaneously drives the plurality of pixels at the time of normal imaging, and reduces the driving of the plurality of pixels during illumination measurement, and periodically changes The pixels to be driven are reduced.

In the second aspect of the present disclosure, a plurality of pixels are simultaneously driven during normal imaging, and when the illuminance measurement is performed, the plurality of pixels are reduced in driving, and the pixels to be driven are periodically changed.

According to the first aspect of the present disclosure, an image sensor output can be used to perform illuminance measurement with higher precision with less power consumption.

According to the second aspect of the present disclosure, pixel values usable for illuminance measurement can be output.

10‧‧‧Electronic devices

11‧‧‧Control Department

12‧‧‧ Input Department

13‧‧‧Image sensor

14‧‧‧Image Processing Department

15‧‧‧Timer

16‧‧‧Microphone

17‧‧‧User interface

18‧‧‧Gyro sensor

19‧‧‧Setting Department

20‧‧‧ memory

21‧‧‧Communication Department

22‧‧‧ Drive Department

23‧‧‧Power Supply Department

24‧‧‧Memory Department

25‧‧‧Output interface

26‧‧‧Motor

27‧‧‧Display Department

28‧‧‧Speakers

29‧‧‧Lighting Department

Lux‧‧‧ illumination

G‧‧‧ pixel value

R, Gr, Gb, B‧‧‧ short-term storage pixels

R, gr, gb, b‧‧‧ long-term storage pixels

Y‧‧‧ brightness

1 is a block diagram showing a configuration example of an electronic device to which the present disclosure is applied.

2A and 2B are diagrams for explaining an example of the first reduced driving of the image sensor.

Fig. 3 is a view for explaining an example of the first reduced driving of the image sensor.

4 is a graph showing the relationship between the storage time and the pixel value of each of the short-term storage pixels and the long-term storage pixels.

Fig. 5 is a view showing an example of synthesizing respective outputs of short-time storage pixels and long-term storage pixels.

Fig. 6 is a view showing the arrangement of short-time storage pixels and long-term storage pixels in each of the strip-shaped regions.

Fig. 7 is a view showing an example of converting the R, G, and B pixel values of the Bayer arrangement into luminance.

Fig. 8 is a view showing a conversion formula for converting luminance Y into illuminance Lux and an example thereof.

Fig. 9 is a view showing an example in which the illuminance of each frame is moved and averaged between frames.

10A and B are views for explaining an example of the second reduction drive of the image sensor.

Fig. 11 is a view for explaining an example of the second reduced driving of the image sensor.

Fig. 12 is a view for explaining an example of the third reduction driving of the image sensor.

13A and 13B are views for explaining a variation of the first to third reduction driving.

Fig. 14 is a view showing an example of a case where the image sensor includes image plane phase difference detecting pixels.

Hereinafter, the best mode for carrying out the present disclosure (hereinafter referred to as an embodiment) will be described in detail with reference to the drawings.

"Example of the structure of electronic devices"

Fig. 1 is a view showing an example of a configuration of an electronic device that performs illumination measurement using an image sensor output and controls an operation based on the measured illuminance according to an embodiment of the present disclosure.

The electronic device 10 includes a control unit 11, an input unit 12, a setting unit 19, and a drive unit 22. The control unit 11 controls the operation of the input unit 12, and controls the drive unit 22 based on the data from the input unit 12. More specifically, the operation of the drive unit 22 is controlled based on the illuminance input from the input unit 12.

The input unit 12 has an image sensor 13 that generates an image signal, and an image processing unit 14 that processes an image signal. The image sensor 13 includes a plurality of photoelectric conversion elements (hereinafter referred to as pixels), and simultaneously drives all pixels at the time of normal imaging according to control from the control unit 11, and outputs a pixel signal as its output to image processing. Part 14. Further, the image sensor 13 subtracts one of the driving pixels in the illuminance measurement (details are described later), and outputs the pixel signal as the output thereof to the image processing unit 14.

Further, in the present embodiment, the image sensor 13 is driven under the control of the control unit 11, but the image sensor 13 itself may be provided with a control unit, thereby causing the image sensor 13 Self-controls the action when shooting normally and when measuring illuminance.

In the illuminance measurement, the image processing unit 14 calculates the illuminance based on the image signal input from the image sensor 13 as an illuminance calculation unit, and outputs the illuminance to the control unit 11. The illuminance calculation in the image processing unit 14 calculates the illuminance of each frame based on the reduction of the output of the image sensor 13 driving the different regions, and the illuminance is performed between the frames for each frame. The moving average is used for the control of the drive unit 22.

Further, the input unit 12 may include a timer 12 for counting, a microphone 16 for inputting a voice, a user interface (I/F) 17 for inputting a user's operation, a keyboard, a touch panel, and the like, and a detecting electronic device 10. Position or displacement of the gyro sensor 18 and the like.

The setting unit 19 has a memory 20 that holds a control program or various coefficients, and a communication unit 21 that connects to a network and communicates data. The drive unit 22 may include a power supply unit 23, a storage unit 24, an output interface (I/F) 25, a motor 26, a display unit 27, a speaker 28, an illumination unit 29, and the like.

"Example of the first reduction drive of the image sensor 13"

2 and 3 are diagrams for explaining an example of the first reduced driving of the image sensor 13. In addition, the hatched area in the figure indicates the area of the pixel to be driven, and FIG. 2 shows the driving area of the first frame in the case where the frame is one cycle, and FIG. 3 shows that the frame is one cycle. The driving area of the second frame in the case. Further, the minimum rectangle of A of FIG. 2 represents 1 pixel, and the minimum rectangle of B of FIG. 2 and FIG. 3 represents 2 × 2 pixels.

In the example of the first reduction driving, all the pixels of the image sensor 13 are divided into long strips of a length of 2 pixels and a vertical width of 16 pixels, and each of the 8 frames only drives the image. The area is 1 time. The order in which the respective regions are driven is as shown by the number indicated on the upper side of the image sensor 13 in the figure, and moves not only in one direction toward the adjacent region but also in the left and right intervals.

Furthermore, as shown in FIG. 2A, the arrangement of the colors of the pixels constituting the image sensor 13 is a Bayer arrangement, and pixels of various colors are arranged until storage until saturation. The pixels R, Gr, Gb, B or the long-term storage pixels r, gr, gb, b are stored for a short time. Each of the short strip-shaped regions is alternately arranged with a short-time storage pixel of 2×4 pixels sharing a FD (Floating Diffusion) and a long-time storage pixel of 2×4 pixels.

In this way, by alternately arranging the FD of the short-term storage pixel and the FD of the long-term storage pixel, the center of gravity of the short-time storage pixel and the long-term storage pixel respectively adding the pixel values respectively can be shortened before the same imaging area. The center of the strip. Furthermore, as for the arrangement of the short-term storage pixels and the long-term storage pixels, as long as the center of gravity of the pixels is the same, in addition to the above, the pixels may be stored for a short period of time, or the pixels may be stored for a long time. Store pixels for a short time. Further, it may be arranged not in the vertical direction but in the horizontal direction. Further, it is not necessary to separately store the long-term storage and the short-term storage in the FD sharing unit, and if there is a mechanism that can change the storage time by the pixel, the long-term storage pixel and the short-time storage pixel can be interposed in the FD sharing.

FIG. 4 shows the relationship between the storage time and the pixel value of each of the short-term storage pixels and the long-term storage pixels. As shown in the figure, by synthesizing the respective outputs of the short-time storage pixels and the long-term storage pixels having different characteristics, a pixel with a wide dynamic range and high precision can be obtained.

FIG. 5 shows an example in which the output of the short-term storage pixel is converted into the output of the long-time storage pixel when synthesizing the output of each of the short-term storage pixels and the long-term storage pixels. Specifically, the saturation point of the long-term storage pixel is switched to the output of the short-time storage pixel multiplied by the storage time ratio. The threshold value can be set by color according to the pixel characteristics, and can also be set externally in a finely adjustable manner.

Fig. 6 shows the arrangement of short-time storage pixels and long-term storage pixels in each of the strip-shaped regions. As shown in the figure, before synthesizing the output of each of the short-term storage pixels and the long-term storage pixels, in the short strip-shaped regions, 8 pixels of the short-time storage pixels and 8 pixels of the long-term storage pixels in the shared FD. Each of them adds pixel values to each color.

The pixel values R, G, and B of 2 × 2 pixels synthesized from 2 × 16 pixels in this manner are converted into luminance values Y. Figure 7 shows the conversion of the pixel values R, G, B of the Bayer arrangement to the brightness Y The conversion formula and its examples. The coefficient of the conversion formula is recorded in the memory 20 or the like at the time of pre-production, but these coefficients can be set externally via the communication unit 21.

Further, the luminance Y is converted into the illuminance Lux. Fig. 8 shows a conversion formula for converting luminance Y into illuminance Lux and an example thereof. The coefficient of the conversion formula is recorded in the memory 20 or the like at the time of pre-production, but these coefficients can be set externally via the communication unit 21.

9 is a illuminance on the vertical axis and a frame on the horizontal axis, and shows that the illuminance calculated for each frame is moved between frames by an IIR (Infinite Impulse Response) filter having a feedback rate of 50%. The average case. As shown in the figure, even if the illuminance of each frame is not uniform, by taking a moving average between the frames, even if the illuminance of the specific frame is prominent, the influence can be suppressed.

For example, as shown in FIG. 2 and FIG. 3, when the light from the high-brightness light source is irradiated to the area driven by the sixth frame and the eighth frame, the sixth frame and the eighth are displayed. The illumination of the frame is highlighted and becomes higher than the illumination of other frames. However, since the illuminance after the moving average between the frames is finally used for the control of the driving portion 22, the influence of the high-intensity light source can be suppressed.

Furthermore, the number of TAPs (tap) of the moving average can also be changed according to the requirements for stability and responsiveness of the moving average. Further, instead of the IIR filter, an FIR (Finite Impulse Response) filter can be used.

Further, since the driving area of the pixel is moved for each frame so that the number of accesses per pixel is made uniform, a secondary effect of uniformizing the deterioration of the pixel caused by the driving time can be expected. Furthermore, the movement of the area in which the pixels are driven can also be changed each time the image sensor 13 is activated, rather than being changed for each frame.

"Example of the second reduction drive of the image sensor 13"

Next, FIG. 10 and FIG. 11 are diagrams for explaining an example of the second reduction drive of the image sensor 13. The example of the second reduction drive is that the first reduction drive described above is only a lateral reduction and further vertical reduction.

In addition, the hatched area in the figure indicates the area of the pixel to be driven, and FIG. 10 shows the driving area of the second frame when the 16 frame is one cycle, and FIG. 11 shows the case where the 16 frame is one cycle. The driving area of the third frame at the time. Further, the minimum rectangle of A of FIG. 10 represents 1 pixel, and the minimum rectangle of B of FIG. 10 and FIG. 11 represents 2 × 2 pixels.

In the example of the second reduction driving, all the pixels of the image sensor 13 are divided into long strips of a length of 2 pixels and a vertical width of 16 pixels, and each of the 16 frames only drives the pixel. The area is 1 time. The order in which the respective regions are driven is as shown by the number indicated on the upper side of the image sensor 13 in the figure, and moves not only in one direction toward the adjacent region but also in the left and right intervals.

By implementing the second reduction drive, the power consumption can be further suppressed to half as compared with the case where the first reduction drive is implemented.

However, when the vertical reduction is added as in the case of the second reduction drive, the OB (Optical Black) region (not shown) of the image sensor 13 may be insufficient. In this case, a frame that drives or does not drive the same pixel is set in the vertical direction. When the frame that is not driven is set, the black level adjustment is performed using the black level of the previous frame, or the black level is not changed.

Further, as shown in FIG. 10A, the arrangement of the colors of the pixels constituting the image sensor 13 is the same as the example of the first reduction drive described with reference to A of FIG. 2, and therefore the description thereof will be omitted.

"Example of the third reduction drive of the image sensor 13"

Fig. 12 is a view for explaining an example of the third subtraction driving of the image sensor 13, showing the entire area of the image sensor 13, and the hatched area in the figure indicates the area of the driven pixel. That is, the third reduction driving example divides the entire area of the image sensor 13 into a plurality of rectangular areas, and performs the first or second reduction described above only in a specific rectangular area in the rectangular area. drive.

Specifically, the entire area of the image sensor 13 is divided into 1920 pixels×1080 pixels. In the case of a rectangular area of 4 × 4, in this case, each bit of the 16 bit (bit) ALS_GRID_EN can be used to set the driving of each rectangular area to ON.

By implementing the third reduction drive, power consumption can be further suppressed as compared with the case where the first or second reduction drive is implemented.

The specific rectangular area of the driving pixel can be specified from the outside, and can be changed at any time for each frame.

Furthermore, of course, the image obtained when the first to third reduction driving examples are performed is smaller than the image of all the pixels that are normally driven, and the resolution is poor, but it is not The calculation of the contrast degree, the position of the light source, and the judgment of the action object cause obstacles.

"About the variation of the first to third reduction drivers of the image sensor 13"

A of FIG. 13 shows an example of the first to third subtraction driving of the image sensor 13 described above, and FIG. 13B shows an example of driving the pixels of the R row and the pixels of the B row individually as the first to third subtractions. Province-driven changes. In the case of the example shown in FIG. 13B, the number of pixels to be driven can be made half in a state in which the spatial frequency of the obtained image is maintained, and power consumption can be suppressed to a corresponding extent. Furthermore, since the pixels of the R line are spatially separated from the pixels of the B line, false colors may be generated, but there is no problem in the use of the illumination measurement.

"Illumination operation in the case where the image sensor 13 is provided with the image plane phase difference detecting pixel"

FIG. 14 shows an arrangement example in the case where the image sensor 13 mounted on a digital camera having an image plane phase difference AF (Auto Focus) function is provided with an image plane phase difference detecting pixel, and corresponds to the situation. An example of illuminance calculation.

When the image sensor phase difference detecting pixel is provided in the image sensor 13, the image plane phase difference detecting pixel is disposed at the position of Gr and gr of the Bayer array, and the outputs of Gr and gr are not used in the illuminance calculation. Further, in the case where the infrared ray detecting pixels are provided in the image sensor 13, the output of the infrared ray detecting pixels is not used for the illuminance calculation.

"Example of illumination control based on illumination"

The control unit 11 can control the operation of the drive unit 22 based on the illuminance input from the image processing unit 14.

For example, when the electronic device 10 is applied to a fire alarm or a monitor, it is possible to perform zoom photography or the like in a portion where the brightness is suddenly increased and brightened, and the self-illumination measurement mode is switched to the normal photography mode or the photography mode is started. Video recording of the video taken, or an alarm sounds.

For example, when the electronic device 10 is applied to an in-vehicle camera, the motor that rotates the illumination shaft of the light can be controlled in such a manner that the direction of the light of the self-driving vehicle is shifted from the opposite side of the vehicle according to the position of the light of the opposite vehicle.

For example, when the electronic device 10 is applied to a home appliance, the illuminance measured against the control is also referred to as time. When the time period is night and the surroundings are dark and there is no moving object, the power saving mode for suppressing power consumption can be suppressed. The way of action is controlled.

For example, when the electronic device 10 is applied to a mobile phone, a smart phone, or the like, when the time period is daytime but the illumination is low, it is determined that the mobile phone or the like has been placed in a pocket or a bag, for the purpose of preventing malfunction. Turn off the touch panel or increase the volume of the call. Conversely, when the time period is nighttime and the illumination is low, it is judged that the user is sleeping, and the volume of the speaker or the like is lowered. In this way, the control action can be changed according to the surrounding environment.

As described above, according to the electronic device 10 of the present embodiment, the illuminance can be measured without providing a dedicated circuit configuration (illuminance meter or the like) for illuminance measurement. Further, since the pixels of the driving image sensor 13 are reduced in the illuminance measurement, power consumption can be suppressed as compared with the case where all the pixels of the image sensor 13 are driven. Also, the illuminance can be quickly calculated as compared with the case of reading all the pixels.

Further, since the area of the pixel to be driven is changed in each frame or the like, and the illuminance of each frame is moved and averaged between the frames, the illuminance deviation finally obtained can be suppressed, and the accuracy can be maintained. By reducing the area of the driven pixels by changing each frame or the like The domain can also expect a secondary effect of homogenizing the degradation of the pixel.

Further, it is also possible to drive pixels of only a specific region in the entire image, and calculate the illuminance based on the output. Thereby, the illuminance of only a specific region (angle of view) in the imaging range can be obtained.

The pixel value obtained by the illuminance measurement can also be output as an image of low resolution.

In addition, the embodiment of the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit and scope of the invention.

Furthermore, the present disclosure may also adopt the configuration described below.

(1)

An electronic device comprising: an image sensor unit including a plurality of pixels; and a control unit that reduces driving of the plurality of pixels constituting the image sensor unit and periodically changes And a illuminance calculation unit that calculates an illuminance by using an output of the image sensor unit when the plurality of pixels are reduced in driving.

(2)

The electronic device according to (1) above, wherein the illuminance calculation unit calculates the illuminance each time the plurality of pixels are driven to be reduced, and further outputs a value obtained by moving and averaging the calculated illuminance.

(3)

The electronic device according to (1) or (2) above, further comprising: a drive unit that operates based on the control of the illuminance output from the illuminance calculation unit.

(4)

The electronic device according to any one of the above (1), wherein the control unit separates the plurality of pixels constituting the image sensor unit into The area of a particular size is periodically changed for each frame or each time it is started.

(5)

The electronic device according to any one of the above (1), wherein the image sensor unit includes short-time storage pixels and long-term storage pixels, and the illumination calculation unit uses the short-term storage that is simultaneously driven. The illuminance is calculated by the output of the pixel and the output of the long-term storage pixel.

(6)

The electronic device according to any one of the above-mentioned (1), wherein the image sensor unit includes a pixel for detecting an image plane phase difference, and the illumination calculation unit does not use the driven device. The illuminance is calculated by the output of the pixel for the surface phase difference detection purpose.

Claims (7)

An electronic device comprising: an image sensor unit including a plurality of pixels; and a control unit that reduces driving of the plurality of pixels constituting the image sensor unit and periodically changes And a illuminance calculation unit that calculates an illuminance by using an output of the image sensor unit when the plurality of pixels are reduced in driving. The electronic device of claim 1, wherein the illuminance calculation unit calculates the illuminance each time the plurality of pixels are driven to be reduced, and further outputs a value obtained by moving and averaging the calculated illuminance. The electronic device of claim 2, further comprising: a drive unit that operates based on the control of the illuminance output from the illuminance calculation unit. The electronic device of claim 2, wherein the control unit divides the plurality of pixels constituting the image sensor unit into regions of a specific size, and periodically changes for each frame or each time it is started The above area to be driven. The electronic device of claim 2, wherein the image sensor unit comprises a short-term storage pixel and a long-term storage pixel, wherein the illumination calculation unit uses the output of the short-time storage pixel that is simultaneously driven and the long-term storage pixel. The output is calculated to calculate the illuminance. The electronic device according to claim 5, wherein the image sensor unit includes pixels for detecting an image plane phase difference, and the illuminance calculation unit does not use the pixel having the image surface phase difference detection function that is driven. Output and calculate illuminance. An image sensor comprising a plurality of pixels, and simultaneously driving the plurality of pixels during normal imaging, reducing the driving of the plurality of pixels when measuring the illuminance, and periodically changing the driving to be reduced The pixels.