JP2012019429A - Imaging signal processing circuit and imaging signal processing method - Google Patents

Abstract

Image processing is performed in an imaging signal processing circuit for a captured image due to the influence of a flash or the like without manually editing, and an image without a sense of incongruity is output.
An image memory for storing at least two screens of RGB or YPbPr3 channel imaging signals in time series is provided, and an image in the image memory is exchanged only when flash light emission occurs during photographing from the output of the image memory. And a black line processing circuit for concealing a black line in a low luminance part for an image whose overall luminance value is increased due to the influence of flash light emission in the output of the memory processing circuit, , A luminance signal generation circuit that generates a luminance signal only when the input signal is RGB 3 channel, and whether or not a flash has occurred at the time of shooting with respect to the output of the luminance signal generation circuit, and by the flash emission in the generated image A portion where the luminance value increases rapidly is detected as a boundary line, and the memory processing circuit and the black line processing circuit are controlled.
[Selection] Figure 1

Description

  The present invention relates to an imaging signal processing circuit and an imaging signal processing method thereof that process an image captured by a television camera device using a rolling shutter type CMOS image sensor to eliminate the influence of flash emission.

  Television cameras used in broadcasting stations are becoming mainstream television cameras using CMOS (Complementary Metal Oxide Semiconductor) image sensors (hereinafter referred to as CMOS image sensors) instead of CCD (charge coupled device) image sensors. CMOS image sensors are classified into a rolling shutter system and a global shutter system depending on the electronic shutter. The rolling shutter method is an exposure method in which the exposure timing is different for each horizontal line, and the exposure is sequentially performed while resetting the exposure conditions and the like. The global shutter method is a method in which the exposure timing is processed simultaneously for all horizontal lines. In terms of cost, the rolling shutter system is inexpensive, whereas the global shutter system is expensive, so the rolling shutter system is generally used.

  In a captured image taken with a TV camera using a rolling shutter type CMOS image sensor, if an illumination component with a very short light emission period, such as a flash or strobe, is input to the captured image, flash emission occurs in a part of the screen. The portion exposed after the brightness increases due to the influence of the light and the portion exposed when not affected by the flash emission coexist in the same field image, and a band-like luminance difference may occur in the image. is there. Such a phenomenon is called a flash band, which is due to the sequential exposure method, and does not occur in the case of the global shutter system.

  FIG. 9 shows captured images in chronological order and illustrates the difference between the rolling shutter method and the global shutter method. When there is flash emission, the captured image is a flash band in the captured image in the case of the rolling shutter method. Is generated on two adjacent screens in time series, whereas in the case of the global shutter system, the brightness is increased by the flash emission and the entire image is affected, so that no flash band is generated.

  As is apparent from FIG. 9, when a flash band phenomenon caused by a high luminance portion due to flash emission occurs in the captured image, it becomes difficult to see the image, which is not preferable. Therefore, as disclosed in Japanese Patent Application Laid-Open No. 2009-284199, the applicant of the present application creates and records information about a flash together with image data captured by a television camera, and facilitates subsequent editing in a studio or the like. A TV camera with a recording medium capable of recording was proposed. A conventional television camera apparatus by the present applicant will be described with reference to FIG.

  The television camera device of FIG. 10 includes a video DSP (Digital Signal Processor) 2 that processes a video signal from the solid-state imaging device 1 and whether the output signal of the video DSP 2 includes a flash video of a still image camera. Is detected, a trigger signal indicating that there is a flash is generated, and a flash discrimination circuit 3 that outputs data relating to the flash, and output video data of the video DSP 2 based on the trigger signal from the flash discrimination circuit 3 The CPU 4 is configured to create data related to the flash, and record and control the output video data together with the output video data. The recording medium 5 records the output from the CPU 4. This can be done by work (see, for example, Patent Document 1).

JP 2009-284199 A

  However, when a rolling shutter type CMOS image sensor is used in a conventional television camera device, since it is a sequential exposure method with different exposure timing for each horizontal line, if a flash occurs during shooting, as shown in FIG. In addition, the field image becomes a high-intensity image in the middle of the image, and a phenomenon occurs in which the next field is affected and the image is returned to an image that is high-intensity until the middle and is not affected by the original flash emission. The image in the high luminance state due to the influence of the flash is exactly one field when the high luminance region of the field image where the flash emission is generated and the image of the high luminance region of the next field image are added. Thus, when there is a high luminance region straddling between field images, there is a problem that when viewed as a moving image, images having different luminances exist within one screen, and the user feels uncomfortable.

  Further, when a flash occurs, blackout occurs at a boundary portion where the high luminance image affected by the flash emission changes to the normal image that is not affected by the flash emission. Similarly, blackout occurs when the image changes suddenly from a high luminance image to a low luminance image. Since this blackout is set by setting the exposure time for each horizontal line and resetting it, it cannot cope with a sudden change in light intensity, resulting in insufficient light quantity and blackout. This blackout occurs when there is an extreme difference between the brightness values of the low-brightness area in the captured image and the surrounding high-brightness part, resulting in an image that looks like a black line, and the video feels uncomfortable. There was a problem. At present, when such a high-brightness captured image is acquired, the current situation is that the image is corrected for manual broadcasting and used for broadcasting, which is troublesome.

  The present invention has been made in view of the above-described problems, and does not give a sense of incongruity by performing image processing in a captured image signal processing circuit without performing manual editing operations on captured images due to the influence of flash or the like. An object of the present invention is to provide an imaging signal processing circuit capable of outputting an image and an imaging signal processing method thereof.

The present invention achieves the above-mentioned problems, and the invention according to claim 1 is an imaging signal processing circuit in an imaging apparatus.
A three-channel image memory for storing at least two screens of the latest image and the immediately preceding image of the input image signals from the RGB3 channel or the YPbPr3 channel in time series; Three memory processing circuits that perform replacement processing of images in the image memory only when a flash occurs during shooting from the output;
In the output of the memory processing circuit, a black line, which is a low-luminance portion near the boundary portion, is concealed with respect to an image region whose overall luminance value has increased due to the effect of flash emission 3 Two black line processing circuits,
An RGB3 channel or YPbPr3 channel imaging signal is input as an input signal. When the input signal is an RGB3 channel, a luminance signal is generated.
It is determined whether or not the flash has occurred at the time of photographing with respect to the output of the luminance signal generation circuit, and a portion where the luminance value rapidly increases due to flash emission in an image in which the flash has occurred is defined as a boundary line An imaging signal processing circuit comprising: a flash processing circuit that detects and provides control information to the memory processing circuit and the black line processing circuit.

According to a second aspect of the present invention, in the imaging signal processing circuit according to the first aspect,
The flash processing circuit, in the imaging signal of the same field,
A pixel integration unit that integrates luminance values of pixels for each horizontal line;
An average value calculation unit that integrates the average value of the pixel integration unit;
An average value memory for storing an average value for each line of one field;
An R / W control unit that performs read or write control on an arbitrary address of the average value memory;
A difference detection unit for detecting a difference between an output of the average value calculation unit and an average value of a horizontal line corresponding to a vertical position one field before read from the average value memory;
A comparison unit that compares the output of the difference detection unit with a predetermined reference value;
As a result of the comparison by the comparison unit, when the difference value is equal to or greater than the predetermined reference value, it is determined that flash is present, and the vertical line number at that time is obtained from the R / W control unit, and the flash trigger signal and the vertical line number are obtained. An imaging signal processing circuit comprising: a boundary line detection unit that outputs a boundary line.

According to a third aspect of the present invention, in the imaging signal processing circuit according to the first aspect,
The memory processing circuit includes:
An imaging signal processing circuit comprising: a memory transfer control unit that replaces memory contents according to the flash generation position of the at least two image memories based on a control signal indicating that flash is present by the flash processing circuit. is there.

According to a fourth aspect of the present invention, in the imaging signal processing circuit according to the first aspect,
The black line processing circuit includes an image memory for storing an output image of the memory processing circuit;
A position search process for detecting whether or not the black line is present in the image stored in the image memory, specifying the position in the case, and controlling the concealment processing unit to perform the concealment process And
A concealment processing unit that performs a process of concealing the black line by replacing with a pixel value calculated by pixel calculation of a predetermined region when the image stored in the image memory is a processing target image;
An imaging signal processing circuit comprising: an averaging filter unit that performs an averaging filter process on the concealment processing end image.

The invention of claim 5 is an imaging signal processing method in an imaging device,
Calculating the luminance component from the imaging signal of the RGB3 channel or the YPbPr3 channel by the luminance component calculating means;
The average value for each horizontal line in the same field of the luminance component is calculated and stored by an average luminance value calculation means, and the difference value between the average value and the average value for each horizontal line where the vertical position in the previous field is the same position Calculating by the difference processing means, comparing the difference value with a predetermined reference value by the comparison means, and determining by the flash determination means whether or not the imaging signal includes a luminance change due to flash emission. ,
Storing image signals of RGB3 channel or YPbPr3 channel, and performing image replacement processing of a part of the imaging signals by an image replacement processing means when the flash is included;
A step of concealing a black line existing at the boundary by a black line concealment processing unit with respect to a boundary part of the imaging signal subjected to the image replacement process;
An imaging signal processing method comprising: selectively applying an averaging filter processing means to the imaging signal subjected to the black line concealing process.

  According to the first aspect of the present invention, there is an advantage that an image having an uncomfortable feeling due to a flash band phenomenon due to flash emission can be automatically processed regardless of whether the image pickup signal is an RGB signal or a YPbPr signal. In addition, whether or not the flash is emitted in the captured image is determined by the luminance signal of the imaging signal, and when the RGB signal is input, the luminance signal generation circuit can automatically generate the luminance signal. In the case of the YPbPr signal, since the Y signal is a luminance signal, there is an advantage that the circuit configuration becomes compact. In addition, since the RGB signal or the YPbPr signal input is processed and output simultaneously for three channels, there is an effect that no delay occurs between the channels. In addition, the flash emission region straddling between each field image can be automatically aggregated into one field image, and the aggregated flash image can be automatically cut in editing, which is extremely effective. Further, even if the image pickup signal is an output from a rolling shutter type CMOS image pickup device, there is an advantage that an edit process can be easily performed by image processing when a flash band phenomenon occurs.

  According to the second aspect of the present invention, the flash memory is determined based on the average value of the luminance values for each horizontal line, so that the average value memory for each field can also be configured with a small capacity memory, which is compact and inexpensive. Is possible. Further, since the luminance value in the pixel in the specific area can be calculated by the same vertical line number and horizontal line number between the field images, there is an advantage that the processing time can be shortened.

  According to the invention of claim 3, the memory transfer control unit is controlled by the control signal from the flash processing circuit, and the data transfer is the same by three image memories connected corresponding to the three input signal channels. Since the areas can be transferred at the same timing, there is no need to individually set the transfer processing areas, and there is an advantage that the configuration is simple as a whole.

  Further, according to the invention of claim 4, since the black line that is blacked out is concealed by the average pixel value of the peripheral area, the black line is not clearly noticeable, and an uncomfortable feeling due to image replacement can be eliminated. is there. Since the averaging filter processing can be easily configured to be bypassable, it is possible to easily adjust the degree of replacement processing in the vicinity of the image replacement boundary due to the influence of the black line concealment processing, It is possible to reduce discomfort.

  The invention according to claim 5 is an imaging signal processing method in which an image captured by flash light emission is discriminated by a processor and editing processing can be performed by image processing. The basic processing concept is the same as that of a circuit. Even if an RGB signal or a YPbPr signal is input from a rolling shutter type CMOS image sensor, the type of the image signal is automatically determined to create a luminance signal, and whether or not the flash has emitted is determined by the luminance signal. The luminance value is used to determine the presence or absence of flash, the horizontal line number of the boundary area where the flash band occurs is transmitted as an information signal, and image replacement processing is executed. This is an RGB signal or YPbPr signal. However, there is an advantage that the editing process can be automatically performed without distinction.

It is a block diagram which shows one Example of the imaging signal processing circuit in the imaging device which concerns on this invention. It is a detailed block diagram of the flash processing circuit of the present embodiment. It is a detailed block diagram of the memory processing circuit of the present embodiment. It is a detailed block diagram of the black line processing circuit of a present Example. It is explanatory drawing of the memory processing circuit of a present Example. FIG. 5 is an explanatory diagram of a position search unit in the black line processing circuit of FIG. 4. It is explanatory drawing of the concealment process part in the black line processing circuit of FIG. It is a flowchart regarding the imaging signal processing method according to the present invention. It is operation | movement explanatory drawing regarding the shutter of a CMOS image sensor. It is a block diagram of the conventional television camera apparatus.

  Embodiments of an imaging signal processing circuit and an imaging signal processing method in an imaging apparatus according to the present invention will be described below with reference to the drawings. First, the imaging signal processing circuit of this embodiment will be described with reference to FIG. The imaging signal processing circuit of the present embodiment performs signal processing on an imaging signal from a CMOS imaging device provided in an imaging apparatus such as a television camera apparatus or a video camera apparatus, and generates a flash band phenomenon that occurs in a captured image due to the influence of flash emission. This is an image pickup signal processing circuit for solving the above problem, and is particularly suitable for an image pickup signal from a CMOS image pickup device of a rolling shutter system. In the present embodiment, signal processing is performed on an imaging signal based on an RGB signal or a YPbPr signal.

  In the figure, 100 is an R (Pr) channel input terminal, 101 is an R (Pr) channel memory processing circuit, 102 is an R (Pr) channel black line processing circuit, and 103 is a G (Y) channel input terminal. , 104 is a G (Y) channel memory processing circuit, 105 is a G (Y) channel black line processing circuit, 106 is a B (Pb) channel input terminal, 107 is a B (Pb) channel memory processing circuit, Reference numeral 108 denotes a B (Pb) channel black line processing circuit. 109 is a luminance signal generation circuit, 110 is a flash processing circuit, and 111 to 113 are output terminals for outputting R output (Pr), G output (Y), and B output (Pb) of each channel.

  An R (Pr) channel input terminal 100, a G (Y) channel input terminal 103, and a B (Pb) channel input terminal 106 are input terminals for image pickup signals. Each input terminal 100, 103, 106 is connected to a CMOS image pickup device. An imaging signal (RGB signal or YPbPr signal) is input and input to the R (Pr) channel memory processing circuit 101, the G (Y) channel memory processing circuit 104, and the B (Pb) channel memory processing circuit 107, respectively. And input to the luminance signal generation circuit 109. The output signals of the memory processing circuits 101, 104, and 107 are input to the black line processing circuits 102, 105, and 108, respectively. The output signal of the luminance signal generation circuit 109 is input to the flash processing circuit 110, and the output signals are input to the memory processing circuits 101, 104, 107 and the black line processing circuits 102, 105, 108, respectively. From the output terminals 111, 112, and 113, signal-processed imaging signals (RGB signals or YPbPr signals) are output.

  Next, the operation of this embodiment will be described in detail. Each image pickup signal from the CMOS image pickup device is input to the input terminals 100, 103, and 106, and the three image pickup signals of these RGB signals or YPbPr signals are used as memory processing circuits 101, 104, and 107 and luminance signal generation circuit 109 for each channel. And output respectively.

  The memory processing circuits 101, 104, and 107 store one field image based on each signal of the RGB signal, sequentially transfer it, and output it to the black line processing circuits 102, 105, and 108, respectively, and generate a flash in the stored one field image. If there is a flash image area, the flash image in this flash image area is transferred to the memory to form a flash image with one field image, and the one field image with this flash image is stored and output. A normal image area excluding a flash image in one field image is stored and output as a normal one field image in combination with the immediately preceding normal image. Details thereof will be described later with reference to FIGS. These edited outputs are output to the black line processing circuits 102, 105 and 108 in the same manner as the normal image.

  The black line processing circuits 102, 105, and 108 are circuits that conceal black output generated in the image and output the images created by the memory processing circuits 101, 104, and 107. Details thereof will be described with reference to FIG. The imaging signals black-line processed by the black line processing circuits 102, 105 and 108 are output as imaging signals (RBG signals) from the output terminals 111, 112 and 113 of the respective channels.

  The luminance signal generation circuit 109 generates a luminance signal (Y signal) based on the Y signal generation ratio in the case of the RGB signal input with respect to the RGB signal input or the YPbPr signal input from the input terminals 100, 103 and 106, This output is output to the flash processing circuit 110. The ratio of Y signal creation is, for example, R: G: B = 0.3: 0.59: 0.11 in the NTSC standard, and R: G: B = 0.21: 0.72 in the high-vision standard. : 0.07. As described above, when an RGB signal is input, a luminance signal (Y signal) is generated at a ratio determined by each standard. In the case of YPbPr signal input, the luminance signal generation circuit 109 outputs the Y signal of the input video signal as it is as a luminance signal.

  The flash processing circuit 110 determines whether or not flash light emission has occurred, and at the same time searches for the position that becomes the boundary line and outputs it to the memory processing circuits 101, 104, and 107 and the black line processing circuits 102, 105, and 108. The output of the flash processing circuit 110 is determination information indicating that there is a flash, and position information of a boundary position between an image area affected by flash emission and an image area not affected by flash emission in one field image. The boundary line position information is address information by a line number indicating the horizontal line position of one field image.

  The flash processing circuit 110 has the configuration shown in the functional block diagram of FIG. 2, and the output from the luminance signal generation circuit 109 is input to the input terminal 200 and the output from the input terminal 200 is input to the one-line pixel integration unit 201. Then, the luminance values of one line pixel are integrated and input to the average value calculation unit 202, and the average value of the luminance values of one line pixel is obtained. This average value corresponds to the address (line number) of one field image. And stored in the average value memory 204. The data stored in the average value memory 204 is read / written (R / W) control unit 203 by the vertical line address, and read / write control is performed for each line between adjacent field images in time series of field images. The difference value of the average value is calculated by the difference detection unit 205, and the difference value is input to the comparison unit 206 and compared with the reference value, and the boundary position is detected based on the comparison result. The information signal (horizontal line number) and the flash presence / absence information signal are detected by the unit 207 and output from the output terminal 208 to each of the memory processing circuits 101, 104, 107 and the black line processing circuits 102, 105, 108.

  Next, the operation of the flash processing circuit 110 will be described in detail. A Y signal (luminance signal) that is an output of the luminance signal generation circuit 109 is input to the input terminal 200 and input to the one-line pixel integration unit 201. The 1-line pixel integration unit 201 integrates the luminance values of the pixels on one horizontal line of the field image, and inputs them to the average value calculation unit 202. An average value calculation unit 202 calculates an average value of luminance values integrated for each line from the output of the one-line pixel integration unit 201. This average value is output to the average value memory 204 and the difference detection unit 205 together with the address of the vertical line. The average value memory 204 reads the average value for each line one field before while being read / written controlled by the R / W control unit 203, outputs the average value to the difference detection unit 205, and averages the output of the new average value calculation unit 202. Store in the value memory 204. The difference detection unit 205 detects the difference between the current field value and the previous field with respect to the average value for each line. This difference is compared by the comparator 206, and when it is equal to or greater than a predetermined reference value, it is determined that there is a flash. The output of the comparator 206 is input to the boundary detection unit 207 as an information signal determined to be flashed, detects the line number of the boundary line position between the area determined to be flashed and the normal area, The line number information and an information signal are output. This output is output to the memory processing circuits 101, 104 and 107 and the black line processing circuits 102, 105 and 108.

  Further, the comparator 206 will be described. Since the image that the comparator 206 has determined to have a flash is a high-intensity image, the difference amount is equal to or greater than a predetermined value compared to the previous field image of the normal image, thereby Can be determined. Also, the difference amount is equal to or greater than the predetermined reference value when the change in luminance is low → high and high → low in the comparison between the current field image and the previous image by the comparator 206. There is. In the sequential exposure method of the CMOS image sensor, the comparator 206 discriminates a change in luminance from low to high when the flash is present when the screen is changed from the upper side to the lower side. Once it is determined that there is a flash, it will be determined that the top of the screen is the same flash in the next field continuously, but if there are two flashes in succession, only if it is determined that there is a flash first. A control trigger for a series of image replacement. When the flash is present, the boundary line detection unit 207 obtains the line number from the R / W control unit 203, and outputs the flash line (flash trigger signal) and the vertical line number as the boundary line information signal. Output from 208. The boundary line is the boundary between the area affected by the flash emission and the area of the normal image not affected by the flash emission in the same field image. The line number of the horizontal line of the field image Output as.

  Next, the memory processing circuits 101, 104, and 107 will be described with reference to FIG. As shown in the block diagram of FIG. 3, the memory processing circuits 101, 104, and 107 include an input terminal 300, image memories 301 and 302 configured by a field memory, a memory transfer control unit 304, an output terminal 303, and And an input terminal 305. The image memories 301 and 302 are controlled by the memory transfer control unit 304, and images input to the image memory 301 are sequentially transferred to the image memory 302 and output to the black line processing circuit. The image memory 301 stores the latest field image, and the image memory 302 stores the immediately preceding field image. The memory transfer control unit 304 has a control function for sequentially transferring normal images without flash emission, and one field image with flash at the time of flash emission consists of an image area without flash emission and an image area with flash. In this case, it has a function of controlling the image exchange between the image area without flash emission and the image area with flash based on the information signal of flash presence / absence.

Hereinafter, the replacement control function in the memory transfer control unit 304 will be described with reference to FIG. First, the image replacement control will be briefly described with reference to FIG. In the figure, a field image F (T ₁ ) is a normal image without flash emission, and a field image F (T ₂ ) is an image with flash when the flash is emitted during imaging. The memory transfer control unit 304 applies the memory transfer control circuit 304 to the field image F (T ₂ ) stored in the image memory 301 based on the information signal from the flash processing circuit 110 (information signal indicating whether flash is present). An image G (T ₂ ) subjected to the exchange control is stored in the image memory 302 by performing a control process for exchanging the image areas. In FIG. 5, the position written horizontally by broken lines in each image indicates a luminance boundary line generated by flash emission, and the position of this luminance boundary line is a horizontal line along with an information signal indicating whether or not flashing is performed from the flash processing circuit 110. The line number information signal is added to the memory transfer control circuit 304.

Further, the replacement control process in the transfer control between the image memories 301 and 302 in the memory processing circuits 101, 104 and 107 will be described in detail with reference to FIG. Note that field images F (T ₁ ) to F (T ₄ ) in the figure are input images to the memory processing circuit 101 and represent time-series transitions in the image memory 301, and output images G (T _1). ) To G (T ₄ ) represent images transferred from the image memory 301 to the image memory 302 by time-series transitions, and indicate field images of the image memories 301 and 302 at timings T _{1 to} T ₄ . .

When the input image F (T ₁ ) is read into the image memory 301 at timing T ₁ , the flash processing circuit 110 outputs an information signal indicating no flash to the memory processing circuits 101, 104, and 107 for memory transfer. The control unit 304 transfers the input image F (T ₁ ) as it is to the image memory 302 based on the information signal, and the content of the image memory 302 becomes the output image G (T ₁ ). The image memory 302 is an output image G (T ₁ ) without flash emission, and the output image G (T ₁ ) is output to the next black line processing circuit based on the memory transfer control unit 304. For convenience, the input image F (T ₁ ) of the image memory 301 at the timing T ₁ is represented as normal images f ₁ (T ₁ ) and f ₂ (T ₁ ), and the output image G (T ₁ ) of the image memory 302 is represented by Images f ₁ (T ₁ ) and f ₂ (T ₁ ) are shown.

Subsequently, at timing T ₂ , a new image is transferred and read into the image memory 301, and this image is an input image F (T ₂ ) with flash. The input image F (T ₁ ) of the image memory 301 constitutes a one-field image from the normal image f ₁ (T ₂ ) and the flash image f ₂ (T ₂ ). Input image while the input image F of the image memory 301 to dwell in timing _{T 1 (T 1)} is an input image without flash F _{(T 1),} the image of the image memory 301 to dwell in the timing _{T 2} are flush There Since it is F (T ₂ ), the flash processing circuit 110 provides the information signal with flash to the memory transfer control unit 304, and at the timing T ₂ , the memory transfer control unit 304 replaces the memory based on this information signal. Control processing is started. In the memory replacement control at timing T ₂ , the image transfer from the image memory 301 to the image memory 302 is such that only the normal image f ₁ (T ₂ ) is transferred from the input image F (T ₂ ) of the image memory 301 to the image memory 302. The Therefore, the image memory 302 includes an output image G (T ₂ ) that is a combination of the image f ₁ (T ₂ ) transferred from the image memory 301 and the normal image f ₂ (T ₁ ) that is an image at the timing T _1. Become. The output image G (T ₂ ) of the image memory 302 is output to the black line processing circuit at the next stage under the control of the memory transfer control unit 304. Immediately thereafter, only f ₂ (T ₂ ) of the image memory 301 is transferred to the image memory 302.

At the next timing T ₃ , the input image F (T ₃ ) is transferred and read into the image memory 301, and the input image F (T ₃ ) is a flash image f ₁ (T ₃ ) that is affected by flash light emission. And a normal image f ₂ (T ₃ ) that is not affected. Since the memory transfer control unit 304 has already received the information signal with flash for the image of the input image F (T ₂ ) at the timing T ₂ , the flash image f ₁ (T ₁₎ of the image memory 301 at the timing T _{3. 3} ) Only the image memory 302 is transferred. The image memory 302 becomes an output image G (T ₃ ) based on the flash image f ₁ (T ₃ ) and the flash image f ₂ (T ₂ ), and is output to the next stage.

Subsequently, at timing T ₄ , a new input image F (T ₄ ) without flash emission is read into the image memory 301, and the flash processing circuit 110 determines that there is no flash for this input image F (T ₄ ). The memory transfer control unit 304 acquires this information signal and transfers the input image F (T ₄ ) of the image memory 301 to the image memory 302 as it is. The image memory 302 outputs the output image G (T ₄ ) is read. In this way, when there is no flash emission, it becomes a simple field image memory, and the contents of the image memory are directly output to the next stage without performing the above-described image replacement processing.

  Next, details of the black line processing circuits 102, 105 and 108 will be described with reference to FIG. In the figure, reference numeral 400 denotes an input terminal that receives an output from the memory processing circuit, 401 denotes an input terminal that receives an output from the flash processing circuit, 402 denotes an image memory, 403 denotes an averaging filter unit (with bypass), and 404 denotes an input terminal. A position search processing unit, 405 is a concealment processing unit, and 406 is an output terminal.

The operation of the black line processing circuit 405 will be described. The input terminal 400 inputs the output from the memory processing circuits 101, 104, and 107, and the input terminal 401 inputs the output from the flash processing circuit 110. The image memory 402 stores one field for concealment processing. In FIG. 5, the output image G (T ₂ ) or G (T ₃ ) is an image to be subjected to the concealment process. The position search processing unit 404 acquires the flash presence signal and the line number (address) of the boundary line as information signals from the flash processing circuit 110.

The position search processing unit 404 acquires, for example, the information signal of the flash presence signal and the line number (address) 200 of the horizontal line of the boundary line. Based on the information signal, the position search processing unit 404 performs transfer control so that the image subjected to the replacement control process is stored in the image memory 402 based on the information signal, and then the black line (low luminance portion) is generated from the image. Search for a location. The image subjected to the image replacement control process is an image that has been subjected to the image replacement process by the memory processing circuit at the boundary of the flash emission with the flash as being present, and a black line (low brightness portion) is formed on the boundary line. Position is likely to occur. The image subjected to the image replacement process is a high-intensity image of the output image G (T ₃ ) in FIG. The position search processing unit 404 obtains the boundary line number 200 based on the information signal, and then makes a determination with an arbitrary vertical line centering on the boundary line number of the image stored in the image memory 402. For example, the black line is determined by the upper and lower three lines. That is, since the boundary line number is 200, the position search processing unit 404 assigns the line number of the pixel having the minimum luminance value of the vertical pixel to the pixels included in the 197 to 203 lines in the horizontal direction. Search from the luminance values of all the pixels.

  Next, the position search processing unit 404 will be described with reference to FIG. FIG. 6 shows the main part of the field image of the image memory 402. For example, the luminance value of the pixel in the vertical direction at the tenth pixel in the horizontal direction in FIG. , 240, 245, 250, 30, 235, and 240, it is possible to search for the line number 201 of 30 values having the smallest luminance value. This process is performed for all the pixels in the horizontal direction to perform statistical processing, and the line number having the largest number of minimum pixels is determined as the black line generation area (black line position). The position search processing unit 404 outputs the line number 201 indicated by hatching in FIG. 6 to the concealment processing unit 405 as a black line position, and the concealment processing unit 405 controls to conceal this black line occurrence area.

  Next, the concealment processing unit 405 will be described with reference to FIG. FIG. 7 shows an area necessary for the black line concealment processing, and FIG. As shown in FIG. 7, the black line concealment processing area is, for example, the upper 3 lines and the lower 3 lines in the vertical direction around the line number 201 of the black line generation area (black line position), and the black line generation area A range of 5 pixels in the horizontal direction centering on the processing target pixel is defined as a black line concealment processing region. A local calculation area is defined for all pixels in the black line concealment processing area. In the black line concealment process, the local calculation center of the local calculation area and the local calculation area of an arbitrary size are set as shown in FIG. After setting the local calculation area, the average value of the pixels in the local calculation area is calculated as the pixel value of the pixel at the local calculation center. For each of the averaged pixel values, the average value of the pixel values corresponding to the upper ３ with respect to the luminance value is replaced with the replacement pixel value of the black line region. Alternatively, in the black line concealment process, the pixel value of the pixel having the maximum luminance value in the local calculation area may be replaced with the pixel value of the black line area. The range of the black line concealment processing area may be arbitrary, but it is assumed that the range does not exceed the black line concealment processing area of the local calculation area. After the replacement process is completed, the position search process 404 is notified that the process has been completed and searches for other black lines. For the search end, for example, when a predetermined number of times is repeated as a condition, when the number of pixels counted as the minimum luminance value for each line is compared, or the minimum count number and the count number of other lines The search is terminated when the ratio of the average value to or below becomes a certain value or less.

The averaging filter 403 applies, for example, a 3 × 3 averaging filter to the high-brightness image (the output image G (T ₃ ) in FIG. 5) after the black line concealing process is completed. The processing target pixel is replaced with the average value of other pixels with the center of the 3 × 3 region. This is because when the brightness is increased, the image after replacement is the average of the boundary line portion due to the combination of different field images due to the replacement processing and the black line concealment processing. A false image is generated mainly due to the generation process. This is to make this false image inconspicuous. It should be noted that the size of the processing target pixel range of the averaging filter can be arbitrarily set, and the averaging filter processing can be selectively executed.

  Next, the control flow of an embodiment of the imaging signal processing method of the present invention will be described with reference to the flowchart of FIG. The control flow of this embodiment is for processing the black line processing circuits 102, 105, 108, the luminance signal generation circuit 109, and the flash processing circuit 110 of the imaging signal processing apparatus of FIG. 1 using an arithmetic processing unit (processor). In this embodiment, an image pickup signal is obtained from a CMOS image pickup device, and an RGB three-channel image pickup signal output from each CMOS image pickup device will be described as an example. The arithmetic processing unit (processor) includes a luminance signal creating unit, an average luminance value calculating unit, a difference processing unit, an average luminance value storing unit, a flash presence / absence determining unit using a comparing unit, and an image for configuring the control flow of the imaging signal processing method Each control function includes a replacement processing unit, a black line concealing processing unit, and an averaging filter processing unit.

  An RGB three-channel imaging signal is input to the processor as an input signal. The control flow by this processor consists of step 80 (S80) to step 87 (S87). First, in step 80 (S80), after the standard of the imaging signal is recognized by the luminance signal creating means, the luminance component is calculated for each pixel from each inputted imaging signal in the case of RGB3 channel. The formula for calculating the luminance component is based on the luminance signal generation circuit (luminance signal generation means) 109 described above. When the input signal is a YPbPr3 channel imaging signal, the Y signal is output as it is as the luminance component without performing the arithmetic processing by the luminance signal creating means. Then, it progresses to step S81.

  In step 81 (S81), the average luminance value calculating means calculates the average value of the luminance values for each horizontal line as the average luminance value for the luminance component for each pixel calculated in step 80. In the average luminance value calculating means, the average luminance value is calculated by integrating the luminance values of the pixels for each horizontal line and dividing the integrated luminance value by the number of pixels constituting one horizontal line. . Then, it progresses to step S82.

  In step 82 (S82), the difference processing means calculates the difference from the average value of the luminance values for each horizontal line calculated in step 81 so that the position (address) of the vertical line between adjacent field images in time series is the same. The position is calculated, and the process proceeds to step 83 (S83). In step 81, the average value of the luminance values of each horizontal line is stored for each field image. Therefore, in step 82 (S82), the average value for each horizontal line of the previous field image is read and each horizontal line is read. The difference calculation is performed.

  In step 83 (S83), the average luminance value storage means stores the average luminance value for each horizontal line calculated in step 81.

  In step 84 (S84), the flash presence / absence determining means compares the difference value of the average value of the luminance values for each horizontal line between the field images calculated in step 82 with a predetermined reference value to determine whether or not there is flash. judge. If there is no flash, this step is terminated without performing the image replacement process. If there is a flash, the process proceeds to step 85 (S85), and as described with reference to FIG. Is executed, the image data storage area is updated, and the process proceeds to step 86 (S86).

  In step 86 (S86), the black line concealment processing means executes the black line concealment process for the blackened border line portion, and performs the concealment process described with reference to FIGS. Then, it progresses to step 87 (S87).

  In step 87 (S87), the averaging filter processing means executes the averaging filter processing, performs the averaging filter processing on the image after the black line concealment processing, and ends. This averaging filter process can be skipped for each field image. By executing this step, the image after the black line concealment process is prevented from being uncomfortable, and the image-processed image signals of each RGB3 channel or YPbPr3 channel are output to the next stage.

  The above is the processing flow of an embodiment of the imaging signal processing method according to the present invention. This processing flow can be implemented in various modifications. For example, in this embodiment, the image memory is provided so as to correspond to field images adjacent in time series, but a large-capacity memory is used. Can be used by dividing the memory, and is not limited to this embodiment.

  INDUSTRIAL APPLICABILITY The present invention is useful when editing an image captured by a broadcast television camera device using a rolling shutter type CMOS image sensor.

100 R (Pr) channel input terminal 101 R (Pr) channel memory processing circuit 102 R (Pr) channel black line processing circuit 103 G (Y) channel input terminal 104 G (Y) channel memory processing circuit 105 G ( Y) Channel black line processing circuit 106 B (Pb) channel input terminal 107 B (Pb) channel memory processing circuit 108 B (Pb) channel black line processing circuit 109 Luminance signal generation circuit 110 Flash processing circuit 111 to 113 channels Output terminal 200 input terminal 201 1-line pixel integration unit 202 average value calculation unit 203 R / W control unit 204 average value memory 205 difference detection unit 206 comparison unit 207 boundary line detection unit 208 output terminals 300 and 305 input terminals 301 and 302 Image memory 303 Output terminal 04 memory transfer control unit 400 memory processing circuit 401 flush processing circuit 402 image memory 403 averaging filter unit 404 position search processing unit 405 concealment processing unit 406 output terminal

Claims (5)

In the imaging signal processing circuit in the imaging device,
A three-channel image memory for storing at least two screens of the latest image and the immediately preceding image of the input image signals from the RGB3 channel or the YPbPr3 channel in time series; Three memory processing circuits that perform replacement processing of images in the image memory only when a flash occurs during shooting from the output;
In the output of the memory processing circuit, a black line, which is a low-luminance portion near the boundary portion, is concealed with respect to an image region whose overall luminance value has increased due to the effect of flash emission 3 Two black line processing circuits,
An RGB3 channel or YPbPr3 channel imaging signal is input as an input signal. If the input signal is an RGB3 channel, a luminance signal is generated. If the input signal is a YPbPr3 channel, a luminance signal generation circuit using the Y signal as a luminance signal;
It is determined whether or not the flash has occurred at the time of photographing with respect to the output of the luminance signal generation circuit, and a portion where the luminance value rapidly increases due to flash emission in an image in which the flash has occurred is defined as a boundary line An imaging signal processing circuit comprising: a flash processing circuit that detects and provides control information to the memory processing circuit and the black line processing circuit. The imaging signal processing circuit according to claim 1,
The flash processing circuit, in the imaging signal of the same field,
A pixel integration unit that integrates luminance values of pixels for each horizontal line;
An average value calculation unit that integrates the average value of the pixel integration unit;
An average value memory for storing an average value for each line of one field;
An R / W control unit that performs read or write control on an arbitrary address of the average value memory;
A difference detection unit for detecting a difference between an output of the average value calculation unit and an average value of a horizontal line corresponding to a vertical position one field before read from the average value memory;
A comparison unit that compares the output of the difference detection unit with a predetermined reference value;
As a result of the comparison by the comparison unit, when the difference value is equal to or greater than the predetermined reference value, it is determined that flash is present, and the vertical line number at that time is obtained from the R / W control unit, and the flash trigger signal and the vertical line number are obtained. An imaging signal processing circuit comprising: a boundary line detection unit that outputs a boundary line. The imaging signal processing circuit according to claim 1,
The memory processing circuit includes:
An imaging signal processing circuit comprising: a memory transfer control unit that replaces memory contents corresponding to the flash generation positions of the at least two image memories based on a control signal indicating that flashing is performed by the flash processing circuit. The imaging signal processing circuit according to claim 1,
The black line processing circuit includes an image memory for storing an output image of the memory processing circuit;
A position search process for detecting whether or not the black line is present in the image stored in the image memory, specifying the position in the case, and controlling the concealment processing unit to perform the concealment process And
A concealment processing unit that performs a process of concealing the black line by replacing with a pixel value calculated by pixel calculation of a predetermined region when the image stored in the image memory is a processing target image;
An imaging signal processing circuit comprising: an averaging filter unit that performs an averaging filter process on the concealment processing end image. In an imaging signal processing method in an imaging device,
Calculating the luminance component from the imaging signal of the RGB3 channel or the YPbPr3 channel by the luminance component calculating means;
The average value for each horizontal line in the same field of the luminance component is calculated and stored by an average luminance value calculation means, and the difference value between the average value and the average value for each horizontal line where the vertical position in the previous field is the same position Calculating by the difference processing means, comparing the difference value with a predetermined reference value by the comparison means, and determining by the flash determination means whether or not the imaging signal includes a luminance change due to flash emission. ,
Storing image signals of RGB3 channel or YPbPr3 channel, and performing image replacement processing of a part of the imaging signals by an image replacement processing means when the flash is included;
A step of concealing a black line existing at the boundary by a black line concealment processing unit with respect to a boundary part of the imaging signal subjected to the image replacement process;
An imaging signal processing method comprising: selectively applying an averaging filter processing means to the imaging signal subjected to the black line concealing process.

Images