JP2017011676A - Image processing apparatus and image processing method - Google Patents

Abstract

An object of the present invention is to provide a technique capable of easily obtaining an HDR synthesized moving image in which the degree of blur of the movement of a subject (a feeling of focus on the subject) is favorable.
An image processing apparatus according to the present invention acquires first moving image data including a first frame obtained by imaging and a second frame obtained by imaging with an exposure time shorter than the first frame. 1 acquisition means, second acquisition means for acquiring reproduction information that is information relating to the reproduction speed of the moving image data, and synthesis means for combining the first frame and the second frame based on the reproduction information; And the synthesizing unit changes a synthesis ratio of the first frame and the second frame in accordance with a reproduction speed of the moving image data.
[Selection] Figure 8

Description

  The present invention relates to an image processing apparatus and an image processing method.

  By combining a plurality of images including an image (first frame) obtained by imaging with a relatively long exposure time and an image (second frame) obtained by imaging with a relatively short exposure time, There is a technique for generating an image with a high dynamic range. Such a technique (synthesis) is called “high dynamic range synthesis (HDR synthesis)”, and an image generated by HDR synthesis is called “high dynamic range synthesis image (HDR synthesized image)”. By performing HDR composition, it is possible to generate an HDR composite image in which occurrence of overshoot of received light data due to exposure time is suppressed. That is, it is possible to generate an HDR composite image in which occurrence of overexposure and underexposure is suppressed.

  In recent years, it has become possible to obtain a moving image (HDR combined moving image) in which each frame is an HDR combined image as a captured moving image (moving image obtained by imaging) by performing HDR synthesis in real time during imaging. FIG. 16 shows a state where an HDR synthesized moving image is generated. In the example of FIG. 16, the first frame and the second frame are alternately obtained by imaging, and HDR is obtained by synthesizing the obtained first frame and second frame each time the first frame and the second frame are obtained. A composite image is generated. Thereby, an HDR synthetic moving image is generated.

  In the generation of an HDR video, not only the dynamic range but also the degree of blur of subject movement (subject focus feeling) can be changed by changing the weights of the first frame and the second frame in HDR synthesis. FIG. 17A shows an example of a first frame obtained by imaging a black circular object moving in the horizontal direction. In the first frame, since the exposure time is long, the gradation information of the dark part can be held in detail, but the motion blur is large. FIG. 17B shows an example of a second frame obtained by imaging a black circular object moving in the horizontal direction. In the second frame, since the exposure time is short, the gradation information of the bright part can be held in detail and the motion blur is small. FIGS. 17C and 17D show examples of HDR synthesized images obtained by synthesizing the first frame of FIG. 17A and the second frame of FIG. 17B. FIG. 17C shows an example in which the weight of the first frame is large and the weight of the second frame is small. In this case, as shown in FIG. 17C, an HDR image in which motion blur remains is generated. FIG. 17D shows an example in which the weight of the first frame is small and the weight of the second frame is large. In this case, as shown in FIG. 17D, an HDR image with less motion blur than in FIG. 17C is generated. In general, in reproducing a moving image at a normal reproduction speed, an image (for example, an image as shown in FIG. 17C) in which motion blur remains to some extent is used to express natural motion. Is desirable. On the other hand, it is desirable to use an image with a high focus feeling (for example, an image as shown in FIG. 17D) for moving image playback (slow playback or pause playback) at a playback speed slower than the normal playback speed. .

  In recent years, the size of image sensors in video cameras has increased, and it has become common to mount abundant lens groups prepared for single-lens reflex cameras on video cameras. From such a background, a workflow of acquiring a high-resolution moving image by imaging and extracting one frame of the moving image as a still image has attracted attention. Hereinafter, such a workflow is referred to as a “motion capture workflow”. In the motion capture workflow, for example, a high-resolution moving image for CM (Commercial Message) or PV (Promotion Video) is acquired by imaging, and one frame of the high-resolution moving image is extracted as a still image for a magazine or advertisement.

  FIG. 18 is a diagram illustrating an example of a configuration of an image display system used in the motion capture workflow. The image display system in FIG. 18 includes an imaging device 1100, a playback device 1200, a developing device 1300, and an image display device 1400. The imaging device 1100 generates moving image data by imaging and outputs the generated moving image data. The imaging apparatus 1100 outputs RAW moving image data in which the image data of each frame is RAW image data as moving image data. The playback device 1200 stores the RAW moving image data output from the imaging device 1100. Further, the playback device 1200 outputs the stored RAW moving image data. The developing device 1300 performs development processing on the RAW moving image data output from the reproducing device 1200, and outputs the moving image data after the developing processing. The image display device 1400 displays a moving image based on the moving image data output from the developing device 1300.

  In the image display system of FIG. 18, first, RAW moving image data obtained by the imaging device 1100 is accumulated in the playback device 1200. Thereafter, the RAW moving image data stored in the playback device 1200 is developed by the developing device 1300, and the development result is displayed by the image display device 1400. A user (for example, a cameraman) checks a captured moving image, selects a still image, and the like by looking at the display on the image display device 1400.

  FIG. 19 is a flowchart illustrating an example of a motion capture workflow. Here, an operation from a state in which a captured moving image is normally reproduced will be described. Normal playback is a playback method in which a moving image is played back at a normal playback speed.

  First, the user confirms the display of the image display device 1400 and searches for an image group (scene) that can be used as a still image (S1001; confirmation work). Then, the user performs a user operation (pause operation) for pause reproduction from a frame in the vicinity of the searched image group, and repeatedly performs a user operation (frame advance operation) for frame advance (S1002). Thereby, a frame in the vicinity of the searched image group is displayed, and the display is switched to the next frame every time a frame advance operation is performed. Next, the user confirms the display of the image display device 1400 while performing a frame advance operation, and selects a frame to be used as a still image (S1003; selection work). In general, in addition to the composition of an image, a frame is selected in consideration of the presence or absence of a focus feeling. For example, a frame that has a necessary and sufficient focus feeling for use in a magazine or an advertisement is selected. Then, the user performs a user operation (extraction operation) for extracting the RAW image data of the selected frame from the playback apparatus 1200 as still image data (S1004). Next, the user determines whether or not to end the motion capture workflow (S1005). If the motion capture workflow is to be terminated, this flowchart is terminated. If the motion capture workflow is not to be terminated, the operation returns to S1002, and the frame advance operation is continued. If the motion capture workflow is not terminated, the work may be returned to S1001. Note that the RAW image data is very suitable for image editing. By extracting the RAW image data as the still image data, it is possible to suitably perform an editing operation such as retouching according to the use such as a magazine or an advertisement.

  By the way, when generating an HDR composite video in the motion capture workflow, it is preferable that the HDR composite video is generated using weights suitable for playback of a video at a normal playback speed in the confirmation operation in S1001. However, when such weights are used, a display with insufficient focus is often performed in the selection operation in S1003. It is assumed that the weight is adjusted by the user every time such display is performed.

In the motion capture workflow, it takes a very long time to select (select) a suitable image from a large number of images (frames), and the burden on the user is very large. And if adjustment of a weight is needed in selection work, work time and a user's burden will increase further.

  Conventional techniques related to HDR synthesis are disclosed in, for example, Patent Documents 1 and 2. In the technique disclosed in Patent Document 1, the weights of the long-time exposure image and the short-time exposure image are adjusted based on the presence or absence of the movement of the subject in the HDR synthesized moving image. In the technique disclosed in Patent Document 2, the number of images used for HDR composition and the exposure conditions for images used for HDR composition are adjusted according to a user operation.

  However, in the techniques disclosed in Patent Documents 1 and 2, a weight is uniquely determined for one frame of the HDR synthesized moving image. If the weight is uniquely determined for one frame, an unnatural display with small motion blur is performed during the confirmation operation in S1001, or a display with insufficient focus is performed during the selection operation in S1003. Therefore, it is preferable to use different weights for the same frame between the confirmation work and the selection work. Even if the techniques disclosed in Patent Documents 1 and 2 are used, such processing cannot be realized, and a user operation for adjusting the weight is required.

JP2012-049883A JP 2012-175277 A

  An object of the present invention is to provide a technique capable of easily obtaining a synthesized moving image in which the degree of blur of a subject's movement (the subject's focus feeling) is suitable.

The first aspect of the present invention is:
First acquisition means for acquiring moving image data including a first frame obtained by imaging and a second frame obtained by imaging with an exposure time shorter than the first frame;
Second acquisition means for acquiring reproduction information that is information relating to the reproduction speed of the moving image data;
Combining means for combining the first frame and the second frame based on the reproduction information;
Have
The synthesizing unit is an image processing apparatus that changes a synthesis ratio of the first frame and the second frame in accordance with a reproduction speed of the moving image data.

The second aspect of the present invention is:
A first acquisition step of acquiring moving image data including a first frame obtained by imaging and a second frame obtained by imaging with an exposure time shorter than the first frame;
A second acquisition step of acquiring reproduction information which is information relating to the reproduction speed of the moving image data;
A combining step of combining the first frame and the second frame based on the reproduction information;
Have
In the composition step, the composition ratio of the first frame and the second frame is changed according to the reproduction speed of the moving image data.

  A third aspect of the present invention is a program that causes a computer to execute each step of the above-described image processing method.

  According to the present invention, it is possible to easily obtain a synthesized moving image in which the degree of blur of the subject movement (the subject's focus feeling) is suitable.

1 is a diagram illustrating an example of a configuration of an image display system according to a first embodiment. 1 is a block diagram illustrating an example of a functional configuration of an imaging apparatus according to a first embodiment. The figure which shows an example of the RAW-HDR moving image data which concerns on Example 1. FIG. FIG. 1 is a block diagram illustrating an example of a functional configuration of a playback device according to a first embodiment. The figure which shows an example of the raw-RAW moving image data for reproduction | regeneration which concerns on Example 1. FIG. FIG. 1 is a block diagram illustrating an example of a functional configuration of a developing device according to a first embodiment. FIG. 1 is a block diagram illustrating an example of a functional configuration of an image display device according to a first embodiment. 7 is a flowchart illustrating an example of the operation of the image display apparatus according to the first embodiment. FIG. 10 is a diagram illustrating an example of a display image according to the first embodiment. FIG. 7 is a block diagram illustrating an example of a functional configuration of an image display apparatus according to a second embodiment. 10 is a flowchart illustrating an example of the operation of the image display apparatus according to the second embodiment. The figure which shows an example of the histogram data which concern on Example 2. FIG. 9 is a block diagram illustrating an example of a functional configuration of an image display apparatus according to a third embodiment. FIG. 10 is a diagram illustrating an example of a display image according to the third embodiment. FIG. 9 is a block diagram illustrating an example of a functional configuration of an image display apparatus according to a fourth embodiment. The figure which shows an example of a mode that an HDR synthetic | combination animation is produced | generated. Figure showing an example of an image The figure which shows an example of the image display system Flow chart showing an example of a motion capture workflow

<Example 1>
Hereinafter, an image processing apparatus and an image processing method according to Embodiment 1 of the present invention will be described. Hereinafter, an example in which the image processing apparatus according to the present embodiment is provided in the image display apparatus will be described. However, the image processing apparatus according to the present embodiment may be a separate apparatus from the image display apparatus. Good. The image processing apparatus according to the present embodiment may be provided in an apparatus different from the image display apparatus.

  FIG. 1 is a diagram illustrating an example of the configuration of the image display system according to the present embodiment. The image display system of FIG. 1 is used in, for example, a motion capture workflow. As shown in FIG. 1, the image display system according to the present embodiment includes an imaging device 100, a playback device 200, a developing device 300, and an image display device 400.

  The imaging device 100 generates moving image data by imaging and outputs the generated moving image data. In this embodiment, the imaging apparatus 100 outputs RAW moving image data in which the image data of each frame is RAW image data as moving image data. The playback device 200 records the RAW moving image data output from the imaging device 100 in the storage unit. In addition, the playback device 200 reads RAW video data from the storage unit, generates RAW video data based on the set playback method from the read RAW video data, and outputs the generated RAW video data. The developing device 300 performs a developing process on the RAW moving image data output from the reproducing device 200, and outputs the moving image data after the developing process. The image display device 400 displays a moving image based on the moving image data output from the developing device 300.

  Note that two or more of the imaging device 100, the reproduction device 200, the developing device 300, and the image display device 400 may be integrated. For example, the reproducing device 200 and the developing device 300 may be integrated, the imaging device 100, the reproducing device 200, and the developing device 300 may be integrated, or the developing device 300 and the image display device 400. May be integrated.

  Note that when the imaging apparatus 100 outputs moving image data, the imaging apparatus 100 may perform development processing. When the playback device 200 records moving image data in the storage unit, the playback device 200 may perform development processing. When the playback device 200 generates moving image data based on the set playback method, the playback device 200 may perform development processing. When the playback device 200 outputs moving image data, the playback device 200 may perform development processing. When the developing process is performed by the imaging device 100 or the reproduction device 200, the developing device 300 is not necessary. In this case, it can also be said that “the developing device 300 is provided in the imaging device 100 or the reproduction device 200”.

  FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging apparatus 100. As illustrated in FIG. 2, the imaging apparatus 100 includes at least a lens unit 101, an imaging unit 102, a signal processing unit 103, an output unit 104, a user I / F unit 105, and a control unit 106. These functional units are connected to each other by an internal bus 107. Data transmission / reception between these functional units is performed using the internal bus 107. The operation of each functional unit is controlled by the control unit 106.

  The lens unit 101 is installed in front of the imaging unit 102 and forms incident light, which is light from an imaging target (subject), on the imaging unit 102.

  The imaging unit 102 performs an exposure change process for changing the exposure time of incident light imaged by the lens unit 101, converts the incident light into an analog electric signal (imaging), and outputs the obtained analog electric signal. In the present embodiment, the imaging unit 102 alternately performs the first imaging and the second imaging with an exposure time shorter than that of the first imaging. Then, the analog electrical signal obtained by the first imaging and the analog electrical signal obtained by the second imaging are alternately output from the imaging unit 102.

  The signal processing unit 103 performs signal processing on the analog electrical signal output from the imaging unit 102, thereby converting the analog electrical signal output from the imaging unit 102 into RAW image data that is a digital electrical signal. Then, the signal processing unit 103 outputs RAW image data. The signal processing includes A / D conversion processing for converting an analog electrical signal into a digital electrical signal, correction processing for performing various corrections, and the like. In the correction processing, errors due to defects in the image sensor (CCD, CMOS, etc.) included in the imaging unit 102, errors due to optical variations of the lenses included in the lens unit 101, and the like are reduced.

  As described above, the imaging unit 102 alternately outputs the analog electrical signal obtained by the first imaging and the analog electrical signal obtained by the second imaging. Therefore, RAW-HDR moving image data is output from the signal processing unit 103. The RAW-HDR moving image data is moving image data having RAW image data as image data of each frame. The RAW-HDR moving image data is also moving image data in which a frame pair including the first frame obtained by the first imaging and the second frame obtained by the second imaging is repeated.

The output unit 104 outputs the RAW-HDR moving image data output from the signal processing unit 103 to the playback device 200 at a predetermined frame rate. When outputting the RAW-HDR moving image data, the output unit 104 adds metadata including HDR meta information, which is information about the frame, to each frame (each RAW image data) of the RAW-HDR moving image data. In this embodiment, the HDR meta information includes HDR identification information (moving picture identification information), pair identification information, frame identification information, and exposure time information.

  In the HDR identification information, the moving image data output from the output unit 104 is HDR moving image data (a frame pair including a first frame obtained by the first imaging and a second frame obtained by the second imaging is repeated. This is information indicating whether or not it is moving image data. In this embodiment, since the RAW-HDR moving image data is output from the imaging apparatus 100, the HDR identification information can also be said to be “information indicating that the moving image data output from the output unit 104 is RAW-HDR moving image data”. . The HDR identification information may or may not be added to all frames of moving image data. For example, the HDR identification information may be added only to the first frame of the moving image data. The first frame can be detected, for example, according to a user operation that instructs the start of imaging.

  The pair identification information is information for identifying a frame pair of HDR moving image data (RAW-HDR moving image data) output from the output unit 104. For example, the same identifier is added as pair identification information to each of the first frame and the second frame included in one frame pair. When such an identifier is used, pair identification information is added to all frames of the HDR video data. Information indicating a second frame included in the same frame pair as the first frame may be added to the first frame as pair identification information. Information indicating the first frame included in the same frame pair as the second frame may be added to the second frame as pair identification information. When such information is used, pair identification information is added to at least one of the first frame and the second frame of the HDR video data.

  Frame identification information and exposure time information are added to all frames of the HDR video data. The frame identification information is information for identifying the first frame and the second frame of the HDR moving image data (RAW-HDR moving image data) output from the output unit 104. Specifically, the frame identification information is information indicating whether the corresponding frame is the first frame or the second frame. The exposure time information is information indicating the exposure time of each frame of the moving image data output from the output unit 104. Specifically, the exposure time information is information indicating the exposure time of the corresponding frame.

  The HDR meta information may include information other than HDR identification information, pair identification information, frame identification information, and exposure time information. For example, the HDR meta information may include information such as a time code.

FIG. 3 is a diagram illustrating an example of RAW-HDR moving image data output from the imaging apparatus 100. As shown in FIG. 3, in the RAW-HDR moving image data, the first frame and the second frame are arranged alternately. Also, HDR meta information is added to each frame. The imaging device 100 (output unit 104) and the playback device 200 are connected to each other by wire or wirelessly. As a cable for connecting the imaging device 100 and the playback device 200 to each other, for example, an SDI (Serial Digital Interface) cable, an HDMI (High-Definition Multimedia Interface) cable, or the like is used. When the imaging apparatus 100 is connected to the playback apparatus 200 using an SDI cable, the HDR meta information is added to an ancillary area, which is a vertical blanking area, for example. Note that when the imaging device 100 and the playback device 200 are integrated, the RAW-HDR moving image data may be transmitted from the output unit 104 to the playback device 200 via an internal bus.

  The user I / F unit 105 receives a user operation on the imaging apparatus 100. Then, the user I / F unit 105 outputs operation information indicating a user operation performed on the imaging apparatus 100 to the control unit 106. For example, when a shutter button (not shown) is pressed, the user I / F unit 105 outputs operation information indicating that the shutter button has been pressed to the control unit 106.

  The control unit 106 controls the operation of each functional unit included in the imaging device 100. For example, the control unit 106 controls the operation of each functional unit by outputting an instruction to each functional unit based on operation information from the user I / F unit 105. Specifically, when operation information indicating that the shutter button is pressed is output from the user I / F unit 105, the control unit 106 notifies the imaging unit 102 of the exposure time of the first frame and the second frame. The imaging unit 102 is instructed to start imaging.

  FIG. 4 is a block diagram illustrating an example of a functional configuration of the playback apparatus 200. As illustrated in FIG. 4, the playback device 200 includes at least an input unit 201, a storage unit 202, a playback information addition unit 203, an output unit 204, a user I / F unit 205, and a control unit 206. These functional units are connected to each other by an internal bus 207. Data transmission / reception between these functional units is performed using the internal bus 207. The operation of each functional unit is controlled by the control unit 206.

  The input unit 201 acquires RAW-HDR moving image data output from the imaging device 100 at a predetermined frame rate and records it in the storage unit 202. The storage unit 202 is a storage device that can store moving image data. As the storage unit 202, for example, a magnetic disk, an optical disk, a semiconductor memory, or the like is used. Note that the storage unit 202 may be a storage device that is not detachable from the playback device 200, or may be a storage device that is detachable from the playback device 200.

  The user I / F unit 205 receives a user operation on the playback device 200. Then, the user I / F unit 205 outputs operation information indicating a user operation performed on the playback device 200 to the control unit 206. For example, when a recording start button (not shown) is pressed, the user I / F unit 205 outputs operation information indicating that the recording start button has been pressed to the control unit 206. When a playback start button (not shown) is pressed, the user I / F unit 205 outputs operation information indicating that the playback start button has been pressed to the control unit 206. When a pause button (not shown) is pressed, the user I / F unit 205 outputs operation information indicating that the pause button has been pressed to the control unit 206. When a slow playback button (not shown) is pressed, the user I / F unit 205 outputs operation information indicating that the slow playback button has been pressed to the control unit 206. When a frame advance button (not shown) is pressed, the user I / F unit 205 outputs operation information indicating that the frame advance button has been pressed to the control unit 206. When a playback stop button (not shown) is pressed, the user I / F unit 205 outputs operation information indicating that the playback stop button has been pressed to the control unit 206.

The control unit 206 controls the operation of each functional unit included in the playback device 200. For example, the control unit 206 controls the operation of each functional unit by outputting an instruction to each functional unit based on operation information from the user I / F unit 205. Specifically, when operation information indicating that the recording start button has been pressed is output from the user I / F unit 205, the control unit 206 instructs the input unit 201 to record moving image data. When operation information indicating that the playback start button has been pressed is output from the user I / F unit 205, the control unit 206 instructs the playback information adding unit 203 to perform normal playback. Thereby, normal playback is set as the playback method. When operation information indicating that the pause button has been pressed is output from the user I / F unit 205, the control unit 206 instructs the reproduction information adding unit 203 to perform pause reproduction. Thereby, pause playback is set as the playback method. When operation information indicating that the slow playback button has been pressed is output from the user I / F unit 205, the control unit 206 instructs the playback information adding unit 203 to perform slow playback. Thereby, slow playback is set as the playback method. When operation information indicating that the frame advance button has been pressed is output from the user I / F unit 205, the control unit 206 instructs the reproduction information adding unit 203 to perform frame advance. When operation information indicating that the playback stop button has been pressed is output from the user I / F unit 205, the control unit 206 instructs the playback information adding unit 203 to stop playback.

  The reproduction information adding unit 203 generates reproduction RAW-HDR moving image data from the RAW-HDR moving image data recorded in the storage unit 202 based on the reproduction method set according to the instruction from the control unit 206. Then, the reproduction information adding unit 203 outputs the reproduction RAW-HDR moving image data at a predetermined frame rate. Further, when outputting the reproduction RAW-HDR moving image data, the reproduction information adding unit 203 adds metadata including reproduction information to each frame of the reproduction RAW-HDR moving image data. The reproduction information is information regarding the set reproduction method.

  For example, when normal playback is instructed from the control unit 206, the playback information adding unit 203 sequentially outputs each frame of the RAW-HDR moving image data recorded in the storage unit 202. At this time, reproduction information indicating normal reproduction is added to each frame. When pause playback is instructed, the playback information adding unit 203 repeatedly outputs the frame pair acquired (or output) when pause playback is instructed. At this time, reproduction information indicating pause reproduction is added to each frame. When the frame playback is instructed after the pause reproduction is instructed, the reproduction information adding unit 203 stores the next frame pair of the frame pair acquired (or output) when the pause reproduction is instructed. And repeatedly outputs the read frame pair. Also at this time, reproduction information indicating pause reproduction is added to each frame. When the slow reproduction is instructed, the reproduction information adding unit 203 outputs each frame of the RAW-HDR moving image data recorded in the storage unit 202 N times (N is an integer of 2 or more). At this time, reproduction information indicating slow reproduction is added to each frame. When the reproduction stop is instructed, the reproduction information adding unit 203 stops the output of the reproduction RAW-HDR moving image data.

  Normal playback is a playback method in which a moving image is played back at a normal playback speed (second playback speed). Pause playback and slow playback are playback methods for playing back a moving image at a playback speed (first playback speed) lower than the normal playback speed. Pause playback is a playback method that does not update a frame pair to be played back. As described above, in this embodiment, a plurality of reproduction methods having different reproduction speeds are used. Therefore, it can be said that the reproduction information is “information regarding reproduction speed”. Information indicating the playback speed may be used as the playback information. Note that the playback method and playback information are not limited to these. For example, reverse playback in which playback is performed from a frame with a later imaging time may be used as a playback method.

The output unit 204 outputs the playback RAW-HDR moving image data output from the playback information adding unit 203 to the developing device 300 at a predetermined frame rate. FIG. 5 is a diagram illustrating an example of RAW-HDR moving image data for reproduction output from the reproduction apparatus 200. As shown in FIG. 5, in the RAW-HDR moving image data for reproduction, the first frame and the second frame are alternately arranged. Also, HDR meta information and reproduction information are added to each frame. The playback device 200 (output unit 204) and the developing device 300 are connected to each other by wire or wirelessly. As a cable for connecting the reproducing device 200 and the developing device 300 to each other, for example, an SDI cable, an HDMI cable, or the like is used. When the playback device 200 is connected to the developing device 300 using an SDI cable, the playback information is added to, for example, an ancillary area that is a vertical blanking area. If the playback device 200 and the developing device 300 are integrated, playback RAW-HDR moving image data may be transmitted from the output unit 204 to the developing device 300 via an internal bus.

  FIG. 6 is a block diagram illustrating an example of a functional configuration of the developing device 300. As illustrated in FIG. 6, the developing device 300 includes at least an input unit 301, a developing unit 302, an output unit 303, a user I / F unit 304, and a control unit 305. These functional units are connected to each other by an internal bus 306. Data transmission / reception between these functional units is performed using an internal bus 306. The operation of each functional unit is controlled by the control unit 305.

  The input unit 301 acquires the playback RAW-HDR moving image data output from the playback device 200 at a predetermined frame rate, and outputs it to the development unit 302. The developing unit 302 performs a developing process (Debayer process) on the playback RAW-HDR moving image data output from the input unit 301. The RAW image data has pixel values that depend on the element arrangement of the imaging unit 102. By the development process, the pixel value of the RAW image data is converted into a general pixel value. In the present embodiment, development processing is performed to convert the pixel values of the RAW image data into RGB values (sampling number 4: 4: 4). Thereby, reproduction RGB-HDR moving image data in which pixel values of the reproduction RAW-HDR moving image data are converted into RGB values is generated. Then, the developing unit 302 outputs the reproduction RGB-HDR moving image data. The pixel value after the development processing is not limited to the RGB value. For example, the pixel value after the development processing may be a YCbCr value.

  The output unit 303 outputs the reproduction RGB-HDR moving image data output from the developing unit 302 to the image display device 400 at a predetermined frame rate. The structure of the reproduction RGB-HDR moving image data is the same as that of the reproduction RAW-HDR moving image data (FIG. 5). That is, the HDR meta information and the reproduction information are also added to the reproduction RGB-HDR moving image data. The developing device 300 (output unit 303) and the image display device 400 are connected to each other by wire or wirelessly. As a cable for connecting the developing device 300 and the image display device 400 to each other, for example, an SDI cable, an HDMI cable, or the like is used. When the developing device 300 and the image display device 400 are integrated, the reproduction RGB-HDR moving image data may be transmitted from the output unit 303 to the image display device 400 via an internal bus.

  The user I / F unit 304 receives a user operation on the developing device 300. Then, the user I / F unit 304 outputs operation information indicating a user operation performed on the developing device 300 to the control unit 305. For example, when a development start button (not shown) is pressed, the user I / F unit 304 outputs operation information indicating that the development start button has been pressed to the control unit 305.

  The control unit 305 controls the operation of each functional unit included in the developing device 300. For example, the control unit 305 controls the operation of each functional unit by outputting an instruction to each functional unit based on operation information from the user I / F unit 304. Specifically, when operation information indicating that the development start button has been pressed is output from the user I / F unit 304, the control unit 305 instructs the development unit 302 to start development processing.

  FIG. 7 is a block diagram illustrating an example of a functional configuration of the image display apparatus 400. As illustrated in FIG. 7, the image display apparatus 400 includes an input unit 401, a reproduction information extraction unit 402, an HDR determination unit 403, an HDR composition unit 404, an image processing unit 405, a display unit 406, a control unit 407, and a composition ratio. At least a determination unit 408 is provided. These functional units are connected to each other by an internal bus 409. Data transmission / reception between these functional units is performed using an internal bus 409. The operation of each functional unit is controlled by the control unit 407.

  The input unit 401 acquires the reproduction RGB-HDR moving image data output from the developing device 300 at a predetermined frame rate (first acquisition process). Then, the input unit 401 outputs the acquired reproduction RGB-HDR moving image data to the reproduction information extraction unit 402 and the HDR determination unit 403 at a predetermined frame rate. Note that the input unit 401 may acquire moving image data other than HDR moving image data.

  Each time frame data (image data for one frame) is output from the input unit 401, the reproduction information extraction unit 402 extracts reproduction information from the frame data (second acquisition process), and controls the extracted reproduction information. Output to the unit 407.

  Each time frame data is output from the input unit 401, the HDR determination unit 403 extracts HDR meta information from the frame data. Then, the HDR determination unit 403 acquires HDR identification information from the extracted HDR meta information (sixth acquisition process), and determines whether or not the moving image data acquired by the input unit 401 is reproduction RGB-HDR moving image data. The determination is made according to the HDR identification information (moving image determination process). When the moving image data acquired by the input unit 401 is not reproduction RGB-HDR moving image data, the HDR determination unit 403 outputs the frame data from the input unit 401 to the image processing unit 405. When the moving image data acquired by the input unit 401 is reproduction-use RGB-HDR moving image data, the HDR determination unit 403 outputs the frame data from the input unit 401 to the HDR combining unit 404. If the moving image data acquired by the input unit 401 is reproduction-use RGB-HDR moving image data, the HDR determining unit 403 outputs the extracted HDR meta information to the HDR combining unit 404 and the combining ratio determining unit 408. .

  If it is assumed that the reproduction RGB-HDR moving image data is input to the image display device 400, the sixth acquisition process and the moving image determination process may not be performed. Also, the method of moving image determination processing is not particularly limited. For example, it may be determined whether the moving image data is reproduction-use RGB-HDR moving image data using at least one of pair identification information and frame identification information. That is, at least one of pair identification information and frame identification information may also serve as HDR identification information. Since the exposure time of the first frame is relatively long, the first frame is often bright. Since the exposure time of the second frame is relatively short, the second frame is often dark. Therefore, based on the temporal change in luminance of the moving image data acquired by the input unit 401, it can be determined whether the moving image data is reproduction-use RGB-HDR moving image data.

  The HDR synthesizing unit 404 acquires the HDR meta information output from the HDR determining unit 403, and acquires frame identification information from the acquired HDR meta information (fourth acquisition process). Then, the HDR synthesizing unit 404 determines whether the frame data from the HDR determining unit 403 is the first frame data or the second frame data according to the frame identification information (frame determination process). That is, in this embodiment, the first frame and the second frame of the reproduction RGB-HDR moving image data acquired by the input unit 401 are identified using the frame identification information. Also, the HDR synthesizing unit 404 acquires pair identification information from the acquired HDR meta information (fifth acquisition process). Then, the HDR synthesizing unit 404 identifies the frame pair of the reproduction RGB-HDR moving image data acquired by the input unit 401 using the pair identification information (pair identification process).

Then, every time a frame pair is acquired, the HDR synthesizing unit 404 performs a synthesizing process for synthesizing the first frame and the second frame of the frame pair, and the result of the synthesizing process (composite frame data) is displayed as an image processing unit 405. Output to. In the combining process, the first frame and the second frame are combined with a weight corresponding to the combining ratio output from the combining ratio determination unit 408. In order to perform the combining process, the HDR combining unit 404 has a buffer (frame memory) that stores frame data. By performing the synthesizing process for each frame pair of the reproduction RGB-HDR moving image data, the HDR combined moving image data is generated.

In the combining process, for example, the first frame and the second frame are combined using Expression 1 below. Expression 1 shows a composition process for one pixel position. A composition process using Expression 1 is performed on each pixel position. In Equation 1, K is a composition ratio, L1 is a luminance value of the first frame, L2 is a luminance value of the second frame, and Lc is a composition result (luminance value after composition processing). K is a value of 0 or more and 1 or less.

Lc = L2 * K + L1 * (1-K) (Formula 1)

  Note that the order of the first frame and the second frame in the reproduction RGB-HDR moving image data may be determined in advance. Then, when the image display device 400 knows in advance the order of the first frame and the second frame in the reproduction RGB-HDR moving image data, the fourth acquisition process and the frame determination process may not be performed. Further, the frame determination processing method is not particularly limited. For example, based on the luminance of the frame data from the HDR determination unit 403, it may be determined whether the frame data is the first frame data or the second frame data. The order of the first frame and the second frame included in the frame pair may be determined in advance. Then, when the image display device 400 knows in advance the order of the first frame and the second frame included in the frame pair, the fifth acquisition process and the pair identification process may not be performed.

The image processing unit 405 generates processing frame data by performing predetermined image processing on the frame data output from the HDR determination unit 403 or the combined frame data output from the HDR combining unit 404. Then, the image processing unit 405 outputs the generated processing frame data to the display unit 406. The image processing unit 405 performs the above-described processing every time frame data (combined frame data) is input to the image processing unit 405. As the predetermined image processing, for example, white luminance changing processing for changing white luminance, gradation changing processing for changing gradation values, chromaticity changing processing for changing chromaticity, and the like are performed.
Note that there may be a noticeable luminance change in which the composite frame data becomes darker or brighter than the previous composite frame due to the difference in the composite ratio between the composite frame data. The image processing unit 405 may perform image processing for increasing or decreasing the luminance in order to reduce such a luminance change. Specifically, the luminance average value difference or peak luminance value (maximum luminance value) difference between the synthesized frame data is monitored, and if the difference is greater than or equal to a predetermined difference, the difference falls within the predetermined difference. For example, processing for changing white luminance (white luminance) may be performed. Alternatively, for each combination frame data, a combination ratio may be acquired from the combination ratio determination unit 408, and a process of changing the luminance according to the combination ratio may be performed.

  The display unit 406 displays a moving image corresponding to the processed moving image data output from the image processing unit 405 on the screen. The processed moving image data is moving image data having processed frame data as image data of each frame. Specifically, the display unit 406 includes a buffer (frame memory) that stores frame data and a display panel. The display unit 406 records (overwrites) the processing frame data output from the image processing unit 405 in the buffer every time processing frame data is output from the image processing unit 405. The display unit 406 reads the processing frame data from the buffer at a predetermined refresh rate, and outputs the read processing frame data to the display panel. Each display element of the display panel is driven according to the processing frame data input to the display panel, and an image corresponding to the processing frame data is displayed. As the display panel, for example, a liquid crystal panel, an organic EL panel, a plasma panel, or the like is used.

  The control unit 407 controls the operation of each functional unit included in the image display device 400. For example, the control unit 407 controls the operation of the combination ratio determination unit 408 by outputting the reproduction information output from the reproduction information extraction unit 402 to the combination ratio determination unit 408. Note that the control unit 407 may generate information based on the reproduction information output from the reproduction information extraction unit 402, and output the generated information instead of the reproduction information. For example, the control unit 407 may generate information indicating whether or not the current reproduction method is pause reproduction based on the reproduction information.

The composition ratio determination unit 408 determines the composition ratio K based on the reproduction information from the control unit 407. As shown in Equation 1 described above, in this embodiment, the weight of the second frame is determined as the combination ratio K. In this embodiment, the composition ratio determination unit 408 determines whether the current playback method is pause playback based on the playback information. Then, the composition ratio determination unit 408 determines the composition ratio K as follows according to the determination result of the current reproduction method. The composition ratio determination unit 408 outputs the determined composition ratio K to the HDR composition unit 404.

When the current playback method is normal playback: K = 0.5
When the current playback method is pause playback: K = 0.8

  Thus, in the present embodiment, when the current playback method is pause playback, a larger value is used as the weight of the second frame than when the current playback method is normal playback. A composition ratio K is determined. When the current playback method is pause playback, a large value is used as the weight of the second frame, so that image data representing an image with a high focus on the subject can be obtained as composite frame data. The composition ratio K when the current reproduction method is normal reproduction is a reference value (default value), and is a value that can express the natural movement of the subject in normal reproduction. The value of the composition ratio K corresponding to the reproduction method may be a fixed value determined in advance by the manufacturer, or may be a value that can be changed by the user.

  Note that the value of the composition ratio K is not limited to the above-described values (0.5 and 0.8). When the current reproduction method is normal reproduction, the composition ratio K may be larger or smaller than 0.5. The combination ratio K when the current reproduction method is pause reproduction may be larger or smaller than 0.8. For example, 1 may be used as the composition ratio K when the current reproduction method is pause reproduction. That is, when the current reproduction method is pause reproduction, zero may be used as the synthesis ratio (weight) of the first frame, and 1 may be used as the synthesis ratio of the second frame. In this case, a selection process for selecting and outputting only the second frame from the first frame having a long exposure time and the second frame having a short exposure time may be performed, and the combining process may not be performed.

  Note that a value other than the weight of the second frame may be determined as the composition ratio K. For example, the weight of the first frame may be determined as the combination ratio K. As the combination ratio K, the ratio of the weight of the second frame to the weight of the first frame may be determined. As the combination ratio K, the ratio of the weight of the first frame to the weight of the second frame may be determined.

  It should be noted that the reference value may or may not be used as the composition ratio K when the current reproduction method is different from the pause reproduction. For example, when the current playback method is slow playback, the composition ratio K is determined such that a larger value is used as the weight of the second frame than when the current playback method is normal playback. Good. When the current playback method is slow playback, the composition ratio K may be determined so that the same value as that when the current playback method is pause playback is used as the weight of the second frame. When the current playback method is slow playback, a value larger than that when the current playback method is normal playback and smaller than when the current playback method is pause playback is used as the weight of the second frame. As used, the composition ratio K may be determined.

Note that the determination processing method for determining the composition ratio K is not limited to the above method. For example, when the current playback method is pause playback (or slow playback), the composition ratio K is determined based on the brightness of the first frame and the second frame (the brightness of the frame data from the HDR determination unit 403). (Adjustment) may be performed. When the current playback method is pause playback (or slow playback), the composition ratio K may be determined based on the exposure times of the first frame and the second frame. When the difference in exposure time between the first frame and the second frame is small, the increase in focus feeling due to the increase in the weight of the second frame is small. Therefore, when the difference between the exposure times of the first frame and the second frame is less than the first threshold, the value closer to the reference value than the case where the difference between the exposure times is equal to or greater than the first threshold is the weight of the second frame. The composite ratio K may be determined to be used as Specifically, the composition ratio K may be determined as follows. Thereby, the change of the focus feeling by the difference in the reproduction | regeneration method can be reduced, and a user's discomfort can be reduced. In these methods, for example, exposure time information is acquired from the HDR meta information (third acquisition process), and the exposure time indicated by the exposure time information is used. The first threshold may be a fixed value determined in advance by the manufacturer, or may be a value that can be changed by the user.

When the current playback method is normal playback: K = 0.5
The current playback method is pause playback,
When the difference in exposure time is greater than or equal to the first threshold: K = 0.8
The current playback method is pause playback,
When the difference in exposure time is less than the first threshold: K = 0.6

  FIG. 8 is a flowchart showing an example of the operation of the image display apparatus 400. Here, an example in which the reproduction RGB-HDR moving image data is input to the image display device 400 will be described.

  First, the control unit 407 acquires reproduction information from the reproduction information extraction unit 402, and outputs the acquired reproduction information to the synthesis ratio determination unit 408 (S101). Next, the composition ratio determination unit 408 determines whether or not the current reproduction method is pause reproduction based on the reproduction information output from the control unit 407 (S102). Then, the composition ratio determination unit 408 acquires the exposure time information of the first frame and the second frame from the HDR determination unit 403 (S103). Next, the composition ratio determination unit 408 determines the composition ratio K based on the determination result in S102 and the exposure time information acquired in S103 (S104). Then, the HDR combining unit 404 combines the first frame and the second frame using the combining ratio K determined in S104 (S105). The processing from S101 to S105 is repeatedly performed for each frame pair of the reproduction RGB-HDR moving image data (S106).

  FIGS. 9A and 9B are diagrams illustrating examples of display images (images displayed on the screen) in the present embodiment. The display images in FIGS. 9A and 9B are images displayed according to the composite frame data. In the motion capture workflow, first, normal playback is set, and the moving image is confirmed by the user. At this time, a reference value is used as the composition ratio K. As a result, as shown in FIG. 9A, an image in which some blurring of the movement of the subject remains is displayed. By displaying an image in which motion blur remains to some extent, natural motion can be expressed. Thereafter, in order to select composite frame data to be extracted as still image data, slow playback or pause playback is set, or frame advance is performed. At this time, a value larger than the reference value is used as the composition ratio K (weight of the second frame). As a result, as shown in FIG. 9B, an image with a high focus feeling of the subject is displayed. By displaying an image with a high focus feeling, the user can easily and appropriately select composite frame data to be extracted as still image data. Instead of the synthesized frame data, a frame pair used for generating still image data may be extracted.

  As described above, according to the present embodiment, the combination ratio used when combining the first frame and the second frame is automatically determined based on the method of reproducing HDR video data. As a result, it is possible to easily obtain HDR synthesized moving image data in which the degree of blur of the subject movement (the subject's focus feeling) is suitable. Specifically, it is possible to save the user's trouble of changing the composition ratio every time the reproduction method is changed, and to reduce the user's work time and burden (work efficiency). Further, in the motion capture workflow, the user can select (select) suitable composite frame data and frame pairs in a short time and easily.

  In this embodiment, the example in which HDR identification information, pair identification information, frame identification information, exposure time information, reproduction information, and the like are transmitted as metadata has been described, but the present invention is not limited to this. For example, at least one of HDR identification information, pair identification information, frame identification information, exposure time information, and reproduction information may be transmitted separately from the moving image data. At least one of HDR identification information, pair identification information, frame identification information, exposure time information, and reproduction information may be acquired in response to a user operation. User operations include a user operation on the imaging device, a user operation on the playback device, a user operation on the image display device, and the like.

  In the present embodiment, an example in which two frames of the first frame and the second frame are combined has been described. However, the present invention is not limited to this. The HDR moving image data may be moving image data in which a frame group including three or more frames is repeated. For example, moving image data in which a frame group including three frames including a frame imaged by short exposure, a frame imaged by appropriate exposure, and a frame imaged by long exposure may be used. In that case, three or more frames included in the frame group may be combined. Further, in the HDR moving image data, the first frame and the second frame may not be continuous.

<Example 2>
Hereinafter, an image processing apparatus and an image processing method according to Embodiment 2 of the present invention will be described. In this embodiment, the method for determining the composition ratio K is different from that in the first embodiment. Hereinafter, configurations and processes different from those of the first embodiment will be described in detail, and descriptions of the same configurations and processes as those of the first embodiment will be omitted. The configuration of the image display system according to the present embodiment is the same as that of the first embodiment. The configurations of the imaging device, the reproduction device, and the developing device according to the present embodiment are the same as those of the first embodiment.

  FIG. 10 is a block diagram illustrating an example of a functional configuration of the image display apparatus 500 according to the present embodiment. As illustrated in FIG. 10, the image display apparatus 500 includes an input unit 401, a reproduction information extraction unit 402, an HDR determination unit 403, an HDR synthesis unit 404, an image processing unit 405, a display unit 406, a control unit 407, and a composition ratio determination unit. 508 and a histogram acquisition unit 510 at least. These functional units are connected to each other by an internal bus 409. Data transmission / reception between these functional units is performed using an internal bus 409. The operation of each functional unit is controlled by the control unit 407. In FIG. 10, the same functional parts as those in the first embodiment (FIG. 7) are denoted by the same reference numerals as those in the first embodiment, and the description thereof is omitted.

  In this embodiment, when the moving image data acquired by the input unit 401 is HDR moving image data (playback RGB-HDR moving image data), the HDR determination unit 403 converts the frame data from the input unit 401 into the HDR combining unit 404. And output to the histogram acquisition unit 510. Also, the HDR determination unit 403 outputs the extracted HDR meta information to the HDR synthesis unit 404, the synthesis ratio determination unit 508, and the histogram acquisition unit 510.

The histogram acquisition unit 510 identifies the first frame, the second frame, and the frame pair of the HDR moving image data acquired by the input unit 401 using the same method as the HDR synthesis unit 404. Then, every time a frame pair is acquired, the histogram acquisition unit 510 performs a combining process of combining the first frame and the second frame of the frame pair, and acquires each gradation value of the combined frame data (seventh acquisition). processing). Here, a composition process using the candidate ratio as the composition ratio K is performed. Specifically, for each of the plurality of candidate ratios, a combining process using the candidate ratio as the combining ratio K is performed. Thereby, each gradation value of the composite frame data is acquired for each of the plurality of candidate ratios. The candidate ratio is a candidate for the combination ratio K. In order to perform the synthesis process, the histogram acquisition unit 510 includes a buffer (frame memory) that stores frame data.

  Note that the synthesis process using the candidate ratio as the synthesis ratio K may be performed by the HDR synthesis unit 404. And the process which acquires each gradation value of the synthetic | combination frame data produced | generated by the HDR synthetic | combination part 404 may be performed as a 7th acquisition process. Further, the value of the candidate ratio may be a fixed value determined in advance by the manufacturer, or may be a value that can be changed by the user.

  The histogram acquisition unit 510 outputs the acquired gradation value to the synthesis ratio determination unit 508. In the present embodiment, the histogram acquisition unit 510 generates histogram data representing the number of gradation values for each of a plurality of candidate ratios. Then, the histogram acquisition unit 510 outputs the generated plurality of histogram data to the synthesis ratio determination unit 508. Note that the gradation value (histogram data) acquired by the histogram acquisition unit 510 is used for blackout confirmation, so that it is not necessary to output information regarding a large gradation value to the composition ratio determination unit 508. Therefore, only the gradation value equal to or lower than the fourth threshold value may be output to the composition ratio determination unit 508. Histogram data representing only the number of gradation values equal to or smaller than the fourth threshold value may be output to the composition ratio determination unit 508. The fourth threshold value may be a fixed value determined in advance by the manufacturer, or may be a value that can be changed by the user.

  Similar to the first embodiment, the composition ratio determination unit 508 determines the composition ratio K based on the reproduction information. However, in the present embodiment, based on the HDR video data input to the input unit 401, the composition ratio K is determined so as not to cause blackout (dark portion gradation collapse) due to composition processing (blackout). Suppression process). The method of blackout suppression processing is not particularly limited. In the present embodiment, the composition ratio determination unit 508 determines the number of pixels whose gradation value is equal to or less than the second threshold among the pixels of the composition frame data based on the gradation value (histogram data) acquired by the histogram acquisition unit 510. Is determined so as to be less than or equal to the third threshold. A value smaller than the fourth threshold is used as the second threshold.

  Note that the blackout suppression processing may be performed constantly, or the blackout suppression processing may be performed only when the playback method is a predetermined playback method. When a large value is used as the weight of the second frame, blackout is more likely to occur than when a small value is used as the weight of the second frame. For this reason, the blackout suppression processing may be performed only when the playback method is pause playback or slow playback. The blackout suppression process may be performed only when the playback method is pause playback. Only when the playback method is slow playback, the blackout suppression processing may be performed. The second threshold value may be a fixed value determined in advance by the manufacturer or a value that can be changed by the user. The third threshold value may be a fixed value determined in advance by the manufacturer, or may be a value that can be changed by the user.

  FIG. 11 is a flowchart illustrating an example of the operation of the image display apparatus 500. Here, an example in which HDR moving image data (reproduction RGB-HDR moving image data) is input to the image display device 500 will be described. As shown in FIG. 11, in the present embodiment, the processes of S201 to S207 are performed in order. The processing of S201 to S203 is the same as the processing of S101 to S103 of FIG. 8, and the processing of S206 and S207 is the same as the processing of S105 and S106 of FIG.

  In S <b> 204, the histogram acquisition unit 510 acquires a plurality of histogram data corresponding to a plurality of candidate ratios, and outputs the acquired plurality of histogram data to the synthesis ratio determination unit 508. 12A to 12C show examples of histogram data. 12A to 12C, the vertical axis represents the number of pixels, and the horizontal axis represents the gradation value. FIG. 12A shows histogram data corresponding to the first candidate ratio. FIG. 12B shows histogram data corresponding to a second candidate ratio that is larger than the first candidate ratio. FIG. 12C shows histogram data corresponding to a third candidate ratio that is larger than the second candidate ratio. The first candidate ratio, the second candidate ratio, and the third candidate ratio are the weights of the second frame. 12 (A) to 12 (C), it can be seen that blackout occurs and increases as the weight of the second frame increases.

  In S205, the composition ratio determination unit 508 determines the composition ratio K based on the determination result in S102, the exposure time information acquired in S103, and the histogram data acquired in S204. If the current playback method is not pause playback, the reference value is determined as the composition ratio K. When the current reproduction method is pause reproduction, a value larger than the reference value is set so that the number of pixels whose gradation value is equal to or smaller than the second threshold among the pixels of the composite frame data is equal to or smaller than the third threshold. It is determined as the composition ratio K. The process of S205 when the current playback method is pause playback will be specifically described with reference to FIGS. 12 (A) to 12 (C). In FIG. 12A, there is no pixel having a gradation value in the rectangular area indicated by the broken line. In FIG. 12B, the number of pixels having gradation values in the rectangular area is about half of the number corresponding to the size of the rectangular area. In FIG. 12C, the number of pixels having gradation values in the rectangular area is larger than the number corresponding to the size of the rectangular area. In S205, for example, the composition ratio K is determined so that the number of pixels having gradation values in the rectangular area is equal to or less than half the number corresponding to the size of the rectangular area. Therefore, the second candidate ratio corresponding to the histogram data shown in FIG.

  As described above, according to the present embodiment, since the combination ratio is automatically determined based on the imaging information, the same effect as that of the first embodiment can be obtained. Furthermore, in this embodiment, the composition ratio is determined so that occurrence of blackout is suppressed. Thereby, HDR synthetic moving image data with higher image quality can be obtained.

  In this embodiment, an example in which a plurality of candidate ratios are used has been described, but the number of candidate ratios used may be one. An example in which the number of candidate ratios used is one will be described. The correspondence relationship between the number of pixels whose gradation value is equal to or less than the second threshold, the composition ratio K, and the reproduction method is determined in advance. Then, the composition ratio K is determined based on the reproduction information, the gradation value acquired by the histogram acquisition unit 510, and the correspondence relationship. Specifically, the composition ratio K corresponding to the combination of the reproduction method indicated by the reproduction information and the number of pixels whose gradation value acquired by the histogram acquisition unit 510 is equal to or smaller than the second threshold is the above-described correspondence relationship. Obtained from. When blackout suppression processing is performed only during pause playback, the correspondence between the number of pixels with gradation values equal to or smaller than the second threshold and the composition ratio K may be determined in advance. In that case, the composition ratio K corresponding to the number of pixels for which the gradation value acquired by the histogram acquisition unit 510 is equal to or smaller than the second threshold value may be acquired from the correspondence relationship.

<Example 3>
Hereinafter, an image processing apparatus and an image processing method according to Embodiment 3 of the present invention will be described. Hereinafter, configurations and processes different from those of the first embodiment will be described in detail, and descriptions of the same configurations and processes as those of the first embodiment will be omitted. The configuration of the image display system according to the present embodiment is the same as that of the first embodiment. The configurations of the imaging device, the reproduction device, and the developing device according to the present embodiment are the same as those of the first embodiment. In addition, the structure which becomes the characteristic of a present Example described below can also be combined with the structure of Example 2 or Example 4 mentioned later.

  FIG. 13 is a block diagram illustrating an example of a functional configuration of the image display apparatus 600 according to the present embodiment. As illustrated in FIG. 13, the image display apparatus 600 includes an input unit 401, a reproduction information extraction unit 402, an HDR determination unit 403, an HDR synthesis unit 404, an image processing unit 405, a display unit 406, a control unit 407, and a synthesis ratio determination unit. 408 and an image composition unit 610. These functional units are connected to each other by an internal bus 409. Data transmission / reception between these functional units is performed using an internal bus 409. The operation of each functional unit is controlled by the control unit 407. In FIG. 13, the same functional parts as those in the first embodiment (FIG. 7) are denoted by the same reference numerals as those in the first embodiment, and the description thereof is omitted.

  In this embodiment, the composition ratio determination unit 408 outputs the determined composition ratio K to the HDR composition unit 404 and the image composition unit 610. In addition, the image processing unit 405 outputs the generated processing frame data (first processing frame data) to the image composition unit 610.

  The image composition unit 610 notifies the user of the composition ratio K output from the composition ratio determination unit 408. In this embodiment, the image composition unit 610 has a buffer (frame memory) for storing frame data, and records (overwrites) the first processing frame data output from the image processing unit 405 in the buffer. The image composition unit 610 represents an image obtained by superimposing the composition ratio image on the process frame image based on the composition ratio K output from the composition ratio determination unit 408 and the first processing frame data recorded in the buffer. Frame data (second processing frame data) is generated. The composite ratio image is a graphic image representing the composite ratio, and the processing frame image is an image represented by the first processing frame data. Specifically, the image composition unit 610 generates image data (graphic image data) representing a composition ratio image based on the composition ratio K. Then, the image synthesis unit 610 generates second process frame data by synthesizing the graphic image data with the first process frame data. Thereafter, the image composition unit 610 outputs the generated second processing frame data to the display unit 406. As a result, an image obtained by superimposing the composite ratio image on the processing frame image is displayed, and the composite ratio is notified to the user. The image composition unit 610 performs the above-described process every time processing frame data is output from the image processing unit 405. Thereby, moving image data representing a moving image in which a composition ratio image is superimposed on a moving image based on the HDR combined moving image data is generated and displayed.

  FIG. 14 is a diagram illustrating an example of a display image according to the present embodiment. In FIG. 14, a bar image indicating the composition ratio K is displayed as the composition ratio image. The composite ratio image is not limited to the image shown in FIG. For example, a text image in which a numerical value of the composition ratio K is described may be used as the composition ratio image. An icon corresponding to the value of the composition ratio K may be used as the composition ratio image.

  As described above, according to the present embodiment, since the composition ratio is automatically determined based on the imaging information, it is possible to obtain the same effect as that of the first embodiment. Furthermore, in this embodiment, the current composition ratio is notified to the user. Thereby, the user can grasp the composition ratio of the images that he / she has confirmed. The composition ratio grasped by the user is used in work performed separately, for example. Specifically, the composition ratio grasped by the user is used in a retouching operation using image data extracted as still image data. In order to obtain a plurality of still image data by synthesizing the first frame and the second frame at various synthesis ratios, the synthesis ratio grasped by the user may be used. In this embodiment, the example in which the user is notified of the composition ratio by image display has been described. However, the method of notifying the user is not limited to this. For example, the synthesis ratio may be notified to the user using sound or light from a lamp (light source).

<Example 4>
Hereinafter, an image processing apparatus and an image processing method according to Embodiment 4 of the present invention will be described. Hereinafter, configurations and processes different from those of the first embodiment will be described in detail, and descriptions of the same configurations and processes as those of the first embodiment will be omitted. The configuration of the image display system according to the present embodiment is the same as that of the first embodiment. The configurations of the imaging device, the reproduction device, and the developing device according to the present embodiment are the same as those of the first embodiment. In addition, the structure which is the characteristic of the present embodiment described below can be combined with the structure of the second and third embodiments.

  FIG. 15 is a block diagram illustrating an example of a functional configuration of the image display apparatus 700 according to the present embodiment. As illustrated in FIG. 15, the image display device 700 includes an input unit 401, a reproduction information extraction unit 402, an HDR determination unit 403, an HDR synthesis unit 404, an image processing unit 405, a display unit 406, a control unit 407, and a synthesis ratio determination unit. 408, a ratio information generation unit 710, and an output unit 711 at least. These functional units are connected to each other by an internal bus 409. Data transmission / reception between these functional units is performed using an internal bus 409. The operation of each functional unit is controlled by the control unit 407. In FIG. 15, the same functional parts as those in the first embodiment (FIG. 7) are denoted by the same reference numerals as those in the first embodiment, and the description thereof is omitted.

  In this embodiment, the HDR determination unit 403 outputs the extracted HDR meta information to the HDR synthesis unit 404, the synthesis ratio determination unit 408, and the ratio information generation unit 710. Further, the composition ratio determination unit 408 outputs the determined composition ratio K to the HDR composition unit 404 and the ratio information generation unit 710.

  The ratio information generation unit 710 generates ratio information by adding (associating) the combination ratio K output from the combination ratio determination unit 408 to the HDR meta information output from the HDR determination unit 403. Further, the ratio information generation unit 710 has a buffer for storing ratio information (HDR meta information to which the combination ratio K is added), and records (overwrites) the ratio information in the buffer. Then, the ratio information generation unit 710 reads the ratio information from the buffer and outputs it to the output unit 711 in response to an instruction from the control unit 407. For example, the control unit 407 instructs the ratio information generation unit 710 to output ratio information in response to a user operation. Note that the HDR meta information may not be used. Any information may be generated as the ratio information as long as the information indicates the combination ratio K. However, it is preferable that information in which the composition ratio K is associated with the frame pair is used as the ratio information.

  The output unit 711 outputs the ratio information output from the ratio information generation unit 710 to the outside. For example, the output unit 711 records the ratio information in a storage device that is an external device connected to the image display device 700 by outputting the ratio information to the outside in a file format. As the storage device, for example, a magnetic disk, an optical disk, a semiconductor memory, or the like is used. For example, an SD card is used as the semiconductor memory. The image display device 700 may include a storage unit for ratio information, and the ratio information output from the ratio information generation unit 710 may be recorded in the storage unit. One of the output process for outputting the ratio information to the outside and the recording process for recording the ratio information in a storage unit (a storage unit provided in the image display device 700, a storage device that is an external device, etc.) are performed. Good or not. Both output processing and recording processing may be performed. Note that ratio information regarding each of a plurality of frame pairs may be output or recorded.

  As described above, according to the present embodiment, since the combination ratio is automatically determined based on the imaging information, the same effect as that of the first embodiment can be obtained. Furthermore, in the present embodiment, ratio information indicating the combination ratio is output to the outside or recorded in the storage unit. The output or recorded composition ratio is used, for example, in work performed separately. Specifically, in the retouching operation using image data extracted as still image data, the output or recorded composition ratio is used. In order to obtain a plurality of still image data by synthesizing the first frame and the second frame at various synthesis ratios, an output or recorded synthesis ratio may be used.

<Other examples>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

400, 500, 600, 700: image display device 401: input unit 402: reproduction information extraction unit 404: HDR synthesis unit 408, 508: synthesis ratio determination unit

Claims (19)

First acquisition means for acquiring moving image data including a first frame obtained by imaging and a second frame obtained by imaging with an exposure time shorter than the first frame;
Second acquisition means for acquiring reproduction information that is information relating to the reproduction speed of the moving image data;
Combining means for combining the first frame and the second frame based on the reproduction information;
Have
The image processing apparatus according to claim 1, wherein the combining unit changes a combining ratio of the first frame and the second frame in accordance with a reproduction speed of the moving image data. In the synthesizing unit, when the reproduction information related to the first reproduction speed is acquired by the second acquisition unit, the reproduction information related to the second reproduction speed higher than the first reproduction speed is acquired by the second acquisition unit. The image processing apparatus according to claim 1, wherein a larger value than the case is determined as a composition ratio of the second frame. The image processing apparatus according to claim 2, wherein the reproduction information related to the first reproduction speed is reproduction information indicating a pause of the moving image data. The synthesis means determines zero as the synthesis ratio of the first frame and 1 as the synthesis ratio of the second frame when reproduction information related to the first playback speed is acquired by the second acquisition means. The image processing apparatus according to claim 3, wherein: The combining means determines the combining ratio based on exposure times of the first frame and the second frame when reproduction information related to the first reproduction speed is acquired by the second acquisition means. The image processing apparatus according to any one of claims 2 to 4. The synthesis means includes
When reproduction information related to the second reproduction speed is acquired by the second acquisition unit, a reference value is determined as a synthesis ratio of the second frame;
When the reproduction information related to the first reproduction speed is acquired by the second acquisition means, and the difference in exposure time between the first frame and the second frame is less than a first threshold, the first frame and the 6. The image according to claim 5, wherein a value closer to the reference value compared to a case where a difference in exposure time of the second frame is equal to or greater than the first threshold is determined as a composition ratio of the second frame. Processing equipment. A third acquisition means for acquiring time information indicating an exposure time of each frame of the moving image data;
7. The composition unit according to claim 5 or 6, wherein when the reproduction information related to the first reproduction speed is acquired by the second acquisition unit, the combination unit determines the combination ratio using the time information. Image processing apparatus. A fourth acquisition means for acquiring frame identification information for identifying the first frame and the second frame;
The image processing apparatus according to claim 1, wherein the synthesizing unit identifies the first frame and the second frame using the frame identification information. A fifth acquisition means for acquiring pair identification information for identifying a frame pair including the first frame and the second frame;
The image processing apparatus according to claim 1, wherein the synthesizing unit identifies the frame pair using the pair identification information. 10. The apparatus according to claim 1, further comprising a determination unit that determines whether the moving image data acquired by the first acquisition unit includes the first frame and the second frame. The image processing apparatus described. Further comprising sixth acquisition means for acquiring moving image identification information indicating whether or not the moving image data acquired by the first acquisition means includes the first frame and the second frame;
11. The determination unit according to claim 10, wherein the determination unit determines whether the moving image data acquired by the first acquisition unit includes the first frame and the second frame according to the moving image identification information. An image processing apparatus according to 1. The composition unit determines the composition ratio based on the moving image data so as not to cause blackout due to composition of the first frame and the second frame. The image processing apparatus according to any one of the above. A seventh acquisition means for acquiring each gradation value of a frame obtained by combining the first frame and the second frame according to a candidate ratio that is a candidate for the combination ratio;
The synthesizing unit is configured such that, based on each tone value acquired by the seventh acquiring unit, the number of pixels whose tone value after combining the first frame and the second frame is equal to or less than a second threshold is third. The image processing apparatus according to claim 12, wherein the composition ratio is determined so as to be equal to or less than a threshold value. The image processing apparatus according to claim 1, further comprising notification means for notifying a user of the composition ratio. A moving image based on the combined moving image data based on the combined moving image data based on the combined moving image data obtained by combining the first frame and the second frame and the combined ratio. The image processing apparatus according to claim 1, further comprising generation means for generating moving image data representing a moving image superimposed on the screen. The image processing apparatus according to claim 1, further comprising an output unit configured to output ratio information representing the synthesis ratio to the outside. The image processing apparatus according to claim 1, further comprising a recording unit that records ratio information representing the synthesis ratio in a storage unit. A first acquisition step of acquiring moving image data including a first frame obtained by imaging and a second frame obtained by imaging with an exposure time shorter than the first frame;
A second acquisition step of acquiring reproduction information which is information relating to the reproduction speed of the moving image data;
A combining step of combining the first frame and the second frame based on the reproduction information;
Have
In the composition step, the composition ratio of the first frame and the second frame is changed according to the reproduction speed of the moving image data.   A program for causing a computer to execute each step of the image processing method according to claim 18.

Images