JP2008298918A - Image processing apparatus and method, and program - Google Patents

Abstract

To improve the visual movement of a user.
For frame number k (k is n-3 to n + 2), frame image data is obtained when the image information incident on the camera during the shutter opening time Tsk is time-integrated. When a recording data sequence that is an aggregate of these frame video data is to be displayed, the display time Tik of frame k is set to a time corresponding to the shutter opening time Tsk in the frame period Tf, and darkness is used for the remaining time Tmk. Display control for displaying video is performed. The present invention is applicable to an image processing apparatus that performs display control such as a liquid crystal display.
[Selection] Figure 3

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program capable of improving a user's visual movement without using an intermediate image interpolation technique. .

  In recent years, the spread of liquid crystal display devices has been remarkable (for example, see Patent Document 1). However, when the liquid crystal display device displays an image having a frame rate that is not so high, such as a movie, there arises a problem that the visual movement of the user is deteriorated. The “visual movement of the user” means that the user visually recognizes as “movement” when an image with “movement” is displayed on a liquid crystal display device or the like. That is, such a problem arises that, when a scene with “motion” is displayed, the user may visually recognize that it is a discrete “motion”.

Therefore, in order to solve this problem, conventionally, a technique has been employed in which an intermediate image between frames is interpolated and inserted, and smooth motion is realized by increasing the frame rate. ing.
JP 2001-118396 A

  However, an interpolated image is an image created by estimating a video at the interpolation timing, and is an image different from an actual one. This means that the original video signal is processed, that is, video information not intended by a video producer such as a movie producer is inserted, which is not always preferable.

  The present invention has been made in view of such circumstances, and is intended to improve the visual movement of the user without using an intermediate image interpolation technique.

  An image processing apparatus according to an aspect of the present invention uses, as unit image information, information obtained as a result of integrating video information of a subject during each sampling time, on moving image information including a plurality of unit image information. In the image processing apparatus for performing image processing, display of each of the plurality of unit image information is started for each predetermined cycle time as display control of the moving image information, and the display time is set to the sampling time. A display control circuit for performing control based on the control.

  The display control circuit further controls the luminance at the time of displaying the unit image information based on the display time of the unit image information to be controlled.

  The display circuit performs control to correct the luminance to a level inversely proportional to the display time of the unit image information.

  The display control circuit performs control to match the display time of the unit image information to be controlled with the sampling time.

  The display control circuit performs control to match all display times of the plurality of unit image information with a predetermined fixed time variably set based on the sampling time.

  The moving image information includes specific information that specifies the sampling time for each of the plurality of unit image information, and the display control circuit determines a display time of the unit image information to be controlled as the specific information. Control to change based on the sampling time specified from the above.

  The display control circuit further performs control for estimating the sampling time of the unit image information to be controlled and changing the display time of the unit image information based on the estimation result.

  The display control circuit further controls display of a dark image or prohibition of display itself until the display of the next unit image information is started after the display time of the unit image information to be controlled has elapsed.

  The moving image information is displayed by a display device that displays a video based on an electrical video drive signal, and the display control circuit controls the video drive signal applied to the display device, thereby controlling the video to be controlled. It controls the display time of the unit image information and the time for displaying the dark image or prohibiting the display itself.

  The moving image information is displayed by a display device that projects an image through a lens, and the display control circuit controls the display time of the unit image information to be controlled and the darkness by controlling the opening of the lens. Controls the time to display an image or to prohibit the display itself.

  An image processing method and program according to one aspect of the present invention are a method and program corresponding to the above-described image processing apparatus according to one aspect of the present invention.

  In the image processing apparatus, method, and program according to one aspect of the present invention, information obtained as a result of integrating video information of a subject during each sampling time is used as unit image information, and is configured from a plurality of unit image information. The following image processing is performed on the moving image information. That is, as the display control of the moving image information, the display of each of the plurality of unit image information is started every predetermined cycle time, and the display time is changed based on the sampling time.

  As described above, according to the present invention, display control of a hold type display device or the like can be realized. In particular, it is possible to improve the visual movement of the user in the hold type display device or the like without using an intermediate image interpolation technique.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. Not something to do.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

The image processing apparatus according to one aspect of the present invention (for example, the image processing apparatus 11 in FIG. 8 or the image processing apparatus 51 in FIG. 12)
The result of integrating the video information of the subject during each sampling time (for example, each shutter opening time Tsk in the upper diagram of FIG. 3, where k is one of frame numbers n-3 to n + 2) The obtained information (for example, the frame image data in the central diagram in FIG. 3) is used as unit image information, and the moving image information (for example, the recording data series in the central diagram in FIG. 3) composed of the plurality of unit image information is used. In an image processing apparatus that performs image processing,
As the display control of the moving image information, display of each of the plurality of unit image information is started at every predetermined cycle time (for example, the frame cycle Tf in the lower diagram of FIG. 3), and the display time is sampled. A display control circuit (for example, the dark image insertion control unit 21 in FIG. 8 or the dark section insertion control unit 62 in FIG. 12) that performs control based on the time (for example, see the display time Tik in the lower diagram in FIG. 3).
Is provided.

  Here, with reference to FIG. 1 thru | or FIG. 4, the outline of the method to which this invention is applied is demonstrated first.

  As described above in the section “Problems to be Solved by the Invention”, when a scene with “movement” is conventionally displayed, the “movement” may be visually recognized by the user as an awkward movement. The problem has arisen.

  However, the present inventor has found that such a problem occurs in a liquid crystal display device such as a flat panel display, but the impression that it has occurred in a conventional cathode ray tube type display is thin. Based on this fact, the present inventor made the present invention to find out the cause of the above-mentioned problem and to eliminate the cause of the problem, that is, to solve the above-mentioned problem.

  In order to facilitate understanding of the present invention through such circumstances, first, an outline of the difference between a liquid crystal display device and a conventional cathode ray tube display, that is, a CRT (Cathode Ray Tub) will be described. To do.

  Note that various processing units such as a frame and a field can be used as a processing unit of a display device such as a liquid crystal display device or a CRT. However, for the sake of simplicity, it is assumed that a frame is used. Of course, by replacing the phrase “frame” appearing in the following explanatory text with a phrase indicating another processing unit such as “field”, the following explanation content is also basic for other processing units such as “field”. Needless to say, this is true.

  When the display of a predetermined one of a plurality of frames constituting a moving image is instructed, the CRT moves the built-in electron gun along each of a plurality of horizontal lines (scanning lines) constituting the CRT screen. By sequentially scanning, the designated frame is displayed on the screen.

  Here, a frame for which display is instructed (hereinafter referred to as a display target frame) is composed of a plurality of pixels. Therefore, when viewed in pixel units, the pixel arranged at a predetermined position in the display target frame is displayed at the moment when the electron gun scans the predetermined position in the time direction, that is, in an impulse shape. Will be displayed. Hereinafter, display devices that employ a display method similar to that of the CRT are collectively referred to as an impulse display device.

  In contrast to such an impulse display device, the liquid crystal display device holds the display of all the liquid crystals constituting the screen during the period from the display instruction of the display target frame to the display instruction of the next frame, Display the display target frame on the screen.

  Specifically, for example, it is assumed that one pixel is associated with one liquid crystal. In this case, for example, an instruction based on the pixel value of each pixel constituting the display target frame can be adopted as the display instruction for the display target frame. In this case, the liquid crystal display device applies a voltage corresponding to the instructed pixel value to each liquid crystal (each pixel) constituting the screen. Then, each liquid crystal emits light at a light level corresponding to the applied voltage value. Thereafter, since the applied voltage value of each liquid crystal is kept constant until at least the display instruction of the next frame, each liquid crystal continues to emit light at a constant light level. As a result, pixels having the designated pixel value are continuously displayed on each liquid crystal.

  Thereafter, when a period corresponding to a predetermined frame period elapses, the next frame becomes a new display target frame, and the display instruction is issued. In this case, when the pixel value of the predetermined pixel is changed, a voltage corresponding to the changed pixel value is applied to the liquid crystal corresponding to the predetermined pixel, that is, the applied voltage value changes. As a result, the light level emitted from the liquid crystal also changes.

  Thus, since the liquid crystal display device employs a display method different from the impulse display device such as a CRT, the installation space is small and the power consumption is small compared to the impulse display device. It has various features such as less distortion.

  Although only the liquid crystal display device has been described above, the above description generally applies to the following devices. That is, a display method is adopted in which at least a part of the display elements constituting the screen is held for a predetermined period after the display instruction of the display target frame is given (for example, a period corresponding to the above-described frame period). There is a display device. Hereinafter, such a display device is referred to as a hold type display device. Among such hold-type display devices, a display device using liquid crystal as a display element can be said to be a liquid crystal display device. Therefore, the above-described content applies to the entire hold type display device.

  In summary, the impulse-type display device has the characteristics that light emission is basically short afterglow and the light emission time is instantaneously fixed. On the other hand, the hold-type display device has a feature that it is basically continuous light emission (or display) and there is no dark section between frames.

  Here, let us consider a process of producing video information that can be displayed on either the impulse-type display device or the hold-type display device. Specifically, for example, let us consider the production process of video information shot and recorded by a camera capable of shooting a moving image.

  For example, as shown in FIG. 1, the camera used when producing such an image opens the shutter for a predetermined shutter opening time Ts every predetermined frame period Tf (for example, 1/24 Sec), The video information (subject light information) captured through the lens during the shutter opening time Ts is time-integrated, and the video information obtained as a result of the time integration is output as frame video data. Is recorded.

  As shown in FIG. 1, when the shutter opening time Ts is shorter than the frame period Tf, the shutter is opened again to shoot the next frame after the camera shutter has been opened. There is a time Tb until the shutter is closed, that is, a time Tb when the shutter is closed. At the time Tb, no video information (light information) is captured by the camera. That is, an event occurring in the real world during the time Tb when the shutter is closed is not photographed by the camera. Therefore, video information about an event occurring in the real world during the time Tb is not output from the camera and is not recorded.

  As the shutter opening time Ts, a single time length may be used, but in many cases, a plurality of discrete time lengths are properly used. That is, as the shutter speed corresponding to the shutter opening time Ts, various speeds are properly used based on the actual situation in the field and the intention of the photographer.

  For example, as shown in FIG. 2, let us consider a case where a moving image including a scene 1 where a train arrives at a station and a scene 2 where a door of the train is opened and a passenger gets off is shot.

  The upper view of FIG. 2 shows a timing chart at the time of photographing with the camera.

  Assume that frame numbers n-3 to n + 2 are assigned to each frame in order to distinguish the predetermined frame from other frames. However, n is an arbitrary integer value.

  As shown in the upper diagram of FIG. 2, shooting is performed at the same shutter opening time Ts in the same scene, whereas shooting is performed at different shutter opening times Ts in another scene. is there. Here, for ease of explanation, for example, the shutter opening times Tsn-3 to Tsn-1 of the frames n-3 to n-1 constituting the scene 1 are set to the same 1/60 sec. To do. On the other hand, it is assumed that the shutter opening times Tsn to Tsn + 2 of the frames n to n + 2 constituting the scene 2 are the same 1/30 sec.

  In this case, for example, it is assumed that a data series of video information recorded by the camera (hereinafter referred to as a recording data series) is as shown in the central view of FIG.

  In this example, the recording data of frame number k (k is an integer value indicating any one of n−3 to n + 2) includes frame video data of frame number k and shutter speed data for frame number k. It consists of and. Here, the shutter speed data is information on all or a part of the metadata attached to each frame, and is specific information that can specify the shutter speed corresponding to the shutter opening time Ts for the corresponding frame. The shutter speed data is not essential data for the recording data series.

  In this case, when a moving image corresponding to the recording data series, that is, a moving image including the scene 1 and the scene 2 in the upper diagram of FIG. 2 is displayed by a predetermined display device, conventionally, the moving image shown in FIG. It was displayed as shown in the lower figure.

  As shown in the lower side of FIG. 2, the display time Tik of the frame of frame number k (hereinafter abbreviated as “frame k”) is constant at the same time as the frame period Tf. That is, the display time Tik is set to a fixed time without any change regardless of whether the shutter opening time Tsk is the same or different.

  Here, the frame k is a video corresponding to the frame video data of the frame number k. The frame video data of frame number k is the time integration of the video information (light information) captured by the camera during the shutter opening time Tsk, which is a part of the entire period of the frame period Tf. It has been done. That is, the “movement” generated in the real world during the shutter opening time Tsk is different from the “movement” generated during the display time Tik on the display device, that is, the shutter opening time Tsk (this example). Then, it is displayed as “movement” generated in the period of the frame period Tf (which is longer than that).

  In other words, each frame video data recorded as a time integral of video information (light information) captured by the camera during a different shutter opening time Tsk is changed to a frame period Tf different from that on the display device side. It is regarded as the opening time (period), and the opening time is uniformly displayed as the display time Tik.

  In this case, when the display device is an impulse-type display device, as described above, in the pixel unit, it does not continue to emit light during the display time Tik, but only emits light for a moment. is there. Therefore, even if the display time Tik is uniform in all frames, the fact that the shutter opening time Tsk differs between frames hardly affects, and as a result, the “movement” seen by the user's eyes is very uncomfortable. There will be no.

  On the other hand, when the display device is a hold-type display device, as described above, even when viewed in units of pixels, the pixels (liquid crystals) continue to emit light during the display time Tik. That is, when the frame period Tf, which is clearly longer than the shutter opening time Tsk in which video information (light information) is actually captured by the camera, is the display time Tik, the user can display the period of the display time Tik. A frame k composed of each pixel that continues to be displayed (emitted) in the middle is viewed. This contributes to the feeling of unnaturalness in the process of visually recognizing “movement”, that is, the cause of the above-described problem.

  Accordingly, the present inventors have invented a technique as shown in FIG. 3 in order to solve such a problem. That is, FIG. 3 is a diagram for explaining the outline of the technique to which the present invention is applied while comparing with the conventional technique. For this reason, in FIG. 2 and FIG. 3, the same figure is employ | adopted for an upper side figure and a center figure. That is, the recording data series to be displayed (refer to the central diagrams in FIGS. 2 and 3) are the same.

  In this case, as shown in the lower diagram of FIG. 3, the display time Tik of the frame k is not a fixed time that is the same as the frame period Tf, but is a variable time corresponding to the shutter opening time Tsk, and the remaining time In the period of time Tmk, that is, in the period from the elapse of the display time Tik to the display of the next frame k + 1, an image that is almost black or black, i.e., a dark image that has an extremely low luminance (hereinafter, A technique of performing display control for displaying a dark image) is an example of a technique invented by the present inventors.

  For example, in the example of FIG. 3, since the shutter speed data is added to the frame image data of the frame number k, the display time Tik of the frame k is the same as the shutter opening time Tsk by using this shutter speed data. Can be easily realized.

  In this case, the time Tmk coincides with the time Tbk in the upper diagram of FIG. 3, that is, the time Tbk when the shutter of the camera is closed at the time of photographing. That is, that the shutter of the camera is closed means that video information (light information) has not been captured by the camera during that period. Therefore, in this method, a dark image is displayed during the period of time Tbk. Hereinafter, the time Tmk for displaying such a dark image is referred to as a dark section Tmk.

  That is, in the example of FIG. 3, when the display device is a hold-type display device, each frame k (k = n−3 to n−1) corresponding to the scene 1 is actually measured in terms of pixels. Light is emitted (or displayed) for almost the same display time Tik as the shutter opening time Tsk = 1/60 of the shutter, and no light is emitted during the dark period Tmk that is substantially the same as the time Tbk when the actual shutter is closed (or It will be darkly displayed.

  That is, the user sees a frame k composed of pixels that continue to display (emit light) during the display time Tik that is substantially the same as the actual shutter opening time Tsk = 1/60. In the process of visually recognizing "", there is no sense of incongruity.

  Incidentally, in the recording data series in the example of FIG. 3, shutter speed data of each frame is added, but in the actual recording data series, there is specific information that can specify such a shutter release time Tsk. Some are not added.

  When such a recording data series is given as a display target, the shutter opening time Tsk may be estimated as follows on the display device side, for example.

  In other words, the display device may estimate the shutter opening time Tsk by executing, for example, the following steps S1 to S3.

  The process of step S1 refers to an inter-frame motion detection process.

  That is, each frame constituting the moving image is divided into blocks each having a small area of about 8 × 8 pixels, for example, and the similarities between the blocks of two frames (one of which is the display target frame k) are compared. The process of obtaining the amount of motion in units of blocks by detecting whether or not the coordinates of the blocks are moving between them is an example of the inter-frame motion detection process in step S1.

  The process of step S2 refers to a shake detection process.

  That is, for example, it is detected whether characteristic pixel data in the display target frame k is continuous in one direction, specifically, for example, a so-called thread pulling state, and based on the detection result. The process of detecting the movement amount (movement direction and distance) of the pixel data is an example of the blur detection process in step S2.

  The process of step S3 refers to a shutter opening time estimation process.

  That is, the process of estimating the shutter opening time Tsk based on the amount of motion obtained in the process of step S1 and the movement amount detected in the process of step S2 is an example of the shutter opening time estimation process of step S3. is there.

  Furthermore, in recent years, not only the recording data series of video shot by cameras, but also the CG (computer graphics) video recording data series that does not depend on camera shooting, and the motion blur function in the production process of digital intermediate, etc. In some cases, a recording data series of a video in which the visual effect is controlled by integrating the time axis component or the like is a display target.

  In such a case, it may be difficult for the display device to estimate the shutter opening time Tsk even if the video analysis processing such as the processing in steps S1 to S3 described above is performed.

  In such a case, as a simple process, for example, as shown in FIG. 4, the display device sets the display time Tik of each frame k regardless of scene switching, that is, regardless of the shutter opening time Tsk. You may perform the process which makes semi-fixed time Tic.

  Here, the reason for describing “semi-fixed” is to indicate that the time Tic is a fixed time at the time of display, but is variable before the display, that is, can be variably set by the user or the like. It is.

  As described above, even when the display time Tik of each frame k becomes the semi-fixed time Tic, the dark interval Tmk of the semi-fixed time Tmc is generated as a result. This means that the above-mentioned effects such as CG and motion blur function are taken into consideration to some extent. Therefore, even if the user visually recognizes that some inconsistency has occurred, the user can visually recognize that the overall moving image display performance is improved.

  It should be noted that the method of setting the display time Tik of each frame k to the semi-fixed time Tic can naturally be employed also when displaying a recorded data series for a natural image taken with a camera. In this case as well, it is preferable to comprehensively determine the characteristics of the entire moving image, and to set the semi-fixed time Tic that is visually felt as the display time Tik of the frame k. As a result, the phenomenon that the shutter of the camera does not continue to be opened at the time of shooting and there is a period during which the shutter is closed is taken into consideration. Can be reduced.

  Further, this method will be further described with reference to FIGS. 5 and 6 from the viewpoint of display of pixels (liquid crystal or the like) of the hold type liquid crystal display device.

  That is, FIG. 5 shows a display timing chart of pixels (liquid crystal or the like) of a conventional hold type liquid crystal display device. On the other hand, FIG. 6 shows a timing chart of display of pixels (liquid crystal or the like) of the hold type liquid crystal display device to which the present method is applied.

  A timing chart described as a frame k (k = 1 to 4 in the examples of FIGS. 5 and 6) indicates a display state (light emission state of a liquid crystal or the like) of a pixel of the hold type liquid crystal display device. It is assumed that the thickness in the direction perpendicular to the time axis t indicates the luminance of the pixel (light emission level of liquid crystal or the like). That is, at the timing described as the frame k, it is indicated that the display is performed with the luminance of the level designated by the frame video data of the frame number k (hereinafter referred to as the designated level). However, there is a response delay in the liquid crystal or the like constituting the pixel. This response delay time Td (= panel response speed / 2) is also considered in the examples of FIGS. That is, at the time when the frame switching command is issued, the luminance of the pixels such as the liquid crystal does not immediately reach the command level, but reaches the first time when the response delay time Td has elapsed. Therefore, the actual time that the user visually recognizes as the display time Tik of the frame k is a time obtained by subtracting the response delay time Td.

  In this case, as apparent from FIG. 5, the display time Tik of the conventional frame k (k = 3 in the example of FIG. 5) is the time obtained by subtracting the response delay time Td from the frame period Tf, and the actual shutter. It was fixed time regardless of the opening time Tsk.

  On the other hand, when this method is applied, if the shutter opening time Tsk at the time of shooting is known or can be estimated, the time obtained by subtracting the response delay time Td from the shutter opening time Tsk is displayed in the frame k. Will be able to be with time Tik. That is, as shown in the upper timing chart of FIG. 6, at the time of slow shutter, for example, when the shutter opening time Tsk is the same as the frame period Tf, the response delay time Td is set from the frame period Tf as in the conventional case. The subtracted time is the display time Tik of frame k. On the other hand, at the time of high-speed shutter, that is, when the actual shutter opening time Tsk is shorter than the frame period Tf, the display time Tik of the frame k can be made shorter than before, That is, it can be a time obtained by subtracting the response delay time Td from the actual shutter opening time Tsk, and dark display can be performed in other time zones.

  In this case, the luminance of the pixel visually recognized by the user depends not only on the light emission level of the pixel but also on the light emission time. The user (natural person) also visually recognizes the luminance by integrating the amount of light continuously captured in the eyes to some extent rather than only by the amount of light instantaneously captured in his / her eyes. It is.

  In this case, as is clear from comparison between the upper and lower timing charts of FIG. 6, the luminance command level for the predetermined pixel of frame k is the same level, while the display time Tik of the predetermined pixel of frame k is the same. Is changed according to the shutter opening time Tsk. Therefore, the shorter the shutter speed, that is, the shorter the shutter opening time Tsk, the shorter the light emission time corresponding to the display time Tik of the predetermined pixel in the frame k. Since the reduction in the light emission time of the pixel extends over the entire frame k, as a result, when the luminance of the entire frame k is low, the user visually recognizes, that is, the phenomenon that the frame k appears dark to the user's eyes. Arise.

  When it is necessary to prevent the occurrence of this phenomenon, in other words, when there is a desire to make the luminance of the frame k seen by the user the same as other frames, that is, to make the luminance of the entire moving image constant. For example, as shown in FIG. 7, a technique of correcting the luminance command level (light emission level) for the predetermined pixel of the frame k according to the display time Tik of the predetermined pixel of the frame k may be adopted.

  In other words, a technique may be adopted in which control is performed to vary the luminance command level (light emission level) for a predetermined pixel in frame k in inverse proportion to the proportion of the display time Tik in the frame period Tf. This is because the time integration amount of the light amount in the user's eyes can be kept substantially constant, and as a result, the user can visually recognize that the luminance is kept almost constant.

  Such a technique can be embodied as follows, for example. That is, if the luminance command level before correction is Lb and the luminance command level after correction is described as La, the luminance command level (light emission level) of a predetermined pixel is corrected according to the following equation (1). Is possible.

  Lb = La × (Tf / Tik) × min (Tik / Tf) (1)

  Here, min (Tik / Tf) indicates a minimum value among possible values taken as a calculated value of Tik / Tf. For example, in this example, the frame period Tf is a fixed value, while the display time Tik of a predetermined pixel in the frame k is a variable value. Therefore, if the shortest time among the display times Tik of the predetermined pixels in the frame k is described as the display time Tikmin, min (Tik / Tf) = Tikmin / Tf.

  Specifically, for example, as shown in the upper diagram of FIG. 7, when the display time Tik of a predetermined pixel of the frame k coincides with the frame period Tf, the display time Tik is described as a, and the frame period If Tf is described as b, it will be understood that b / a = 1 may be substituted for the term (Tf / Tik) in equation (1). For example, as shown in the lower diagram of FIG. 7, when the display time Tik of the predetermined pixel of the frame k is shorter than the frame period Tf, the display time Tik is described as c, and the frame period Tf is If d is described, it is understood that d / c may be substituted for the term (Tf / Tik) in the equation (1).

  FIG. 8 shows an example of the configuration of an image processing apparatus to which the above-described method is adopted, that is, an image processing apparatus to which the present invention is applied.

  In the example of FIG. 8, the image processing apparatus 11 is configured to include a dark image insertion control unit 21 and a hold type display device driving unit 22.

  The image processing device 11 is a device that controls the display of the hold-type display device 12. For example, when a recording data series to be displayed is input in units of frames, the frame video data of the input frame number k is set as a display target, and the luminance command of each pixel constituting the frame k is based on the content. A drive signal including a level is generated and provided to the hold type display device 12. Then, the hold type display device 12 displays the frame k according to the drive signal.

  In this case, a conventional display device driving unit 22 may be used as it is. In other words, when the hold-type display device driving unit 22 does not receive the control of the dark image insertion control unit 21, the same driving signal as that in the conventional case, that is, displays each pixel continuously throughout the entire period of the frame period Tf ( A drive signal including a command to continue to emit light is generated and provided to the hold type display device 12. As a result, the display state of the hold type display device 12 becomes the display state similar to the conventional case, that is, the display state shown in the timing chart of FIG.

  On the other hand, the image processing apparatus 11 in FIG. 8 has a dark image insertion control unit 21. The dark image insertion control unit 21 controls the hold type display device driving unit 22 to thereby set a time corresponding to the shutter opening time Tsk during the frame period Tf (from the frame switching command to the dark interval command in FIG. 6). Each pixel emits light (displays) only for a period of time), and a drive signal including a command to cause each pixel to emit substantially no light (display a dark image) in the subsequent dark section Tmk is generated, and hold type display Output to the device 12. As a result, the display state of the hold type display device 12 becomes the display state shown in the timing chart of FIG.

  In addition, the dark image insertion control unit 21 can also execute the brightness correction processing according to the above-described equation (1) as necessary.

  Note that the form of the output signal of the image processing device 11 when the control by the dark image insertion control unit 21 is performed, that is, the form of the drive signal of the hold type display device 12 is the hold type display device during the dark period Tmk. It is sufficient that 12 can display a dark image (does not emit light). In other words, the image processing apparatus 11 may output a driving signal suitable for the driving method of the hold type display device 12.

  Specifically, for example, when the hold type display device 12 adopts so-called DC (direct current) level driving, the image processing device 11 uses the signal of the form shown in FIG. 9 as a drive signal as the hold type display device 12. Can be output. In both figures of FIG. 9, the horizontal axis indicates the time axis, and the vertical axis indicates the command level (voltage level) of the luminance of the pixel. In this case, sections Tml and Tmh in which the brightness command level (voltage level) is 0 indicate dark sections. That is, the section Tml indicates a dark section at the time of low-speed shutter (when the shutter opening time Tsk is long). On the other hand, a section Tmh indicates a dark section at the time of high-speed shutter (when the shutter opening time Tsk is short).

  Further, for example, when the hold type display device 12 is configured by a liquid crystal device that requires so-called AC inversion, the image processing device 11 outputs a signal of the form shown in FIG. 10 to the hold type display device 12 as a drive signal. do it. That is, as shown in the hatched portion on the upper side of FIG. 10, the pixel luminance command level (voltage level) is inverted so as to be symmetric with respect to the reference potential, that is, as a result, the integration of the DC component. The image processing device 11 may output a drive signal having a value of zero to the hold type display device 12. In both figures of FIG. 10, the horizontal axis represents the time axis, and the vertical axis represents the pixel luminance command level (voltage level) when the reference potential is used as a reference. In this case, sections Tml and Tmh in which the brightness command level (voltage level) is 0 indicate dark sections. That is, the section Tml indicates a dark section at the time of low-speed shutter (when the shutter opening time Tsk is long). On the other hand, a section Tmh indicates a dark section at the time of high-speed shutter (when the shutter opening time Tsk is short).

  Further, for example, when the hold type display device 12 is configured by a so-called PWM (pulse width modulation) drive type element, the image processing device 11 uses the signal of the form shown in FIG. 11 as a drive signal to the hold type display device 12. Just output. In both figures of FIG. 11, the horizontal axis represents the time axis. In this case, the brightness command level (voltage level) varies depending on the density of each pulse. Therefore, the sections Tml and Tmh in which the pulse interval is open for a certain time or more indicate dark sections. That is, the section Tml indicates a dark section at the time of low-speed shutter (when the shutter opening time Tsk is long). On the other hand, a section Tmh indicates a dark section at the time of high-speed shutter (when the shutter opening time Tsk is short).

  As described above, the technique to which the present invention is applied is a display device such as the hold-type display device 12 according to the shutter opening time Tsk at the time of recording (the amount of time Tsk for taking in the light amount of each pixel constituting the frame). Makes the display time Tik of each frame variable when displaying a moving image. Thereby, the user can visually recognize the displayed moving image more naturally.

  That is, video information whose shutter opening time Tsk at the time of shooting is, for example, 1/100 second should be visually recognized as a natural movement for the user when it is displayed for a period of 1/100 second. Further, the shutter opening time Tsk is not fixed at 1/100 seconds, and is appropriately changed according to each scene such as 1/60, 1/100 seconds, 1/250 seconds.

  However, in the conventional hold-type display device, such a shutter opening time Tsk is not considered at all, and the display time Tik of the frame k is a uniform frame period if the response delay of pixels such as liquid crystal is ignored. The time was the same as Tf. In the case of a high-speed shutter or the like, the frame period Tf is clearly longer than the shutter opening time Tsk, which is the time when image information (light information) is captured at the time of shooting. Therefore, for a user who has viewed a moving image displayed by a conventional hold-type display device, a single frame k is viewed over a significantly longer time than when the image information (light information) is captured at the time of shooting. As a result, I felt unnaturalness in the process of motion recognition.

  On the other hand, according to the method to which the present invention is applied, the time when the image information (light information) is actually captured at the time of shooting, that is, the time corresponding to the shutter opening time Tsk is set as the display time Tik of the frame k. It is said. Therefore, it is possible for a user who has viewed a moving image displayed by the hold-type display device to view one frame k in substantially the same time as the time when image information (light information) is captured at the time of shooting. Therefore, the unnatural degree felt in the process of motion recognition can be reduced.

  In other words, when the user visually recognizes the moving image, the display time Tik of the frame k is a time corresponding to the time when the image information (light information) is actually captured at the time of shooting, that is, the time corresponding to the shutter opening time Tsk. It will be good.

  Therefore, as shown in FIG. 12, when the hold-type display device 12 is constituted by a projector or the like, the light emission (or display) of each pixel such as the liquid crystal of the hold-type display device 12 is the same as the conventional one. In other words, light is emitted (or displayed) for a period of the frame period Tf, but instead, a frame projected on the screen 53 by controlling the opening time of the lens of the projector with the rotary shutter 52 or the like. The display time Tik of k, that is, the display time Tik of the frame k visually recognized by the user may be set to a time corresponding to the shutter opening time Tsk.

  In other words, the image processing apparatus 51 of FIG. 12 is configured to include a hold type display device driving unit 61 and a dark section insertion control unit 62 so as to employ such a method. That is, the image processing apparatus 51 is an image processing apparatus according to another embodiment of the present invention.

  The hold type display device driving unit 61 has basically the same function and configuration as the hold type display device driving unit 22 of FIG. In other words, the same type as the conventional one may be adopted as the hold type display device driving unit 61.

  The rotary shutter 52 can be configured by stacking a plurality of rotary shutters such as rotary shutters 52-1 and 52-2, for example, as shown in the upper diagram of FIG. 13.

  In this case, the dark interval insertion control unit 62 controls the rotation phases φ1 and φ2 of the rotary shutters 52-1 and 52-2, respectively, thereby controlling the video occlusion time and eventually the frame that the user visually recognizes. The display time Tik of k can be set. The number of rotations in this case is preferably synchronized with the frame period Tf, for example.

  FIG. 13 is a diagram schematically illustrating the principle of the rotary shutter 52. That is, the display target frame k is actually projected from the entire lens, but is shown in the right part of the rotary shutter 52 in the two lower diagrams of FIG. A pattern in which the frame k to be displayed is shielded by the rotary shutter 52 is illustrated by a gray fan shape. In other words, an increase in the area of the gray fan shape means that the image shielding time becomes longer.

  For example, a state in which the rotational phases φ1 and φ2 of the rotary shutters 52-1 and 52-2 are set to the same phase is shown in the lower left side of FIG. In this case, the time for which the frame k is shielded by the rotary shutters 52-1 and 52-2 is the same.

  On the other hand, the state when the rotational phases φ1 and φ2 of the rotary shutters 52-1 and 52-2 are set to different phases is shown in the lower right side view of FIG. In this case, the time during which the frame k is shielded by the rotary shutters 52-1 and 52-2 is different. As a result, the rotary phases φ1 and φ2 of the rotary shutters 52-1 and 52-2 are set to the same phase. Compared to the case, the image shielding time becomes longer.

  This video occlusion time is a time corresponding to the dark interval Tmi for the user who is viewing the frame k projected on the screen 53. That is, the dark section insertion control unit 62 controls the rotational phases φ1 and φ2 of the rotary shutters 52-1 and 52-2 based on the actual shutter opening time Tsk, so that the frame k projected on the screen 53 is displayed. The display time Tik of the frame k seen by the user who is watching the video can be set to a time corresponding to the shutter opening time Tsk.

  In the above description, each display element (liquid crystal in the case of a liquid crystal display device) constituting the screen of the hold type display device 12 has a predetermined one of a plurality of pixels constituting the frame. However, a plurality of display elements may be associated with one pixel. That is, one pixel may be displayed by a plurality of display elements.

  In the above description, the reason why the hold type display device 12 is used as the display device whose display is controlled by the image processing device to which the present invention is applied is that the problem described above in [Problems to be solved by the invention]. This is to clarify that the point can be solved. However, there is no particular reason for limiting the application destination of the present invention to the hold type display device 12. That is, the present invention may be applied to other various display devices, such as an impulse display device.

  Furthermore, the image processing apparatus to which the present invention is applied is not limited to the configuration of FIG. 8 and FIG. 12 described above, and can take various embodiments.

  For example, the image processing device 11 and the hold type display device 12 in FIG. 8 are regarded as one image processing device, or the image processing device 51, the hold type display device 12, and the rotary shutter 52 in FIG. It can also be regarded as one display part.

  In other words, the present invention can be applied to, for example, an image processing element device having the following functions, and other elements such as the configuration are not particularly limited as long as such functions are provided.

  That is, information obtained as a result of integrating the video information of the subject during each sampling time (in the above example, the shutter opening time Ts of the camera) is unit image information (in the above example, frame video data), and a plurality of the above-mentioned information is obtained. The present invention can be applied to an image processing apparatus having a function of performing image processing on moving image information composed of unit image information.

  In this case, as the display control of the moving image information, the image processing apparatus starts display of each of the plurality of unit image information at predetermined cycle times, and the display time is based on the sampling time. It is preferable to have a display control function for performing control to be changed.

  Furthermore, it is preferable that the display control function of the image processing apparatus has a function of controlling the luminance at the time of displaying the unit image information based on the display time of the unit image information to be controlled.

  The series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, all or a part of the image processing apparatus is constituted by a personal computer as shown in FIG. 14, for example.

  In FIG. 14, a CPU (Central Processing Unit) 101 executes various processes according to a program recorded in a ROM (Read Only Memory) 102 or a program loaded from a storage unit 108 to a RAM (Random Access Memory) 103. To do. The RAM 103 also appropriately stores data necessary for the CPU 101 to execute various processes.

  The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104.

  Connected to the input / output interface 105 are an input unit 106 such as a keyboard and a mouse, an output unit 107 composed of a display, a storage unit 108 composed of a hard disk, and a communication unit 109 composed of a modem, a terminal adapter, and the like. ing. The communication unit 109 performs communication processing with other information processing apparatuses via a network including the Internet.

  In this case, the output unit 107 itself may be a hold-type display device, or an external hold-type display device is connected to a connection unit (not shown) connected to the input / output interface 105 as necessary. May be connected.

  A drive 110 is connected to the input / output interface 105 as necessary, and a removable medium 111 made up of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached, and a computer program read from them is loaded. These are installed in the storage unit 108 as necessary.

  When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer is installed from a network or a recording medium.

  As shown in FIG. 14, the recording medium including such a program is distributed to provide a program to the user separately from the apparatus main body, and a magnetic disk (including a floppy disk) on which the program is recorded. , Removable media (package media) consisting of optical disks (including compact disk-read only memory (CD-ROM), DVD (digital versatile disk)), magneto-optical disks (including MD (mini-disk)), or semiconductor memory ) 111 as well as a ROM 102 on which a program is recorded and a hard disk included in the storage unit 108 provided to the user in a state of being incorporated in the apparatus main body in advance.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the order, but is not necessarily performed in chronological order, either in parallel or individually. The process to be executed is also included.

  In addition, as described above, the moving image handled by the image processing apparatus according to the present embodiment can be in units of frames or in units of fields. Therefore, in this specification, such a unit is also referred to as an access unit.

It is a figure explaining the imaging | photography method of the moving image by a camera. It is a figure which shows the example of the method of the conventional display control in the case of displaying a moving image on a hold type display apparatus. It is a figure which shows the example of the method of the display control to which this invention is applied when displaying a moving image on a hold type display apparatus. FIG. 4 is a diagram showing an example of a display control method to which the present invention is applied when a moving image is displayed on a hold-type display device, and shows an example different from FIG. 3. It is the figure which showed the example of the method of the conventional display control of FIG. 2 from the viewpoint of the display of a hold type display apparatus. It is the figure which showed the example of the method of the display control to which this invention of FIG. 3 is applied shown from the viewpoint of the display of a hold type display apparatus. FIG. 7 is a diagram illustrating an example of a display control technique to which the present invention is applied when a moving image is displayed on a hold-type display device, and illustrates an example different from the example of FIG. 6. It is a functional block diagram which shows the functional structural example of the image processing apparatus to which this invention is applied. It is a figure which shows the example of the output signal of the image processing apparatus of FIG. FIG. 10 is a diagram illustrating an example of an output signal of the image processing apparatus in FIG. 8, which is an example different from FIG. 9. It is an example of the output signal of the image processing apparatus of FIG. 8, Comprising: It is a figure which shows the example different from FIG. 9 and FIG. FIG. 9 is a functional block diagram illustrating an example of a functional configuration of an image processing apparatus to which the present invention is applied, and illustrating an example different from FIG. 8. It is a figure explaining the operation example of the image processing apparatus of FIG. FIG. 11 is a block diagram illustrating a configuration example of a computer as a hardware resource for executing software when the present invention is implemented by software.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Image processing apparatus, 12 Hold type display apparatus, 21 Dark image insertion control part, 22 Hold type display apparatus drive part, 51 Image processing apparatus, 52 Rotating shutter, 53 Screen, 61 Hold type display apparatus drive part, 62 Dark section insertion Control unit, 101 CPU, 107 output unit, 110 drives, 111 removable media

Claims (12)

In an image processing apparatus for performing image processing on moving image information composed of a plurality of unit image information, using information obtained as a result of integrating video information of a subject during each sampling time as unit image information,
A display control circuit for controlling the display of the moving image information to start display of each of the plurality of unit image information at predetermined cycle times and to change the display time based on the sampling time. An image processing apparatus. The image processing apparatus according to claim 1, wherein the display control circuit further controls luminance at the time of displaying the unit image information based on a display time of the unit image information to be controlled. The image processing apparatus according to claim 2, wherein the display control circuit performs control to correct the luminance to a level inversely proportional to the display time of the unit image information. The image processing apparatus according to claim 1, wherein the display control circuit performs control to match a display time of the unit image information to be controlled with the sampling time. The image processing apparatus according to claim 1, wherein the display control circuit performs control to match all display times of the plurality of unit image information with a predetermined fixed time that is variably set based on the sampling time. The moving image information includes specific information for specifying the sampling time for each of the plurality of unit image information,
The image processing apparatus according to claim 1, wherein the display control circuit performs control to change a display time of the unit image information to be controlled based on the sampling time specified from the specific information. The image according to claim 1, wherein the display control circuit further estimates the sampling time of the unit image information to be controlled and changes the display time of the unit image information based on the estimation result. Processing equipment. The display control circuit further controls display of a dark image or prohibition of display itself until the display of the next unit image information is started after the display time of the unit image information to be controlled elapses. Item 8. The image processing apparatus according to Item 1. The moving image information is displayed by a display device that displays a video based on an electrical video drive signal,
The display control circuit controls the display time of the unit image information to be controlled and the time for displaying the dark image or prohibiting the display itself by controlling the video drive signal applied to the display device. The image processing apparatus according to claim 8. The moving image information is displayed by a display device that projects an image through a lens,
9. The display control circuit controls a display time of the unit image information to be controlled and a time for displaying the dark image or prohibiting display itself by controlling opening of the lens. Image processing apparatus. Image processing of an image processing apparatus that performs image processing on moving image information composed of a plurality of unit image information using information obtained as a result of integrating video information of a subject during each sampling time as unit image information In the method
As the display control of the moving image information, an image including a step of starting display of each of the plurality of unit image information at predetermined cycle times and changing the display time based on the sampling time. Processing method. A computer that performs image processing on moving image information composed of a plurality of unit image information, using information obtained as a result of integrating video information of a subject during each sampling time as unit image information,
As a control of the display of the moving image information, a step is executed in which display of each of the plurality of unit image information is started at predetermined cycle times, and the display time is changed based on the sampling time. program.

Images