JP2018011098A - Video processing apparatus, imaging apparatus and video processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform a satisfactory vibration-proof process near the beginning and termination of a playback video.SOLUTION: A video processing apparatus performs a processing for recording video data produced by imaging, and shack information about shake during imaging. The device has: providing means that provides video data with playback valid information indicating playback is possible and with playback invalid information indicating playback is impossible; and processing means that produces and records recording data including video data, shake information and playback identification information. According to an instruction to start recording, the processing means produces first data including video data in a first period before the start of recording and shake information and playback invalid information; second data including video data in a second period from the start to the end of recording corresponding to an instruction to end the recording and shake information and playback valid information; produces third data including video data in a third period after the recording and shake information and playback invalid information; and produces recording data integrating the first to the third data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video processing technique for performing reproduction image stabilization processing.

  When recording an image using an imaging device such as a video camera, the image may be shaken due to camera shake due to camera shake or the like. In Patent Document 1, in order to correct (reduce) such image shake at the time of video reproduction, reproduction image stabilization that cuts out and expands a partial area in the video in accordance with a motion vector amount as shake information acquired from the video. A process is disclosed. Patent Document 2 discloses a reproduction image stabilization process in which shake information based on the angular velocity of the imaging device detected by the angular velocity sensor at the time of recording is recorded together with a video, and the image stabilization process is performed using the shake information at the time of video reproduction. Has been. Further, Patent Document 3 discloses a reproduction anti-shake process in which pre-recorded shake information is pre-read and the anti-shake process is performed with a shake correction amount determined so that the enlargement rate of the partial area changes gradually.

JP-A-7-143380 Japanese Patent Laid-Open No. 10-42233 JP 2012-44418 A

  However, in the conventional reproduction image stabilization processing disclosed in these patent documents, shake information before the start of video recording and shake information after the end of video recording are not recorded. Good anti-vibration processing may not be performed in the vicinity or near the end. That is, image blur that should not be corrected originally is corrected, and unnatural fluctuations may occur in the reproduced video.

  The present invention provides a video processing apparatus and the like in which good image stabilization processing is performed near the beginning and end of a reproduced video.

  A video processing apparatus according to one aspect of the present invention performs processing for recording video data generated by imaging and shake information regarding shake during imaging. The video processing apparatus includes, as playback identification information indicating whether or not video data can be played back, providing means for giving playback valid information indicating that playback is possible and playback invalid information indicating that playback is not possible to video data, video data, and shake information And processing means for generating and recording recording data including reproduction identification information. In response to the recording start instruction, the processing means generates first data including video data, shake information, and reproduction invalid information in the first period before the recording start, and performs recording according to the recording end instruction from the recording start. Second data including video data, shake information, and reproduction valid information in the second period until the end is generated, and second data including video data, shake information, and reproduction invalid information in the third period after the recording ends. 3 is generated, and recording data obtained by integrating the first, second and third data is generated.

  A video processing apparatus according to another aspect of the present invention performs processing for dividing video data generated by imaging and recorded together with shake information regarding shake during imaging at an instructed division position. The video processing apparatus is provided with a granting unit that gives to the video data playback valid information indicating that playback is possible and playback invalid information indicating that playback is not possible as playback identification information indicating whether or not video data can be played back. And processing means for generating first divided data and second divided data including video data, shake information, and reproduction identification information. The processing means generates first data including a portion of the first period preceding the division position and the reproduction identification information among the video data and the shake information, and the first data following the division position of the video data and the shake information. The second data including the portion of the period 2 and the reproduction invalid information is generated, and the first divided data obtained by integrating the first and second data is generated. Further, the processing means generates video data and third data including shake information and reproduction invalid information in a third period preceding the division position, and fourth of the video data and shake information following the division position. The fourth data including the period portion and the reproduction identification information is generated, and the second divided data obtained by integrating the third and fourth data is generated.

  According to another aspect of the present invention, there is provided a video processing apparatus that includes first video data and a first video data that are generated by dividing video data that is generated by imaging and recorded together with shake information relating to shake during imaging. The processing of combining the two video data together with the shake information is performed. The video processing apparatus is playback identification information indicating whether or not each video data is given to each of the first and second video data, and includes playback valid information indicating that playback is possible and playback invalid information indicating that playback is not possible. And processing means for generating combined data obtained by combining the first and second video data, shake information, and reproduction identification information. Reproduction identification information for each of the first period portion of the first video data and the second period portion of the second video data that is the start period and precedes the first period. Is reproduction invalid information, each of a portion of the third period corresponding to the second period of the first video data and a portion of the fourth period corresponding to the first period of the second video data. The reproduction identification information for the first period information, the second period part, the third period part, and the fourth period part are referred to as first shake information, The second shake information, the third shake information, and the fourth shake information are used. The processing means compares the first shake information and the fourth shake information, compares the second shake information and the third shake information, and obtains the first shake information and the fourth shake information. When the second shake information matches the third shake information, the first and second video data, the shake information, and the reproduction identification information include the first and second period portions. The removed data is combined to generate combined data.

  Also, the video processing apparatus according to another aspect of the present invention provides reproduction valid information or reproduction indicating that reproduction is possible as shake information generated by imaging and relating to shake during imaging and reproduction identification information indicating whether or not the video data can be reproduced. A process of reproducing the video data recorded together with the reproduction invalid information indicating the impossibility is performed. The video processing apparatus reduces image blur when reproducing unit image data for each of a plurality of unit image data constituting video data based on shake information for a predetermined period prior to the unit image data. Therefore, the image forming apparatus includes a generation unit that generates shake correction information used in the image stabilization process, and a reproduction unit that reproduces the unit image data subjected to the image stabilization process using the shake correction information. The reproduction means does not reproduce the unit image data when the reproduction identification information for the unit image data is reproduction invalid information, and reproduces the unit image data when the reproduction identification information for the unit image data is reproduction valid information. It is characterized by that.

  Note that an imaging device including the video processing device also constitutes another aspect of the present invention. Further, a video processing program as a computer program that causes a computer to operate as the video processing apparatus constitutes another aspect of the present invention.

  According to the present invention, by using the shake information before the start of video recording and the shake information after the end of video recording, good image stabilization processing can be performed near the start and end of the video to be played back. It can be performed.

1 is a block diagram illustrating a configuration of an imaging apparatus including a video processing apparatus that is Embodiment 1 of the present invention. 3 is a flowchart showing video recording processing in Embodiment 1. FIG. 3 is a conceptual diagram illustrating video recording processing according to the first embodiment. The block diagram which shows the structure of the video processing apparatus which is Example 2 of this invention. 9 is a flowchart illustrating video division processing according to the second embodiment. FIG. 9 is a conceptual diagram illustrating video division processing in the second embodiment. 9 is a flowchart illustrating video combining processing according to the second embodiment. FIG. 9 is a conceptual diagram illustrating video combining processing in the second embodiment. The block diagram which shows the structure of the video reproduction apparatus provided with the video processing apparatus which is Example 3 of this invention. 10 is a flowchart illustrating video reproduction processing according to the third embodiment. FIG. 10 is a conceptual diagram illustrating video reproduction processing according to the third embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration of an imaging apparatus (video recording apparatus) 100 including a video processing apparatus that is Embodiment 1 of the present invention. The lens unit 101 forms an object image on the image sensor 102 by forming an image of light from an object (not shown). The imaging element 102 is configured by a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and photoelectrically converts (captures) a subject image.

  The analog signal processing unit 103 generates an analog imaging signal by performing predetermined processing on the electrical signal output from the imaging element 102. The analog signal processing unit 103 includes, for example, a CDS (Co-related Double Sampling) circuit and an AGC (Automatic Gain Control) circuit.

  The camera signal processing unit 104 includes an A / D converter, and converts the analog imaging signal generated by the analog signal processing unit 103 into a digital video signal. The recording control unit 105 delivers the digital video signal and video data to the encoding unit 106, the memory 107, the recording medium 116, and the motion vector amount acquisition unit 112 and the metadata setting unit 114 in the microcomputer 115. The recording medium 116 is a semiconductor memory or an optical disk. Delivery of video signals and video data by the recording control unit 105 is controlled by a microcomputer 115 (hereinafter abbreviated as a microcomputer).

  The encoding unit 106 encodes the digital video signal generated by the camera signal processing unit 104 and received from the recording control unit 105 into MPEG2 format video data, and outputs the encoded video data to the memory 107 via the recording control unit 105. The memory 107 is a volatile storage medium that temporarily records video data. The video data generated by the encoding unit 106 is buffered in the memory 107 for several seconds. Further, metadata output from a metadata setting unit 114 described later is also buffered in the memory 107 in the same manner. Note that the metadata is metadata (higher-order) data that is attached to the video data and includes information about the video data.

  The switch operation unit 108 includes a plurality of switches operated by the user, and inputs an instruction signal to the microcomputer 115 when a recording start / end switch for instructing start and end of video recording is operated. Receiving the instruction signal for instructing recording (recording start instruction signal), the microcomputer 115 causes the recording control unit 105 to generate video data and metadata, which are output to the recording medium 116 and recorded, as will be described later. Further, the microcomputer 115 that has received the instruction signal for instructing the end of recording (recording end instruction signal) ends the recording of the video data and metadata after recording the video data and metadata for a predetermined period as will be described later. To do. The recording control unit 105 and the microcomputer 115 constitute processing means.

  The microcomputer 115 includes a reproduction identification information setting unit 109, an angular velocity shake amount setting unit 111, and a motion vector setting unit 113 in addition to the motion vector amount acquisition unit 112 and the metadata setting unit 114 described above. The reproduction identification information setting unit (giving means) 109 sets reproduction identification information indicating whether or not the video data output to the recording medium 116 can be reproduced in the metadata setting unit 114 in response to the recording start / end instruction signal described above. . The angular velocity sensor 110 detects an angular velocity of shake (hereinafter referred to as camera shake) applied to the imaging apparatus due to camera shake or the like during imaging. The angular velocity shake amount setting unit 111 calculates an angular velocity shake amount that is a shake amount of the camera shake based on the output of the angular velocity sensor 110 and sets it in the metadata setting unit 114.

  The motion vector amount acquisition unit 112 acquires a motion vector amount generated during imaging from the video data stored in the memory 107 and sets it in the motion vector amount setting unit 113. The motion vector amount setting unit 113 sets the motion vector amount set by the motion vector amount acquisition unit 112 in the metadata setting unit 114.

  The metadata setting unit 114 controls recording of metadata including the reproduction identification information, the angular velocity shake amount, and the motion vector amount set by the reproduction identification information setting unit 109, the angular velocity shake amount setting unit 111, and the motion vector amount setting unit 113. The data is output to the memory 107 via the unit 105. In the following description, the angular velocity shake amount and the motion vector amount, which are information relating to camera shake, are collectively referred to as shake information.

  The flowchart of FIG. 2 shows the flow of video recording processing performed by the microcomputer 115 in this embodiment. The microcomputer 115 executes this processing according to a video processing program that is a computer program. Although this processing includes processing performed by the respective units 109 and 111 to 114 included in the microcomputer 115, the processing will be described as being performed by the microcomputer 115 collectively.

  In step S <b> 201 of FIG. 2, the microcomputer 115 causes the recording control unit 105 to output video data and metadata for several seconds to the memory 107. At this time, the microcomputer 115 sets the reproduction identification information included in the metadata to reproduction invalid information indicating that the reproduction of the video is invalid. The term “several seconds” as used herein refers to a time that is at least equal to the time that can be taken by the predetermined period T1 determined in step S202, which will be described later. Note that video data and metadata for several seconds are output using a part of the memory 107 as a ring buffer. That is, after video data and metadata for several seconds are accumulated, the oldest video data and metadata in time series are overwritten with the latest video data and metadata.

  Next, in step S202, the microcomputer 115 sets a predetermined period T1, which is a first period, based on shake information (an angular velocity shake amount and a motion vector amount) included in the metadata. In general, during panning (including horizontal panning and vertical tilting), the amount of shake, which is the amount of shake indicated by the shake information, increases. Therefore, the amount of shake correction for correcting (reducing) image shake is the upper limit. There is a risk of reaching the value. For this reason, the microcomputer 115 performs control so as to stop the image stabilization processing for correcting image blur in the video during panning. However, if the predetermined period T1 is not a sufficient time for determining whether panning or camera shake, image blur that should not be corrected may be corrected. Therefore, the microcomputer 115 sets the predetermined period T1 to be longer as the shake amount is larger (shorter as the shake amount is smaller). This also applies to a predetermined period T3 described later.

  In step S203, the microcomputer 115 determines whether there is an instruction to start recording a video (input of a recording start instruction signal). The microcomputer 115 proceeds to step S204 when instructed to start video recording, and returns to step S201 when not instructed.

  In step S204, the microcomputer 115 generates pre-recording start data (first data) including a predetermined period T1 of the video data and metadata for several seconds output to the memory 107 in step S201 in the recording control unit 105. And output to the recording medium 116. The reproduction identification information of the metadata at this time remains the reproduction invalid information set in step S201.

  In step S205, the recording control unit 105 starts generation of post-recording data (second data) including video data and metadata from the start of recording to the end of recording described later, and outputs the data to the recording medium 116. . At this time, the microcomputer 115 sets the reproduction identification information of the metadata to reproduction valid information indicating that the video data can be reproduced.

  Next, in step S206, the microcomputer 115 sets a predetermined period T3, which is a third period, similarly to the setting of the predetermined period T1 in step S202.

  In step S207, the microcomputer 115 determines whether or not there is an instruction to end video recording (input of a recording end instruction signal). If the end of video recording is instructed, the microcomputer 115 proceeds to step S208. If the end of recording is not instructed, the microcomputer 115 returns to step S205.

  In step S208, the microcomputer 115 causes the recording control unit 105 to finish generating and outputting the data after the start of recording, and further, the data after the end of recording including the video data and metadata for a predetermined period T3 after the end of recording (the third data Data) is generated and output to the recording medium 116. The microcomputer 115 sets the reproduction identification information of the metadata included in the post-recording data to reproduction invalid information indicating that reproduction is impossible.

  Next, in step S209, the microcomputer 115 causes the recording control unit 105 to record the metadata-added video data (recording data) obtained by integrating the data before the recording start, after the recording start, and after the recording ended, which have been output to the recording medium 116 so far. Recording is performed on the recording medium 116. The microcomputer 115 that has generated and recorded (saved) the video data with metadata in this way ends this processing.

  FIG. 3 shows the contents of the video recording process described above. First, when there is a recording start instruction in step S203 in FIG. 2, the data before recording start (video data and metadata) for the predetermined period T1 described in step S204 is output to the recording medium 116. The reproduction identification information of the metadata at this time is the reproduction invalid information set in step S201.

  Subsequently, post-recording start data (video data and metadata) for the recording period (second period) T2 from the recording start described in step S205 to the recording end in response to the recording end instruction is output to the recording medium 116. The The reproduction identification information of the metadata at this time is reproduction valid information.

  When a recording end instruction is issued in step S207, post-recording data (video data and metadata) for the predetermined period T3 described in step S208 is output to the recording medium 116. The reproduction identification information of the metadata at this time is reproduction invalid information. Finally, the data before the start of recording, after the start of recording and after the end of recording output to the recording medium 116 in step S209 is recorded as a set of video data with metadata.

  As described above, in this embodiment, recording pre-recording data including shake information indicating angular velocity shake amount and motion vector amount before video recording start and post-recording data including shake information after video recording end are recorded. Record with data after start. For this reason, when reproducing the video data included in the data after the start of recording, it is possible to use the shake information included in the data before the start of recording and after the end of recording.

  FIG. 4 shows the configuration of a video processing apparatus 400 that is Embodiment 2 of the present invention. The video processing device 400 may be configured by a personal computer, or may be built in an imaging device or a video playback device.

  Video data with metadata recorded by the imaging apparatus 100 described in the first embodiment is recorded on the recording medium 401. As described in the first embodiment, the metadata includes angular velocity shake amount and motion vector amount as shake information, and reproduction valid information and reproduction invalid information as reproduction identification information.

  The recording / playback control unit 402 delivers video data and metadata to the decoding unit 403, the memory 404, the recording medium 401, and the metadata reading / setting unit 402 in the microcomputer 409. Delivery of video data and metadata by the recording / playback control unit 402 is controlled by a microcomputer (hereinafter abbreviated as “microcomputer”) 409. The recording medium 401 is configured by a semiconductor memory, an optical disk, or the like.

  The decoding unit 403 decodes the video data recorded on the recording medium 401 and outputs the decoded digital video signal to the memory 404 via the recording / playback control unit 402. The memory 404 is a volatile storage medium that temporarily records a digital video signal.

  The switch operation unit 405 includes a plurality of switches operated by the user, and inputs a division instruction signal to the microcomputer 409 when a division instruction switch for instructing video division processing is operated. Receiving the division instruction signal, the microcomputer 409 decodes the video data recorded on the recording medium 401 and causes the display device (not shown) to display the decoded video signal. The display device may be provided in the video processing device 400 or may be provided outside the video processing device 400. The video signal is displayed in order to instruct the user to divide the video. Then, the microcomputer 409 performs processing shown in FIG. 5 described later in response to the instruction of the division position by the user, causes the recording / playback control unit 402 to generate the divided video data with metadata and record it on the recording medium 116.

  The switch operation unit 405 inputs a display instruction signal to the microcomputer 409 when a display instruction switch for instructing display of the video divided by the video division processing is operated among the plurality of switches. Receiving the display instruction signal, the microcomputer 409 decodes the video data recorded on the recording medium 401 and causes the display device to display the decoded video signal. Then, in response to an instruction for video combination by the user, the microcomputer 115 performs processing of FIG. 7 described later, and causes the recording / playback control unit 402 to record the combined video data with metadata on the recording medium 116. The recording / playback control unit 402 and the microcomputer 409 constitute processing means.

  A metadata reading / setting unit (providing unit and acquiring unit) 406 reads (acquires) metadata associated with video data recorded on the recording medium 401 and associates it with video data to be recorded on the recording medium 401. Set (grant) metadata. The reproduction identification information reading / setting unit 407 reads reproduction identification information from the metadata read by the metadata reading / setting unit 406. Further, the reproduction identification information reading / setting unit 407 sets the reproduction identification information in the metadata reading / setting unit 406 in response to the video division processing and video combination processing instructions notified from the microcomputer 115. The shake information reading / setting unit 408 reads shake information from the metadata read by the metadata reading / setting unit 406. Also, shake information is set in the metadata read / set unit 406.

  The flowchart of FIG. 5 shows the flow of the video dividing process performed by the microcomputer 409 in this embodiment. The microcomputer 409 executes this processing according to a video processing program that is a computer program. This processing includes processing performed by each of the units 406 to 408 included in the microcomputer 409, and will be described as being performed collectively by the microcomputer 409. These are the same in the video combining process described later.

  In step S501, the microcomputer 409 decodes the video data before division recorded (saved) on the recording medium 401 in accordance with the division instruction signal, and displays the video on the display device. At this time, a user interface (video division UI) for instructing a division position (time position) for video division is also displayed. The division position can be specified in the reproduction effective portion in which the reproduction identification information included in the metadata is the reproduction effective information in the video data before division.

  In step S <b> 502, the microcomputer 409 determines whether there is a division position instruction through the video division UI. The microcomputer 409 proceeds to step S503 when the division position is instructed, and returns to step S501 when the division position is not instructed.

  In step S503, the microcomputer 409 sets a predetermined period (second period) TD2 subsequent to the division position based on the shake information before and after the division position. As will be described later in steps S504 and S505, the entire period preceding the division position (first period) is assumed to be TD1. The predetermined period TD2 is a period of video data and metadata subsequent to the video data and metadata for the period TD1, and is a period for using shake information in the vicinity of the end of the period TD1.

  As described in the first embodiment, since the shake amount increases during panning and the shake correction amount may reach the upper limit value, the microcomputer 409 controls to stop the image stabilization process during panning. However, if the predetermined period TD2 is not a sufficient time for determining whether panning or camera shake, there is a risk of correcting image blur that should not be corrected. For this reason, the microcomputer 409 sets the predetermined period TD2 to be longer as the shake amount is larger (shorter as the shake amount is smaller).

  In step S504, the microcomputer 409 causes the recording / playback control unit 402 to generate divided position preceding data (first data) including video data and metadata for the entire period TD1 preceding the divided position, and the recording medium 401. To output. The reproduction identification information of the metadata at this time is reproduction invalid information in the portion (reproduction invalid portion) corresponding to the pre-recording data described in the first embodiment, and the portion (reproduction valid portion) corresponding to the data after the recording start. Then, it is reproduction effective information.

  In step S505, the microcomputer 409 causes the recording / playback control unit 402 to generate divided position subsequent data (second data) including video data and metadata for a predetermined period TD2 subsequent to the divided position, and to record the recording medium. 401 is output. At this time, the microcomputer 409 sets the reproduction identification information of the metadata as reproduction invalid information.

  Next, in step S506, the microcomputer 409 causes the recording / playback control unit 402 to combine the divided position preceding data and the divided position succeeding data output to the recording medium 116 in steps 504 and S505, and provide the divided video data with metadata. (First divided data) A is recorded. The microcomputer 409 that has generated and recorded (saved) the divided video data A with metadata in this way proceeds to step S507.

  Next, in step S507, the microcomputer 409 sets a predetermined period (third period) TD3 preceding the division position, similarly to the predetermined period TD2 set in step S503. As will be described later in steps S508 and S509, the entire period (fourth period) following the division position is TD4. The predetermined period TD3 is a period of video data and metadata preceding the video data and metadata for the entire period TD4, and is a period for using shake information in the vicinity of the start end of the period TD4.

  In step S508, the microcomputer 409 causes the recording / playback control unit 402 to generate divided position preceding data (third data) including video data and metadata for a predetermined period TD3 preceding the divided position, and recording media. 401 is output. At this time, the microcomputer 409 sets the reproduction identification information of the metadata as reproduction invalid information.

  In step S509, the microcomputer 409 causes the recording / playback control unit 402 to generate divided position subsequent data (fourth data) including video and metadata for the entire period TD4 preceding the divided position, and the recording medium 401. To output. In the reproduction identification information of the metadata at this time, the portion corresponding to the data after the recording start described in the first embodiment is the reproduction valid information, and the portion corresponding to the data after the recording is the reproduction invalid information.

  Next, in step S510, the microcomputer 409 causes the recording / playback control unit 402 to add video data with metadata (second division) by integrating the division position preceding and division position subsequent data output to the recording medium 116 in steps 508 and S509. Data) B is recorded. In this way, the microcomputer 409 that has generated and recorded (saved) the video data B with metadata after the division ends this processing.

  FIG. 6 shows the contents of the above-described video dividing process. First, in step S501 in FIG. 5, the video corresponding to the video data with metadata before division is displayed on the display device in accordance with the division instruction signal. When the division position is instructed in step S502, the division position preceding data (video data and metadata) for the period TD1 described in step S504 is output to the recording medium 401. The reproduction identification information of metadata is reproduction invalid information of a part corresponding to data before recording start and reproduction valid information of a part corresponding to data after recording start.

  Subsequently, the division position subsequent data (video data and metadata) for the predetermined period TD2 described in step S505 is output to the recording medium 401. The reproduction identification information of the metadata is reproduction invalid information. Then, the divided position preceding data for the period TD1 and the divided position subsequent data for the predetermined period TD2 output to the recording medium 401 are recorded on the recording medium 401 as one video data A with metadata.

  Next, the division position preceding data (video data and metadata) for the predetermined period TD3 described in step S508 is output to the recording medium 401. The reproduction identification information of the metadata is reproduction invalid information.

  Subsequently, the division position subsequent data (video data and metadata) for the period TD4 described in step S509 is output to the recording medium 401. In the reproduction identification information of the metadata, the portion corresponding to the data after the start of recording is the reproduction valid information, and the portion corresponding to the data after the recording is the reproduction invalid information. Finally, the division position preceding data for the predetermined period TD3 and the division position subsequent data for the period TD4 output to the recording medium 401 are recorded as one video data B with metadata.

  The flowchart of FIG. 7 shows the flow of video combining processing performed by the microcomputer 409 in this embodiment. In step S701, the microcomputer 409 decodes the video data A and B with metadata (hereinafter simply referred to as video data A and B) before being combined, which is recorded (saved) in the recording medium 401, in accordance with the display instruction signal. And the corresponding video is displayed on the display device. At this time, a user interface (video combination UI) for instructing video combination is also displayed.

  In step S <b> 702, the microcomputer 409 determines whether there is an instruction to combine images through the image combining UI. If the video combination is instructed, the microcomputer 409 proceeds to step S703, and if the video combination is not instructed, the microcomputer 409 returns to step S701.

  In step S703, the microcomputer 409 determines whether shake information included in the metadata of the two pieces of video data A and B to be combined has an overlap. A period (first period) in which the reproduction identification information is reproduction invalid information on the end side of the video data A is TC1, and a period (second second) in which the reproduction identification information is reproduction invalid information on the start side of the video data B. (Period) is TC2. Further, a period (third period) corresponding to the period TC2 excluding the period TC1 on the end side of the video data A is defined as TC3, and a period (fourth period) on the start end side of the video data B excluding the period TC2 is defined. ) Let TC4 be a period corresponding to TC1.

  The microcomputer 409 compares the shake information (first shake information and fourth shake information) of the periods TC1 and TC4, and also shake information (second shake information and third shake information) of the periods TC2 and TC3. ) To determine whether they match. If the shake information of the period TC1 and the period TC4 match and the shake information of the period TC2 and the period TC3 match, the microcomputer 409 proceeds to step S704. On the other hand, if the shake information of the period TC1 and the period TC4 do not match and the shake information of the period TC2 and the period TC3 do not match, the process proceeds to step S705.

  In step S704, the microcomputer 409 determines the video data and metadata of which the reproduction identification information in the metadata including the overlapping shake information is the reproduction invalid information among the video data A and B, that is, the videos in the period TC1 and the period TC2. Delete data and metadata. Then, the remaining video data B is combined with the remaining video data A, and the recording / playback control unit 402 outputs the combined video data with metadata (combined data) to the recording medium 401.

  In step S705, the microcomputer 409 combines the video data B with the video data A without deleting the data in step S704. Then, the recording / playback control unit 402 outputs the combined video data with metadata to the recording medium 401. The combined video data with metadata (combined data) generated in this step includes video data and metadata for the periods TC1 and TC4.

  In step S706, the microcomputer 409 causes the recording / playback control unit 402 to record the combined video data with metadata on the recording medium 401.

  FIG. 8 shows the contents of the above-described video combining process. First, in step S701 in FIG. 7, the video corresponding to the video data A and B before combination is displayed according to the display instruction signal. When there is an instruction for video combination, the shake information of the periods TC1 and TC4 and the shake information of the periods TC2 and TC3 are compared between the video data A and B, respectively.

  When the shake information of the periods TC1 and TC4 and the shake information of the periods T2 and T3 match each other (CASE1), the video data and the metadata of the video data A in the period TC1 are deleted in step S704, and the video data B has the period TC2 Video data and metadata are deleted. The reproduction identification information of the metadata of the deleted periods TC1 and TC2 is reproduction invalid information. Then, the remaining video data B is combined with the remaining video data A, and the combined video data with metadata is recorded (saved) on the recording medium 401.

  On the other hand, when the shake information of the periods TC1 and TC4 and the shake information of the periods TC2 and TC3 do not match (CASE2), the video data B as it is is combined with the video data A as it is at step S705. Then, the combined video data with metadata is recorded (saved) on the recording medium 401.

  According to the present embodiment, even when video data divided by the video dividing process or video data combined by the video combining process is reproduced, the shake information included in the data before recording start and the data after recording end in the video data Can be used.

  FIG. 9 shows the configuration of a video processing apparatus 900 that is Embodiment 3 of the present invention. The video processing device 900 may be configured by a personal computer, or may be incorporated in an imaging device or a video playback device.

  Video data with metadata generated by the imaging apparatus 100 described in the first embodiment is recorded (saved) on the recording medium 901. This video data includes video data and metadata included in the data before the start of recording, after the start of recording, and after the end of recording described in the first embodiment. The metadata includes angular velocity shake amount and motion vector amount as shake information, and reproduction valid information and reproduction invalid information as reproduction identification information.

  The playback control unit 902 sends video data and metadata to the decoding unit 903, the memory 904, the recording medium 901, the image processing unit 911, and the metadata reading unit 906 and the shake correction amount determination unit 909 in the microcomputer 910. Deliver. Delivery of video data and metadata by the reproduction control unit 902 is controlled by a microcomputer (hereinafter abbreviated as “microcomputer”) 910. The recording medium 901 is configured by a semiconductor memory, an optical disk, or the like.

  The decoding unit 903 decodes the video data recorded on the recording medium 901 and outputs the decoded digital video signal to the memory 904 via the reproduction control unit 902. The memory 904 is a volatile storage medium that temporarily holds a digital video signal. The video data and metadata decoded by the decoding unit 903 are buffered in the memory 904 for several seconds.

  The switch operation unit 905 includes a plurality of switches operated by the user, and inputs a playback preparation instruction signal to the microcomputer 910 when a playback preparation switch for starting preparation for video playback is operated. Receiving the reproduction preparation instruction signal, the microcomputer 910 performs processing shown in FIG. 10 to be described later, and reproduces a video based on the video data recorded on the recording medium 901 through the image processing unit 911.

  A metadata reading unit 906 reads metadata associated with video data recorded on the recording medium 901. The reproduction identification information reading unit 907 reads reproduction identification information from the metadata read by the metadata reading unit 906. The shake information reading unit 908 reads shake information from the metadata read by the metadata reading unit 906.

  Based on the shake information read by the shake information reading unit 908, the shake correction amount determination unit (generation unit) 909 calculates the position and size of the cutout range as shake correction information for correcting image shake in the video data. (Generate) and set them in the image processing unit 911. The image processing unit 911 reproduces video data while performing image stabilization processing according to the position and size of the cutout range determined by the shake correction amount determination unit 709. Specifically, the image processing unit 911 is determined by the shake correction amount determining unit 709 for each of a plurality of unit image data (field image or frame image data: hereinafter simply referred to as a unit image) constituting the video data. The partial images within the cutout range are cut out sequentially. Then, the image processing unit 911 sequentially outputs the reproduction unit images generated by performing the electronic zoom process on each of the cut out partial images to the display device 912. The display device 912 reproduces and displays the video by sequentially displaying the input reproduction unit images. The image processing unit 911, the reproduction control unit 902, and the microcomputer 910 constitute a reproducing unit.

  The flowchart of FIG. 10 shows the flow of video reproduction processing performed by the microcomputer 910 in this embodiment. The microcomputer 910 executes this processing according to a video processing program that is a computer program. This processing includes processing performed by each of the units 906 to 909 included in the microcomputer 910, but will be described as being performed collectively by the microcomputer 910. In the following description, video data is simply referred to as video.

  In step S1001, the microcomputer 910 displays a user interface (video playback UI) for video playback processing on the display device 912 in response to the input of the playback preparation instruction signal.

  In step S <b> 1002, the microcomputer 910 determines whether or not a reproduction start is instructed through the video reproduction UI. The microcomputer 910 proceeds to step S1003 when the reproduction start is instructed, and returns to S1001 when the reproduction start is not instructed.

  In the following description, the Nth (1 ≦ n ≦ N) frame among the N (N ≧ 1) unit images constituting the video V to be reproduced is denoted as F (n), and the frame The position of the cutout range from the image F (n) is denoted as P (n) and the size is denoted as S (n). Also, the shake information of the metadata corresponding to the frame image F (n) is expressed as B (n), and the reproduction identification information is expressed as I (n).

  In step S1003, the microcomputer 910 acquires shake information for a predetermined period TR from the shake information B (n) for the frame image F (n) in the video V. Usually, since the camera shake has a frequency of 4 to 9 Hz, the predetermined period TR is set to about 0.5 seconds corresponding to two cycles of the frequency, for example.

  In step S1004, the microcomputer 910 determines whether panning has been performed (camera shake has occurred). Specifically, the microcomputer 910 determines that the shake amount (angular velocity shake amount and motion vector amount) for the predetermined period TR acquired in step S1003 exceeds a predetermined threshold and increases or decreases in one direction. It is determined that panning has occurred. If the microcomputer 910 determines that panning is not performed, the microcomputer 910 proceeds to step S1005. If the microcomputer 910 determines that panning is performed, the microcomputer 910 proceeds to step S1006. In general, since the amount of shake increases during panning, the position of the cutout range may reach the limit at the end of the frame image. For this reason, the microcomputer 910 controls to stop the image stabilization process when it is determined that panning is being performed.

  In step S1005, the microcomputer 910 determines the position P of the cutout range in the frame image F (n) so as to correct the image shake of the video V to be reproduced and displayed based on the shake amount in the predetermined period TR acquired in step S1003. (N) and size S (n) are set.

  On the other hand, in step S1006, the microcomputer 910 sets the position P (n) and size S (n) of the cutout range in the frame image F (n) so as to stop the correction of the image blur of the video. Specifically, the microcomputer 910 determines the position P (n) and size S (n) of the cutout range in the frame image F (n) as the position P of the cutout range in the previous frame image F (n−1). The same as (n-1) and size S (n-1). In the case of n = 1, the microcomputer 910 sets the cutout position P (n) so as to cut out the center of the frame image F (n), and sets the shake amount corresponding to the threshold value in step S1004. Based on this, the cutout range size S (n) is set.

  In step S1007, it is determined whether or not the reproduction identification information I (n) in the video V is reproduction valid information (reproduction invalid information). The microcomputer 910 proceeds to step S1008 when it is reproduction valid information, and proceeds to step S1009 when it is reproduction invalid information.

  In step S1008, the microcomputer 910 sets the position P (n) and size S (n) of the cutout range in the frame image F (n) set in step S1005 or S1006 in the image processing unit 911. As a result, the video V is reproduced and displayed on the display device 912. Thereafter, the microcomputer 910 proceeds to step S1009.

  In step S1009, the microcomputer 910 determines whether all the frame images F (1) to F (N) in the video V have been reproduced and displayed. The microcomputer 910 ends this process when all the frame images are reproduced and displayed. If all the frame images have not been displayed yet, the microcomputer 910 proceeds to step S1003 and performs the above-described processing for the next frame image F (n + 1).

  FIG. 11 shows an outline of the above-described video reproduction process. First, in response to an instruction to start reproduction, shake information B (n) corresponding to the frame image F (n) of the video V is added to the frame image F (n) of the video V as described in step S1003 of FIG. Acquired for a predetermined period TR.

  Next, as described in step S1004, it is determined whether panning (camera shake) is performed using the obtained shake information for a predetermined period TR. Based on the result of the panning determination, the position P (n) and the size S (n) of the cutout range for the frame image F (n) are set as described in steps S1005 and S1006.

  Next, as described in step S1007, it is determined whether or not the reproduction identification information I (n) corresponding to the frame image F (n) is reproduction valid information. If it is the reproduction valid information, as described in step S1008, a partial image is cut out from the frame image F (n) at the position P (n) and the size S (n) of the cutout range, and electronic zoom processing is performed. Display playback.

  As described above, according to the present embodiment, when the recorded video data is reproduced, the start or end of the video data is obtained by using the shake information included in the pre-recording data and the post-recording data in the video data. Good anti-vibration processing (reproduction anti-vibration processing) can be performed in the vicinity.

In each of the above-described embodiments, the case where shake information and reproduction identification information are recorded as metadata accompanying video data has been described. However, shake information and reproduction identification information are not necessarily recorded as metadata, and may be recorded in any form as long as they are recorded in association with video data. In each embodiment, the case where the video data with metadata is recorded or recorded on the recording medium has been described, but the video data with metadata may be recorded or recorded on the network.
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

DESCRIPTION OF SYMBOLS 100 Imaging device 115,409,910 Microcomputer 110 Angular velocity sensor 112 Motion vector amount acquisition part 400,900 Video processing apparatus

Claims (14)

A video processing apparatus that performs processing for recording video data generated by imaging and shake information relating to shake during imaging,
A granting unit for giving to the video data playback valid information indicating that playback is possible and playback invalid information indicating that playback is not possible, as playback identification information indicating whether or not the video data can be played back,
Processing means for generating and recording recording data including the video data, the shake information and the reproduction identification information;
The processing means includes
In response to the recording start instruction, the first data including the video data and the shake information and the reproduction invalid information in the first period before the recording start is generated.
Generating the second data including the video data and the shake information and the reproduction valid information in a second period from the start of recording to the end of recording according to the recording end instruction;
Generating third data including the video data and the shake information and the reproduction invalid information in a third period after the end of recording;
An image processing apparatus that generates the recording data obtained by integrating the first, second, and third data.   The video processing apparatus according to claim 1, wherein the processing unit sets the first and third periods to be longer as the shake indicated by the shake information is larger. A video processing device that performs processing to divide video data generated by imaging and recorded together with shake information about shake during imaging at an instructed division position,
A granting unit for giving to the video data playback valid information indicating that playback is possible and playback invalid information indicating that playback is not possible, as playback identification information indicating whether or not the video data can be played back,
Processing means for generating first divided data and second divided data each including the divided video data, the shake information, and the reproduction identification information,
The processing means includes
Generating first data including a portion of a first period preceding the division position and the reproduction identification information of the video data and the shake information;
Generating second data including a portion of a second period following the division position and the reproduction invalid information of the video data and the shake information;
Generating the first divided data obtained by integrating the first and second data;
Generating third data including the video data and the shake information and the reproduction invalid information in a third period preceding the division position;
Generating fourth data including a portion of a fourth period following the division position and the reproduction identification information among the video data and the shake information;
An image processing apparatus that generates the second divided data obtained by integrating the third and fourth data. The division position is indicated in a reproduction effective portion in which the reproduction identification information is the reproduction effective information in the video data before division,
The video processing apparatus according to claim 3, wherein the processing means uses the reproduction identification information in a period corresponding to the reproduction effective portion in the first and fourth data as the reproduction effective information. The video data before the division includes a reproduction invalid portion whose reproduction identification information is the reproduction invalid information,
5. The video processing apparatus according to claim 4, wherein the processing means uses the reproduction identification information in a period corresponding to the reproduction invalid portion in the first and fourth data as the reproduction invalid information.   6. The video processing apparatus according to claim 3, wherein the processing unit sets the second and third periods longer as the shake indicated by the shake information is larger. The first video data and the second video data generated by dividing the video data generated by the imaging and recorded together with the shake information regarding the shake during the imaging are combined with the shake information. A video processing device,
Acquisition means for acquiring reproduction identification information indicating whether or not each of the video data assigned to each of the first and second video data is reproducible, wherein the reproduction valid information indicating that reproduction is possible and reproduction invalid information indicating that reproduction is impossible When,
Processing means for generating combined data obtained by combining the first and second video data, the shake information, and the reproduction identification information,
For each of the first period portion of the first video data in the first period and the second period portion of the second video data in the start period and preceding the first period. The reproduction identification information is the reproduction invalid information, and a portion of a third period corresponding to the second period of the first video data and a first period of the second video data. The reproduction identification information for each of the portions of the fourth period is the reproduction valid information,
The shake information for each of the first period part, the second period part, the third period part, and the fourth period part is referred to as first shake information, second shake information, When setting the third shake information and the fourth shake information,
The processing means includes
Comparing the first shake information and the fourth shake information, comparing the second shake information and the third shake information,
When the first shake information and the fourth shake information match, and when the second shake information and the third shake information match, the first and second video data and the An image processing apparatus, wherein the combined data is generated by combining portions of the shake information and the reproduction identification information excluding the portions of the first and second periods.   When the first shake information and the fourth shake information do not match, and the second shake information and the third shake information do not match, the processing means does not match the first shake information and the fourth shake information. 8. The video according to claim 7, wherein the combined data is generated by combining the video data of 2, the shake information, and the reproduction identification information including the first and second periods. Processing equipment. The video data generated by the imaging and recorded with the shake information relating to the shake during the imaging and the playback identification information indicating whether or not the video data can be played back is displayed together with the playback valid information indicating that playback is possible or the playback invalid information indicating that playback is not possible. A video processing device that performs processing,
Anti-shake to reduce image shake when reproducing unit image data based on the shake information for a predetermined period prior to the unit image data for each of a plurality of unit image data constituting the video data Generating means for generating shake correction information used for processing;
Playback means for playing back the unit image data subjected to the image stabilization processing using the shake correction information;
The reproduction means does not reproduce the unit image data when the reproduction identification information for the unit image data is the reproduction invalid information, and when the reproduction identification information for the unit image data is the reproduction valid information. A video processing apparatus for reproducing the unit image data.   An image pickup apparatus comprising the video processing apparatus according to claim 1. A computer program for causing a computer to execute processing for recording video data generated by imaging and shake information relating to shake during imaging,
The process is
A first process of giving reproduction valid information indicating that reproduction is possible and reproduction invalid information indicating that reproduction is impossible to the video data as reproduction identification information indicating whether the video data can be reproduced;
A second process for generating and recording recording data including the video data, the shake information, and the reproduction identification information;
In the second process,
In response to the recording start instruction, the first data including the video data and the shake information and the reproduction invalid information in the first period before the recording start is generated.
Generating the second data including the video data and the shake information and the reproduction valid information in a second period from the start of recording to the end of recording according to the recording end instruction;
Generating third data including the video data and the shake information and the reproduction invalid information in a third period after the end of recording;
An image processing program for generating the recording data obtained by integrating the first, second and third data. A computer program for causing a computer to execute a process of dividing video data generated by imaging and recorded together with shake information relating to shake during imaging at an instructed division position,
The process is
A first process of giving reproduction valid information indicating that reproduction is possible and reproduction invalid information indicating that reproduction is impossible to the video data as reproduction identification information indicating whether the video data can be reproduced;
A second process for generating first divided data and second divided data each including the divided video data, the shake information, and the reproduction identification information;
In the second process,
Generating first data including a portion of a first period preceding the division position and the reproduction identification information of the video data and the shake information;
Generating second data including a portion of a second period following the division position and the reproduction invalid information of the video data and the shake information;
Generating the first divided data obtained by integrating the first and second data;
Generating third data including the video data and the shake information and the reproduction invalid information in a third period preceding the division position;
Generating fourth data including a portion of a fourth period following the division position and the reproduction identification information among the video data and the shake information;
An image processing program for generating the second divided data obtained by integrating the third and fourth data. A process for combining the first video data and the second video data generated by dividing the video data generated by the imaging and recorded together with the shake information regarding the shake during the imaging together with the shake information. A computer program to be executed by
The process is
Reproduction identification information indicating whether or not each piece of video data assigned to each of the first and second video data is reproducible, and includes first reproduction valid information indicating that reproduction is possible and reproduction invalid information indicating that reproduction is impossible. And processing
A second process for generating combined data obtained by combining the first and second video data, the shake information, and the reproduction identification information,
For each of the first period portion of the first video data in the first period and the second period portion of the second video data in the start period and preceding the first period. The reproduction identification information is the reproduction invalid information, and a portion of a third period corresponding to the second period of the first video data and a first period of the second video data. The reproduction identification information for each of the portions of the fourth period is the reproduction valid information,
The shake information for each of the first period part, the second period part, the third period part, and the fourth period part is referred to as first shake information, second shake information, When setting the third shake information and the fourth shake information,
In the second process,
Comparing the first shake information and the fourth shake information, comparing the second shake information and the third shake information,
When the first shake information and the fourth shake information match, and when the second shake information and the third shake information match, the first and second video data and the A video processing program for generating the combined data by combining portions of the shake information and the reproduction identification information excluding the portions of the first and second periods. The video data generated by the imaging and recorded with the shake information relating to the shake during the imaging and the playback identification information indicating whether or not the video data can be played back is displayed together with the playback valid information indicating that playback is possible or the playback invalid information indicating that playback is not possible. A computer program for causing a computer to execute processing,
The process is
Anti-shake to reduce image shake when reproducing unit image data based on the shake information for a predetermined period prior to the unit image data for each of a plurality of unit image data constituting the video data A first process for generating shake correction information used in the process;
A second process for reproducing the unit image data subjected to the image stabilization process using the image stabilization information,
In the second process, when the reproduction identification information for the unit image data is the reproduction invalid information, the unit image data is not reproduced, and the reproduction identification information for the unit image data is the reproduction valid information. In this case, a video processing program for reproducing the unit image data.