JP2002125199A - Frame information description method, frame information generation device and method, video reproduction device and method, and recording medium - Google Patents

Abstract

(57) [Problem] To provide a special reproduction control information generation device for generating special reproduction control information that enables more effective special reproduction for a user. SOLUTION: In a special reproduction control information generating device, target video data is read from a video data storage unit 2, and a video position information processing unit 11 converts a part of frames to be used for special reproduction along a frame sequence. To selectively extract
Image position information indicating the position of each extracted frame in the original image data is created, and the display time control information processing unit 12 creates display time control information indicating the display time of each frame. Then, special reproduction control information in which video position information and display time control information of each frame are arranged as frame information is created, and this is stored in the special reproduction control information storage unit 3.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for describing frame information for specially reproducing digital contents, for example, video contents, a frame information generating apparatus and method, a video reproducing apparatus and method, and a recording medium.

[0002]

2. Description of the Related Art In recent years, an environment in which a video can be reproduced at random has been established by compressing a moving image as a digital video and storing it on a disk medium such as a DVD or an HDD. In this environment, midway reproduction is possible from a predetermined location with little waiting time. As with the conventional tape media, it is also possible to perform fast and reverse rotation at 2 to 4 times speed.

[0003]

However, the video is often long, and there is a problem that the time is not sufficiently reduced in order to view all the contents even at the reproduction speed of 2 to 4 times. If the playback speed is too large, the screen changes become larger than the viewing ability, making it difficult to grasp the contents. There is.

[0004] Such a problem applies not only to video contents but also to contents composed of sound and text or multimedia contents.

An object of the present invention relates to a frame information description method, a frame information generation apparatus and method, a video reproduction apparatus and method, and a recording medium that enable more effective special reproduction for a user.

[0006]

In order to solve the above-mentioned problems and achieve the object, the present invention uses the following means.

A frame information description method according to the present invention relates to a step of describing first information for specifying a position in a source video data of a frame extracted from a plurality of frames of the source video data, and a display time of the extracted frame. Writing the second information.

[0008] The extracted frame comprises a frame group,
The first information may include information for specifying a position of the extracted frame group in the original video data.

The method may further include a step of describing third information relating to the importance of the extracted frame.

[0010] The first information may include information for specifying an image data file corresponding to the extracted frame created from the original video data.

[0011] The extracted frame may include a frame extracted from a plurality of frames in a certain temporal section of the original video data, and may further include a step of describing fourth information for specifying the temporal section.

[0012] The first information may include information for specifying an image data file corresponding to the extracted frame created from the original video data.

[0013] The second information may include information relating to a display time so that the screen change amount during reproduction is substantially constant.

The method may further include a step of describing fifth information for instructing reproduction or non-reproduction of the extracted frame.

The first information is information indicating a position of the extracted frame or a position of image data corresponding to the extracted frame in an image data file generated from the original video data and stored separately from the original video data. May be provided.

[0016] The method may further include the step of describing information indicating the position of the media data and information relating to the display time of the media data in media data other than the original video data including the extracted frame.

In a computer-readable recording medium for storing frame information relating to frames extracted from a plurality of frames of original video data according to the present invention, the frame information specifies a position of the extracted frame in the original video data. 1 information and second information about the display time of the extracted frame.

The extracted frame is composed of a frame group,
The first information may include information for specifying a position of the extracted frame group in the original video data.

[0019] The frame information may further include third information relating to the importance of the extracted frame.

[0020] The first information may include information for specifying an image data file corresponding to the extracted frame created from the original video data.

[0021] Along with the frame information, the original video data and an image data file corresponding to the extracted frame created from the original video data may be further stored.

A frame information description device according to the present invention includes means for describing first information for specifying a position in a source video data of a frame extracted from a plurality of frames of the source video data, and a second information relating to a display time of the extracted frame. 2 means for describing information.

According to the frame information generating method of the present invention, there is provided a step of generating first information for specifying a position of a frame extracted from a plurality of frames of original video data in the original video data; Generating two pieces of information.

The video reproducing apparatus of the present invention refers to the first information for specifying the position in the original video data of the frames extracted from the plurality of frames of the original video data, and the second information for the display time of the extracted frame. Means for obtaining the original video data of the extracted frame based on the first information; means for determining a display time for reproducing the original video data of the extracted frame based on the second information; Means for reproducing the determined original video data for the determined display time.

In the video reproducing method according to the present invention, reference is made to first information for specifying the position of a frame extracted from a plurality of frames of the original video data in the original video data, and second information relating to the display time of the extracted frame. Obtaining the original video data of the extracted frame based on the first information; determining a display time for reproducing the original video data of the extracted frame based on the second information; Reproducing the determined original video data for the determined display time.

In a computer-readable recording medium storing a video reproduction program according to the present invention, the video reproduction program causes a computer to specify a position in the original video data of a frame extracted from a plurality of frames of the original video data. 1 information and program code for referring to second information relating to the display time of the extracted frame; program code for causing a computer to acquire the original video data of the extracted frame based on the first information; A program code for determining a display time for reproducing the original video data of the extracted frame based on information; and a program code for reproducing the display time for the determined original video data obtained by a computer. And

The frame information description method according to the present invention relates to a step of describing first information for specifying a position in an original sound data of a sound frame extracted from a plurality of frames of the original sound data, and a reproduction time of the extracted frame. Writing the second information.

In a computer-readable recording medium for storing frame information relating to sound frames extracted from a plurality of frames of original sound data according to the present invention, the frame information specifies a position of the extracted frame in the original sound data. It is characterized by comprising first information and second information relating to the reproduction time of the extracted frame.

According to the frame information description method of the present invention, first information for specifying the position of a text frame extracted from a plurality of frames of original text data is described, and second information relating to the display time of the extracted frame is described. And a step.

In a computer-readable recording medium for storing frame information relating to a text frame extracted from a plurality of frames of original text data according to the present invention, the frame information specifies a position of the extracted frame in the original text data. First
Information and second information relating to the display time of the extracted frame.

According to the present invention, there are provided a frame information description method, a frame information generation apparatus and method, a video reproduction apparatus and method, and a recording medium which enable more effective special reproduction for a user.

[0032]

Embodiments of the present invention will be described below with reference to the drawings.

Although the present invention relates to all digital contents, as an embodiment, reproduction of video contents having video data will be described. The video data is assumed to be composed of a set of video frames (video frame group) constituting a moving image.

First, special reproduction control information that plays an important role in the present embodiment will be described.

The special reproduction control information is control information for special reproduction of the target video data, and is created by the special reproduction control information generating device based on the video data and added to the video data. Control information to be performed or associated. The special reproduction is a reproduction by a method other than the normal reproduction, for example, double-speed reproduction (or high-speed reproduction), interlaced reproduction (or intercontinuous continuous reproduction),
There is trick play. There are various types of trick playback, such as replacement playback, overlap playback, and slow playback. The special reproduction control information is referred to when performing special reproduction in, for example, a video reproduction device that reproduces video data.

FIG. 1 shows an example of a basic data structure of trick play control information created based on video data to be trick played.

This data structure includes a plurality of pieces of frame information i.
(I = 1 to N) are described in association with the frame appearance order in the original video data, and each frame information is an original video (one frame or a plurality of continuous frames) to be displayed during special reproduction. Group or a plurality of adjacent frames, that is, a frame group consisting of a part of a plurality of continuous frames) and video position information 101 including information indicating the location of the original frame, and should be displayed during special reproduction of the original video. Display time control information 10 including information indicating the display time and / or information used to calculate the display time
Includes two sets.

FIG. 1 describes the frame information i arranged in the order in which the frames appear in the video data. If information indicating the order of the frame information is described in the frame information i, the frame information i is described. They may be arranged and described in any order.

The reproduction magnification information 103 attached to the frame information group composed of the plurality of frame information i indicates the magnification of the reproduction speed of special reproduction. It is used to shorten the display time and specify that playback is to be performed several times faster. However, the reproduction magnification information 103 is not essential information. There are a configuration that is always attached, a configuration that is not always attached, and a configuration that allows the user to individually select whether to attach. Even if the reproduction magnification information 103 is attached, the reproduction magnification information 103 does not necessarily need to be used in the special reproduction. There are a configuration that is always used, a configuration that is not always used, and a configuration that allows individual selection of whether to use it.

In FIG. 1, it is also possible to adopt a configuration in which other control information is added to the frame information group together with the reproduction magnification information or in place of the reproduction magnification information. In FIG. 1, a configuration in which other control information is added to each frame information i is also possible. In these cases, the video playback device may use all of the information included in the special playback control information, or may use only some of the information.

FIG. 2 shows a configuration example of such a device for generating special reproduction control information.

As shown in FIG. 2, this special reproduction control information generation apparatus includes a video data storage unit 2, a video data processing unit 1 including a video position information processing unit 11 and a display time control information processing unit 12, a special reproduction The control information storage unit 3 is provided. As will be described in detail later, in the configuration of FIG. 2, when displaying video data, original video data (encoded) is displayed.
Is decoded and displayed as image data, so that it takes a processing time for decoding from when the display is instructed until the image is actually displayed. In order to shorten this time, a method of decoding video data used for special reproduction in advance and storing an image data file is conceivable. When such an image data file is used (when an image data file is always used or when it is possible to select whether to use an image data file), as shown in FIG. The processing unit 1 further includes an image data file creation unit 13,
The image data file storage unit 4 is connected to the video data processing unit 1. When adding other control information obtained based on video data to the special reproduction control information, a corresponding function is added to the video data processing unit 1 as appropriate.

In the present process, in the case of taking the form of intervening a user's operation, a GUI that provides a function of, for example, displaying video data in frame units and receiving an instruction input of the user is used (see FIG. 2 and FIG. 2). It is omitted in FIG. 3).

2 and 3, a CPU, a memory,
External storage device and network communication device provided as needed, driver software and OS used as needed
The software such as is omitted.

The video data storage unit 2 stores special reproduction control information or special reproduction control information and video data to be processed for generating an image data file. The special reproduction control information storage unit 3 stores the generated special reproduction control information.

The image data file storage unit 4 stores the image data file created by the image data file creation unit 13.

Each of the video data storage unit 2, the special reproduction control information storage unit 3, and the image data file storage unit 4 is composed of, for example, a hard disk, an optical disk, a semiconductor memory, or the like. The video data storage unit 2, the trick play control information storage unit 3, and the image data file storage unit 4 may be configured by separate storage devices, but all or a part of them is configured by the same storage device. You may.

The video data processing section 1 is for generating special reproduction control information (or special reproduction control information and an image data file) based on video data to be processed.

The video position information processing section 11 determines (extracts) video frames (group) to be displayed or displayable at the time of special reproduction, and outputs the information 10 to be described in each frame information i.
1 is performed.

The display time control information processing section 102 performs processing for creating information 102 relating to the display time of the video frame (group) related to each frame information.

The image data file creating section 13 performs a process of creating each image data file from video data.

The trick play control information generating device can be realized, for example, by executing software on a computer. It may be realized as a dedicated device for generating special reproduction control information.

FIG. 4 shows an example of a control information generation procedure in the case of the configuration of FIG. Image data is read from the storage unit 2 (step S11), image position information is created (step S12), and display time control information is created (step S1).
3) The special reproduction control information including the video position information and the display time control information is stored in the storage unit 3 (step S14).
The procedure of FIG. 4 may be performed sequentially for each frame information, or each process may be performed in a batch. Other procedures are possible.

FIG. 5 shows an example of a control information generation procedure in the case of the configuration of FIG. A procedure for creating and storing an image data file is added to the procedure of FIG. 4 (step S22). Here, the creation and / or saving of the image data file is performed together with the creation of the video position information. However, the creation and storage can be performed at a timing different from that in FIG. As in the case of FIG. 4, the procedure of FIG. 5 may be performed sequentially for each frame information, or each process may be performed in a batch. Other procedures are possible.

Next, FIG. 6 shows a configuration example of a video reproducing apparatus.

As shown in FIG. 6, the video reproducing apparatus includes a control unit 21, a normal reproduction processing unit 22, a special reproduction processing unit 23, a display unit 24, and a content storage unit 25. When handling content in which sound such as audio is added to video data, it is desirable to have an audio output unit. When handling content in which text data is added to video data, the text may be displayed on the display unit 24 or output from the audio output unit. When handling contents to which a program is attached, an attached program execution unit may be provided.

The content storage unit 25 stores at least video data and special reproduction control information. As will be described later in detail, in a case where an image data file is used, an image data file is further stored. Voice data, text data, and an attached program may be further stored.

The content storage unit 25 may be centrally arranged at one place or may be dispersedly arranged at a plurality of places. In short, the contents storage unit 25 only needs to be accessible by the normal reproduction processing unit 22 and the special reproduction processing unit 23. . The video data, the special reproduction control information, the image data file, the audio data, the text data, and the attached program may be stored on different media, or may be stored on the same medium. As the medium, for example, a DVD or the like is used. They may be data transmitted over a network.

The control unit 21 basically receives an instruction such as normal reproduction or special reproduction for a content from a user via a user interface such as a GUI, and designates the designated processing portion with respect to the designated content. Control such as instructing reproduction by the specified method.

The normal reproduction processing section 22 is for normal reproduction of the designated content.

The special reproduction processing section 23 refers to the special reproduction control information for the specified content, and performs special reproduction (for example, double speed reproduction, jump reproduction,
Trick play).

The display unit 24 is for displaying an image.

The video reproducing apparatus can be realized, for example, by executing software on a computer (of course, hardware may be partially used (for example,
Decode board (MPEG-2 decoder) etc.)). It may be realized as a dedicated device for video reproduction.

FIG. 7 shows an example of a reproduction processing procedure of the video reproduction apparatus of FIG. In step S31, it is determined whether the user has requested normal reproduction or special reproduction. If there is a request for normal reproduction, the designated video data is read in step S32, and normal reproduction is performed in step 33. If there is a special reproduction request from the user, step S34
The special reproduction control information corresponding to the video data designated by the step is read, the position of the video to be displayed is specified in step 35, and the display time is determined. In step 36, the corresponding frame (group) is included in the video data ( Or, it reads from the image data file) and performs special reproduction of the content designated in step 37. The specification of the position of the video to be displayed and / or the determination of the display time can be performed at a timing different from that in FIG. The special reproduction procedure of FIG. 7 may be performed sequentially for each frame information, or each processing may be performed in a batch. Other procedures are possible. For example, in the case of a reproduction method in which the display time of each frame is set to a constant value, the display time is not determined.

In both normal playback and special playback,
The user can make various designations (e.g., a playback start point and / or a playback end point for content, a playback speed for double speed playback, a playback time for double speed playback,
It is more effective to be able to request other special reproduction methods.

Next, an algorithm for generating the frame information of the special reproduction control information, an algorithm for determining the display time during the special reproduction, and the like will be schematically described.

At the time of generating frame information, a frame to be used in special reproduction is determined from video data, video position information is generated, and display time control information is generated.

The frame is determined based on (1) a method based on some characteristic amount of the video data (for example, a specific characteristic amount between adjacent frames (for example, a screen change amount between frames) is determined between each extracted frame). To make the sum constant, and to make the sum of the importance of all the frames between the extracted frames constant),
(2) A method based on a uniform standard (for example, a method of extracting at random, a method of extracting at equal intervals), and the like.

The display time control information is created by (i) a method of obtaining an absolute value or a relative value of the display time or the number of display frames, and (ii) information serving as a reference of the display time or the number of display frames (for example, a user-specified value). , A character in a video, a sound synchronized with the video, a person in the video, or a degree of importance obtained based on a specific pattern in the video, etc.), (iii)
) There is a method of describing both (i) and (ii) above.

(1) or (2), (i) or (i)
It can be appropriately combined with i) or (iii). Of course, other methods are possible. Only one specific combination among them may be possible, or a plurality of combinations among them may be made possible to be appropriately selected.

As a special case, there is a method in which the relative value of the display time or the number of display frames is obtained at the same time as the frame determination by the method (1). If this method is always used, the display time control information processing unit 102 can be omitted.

At the time of special reproduction, it is assumed that the reproduction is performed with reference to the display time control information (i), (ii), or (iii) included in the frame information. May be used by modifying the described value.In addition to the stated value or the modified value, other information prepared by the user or information input by the user may be used. May be used, or only other information prepared independently or information input by the user may be used. A plurality of methods among them may be made possible and may be appropriately selected.

Next, an outline of the trick play will be described.

The double-speed reproduction (or high-speed reproduction) reproduces a part of all the frames constituting the video data content in a shorter time than the time required for normal reproduction of the original content. It is for regenerating. For example, the frames indicated by the frame information are displayed in chronological order by the display time indicated by the frame information. From the user, a double speed specification for specifying how many times the normal content playback speed is to be played (or a fraction of the time required for normal playback of the original content) or , And accepts requests to specify how long it takes to play,
The display time of each frame (group) may be obtained and reproduced so as to satisfy the request. Therefore, this double speed reproduction is also referred to as summary reproduction.

In the interlaced playback (or interleaved continuous playback), in double-speed playback, for example, a part of the frame indicated by the frame information is not played back based on play / non-playback information described later. The frame is reproduced at double speed except for the frame indicated by the frame information except the non-reproduced frame.

The trick reproduction is a reproduction other than the normal reproduction except for the double speed reproduction and the jump reproduction. For example, when reproducing the frame indicated by the frame information, the switching reproduction is performed by changing the chronological order for a certain portion, and when reproducing the frame indicated by the frame information, the certain portion is repeatedly reproduced a plurality of times. When reproducing the frame indicated by the reproduction and frame information, a certain portion is reproduced at a lower speed than other portions (including a case where the speed is set to the normal reproduction speed and a case where the speed is set lower than the normal reproduction). Or variable-speed playback that reproduces at a higher speed than other parts, or displays for a fixed time and stops, or appropriately combines them, and random playback that plays back a time series at random for a certain group of frames indicated by frame information Various forms are conceivable.

Of course, a combination of a plurality of types of methods may be used. For example, at the time of double-speed reproduction, various variations can be considered, such as a method of reproducing an important portion a plurality of times and setting a reproduction speed to a normal reproduction speed.

Hereinafter, the present embodiment will be described in more detail.

First, a case where the amount of screen change between adjacent frames is used as a characteristic value of image data for determining a frame will be described.

Here, a case will be described in which one frame corresponds to one frame information.

FIG. 8 shows an example of the data structure of the special reproduction control information created based on the target video data.

This data structure is used as display time control information 102 in FIG. 1 (or display time control information 102).
), The display time information 121 which is information indicating the absolute or relative display time is described. A configuration for describing the importance in the display time control information 102 will be described later.

The video position information 101 is information that enables the position of the video in the original video frame to be specified. One frame in the stream such as a frame number (for example, a sequence number from the first frame) or a time stamp is used. Anything that can be specified may be used. When the image data corresponding to the frame extracted from the original video stream is set as another file, a URL or the like may be used as information for specifying the file position.

The display time information 121 is information that can specify the time or the number of frames for displaying the video, and is described in a method of actually describing the time or the number of frames as a unit and in other frame information. There is a method of describing a relative value (for example, a normalized numerical value) so that the relation of the relative time length with the display time information can be understood. In the latter case, the actual playback time of each video is calculated from the entire total playback time. For each video,
Rather than describing the duration of the display, a description of the combination of the start time and the end time starting from a specific timing (for example, the start time of the first video is set to 0), or the combination of the start time and the duration May be used.

In the special reproduction, the video present at the position specified by the video position information 101 is displayed on the display time information 121.
Is basically performed sequentially for the number of frame information included in the array.

In the case where the start time and the end time or the continuation time are specified, and in accordance with the specification, the video present at the position specified by the video position information 101 is identified by the start time specified by the display time information 121. Reproduction from time to end time is basically performed sequentially by the number of frame information included in the array.

By using parameters such as the reproduction magnification and other additional information, the described display time can be processed and reproduced.

Next, a description will be given of a method of describing the position information of a video with reference to FIGS.

FIG. 9 is a diagram for explaining a description method of video position information referring to an original video frame.

In FIG. 9, a time axis 200 corresponds to an original video stream for which frame information for special reproduction is created, and an image 201 corresponds to one frame to be described in the video stream. The time axis 202 corresponds to the video playback time when performing special playback using the image 201 extracted from the original video stream.
03 is a section corresponding to one image 201 included therein. In this case, for example, a set of video position information 101 indicating the position of the image 201 and video display time 121 indicating the length of the display time 203 is described as frame information. As described above, the description of the position of the image 201 may be anything that can specify one frame in the original video stream, such as a frame number or a time stamp. This frame information is similarly described for other images.

FIG. 10 is a diagram for explaining a description method of video position information referring to an image data file.

The method of describing the video position information shown in FIG. 9 directly refers to the frame in the original video data to be trick-played, but the method of describing the video position information shown in FIG. The image data 300 corresponding to the single frame 302 extracted from the original video stream is prepared in another file, and its position is described. The description method of the file position is, for example, URL
By using such a method, it is possible to handle the case where it exists on a local storage device and the case where it exists on a network in the same way. Video position information 101 indicating the position of the image data file and the corresponding display time 30
A set with the video display time 121 indicating the length of 1 is described as frame information.

If correspondence with the original video frame is required,
Information indicating a single frame 302 of the original video corresponding to the described frame information (for example, the same as the video position information in the case of FIG. 9) may be included in the frame information. In this case, the frame information is composed of video position information, display time information, and original video information. of course,
Original video information need not be described if it is not necessary.

The form of the image data described by the method shown in FIG. 10 is not particularly limited. For example, the frame of the original video may be used as it is, or may be used after being reduced. This is effective for performing high-speed playback processing because it is not necessary to develop the original video.

The creation of a reduced image is performed when the original video stream is M
When compressed by PEG-1 or MPEG-2, the stream is only partially decoded,
Can be created at high speed. In this method, a reduced image is created by decoding only DCT (discrete cosine transform) coefficients of an intra-frame encoded I picture frame (intra-frame encoded frame) and using its DC component.

In the description method shown in FIG. 10, the image data is stored in separate files, but these files are stored in a randomly accessible video format (for example,
(Motion JPEG). In this case, the position of the image data is a URL indicating the position of the image data group storage file.
And a combination of a frame number or a time stamp indicating a position in the image data group storage file. UR indicating the location of the image data group storage file
L information may be described in individual frame information,
It may be described as additional information outside the array of frame information.

Various methods can be used to select which frame of the original video is to be selected to create the image data and describe it in the video position information. For example, image data may be extracted from the original video at equal intervals, a large amount of image data may be extracted at narrow intervals in areas where the screen moves a lot, and a small amount of image data may be extracted at wide intervals where there is little movement. You may.

Referring to FIG. 11, as an example of a frame selection method, a lot of image data is extracted at a narrow interval in a place where the screen moves a lot, and a wide interval is extracted in a place with a little movement according to the movement of the screen. A method for extracting a small amount of image data will be described.

In FIG. 11, the horizontal axis represents the frame number, and the curve 800 represents the change in the screen change amount (between adjacent frames). The calculation method of the screen change amount of each frame is the same as the method of obtaining display time information described later. Here, in order to determine the extraction interval in accordance with the movement of the screen, a method of obtaining an interval at which the image change amount between video frames from which image data is extracted becomes constant will be described. The sum of the screen changes between the video frames from which the image data is extracted is S _i , and the sum of the screen changes of all the frames is S (= ΣS _i ).
And the number of image data to be extracted is n. In order to make the image change amount between the image data extraction source frames constant, S _i =
S / n is sufficient. In FIG. 11, the curve 8 of the screen change amount
00 the area S _i of the interval delimited by the dashed line corresponds to a constant. Therefore, for example, from the head frame, and sequentially adds screen variation, the value to the frames that exceed the S / n with the image data extraction source of the video frame F _i.

When image data is generated from an MPEG I-picture frame, the calculated image data generation source frame is not necessarily an I-picture. Therefore, image data is generated from a nearby I-picture frame.

By the way, in the method described with reference to FIG. 11, the video belonging to the section where the screen change amount = 0 is skipped. However, for example, when a still image continues, it is often an important scene. Therefore, when the screen change amount = 0 has passed for a predetermined time or more, the frame at that time may be extracted. In this case, for example, a screen change amount is sequentially added from the first frame, and a frame whose value exceeds S / n or a frame in which the screen change amount = 0 has passed for a predetermined time or more is determined as the image data extraction source image. The frame F _i may be used. When a frame is extracted after a predetermined time has elapsed since the screen change amount = 0, there are a method of clearing the added value of the screen change amount to 0 and a method of holding the value without clearing. Whether or not to use this method may be selectable.

In the case of the example shown in FIG. 11, the display time information 121 is set so that the display time is the same for all the frames.
Is assumed to be described (this display time information 1
The display time information 121 is not constant, and may be obtained and described by another method, when the reproduction is performed at a constant time interval according to the method described in FIG.

Next, a case where one or a plurality of frames correspond to one frame information will be described.

An example of the data structure of the special reproduction control information in this case is the same as in FIG.

Hereinafter, a description method of the positional information of the video will be described with reference to FIGS.

FIG. 12 is a diagram for explaining a method of describing video position information referring to successive frames of an original video.

The description method of the video position information shown in FIG. 9 refers to one frame in the original video to be trick-played, but the description method of the video position information shown in FIG. It describes a set 500 of a plurality of continuous frames in the original video. The frame set 500 may be obtained by extracting a part of a plurality of continuous frames in the original video. Further, there may be a frame set 500 that includes only one frame.

When the set 500 of frames includes a plurality of continuous frames or one frame in the original video, the description of the frame position may describe the position of the start frame and the end frame, or may be the description of the start frame. Describe the position and the duration of the description section. In the case of including one frame, for example, the positions of the start frame and the end frame may be the same. As the description of the position and the time, a description such as a frame number and a time stamp that can specify a frame in the stream is used.

When the frame set 500 is a part of a plurality of continuous frames in the original video, information is described so that the frame can be specified. A method for extracting a frame is determined. For example, if the position of a start frame and an end frame can be specified by specifying the position, the frame may be specified.

The display time information 501 in FIG. 12 indicates the total display time corresponding to the entire frame group included in the corresponding original video frame set 500. The display time of each frame included in the original video frame set 500 can be appropriately determined by the device that performs special reproduction. As a simple method, there is a method of equally dividing the total display time by the total number of frames to obtain a display time of one frame. Of course, there are various other methods.

FIG. 13 is a view for explaining a method of describing video position information referring to an image data file.

Although the description method of the video position information shown in FIG. 12 directly refers to a continuous frame in the original video to be reproduced, the description method of the video position information shown in FIG. A frame set 600 of image data corresponding to the frame set 602 extracted from the original video stream is prepared in another file, and its position is described. By using a URL or the like, for example, the description method of the file position can be handled in the same manner whether the file exists on a local storage device or on a network. A set of the video position information 101 indicating the position of the image data file and the video display time 121 indicating the length of the corresponding display time 601 is described as frame information.

If correspondence with the original video frame is required,
Information indicating the frame set 602 of the original video corresponding to the described frame information (for example, the same as the video position information in the case of FIG. 12) may be included in the frame information.
In this case, the frame information is composed of video position information, display time information, and original video information. of course,
Original video information need not be described if it is not necessary.

The form of image data, creation of image data,
The method of creating a reduced image, the method of storing image data, and the method of describing positional information such as a URL are the same as those described above.

Various methods can be used to select which frame of the original video to create the image data and describe it in the video position information, as described above. Data may be extracted, or a large amount of image data may be extracted at a small interval at a place where the screen moves a lot, and a small amount of image data may be extracted at a large interval at a place where there is little movement.

In the above-described embodiment, the image data file 300 and the original video 302 are associated with each other on a frame-by-frame basis. However, the positional information of the frame described as the original video information may have a time width. is there. The data structure of the frame information for that purpose is, for example, as shown in FIG. In FIG. 14, the original video information 37 is added to the frame information of FIG.
01 has been added. In the original video information 3701, the starting point position and the section length of the corresponding section of the original video to be specially reproduced are included.
These are described as start point information 3702 and section length information 3703, respectively.

The information described as the original video information may be any information that can specify the section. Although the start point position and the section length are used here, the start point position and the end point position may be used as the original video information instead. FIG. 15 shows an example in which the original video information has a temporal width in FIG. In this case, for example, as the video position information, display time information, and original video information included in the same frame information, respectively, the position of the original video frame 3801, the display time 3802, and the original video frame section 3803 (the start frame position and the section length) )
To indicate that they correspond to each other. That is,
The original video frame 3801 described in the video position information is displayed as an image representing the original video frame section 3803.

FIG. 16 is an example in which the original video information is given a temporal width as compared with FIG. In this case, for example, as the video position information, display time information, and original video information included in the same frame information, respectively, the storage location of the display image data file 3901, the display time 3902, the original video frame section 3903 (the start frame position) And section length) to indicate that they correspond to each other. That is, the image 390 of the image data file described in the video position information is represented as the image representing the original video frame section 3903.
1 will be displayed.

When a set of frames is used as a display image as shown in FIGS. 12 and 13, a section different from the original image frame section used for the display image is associated with the original image information. It does not matter.

FIG. 17 shows an example in which the original video information is given a temporal width as compared with FIG. In this case, for example, as the video position information, display time information, and original video information included in the same frame information, a set 4001 of frames in the original video, a display time 4002, and an original video frame section 4003 (the starting frame position and Section length) to indicate that they correspond to each other. At this time, the section 4001 of the set of frames described as the video position information and the original video frame section 4003 described as the original video information need not necessarily match, and different sections may be used for display.

FIG. 18 shows an example in which the original video information has a temporal width as compared with FIG. In this case, for example, as the video position information, the display time information, and the original video information included in the same frame information, respectively, the storage location of the frame set 4101 used for display, the display time 4102, the original video frame section 4103 (the starting frame position And section length) to indicate that they correspond to each other.

At this time, the section of the frame set 4101 described as the video position information and the original video frame section 4103 described as the original video information do not necessarily have to match. That is, the section of the set of display frames 4101 may be shorter than the original video frame section 4103,
May be longer. Also, an image having completely different contents may be included. In addition, a method of extracting only a particularly important section from the sections described in the original video information as the video data file and using the collected video data file is also conceivable.

When browsing a summary reproduction (special reproduction) video using such frame information, for example, there are cases where it is desired to refer to a corresponding frame in the original video.

FIG. 19 is a flowchart for starting the reproduction from the frame of the original video corresponding to the frame of the video displayed in summary. In step S3601, a playback start frame is specified in the summary video. In step S3602, an original video frame corresponding to the specified frame is calculated by a method described later. In step S3603, the original video is reproduced from the calculated frame.

Of course, this flow can be used to refer to the corresponding position of the original video in addition to the reproduction.

As an example of a method of calculating the corresponding original video frame in step S3602, a method of using a proportional distribution with respect to the display time of the frame specified by the summary video will be described. The display time information including the i-th frame information is set to _Di seconds, and the section start position of the original video information is set to t _i.
Second, the section length and _{d i} seconds. When a position at which t seconds have elapsed since the start of reproduction using the i-th frame information is specified, the position of the corresponding frame of the original video is T = t _i +
d _i × t / D _i .

Next, a method of selecting a frame for special reproduction and summary reproduction will be described.

Referring to FIGS. 20 and 21, as an example of a frame selection method, a lot of image data is extracted at a small interval in a place where the screen moves a lot, and a lot of image data is extracted in a place with a little movement according to the movement of the screen. A method of extracting a small amount of image data at wide intervals will be described. 20, the horizontal axis and the curve 800 and _{S i} and _{F i} of Figure 21 is similar to FIG. 11.

In the example of FIG. 11, frames are extracted frame by frame at intervals such that the amount of change in the image between image data extraction source frames is constant. FIGS. 20 and 21 show frame numbers F _i.
2 shows an example in which a set of a plurality of frames is extracted on the basis of. In this case, for example, may be extracted successive frames of a certain number from the frame number F _i, as shown in FIG. 20 (frame length 811 and the frame length 81
2 are the same), the frame number F may also be image variation amount sum extracts the number of consecutive frames corresponding respectively to be constant from _i (area 813 and the area 814 as shown in FIG. 21 Same). Of course, various other methods are conceivable.

[0130] Of course, it is possible to use also extraction of F _i in the case where the screen change amount = 0 as described above has passed a predetermined time or more.

As in the case of FIG. 11, in the case of the examples of FIGS. 20 and 21, the display time information 121 may be described so that the same display time is obtained for any of the frame sets. Alternatively, the display time may be obtained and described.

Next, an example of the processing for determining the display time will be described.

FIG. 22 shows a display time such that when the video described in the video position information is continuously reproduced at the time described in the display time information, the amount of change in the screen is as constant as possible. It is an example of a basic processing procedure for obtaining.

This processing can be applied to the case where the frame is extracted by any method.
This process can be omitted when a frame is extracted by the method as described above. This is because the frame is selected in FIG. 11 so that the amount of change in the screen is as constant as possible for a constant display time.

In step S71, the amount of screen change between adjacent frames for all frames of the original video is determined. When each frame of the video is represented by a bitmap, the difference between pixels between adjacent frames can be used as a screen change amount. When the video is compressed by MPEG, it is possible to obtain the amount of screen change using the motion vector.

An example of how to determine the screen change amount will be described.

FIG. 23 shows an example of a basic processing procedure for obtaining the screen change amounts of all the frames from a video stream compressed by MPEG.

In step S81, a motion vector is extracted from a P picture frame. As shown in FIG. 24, video frames compressed by MPEG include an I picture (intra-frame coded frame), a P picture (forward predicted inter-frame coded frame), and a B picture (bidirectional predicted inter-frame coded frame). ). Among them, the P picture contains a motion vector corresponding to the motion from the immediately preceding I picture or P picture.

In step S82, the magnitude (intensity) of each motion vector included in one P-picture frame is determined, and the average is taken as the amount of screen change from the immediately preceding I-picture or P-picture.

In step S83, a screen change amount for each frame corresponding to all frames including frames other than the P picture is calculated based on the screen change amount obtained for the P picture frame. For example, if the average value of the motion vector of the P picture frame is p and the interval from the I or P picture frame immediately before the reference source is d, the amount of screen change per frame of each frame between them Is p / d.

Subsequently, step S in the procedure of FIG.
At 72, the total sum of the screen change amounts of the frames between the next description target frames is obtained from the description target frame described in the video position information.

FIG. 25 shows the amount of screen change for each frame.
It is a figure which described change. The horizontal axis corresponds to the frame number
A curve 1000 represents a change in the screen change amount. flame
Position F _iTo find the display time of video with different location information
In the case, F is the next frame to be described._{i + 1}Ward to
The screen change amount during the interval 1001 is cumulatively added. This is a diagonal line
Area S of part 1002_iAnd the frame position F_iMovement
Can be thought of as the size of.

Subsequently, step S in the procedure of FIG.
At 73, the display time of each frame is obtained. In order to keep the amount of change in the screen as constant as possible, it is necessary to allocate more display time to frames with larger screen movement,
The ratio of the display time allocated to the video at each frame position F _i to the reproduction time may be S _i / ΣS _i . Assuming that the total playback time is T, the display time of each video is D _i = T
S _i / ΣS _i . The value of the total reproduction time T is a standard reproduction time and is defined as the total reproduction time of the original video.

When there is no screen change and S _i = 0, a predetermined lower limit (for example, 1) may be entered,
The frame information need not be described. Until S _i = 0, the lower limit may be substituted for a frame whose screen change is very small and is expected to be hardly displayed in actual reproduction, and frame information may not be described. Good. If not describe the frame information, the value of S _i is may be added to the S _{i + 1,} or not.

The processing for obtaining the display time can be performed by the special reproduction control information generating apparatus for creating frame information, but can also be performed by the video reproducing apparatus at the time of special reproduction.

Next, an example of processing for performing special reproduction will be described.

FIG. 26 shows an example of a processing procedure for performing N-times speed reproduction based on the described special reproduction control information.

In step S111, a display time D ′ _i during reproduction is calculated based on the reproduction magnification. Since the display time information described in the frame information is a standard display time, when the reproduction is performed at N times speed, the display time of each frame is D ′ _i = D _i / N.

In step S112, initialization for display is performed. That is, i = 0 is set so as to display the first frame information.

In step S113, it is determined whether or not the display time D ′ _i of the i-th frame information is longer than a preset display time threshold.

[0151] If so, at step S114, the image of the image position information contained in the i-th frame information _{F i} is displayed D _'i seconds.

If not (lower), step S
Proceeding to 115, the i-th frame information that does not fall below the display time threshold is searched for in the forward direction. During this time, the display time of the frame information that is less than the display time threshold is added to the display time of the i-th frame information obtained as a result of the search, and the display time of the frame information that is less than the display time threshold is 0. I do. The reason why such processing is performed is that if the display time at the time of reproduction is extremely short, the time required to prepare a video to be displayed becomes longer than the display time, and the display may not be in time. Therefore, if the display time is very short, the process proceeds without displaying. At that time, the display time of the video not displayed is added to the display time of the displayed video so that the total playback time does not change.

In step S116, in order to determine whether or not frame information that has not been displayed remains.
It is determined whether i is less than the total number of frame information. If it is lower, the process proceeds to step S117, where i is incremented by 1 to prepare for displaying the next frame information. If i has reached the total number of frame information, the reproduction process ends.

FIG. 27 shows a predetermined display cycle (for example,
In the case of displaying 30 frames per second, one display cycle is 1/30 second). This is an example of a processing procedure for performing N-times speed reproduction based on the described special reproduction control information.

In the step S121, at the time of the N-times speed reproduction,
The display time D ′ _i of each frame is obtained as D ′ _i = D _i / N. Since the display time calculated here actually has a relationship with the display cycle, it is not always possible to display an image with the calculated display time.

FIG. 28 is a diagram showing the relationship between the calculated display time and the display cycle. A time axis 1300 indicates the calculated display time, and a time axis 1301 indicates a display cycle based on the display rate. When the display rate is f frames / sec, the display cycle interval is 1 / f sec.

[0157] Therefore, in step S122, searches the frame information _{F i} including the start point of the display cycle, in step S123, 1 display cycle (1 / f sec) the video included in the frame information _{F i} is displayed.

For example, in the display cycle 1302, since the display start point 1303 is included in the calculated display time 1304, the video of the frame information corresponding to the display time is displayed.

As a method of associating the display cycle with the frame information, as shown in FIG. 29, the video closest to the display cycle start point may be displayed. When the display time becomes shorter than the display cycle as in the display time 1305 in FIG. 28, the display of the video may be omitted or the display may be forcibly performed. In the case of forcible display, the display time before and after is shortened so that the total display time does not change.

In step S124, it is determined whether the current display is the last display cycle. If the current display is the last display cycle, the process is terminated. If not, the process proceeds to step S125 to process the next display cycle. move on.

Another example of the frame information description will be described.

The frame information included in the data structure of FIG. 8 or FIG. 14 deals with the case of summarizing a single original video. By expanding the frame information, a plurality of original videos are summarized and summarized. can do.
FIG. 30 shows an example of such a structure, in which original video position information 4202 indicating the position of the original video file is added to the original video information 4201 included in each frame information. The file described in the original video position information does not necessarily need to handle the section of the entire file, and may be used with only a part of the section extracted. In this case, not only the file information such as the file name, but also section information for indicating which section of the file is the target is described. A plurality of sections to be selected from the video file may be provided for one video.

In the case where there are several types of original video and individual identification information is given, original video identification information may be described instead of the original video position information.

FIG. 31 is a diagram for explaining an example in which a plurality of original images are summarized and displayed together using the frame information to which the original image position information has been added. In this example, three images (image 1, image 2, and image 3) are combined and one summary image is displayed. For video 2, not all sections,
Two sections 4301 and 4302 are extracted and treated as separate original videos. As the frame information, together with the original video information, the frame position of the representative image (4303 for 4301) is used as the video position information, and the display time (4303 for 4301).
4) is described as display time information.

FIG. 32 is a view for explaining another example in which a plurality of original images are summarized and displayed together using the frame information to which the original image position information has been added. Also in this example, three images are combined and one summary image is displayed. Video 2
As for, not all sections but some sections are extracted and treated as separate original videos. Of course, a plurality of sections may be extracted as shown in FIG. As the frame information, together with the original video information (for example, section information of 4401 in addition to the video 2), the storage location of the representative image file (4402) is stored as the video position information.
The display time (4403) is described as display time information.

The addition of the original video position information to the frame information as described in these examples can be applied in exactly the same way even when a set of frames is used as the video position information. Can be displayed as a summary.

FIG. 33 shows another data structure for describing frame information. In this data structure, motion information 4501 and attention area information 4502 are added in addition to the video position information, display time information, and original video information already described. The motion information describes the magnitude of the motion (the amount of change in the screen) in the section of the original video corresponding to the frame information (that is, the section described in the original video information). The attention area information describes information of an area of particular interest in the image described in the video position information.

The motion information can be used for calculating the display time of the image described in the video position information as used in calculating the display time from the motion of the video in FIG. In this case, even if the display time information is omitted and only the motion information is described, special reproduction such as fast-forwarding can be performed in the same manner as when the display time is described (in this case, the display time is calculated at the time of reproduction). .

It is also possible to describe both display time information and motion information at the same time. In such a case, the application that performs the display may use the one necessary for the processing or use it in combination.

For example, the display time calculated irrespective of the movement is described in the display time information. This corresponds to a method of calculating a display time for cutting out an important scene from the original video. When fast-forwarding the summary display calculated in this way, by using the motion information, the portion with large movement is played back later and the portion with small movement is played back earlier, so that fast-forwarding with less oversight is possible. is there.

The attention area information is used when there is a particularly noticeable area in the image described in the video position information of the frame information. For example, a face of a person deemed important to a viewer corresponds to this. When an image including such attention area information is displayed, a rectangle or the like may be superimposed and displayed so that the area can be recognized. This display is not essential, and the image may be displayed as it is.

The attention area information can be displayed by processing special reproduction information such as frame information. For example, when playing back and displaying only some frame information,
The frame information including the attention area information is preferentially displayed. In addition, the larger the rectangular area with a large area is,
It is also possible to select and display using the interpretation that the importance is high.

The case where the frame to be summarized and reproduced is selected based on the screen change amount has been described above.
A case where a frame is selected using importance information will be described.

FIG. 34 shows an example of the data structure of frame information attached to a video.

This data structure is different from the data structure of the frame information shown in FIG. 1 in that importance information 122 which is information on which display time is based is used as display time control information 102 (or instead of display time control information 102). It is intended to be described.

The importance information 122 indicates the importance of the corresponding frame (or frame set). The importance is expressed, for example, as an integer in a certain range (for example, between 0 and 100) or as a real number in a certain range (for example, between 0 and 1). Alternatively, it may be expressed as an integer or a real number without setting an upper limit. The importance information may be attached to all frames of the video, or only the frames whose importance has changed may be attached.

In this case, the description method of the position information of the video is as follows.
It is possible to take any of the forms of FIGS. 9, 10, 12, and 13. The frame extraction methods of FIGS. 11, 20, and 21 can also be used (in this case, FIGS. 11 and 2
0, the screen change amount in FIG. 21 may be replaced with the importance).

Next, in the example described above, the display time is set according to the amount of change in the screen, but the display time can be set according to the importance information. Hereinafter, a method of setting such a display time will be described.

In the display time setting based on the amount of change of the screen described above, the display time is set to be long where the amount of change is large, and the display time is shortened where the amount of change is small in order to make it easy to understand the video contents. Set. In the display time setting based on the importance, the display time may be set longer for a portion having a higher importance, and the display time may be set shorter for a portion having a lower importance. That is, the method of setting the display time based on the importance is basically the same as the method of setting the display time based on the amount of change in the screen (see FIG. 25), and therefore, will be briefly described here.

FIG. 36 shows an example of the basic processing procedure in this case.

In step S191, the importance of all frames of the original video is obtained. The specific method will be described later.

In step S192, the sum of the importance of the frames between the description target frame described in the video position information and the next description target frame is obtained.

FIG. 37 is a diagram describing a change in importance for each frame. 2200 is the importance. When calculating the display time of the video having the position information of the frame position F _i , the importance of the section 2201 up to the next description target frame position F _{i + 1} is added. The result of the addition is the area S ′ _i of the hatched portion 2202.

In the step S193, the display time of each frame is obtained. The ratio of the display time to the playback time allocated to the video at each frame position F _i is represented by S ′ _i / ΣS ′ _j
And Assuming that the total reproduction time is T, the display time of each video is D _i = T · S ′ _i / ΣS ′ _j . The value of the total reproduction time T is a standard reproduction time and is defined as the total reproduction time of the original video.

The sum of the degrees of importance is S '_i= 0
You can enter a lower limit (for example, 1)
The frame information need not be described. S '_i= 0 and
If not, the importance is very small,
Frame that is expected to be hardly displayed in
The lower limit may be substituted for
Need not be described. When not describing frame information
Is S '_iIs S ' _{i + 1}May be added to
It is not necessary.

As shown in FIG. 35, in the data structure of the frame information shown in FIG. 1, video position information 101, display time information 121, and importance information 122 may be described in each frame information i. In this case, during the special reproduction, the display time information 121 is used, but the importance level information 1 is used.
There is a method that does not use 22, a method that uses the importance information 122 but does not use the display time information 121, a method that uses both, and a method that does not use both.

The process for obtaining the display time can be performed by the special reproduction control information generating device for creating frame information, but can also be performed by the video reproducing device at the time of special reproduction.

Next, each frame or scene (video section)
(For example, step S19 in FIG. 36)
1) will be described.

It is usually determined whether a scene in a video is important or not.
As various factors are intertwined, the most plausible way to determine importance is by humans. In this method, the importance evaluator evaluates the importance at each scene of the video or at regular time intervals, and inputs the importance data. The importance data referred to here is a correspondence table between a frame number or time and a value of the importance at that time. In order to prevent the importance evaluation from becoming subjective, a plurality of importance evaluators evaluate the same video, and an average value (or a median or the like) may be used for each scene or each video section. ) To determine the final importance. Such manual input of importance data can add an ambiguous impression or a plurality of factors that cannot be expressed in words to the importance.

In order to save the trouble of human decision, consider the events likely to appear in a video scene that is considered important,
It is preferable to use a process of automatically evaluating such an event and converting it to importance. Hereinafter, some examples of the automatic generation of the importance are shown.

FIG. 38 shows an example of a processing procedure for automatically calculating importance data assuming that a scene with a high audio level is important (FIG. 38 is also established as a functional block diagram).

In the audio level calculation processing in step S210, when audio data accompanying the video is input, the audio level at each time is calculated. Since the audio level greatly changes instantaneously, processing such as smoothing may be performed in the audio level calculation processing in step S210.

In the importance calculation process in step S211,
A process of converting the audio level output as a result of the audio level calculation process into the importance level is performed. For example, a predetermined minimum audio level is set to 0 and a maximum audio level is set to 100, and the input audio level is linearly converted from 0 to 100. The value is set to 0 when the audio level is lower than the minimum audio level, and to 100 when the audio level is higher than the maximum audio level. As a result of the importance calculation process, the importance at each time is determined and output as importance data.

FIG. 39 shows an example of a processing procedure of another importance level automatic determination method (FIG. 39 is also realized as a functional block diagram).

The processing shown in FIG. 39 is for judging that a scene in which many pre-registered important words appear in the audio accompanying the video is important.

In the voice recognition processing of step S220, when voice data accompanying a video is input, words (words) spoken by a person are converted into text data by the voice recognition processing.

In the important word dictionary 221, words likely to appear in important scenes are registered. If the registered words have different degrees of importance, a weight is added to each registered word.

In the word matching process in step S222, text data output from the voice recognition process is compared with words registered in the important word dictionary 221 to determine whether an important word has been spoken.

In the importance calculating process in step S223,
The importance of each scene and each time of the video is calculated from the result of the word matching process. This calculation includes the number of important words,
The weight of the important word is used. For example, a process of increasing the importance around the time (or the scene where the important word appears) by a certain value or a value proportional to the weight of the important word is performed. As a result of the importance calculation process, the importance at each time is determined and output as importance data.

When the weights of all words are the same,
The important word dictionary 221 becomes unnecessary. This is equivalent to assuming that a scene where many words are spoken is an important scene. At this time, in the word matching process of step S222, a process of simply counting the number of words output from the voice recognition process is performed. The number of characters may be counted instead of the number of words.

FIG. 40 shows a processing procedure example of still another importance level automatic determination method (FIG. 40 is also realized as a functional block diagram).

The processing shown in FIG. 40 is for judging that a scene in which many important words registered in advance appear in a telop appearing in a video is important.

In the telop recognition processing of step S230, the character position in the video is specified, and the video area at the character position is binarized to perform the character recognition. The recognized result is output as text data.

The important word dictionary 231 is similar to the important word dictionary 221 in FIG.

In the word collation processing in step S232, text data output from the telop recognition processing is collated with words registered in the important word dictionary 231 as in step S222 in the procedure of FIG. Determine if it has appeared.

In the importance calculating process in step S233,
As in step S223 in the procedure of FIG. 39, the importance at each scene or each time is calculated from the number of appearances of important words and the weight of important words. As a result of the importance calculation process, the importance at each time is determined and output as importance data.

When all the words have the same weight,
The important word dictionary 231 becomes unnecessary. This is equivalent to assuming that a scene where many words appear as a telop is an important scene. At this time, step S23
In the second word matching process, a process of simply counting the number of words output from the telop recognition process is performed. The number of characters may be counted instead of the number of words.

FIG. 41 shows an example of a processing procedure of still another importance level automatic determination method (FIG. 41 is also realized as a functional block diagram).

The processing in FIG. 41 is for judging that the larger the character of the telop appearing in the video, the more important the scene.

In the telop detection processing in step S240, processing for specifying the position of the character string in the video is performed.

In the character size calculation process in step S241, individual characters are cut out from the character string, and the average value or the maximum value of the character size (area) is calculated.

In the importance calculation process of step S242,
The importance is calculated in proportion to the character size which is the output of the character size calculation process. If the calculated importance is too large or too small, a process for keeping the importance within a predetermined range by threshold processing is also performed. As a result of the importance calculation process, the importance at each time is determined,
Output as importance data.

FIG. 42 shows an example of a processing procedure of still another importance level automatic determination method (FIG. 42 is also realized as a functional block diagram).

The processing in FIG. 42 is for judging that a scene where a human face appears in a video is important.

In the face detecting process in step S250, a process for detecting a human face-like region in the video is performed. As a result of the process, the number of areas determined to be human faces (the number of faces) is output. Information on the size (area) of the face may also be output at the same time.

In the importance calculation process in step S251,
The importance is calculated by multiplying the number of faces, which is the output of the face detection processing, by a constant. When the output of the face detection processing includes face size information, the importance is calculated so as to increase with the face size. For example, the importance is calculated by multiplying the face area by a constant. As a result of the importance calculation process, the importance at each time is determined and output as importance data.

FIG. 43 is an example of a processing procedure of still another importance level automatic determination method (FIG. 43 is also realized as a functional block diagram).

The process shown in FIG. 43 is for judging that a scene in which a video similar to a previously registered image appears is important.

In the important scene dictionary 260, images to be determined as important are registered. The image is recorded as raw data or recorded in a data compressed format. Instead of the image itself, the feature amount (color histogram, frequency, etc.) of the image may be recorded.

In the similarity / dissimilarity calculation process of step S261, the similarity or dissimilarity between the image registered in the important scene and the input image data is calculated. As the degree of dissimilarity, a sum of square errors, a sum of absolute value differences, and the like are used. When image data is recorded in the important scene dictionary 260, the sum of the square errors and the sum of the absolute value differences for each corresponding pixel are calculated as the dissimilarity. When the color histogram of the image is recorded in the important scene dictionary 260, the same color histogram is calculated for the input image data, and the sum of square errors and the sum of absolute value differences between the histograms are calculated. To be dissimilarity.

In the importance calculation process of step S262,
The importance is calculated from the similarity or dissimilarity output from the similarity / dissimilarity calculation processing. When the similarity is input, the importance is calculated so that the greater the similarity, the greater the importance is. When the dissimilarity is input, the importance is calculated such that the greater the dissimilarity, the smaller the importance. . As a result of the importance calculation process, the importance at each time is determined and output as importance data.

As another method of automatically determining the importance level, there is a method in which a scene with a high instantaneous rating is important. The instantaneous audience rating data is obtained as an aggregated result of the audience rating survey, and the importance is calculated by multiplying the instantaneous audience rating by a constant. Of course, there are various other methods.

The processing for calculating the importance may be used alone, or the importance may be calculated by using a plurality of importance at the same time. In the latter case, for example, the importance of one video may be calculated by several different methods, and the final importance may be calculated as an average value or a maximum value.

In the above description, the screen change amount and importance have been described as an example. However, one or more types of information are used together with the screen change amount and / or importance, or instead of the screen change amount and importance. (Described in the frame information).

Next, a case where information for controlling reproduction / non-reproduction is added to the frame information (see FIG. 1) will be described.

In a video data, only a specific scene or a part (for example, a highlight scene) is reproduced.
There is a demand to view only a part of a video, such as to reproduce only a scene or a part where a specific person appears.

To satisfy this request, the frame information includes:
Play / non-play information for controlling whether to play or not play may be added. Thereby, on the reproduction side, based on the reproduction / non-reproduction information, it is possible to reproduce only a part of the video or conversely not reproduce only a part of the video.

FIGS. 44, 45 and 46 show examples of data structures to which reproduction / non-reproduction information is added.

FIG. 44 shows an example of the data structure in FIG.
The reproduction / non-reproduction information 123 is added. Of course, FIGS. 45 and 46 are obtained by adding the reproduction / non-reproduction information 123 to the data structure of FIGS. Although not shown, reproduction / non-reproduction information may be added in the data structure example of FIG.

As the reproduction / non-reproduction information 123, there are a method of designating binary information of reproduction or non-reproduction, and a method of designating continuous values such as a reproduction level.

In the latter case, for example, when the reproduction level is higher than a certain threshold value during reproduction, the reproduction is performed, and otherwise, the reproduction is not performed. The threshold may be, for example, directly or indirectly specifiable by the user.

The reproduction / non-reproduction information 123 may be held as independent information. However, in the case where reproduction or non-reproduction is selectively designated, the display time indicated by the display time information 121 has a specific value ( For example, when the value is 0 or -1), it is possible to determine that the reproduction is not performed. Alternatively, when the importance indicated by the importance information 122 is a specific value (for example, 0 or −1), it is possible to determine that the reproduction is not performed. In this case, the reproduction / non-reproduction information 123 need not be added.

In the case where reproduction or non-reproduction is designated by a level value, display time information 121 and / or importance information 122 (when importance is represented by a level value) can be substituted.

In the case where the reproduction / non-reproduction information 123 is held as independent information, the data amount increases accordingly. However, the digest can be viewed without reproducing the non-reproduction designated portion on the reproduction side, or the non-reproduction information can be viewed. It is also possible to play the designated portion and see the entire video (playback / non-playback information 1
23 is not stored as independent information, in order to reproduce the non-reproduction designated portion and to view the entire video, for example, 0
You need to change the display time specified as.)

The reproduction / non-reproduction information 123 may be input by a human, or may be determined based on some condition. For example, if the motion is larger than a certain value from the motion information of the video, it is played, otherwise if it is not played, it can be played back only in places where the movement is intense, and from the color information it is determined whether the skin color is larger or smaller than a certain value That way, you can only play where people are. A method of determining the size of the sound and a method of determining the size based on reproduction program information input in advance may be considered. The importance is determined in some way,
The reproduction / non-reproduction information 123 may be generated from the importance information. When the reproduction / non-reproduction information is a continuous value, the information may be obtained by converting the information into reproduction / non-reproduction information using an appropriate function.

FIG. 47 shows an example in which reproduction is performed by controlling reproduction / non-reproduction based on the reproduction / non-reproduction information 123.

In FIG. 47, the original video 2151 is₁
~ F₆Video frame position information or video frame
Group position information 2153 and D₁~ D₆Display represented by
It is assumed that reproduction is performed based on the time information 2154. This and
The playback / non-playback information 2155 includes display time information 215
4 shall be added. In this example, D₁, D
₂, D₄, D₆Section is played, and other sections are
In the case of non-playback, as the playback video 2152,
D₁, D₂, D₄, D₆Sections are played continuously
(Otherwise, it will not be played).

For example, the frame F of the reproduced video_ismell
Thus, the reproduction / non-reproduction information 123 indicates reproduction.
The display time at the time is D⁺ _iAt the time of non-playback
D between^- _iAnd the total time of the playback part of the original video
If it is T ', then Σ_iD⁺ _i= T '. Normal
Is D⁺ _iSet the display time at the same speed as the original video
Good. As a prerequisite, a fixed fixed speed is implied.
May be described, and information on how many times the speed is set may be described.
No. If you want to play at N times speed, display time D of the playback part⁺
_iIs multiplied by 1 / N. For example, playback at the determined time D '
Is performed, the display time D of each playback portion is required.⁺
_iTo D '/ Σ_iD⁺ _iWhat is necessary is to process it twice and display it.

When the display time of each frame (or frame group) is determined based on the frame information, the determined display time may be adjusted.

In the method in which the determined display time is not adjusted, the determined display time is used as it is without considering the occurrence of the non-reproduction section. Therefore, the display time exceeding 0 is originally assigned to the non-reproduction section. If so, the entire display time is shortened accordingly.

In the method of adjusting the determined display time,
For example, if a display time exceeding 0 is originally assigned to a non-reproduction section, each frame (or frame group) to be reproduced is set so that the entire display time becomes the same as when the non-reproduction section is reproduced. ) Is adjusted by multiplying the display time by a certain number.

The user may be allowed to select whether or not to make adjustments.

[0243] Even when the user specifies N-times speed reproduction,
The process of N-times speed reproduction may be performed without adjusting the determined display time, or the process of N times speed reproduction may be performed based on the display time after the determined display time is adjusted as described above. (The former has a shorter display time).

The user may be able to specify the entire display time. Also in this case, for example, the adjustment may be performed by multiplying the display time of each frame (or frame group) to be reproduced by a certain number so that the designated entire display time is reached.

FIG. 48 shows an example of a processing procedure for reproducing only a part of the video based on the reproduction / non-reproduction information 123.

In step S162, the frame information (image position information and display time information) of the frame is read, and in step S163, whether the frame is to be reproduced or not reproduced is determined from the reproduction / non-reproduction information in the display time information. to decide.

If the judgment result is reproduction, step S16
In step 4, the frame is displayed for the display time. Otherwise, the frame is not reproduced, and the process proceeds to the next frame.

In step S161, it is determined whether or not the video to be reproduced has ended. When the video has ended, the reproduction process ends.

When it is determined in step S163 whether the frame is to be played back or not played back, the playback / non-playback information is simply played back if the playback / non-playback information is not played back. There are times when it is desired to determine whether to reproduce or not reproduce the non-reproducing part according to the user's preference. At this time, whether or not to reproduce the non-reproducing part is determined in advance from the user profile or the like before reproducing the video, and when reproducing the non-reproducing part, the frame is always reproduced in step S164. .

In addition, when the reproduction / non-reproduction information is stored as a continuous value as the reproduction level, a threshold value for distinguishing between reproduction and non-reproduction is obtained from the user profile.
Whether to reproduce or not reproduce may be determined based on whether the reproduction / non-reproduction information exceeds a threshold value. In addition to using the user profile, for example, a threshold may be calculated from the degree of importance set for each frame, or information on whether to reproduce or not reproduce in advance or in real time may be received from the user.

As described above, the reproduction / non-reproduction information 12 for controlling reproduction or non-reproduction is included in the frame information.
By adding 3, it becomes possible to reproduce only a part of the video, and it is possible to reproduce only the highlight scene or to reproduce only the scene in which the person or the object of interest appears. .

Next, in the case where the frame information (see FIG. 1) includes, as the additional information, the position information of a medium (eg, text or sound) other than the video related to the video to be displayed and the time for displaying or reproducing them. A description method will be described.

In FIG. 8, each frame information 100 includes video position information 101 and display time information 102, in FIG. 34, each frame information 100 includes video position information 101 and importance information 103, and in FIG. The information 100 includes the video position information 101, the display time information 102, and the importance information 103, and FIGS. 44, 45, and 46 show examples in which the reproduction / non-reproduction information 123 is further included. In any case, 0 or more sound position information 2703, sound reproduction time information 2704, 0 or more text information 2705, and text display time information 2706 (where one is 1 or more) may be added. Good.

FIG. 49 shows an example of the data structure of FIG. 8 in which one set of sound position information 2703 / sound reproduction time information 2704 and N sets of text information 2705 / text display time information 270
This is an example in the case where 6 is added.

The sound is reproduced from the position stored in the sound position information 2703 for the time stored in the sound reproduction time information 2704. The reproduction target may be sound information attached to the video from the beginning, or background music or the like may be created and newly added.

The text displays the text information stored in the text information 2705 for the time stored in the text display time information 2706. A plurality of text information may be added to one video frame.

[0257] The time at which the reproduction of the sound and the display of the text are started is the same as when the associated video frame is displayed. The sound playback time and the text display time are also within the display time of the associated video frame. To play back a continuous sound over multiple video frames,
The position information of the sound and the reproduction time may be set to be continuous.

By such a method, a summary voice, a summary text, and the like can be obtained.

FIG. 50 shows an example of a method for describing sound information separately from frame information. This is an example of a data structure for reproducing the sound related to the displayed video frame when performing the special reproduction. Position information 2801 indicating the location of the sound to be reproduced, and time 2 at which the reproduction of the sound is started
A set of 802 and a time 2803 during which the reproduction is continued is defined as one sound information 2800, which is described as an array of the sound information.

FIG. 51 shows an example of a data structure for describing text information. It has the same structure as the sound information in FIG. 50, and a set of a character code 2901 of a text to be displayed, a display start time 2902, and a display time 2903 is one text information 2900, and is described as an array of the text information. As information corresponding to 2901, instead of a character code, position information indicating a place where the character code is stored or a place where the character is stored as an image may be used.

The above sound information and text information are synchronized with the display of the video frame, and are displayed as information related to the displayed video frame or a certain video section in which the video frame exists. As shown in FIG. 52, reproduction or display of sound information or text information is started as the time indicated by the time axis 3001 elapses. First, the video 3002 is reproduced by displaying each video frame in the described order in the described display time. Reference numerals 3005, 3006, and 3007 each represent a video frame, and a predetermined display time is assigned to the video frame. The sound 3003 is reproduced at the reproduction start time described in each sound information, and stops reproduction when the reproduction time similarly described has passed. As shown in FIG.
A plurality of sounds 3008 and 3009 may be played at the same time. Similarly to the sound, the text 3004 is displayed at the display start time described in each piece of text information, and stops displaying after the described display time. A plurality of texts 3010 and 3011 may be displayed at the same time.

The sound reproduction start time and the text display start time need not coincide with the video frame display time. The playback time of the sound and the display time of the text need not match the display time of the video frame. These can be freely set, and conversely, the display time of the video frame may be changed according to the sound reproduction time or the text display time.

These can be set manually by a human.

FIGS. 50 and 51 show an example in which reproduction and display are described separately from video frame information and synchronized with video, but only sound and text information are described separately from video. Sound, text summary playback, and summary display may be performed.

In order to save the trouble of human decision, consider an event likely to appear in a video scene that is considered important,
It is preferable to use a process for automatically setting such an event. Hereinafter, some examples of the automatic setting will be described.

FIG. 53 shows an example of a processing procedure in which a continuous video section from a screen change called a shot to the next change is determined, and the sum of the display times of the video frames included in the shot is used as the audio playback time. (FIG. 53 also holds as a functional block diagram).

In step S3101, a shot is detected from the video. This includes a technique for detecting a moving image cut from an MPEG bit stream using a likelihood ratio test (Res. IEICE, Vol. J82-D-II, No. 3, p.
p. 361-370, 1999) ".

In step S3102, it is determined which shot each video frame belongs to by referring to the position information of the video frame. Further, the display time of each shot is obtained by taking the sum of the display times of the video frames.

For example, the sound position information is the position of the sound corresponding to the start of the shot, the sound reproduction start time is set to the display time of the first video frame belonging to each shot, and the sound reproduction time is set to that shot. What is necessary is just to make it equal to display time. Alternatively, the display time of the video frame included in each shot may be modified according to the sound reproduction time. Here, the shot is detected, but when the data structure describes the importance information in the frame information, the importance of the video frame is used to determine a section where the importance is equal to or more than the threshold, and the section is determined. May be reproduced.

If the determined reproduction time is less than the predetermined reference, the sound may not be reproduced.

In FIG. 54, an important word is extracted by voice recognition from voice data corresponding to a shot or a video section of high importance, and the word, a voice including the word, or a voice obtained by combining a plurality of words is reproduced. An example of the processing procedure is shown (FIG. 54 also holds as a functional block diagram).

In step S3201, a shot is detected. Instead of a shot, a video section having a higher importance may be obtained.

In step S3202, speech recognition is performed on the section of the audio data corresponding to the obtained video section.

In step S3203, a speech containing an important word or a speech of an important word portion is obtained from the recognition result. To select an important word, reference is made to the important word dictionary 3204.

In step S3205, a sound for reproduction is created. A continuous voice including important words may be used as it is, or only important words may be extracted. A voice combining a plurality of important words may be created.

[0276] In step S3206, the display time of the video frame is corrected according to the playback time of the created audio. However, the number of selected words may be reduced to shorten the sound reproduction time so that the sound reproduction time is within the display time of the video frame.

FIG. 55 shows an example of a procedure for acquiring text information from a telop (FIG. 55 is also established as a functional block diagram).

In the processing shown in FIG. 55, text information is obtained from telops or audio displayed in a video.

In step S3301, the telop displayed in the video is read. This includes, for example, the document “Hori Osamu:“ Character part extraction method from video for telop area ””, CVIM114-17, pp. 129-136.
(1999) ”, a telop in the original video is automatically extracted, or a human reads the telop and manually inputs the telop.

At step S3302, an important word is extracted from the read telop character string. An important word dictionary 3303 is used to determine an important word. of course,
The read telop character string may be used as text information as it is. The extracted words may be arranged, and a sentence representing the video section may be composed of only the important words, which may be used as text information.

FIG. 56 shows an example of a processing procedure for acquiring text information from voice (FIG. 56 is also established as a functional block diagram).

The voice is recognized by the voice recognition processing in step S3401.

In step S3402, important words are extracted from the recognized voice data. An important word dictionary 3403 is used to determine an important word. Of course, the recognized voice data may be used directly as text information. The extracted words may be arranged, and a sentence representing the video section may be composed of only the important words, which may be used as text information.

FIG. 57 shows an example of a processing procedure for extracting text information from a shot or a video section having high importance by telop recognition and creating text information (FIG. 57 is also established as a functional block diagram).

At step S3501, a shot is detected from the video. Instead of a shot, a section having a high importance may be obtained.

In step S3502, the telop displayed in the video section is recognized.

In step S3503, important words are extracted using the important word dictionary 3504.

At step S3505, a text for display is created. For this purpose, a telop character string including an important word may be used, or a character string using only an important word or a plurality of important words may be used as text information.
When text information is obtained by voice recognition, the part of the telop recognition processing in step S3502 may be replaced with voice recognition processing, and voice data may be input. The text information is displayed in accordance with a video frame in which the text is displayed as a telop or a video frame at the time when the text is reproduced as audio. Alternatively, text information in the video section may be displayed at a time.

FIG. 58 shows a display example of text information. As shown in (a) of FIG. 58, the text information display unit 3601 and the video display unit 3602 may be divided, or (b) of FIG.
, The text information may be displayed on the video display unit 3603 in an overlapping manner.

The display time (reproduction time) of each of the video frame, sound information, and text information is adjusted so that all media information is synchronized. For example, when reproducing a video at double speed, first, important audio is extracted by the above-described method, and audio information for half the time of normal reproduction is obtained. Next, a display time is allocated to a video frame associated with each audio. When the display time of the video frame is determined so that the amount of change in the screen is constant, the audio reproduction time and the text display time are within the display time of the associated video frame. Alternatively, a section including a plurality of video frames such as a shot is obtained, and the sound or text included in the section is reproduced or displayed according to the display time of the section.

In the above, the description has been made centering on video data, but a system dealing mainly with audio data is of course also possible.

In the above, the summary display centering on video has been dealt with, but it is also possible to use sound information and text information in a format having no frame information (that is, no video). In this case, an abstract composed of only sound information and text information is created for the original video. In addition, it is also possible to create a summary composed of only sound information and text information for audio data and music data.

At this time, as in the case of the frame information,
Original data information for describing the correspondence between the original sound and music data may be added to the sound information or text information.

FIG. 59 shows an example of a data structure including original data information 4901 in the sound information having the data structure shown in FIG. When the input is a video, the original data information 4901 indicates the time indicating the video section (starting point information 4902 and section length information 490).
3), and when the input is voice / music, the time indicates a section of voice / music.

FIG. 60 shows an example of a data structure including original data information 4901 in sound information having a data structure corresponding to FIG.

FIG. 61 illustrates an example of summarizing voice / music using voice information. In this example, the original speech / music is divided into several sections, and a part of each section is cut out as a summary speech / music of the section to create a summary. For example, 500 in section 2
1 is cut out as a summary audio / music, and 50
Reproduction is made as a section 02. Examples of the method of dividing the section include a method of dividing music into movements and a method of dividing conversation into contents.

As in the case of the frame information, by including the description of the original data file and the section in the sound information and text information, a plurality of voice and music data can be summarized together. At this time, when identification information is added to each original data, the original data identification information may be used instead of describing the original data file or section.

FIG. 62 illustrates an example of summarizing voice / music using voice information. In this example, a plurality of voice / music data are cut out as a summary voice / music of a partial section, respectively, to create a summary. For example, the section 5101 of voice / music 2 is cut out as a summary voice / music and reproduced as a section 5102 of the summary. It is conceivable, for example, to cut out and collect a part of the music included in one music album and create summary data for trial listening.

When it is better to know the title of a song, such as when summarizing an album or the like, the title of the music data may be included in the sound information. Of course, this information is not required.

Next, a method for providing video data and object area data will be described.

When the special reproduction control information created by the processing of the present embodiment is provided to the user, it is necessary for the creator to provide the special reproduction control information to the user in some way. Various forms of this providing method can be considered as exemplified below. (1) A form in which video data and its special reproduction control information are recorded on one (or more) recording media and provided simultaneously (2) A video data is recorded and provided on one (or more) recording media And separately providing the trick play control information on one (or more) recording media (3) providing the video data and its trick play control information via a communication medium at the same opportunity ( 4) A form in which video data and its special reproduction control information are provided at different occasions via a communication medium. As a result, frames (groups) selectively extracted from an original video as control information for providing special reproduction of video contents. ), And the frame information including the display time information allocated to the frame (group) or the information from which the frame (group) is obtained is described in a plurality of arrays. Effective special reproduction based on the information becomes possible.

As described above, according to the present embodiment, in the special reproduction control information description method for describing the special reproduction control information to be used for the special reproduction of the video content, all of the video data constituting the video content For each one frame selectively extracted from the frame sequence or for each frame group including a plurality of continuous or adjacent frames, first information indicating a position where the data of the one frame or the frame group exists; The second information relating to the display time given to the frame or the frame group and / or the third information indicating the importance given to the one frame or the frame group corresponding to the frame information is represented by frame information. Described as

According to the present embodiment, one frame selectively extracted from the entire frame sequence of the video data constituting the video content or a frame group consisting of a plurality of continuous or adjacent frames is described. , First information indicating a position where data of the one frame or the frame group exists, second information relating to a display time given to the one frame or the frame group, and / or the one frame or the frame There is also provided a computer-readable recording medium storing special reproduction control information including at least frame information including third information indicating importance assigned to a group.

Further, in the embodiment of the special reproduction control information device / generation method for generating special reproduction control information to be used for the special reproduction of the video content, in the special reproduction control information device / generation method, Partial frames to be subjected to trick play are selectively extracted along the frame sequence in a frame group consisting of one frame or a plurality of consecutive or adjacent frames, and the extracted one frame or the frames are extracted. For each group, the video position information indicating the position where the data of the one frame or the frame group exists, and the information of the display time to be assigned to the one frame or the frame group or the information for calculating the same are And display time control information including the video position, the video position generated for each of the one frame or the frame group. By describing the information and the display time control information as the frame information, to create a special reproduction control information for the video content.

Also, in the embodiment of the video reproducing apparatus / method capable of performing the special reproduction for the video content, in the video reproducing apparatus / method, the video data can be selectively selected from the entire frame sequence of the video data constituting the video content accompanying the video content. Described for each frame group consisting of one frame or a plurality of consecutive or adjacent frames extracted in
Video position information indicating the position where the data of the frame or the frame group is present, display time information to be assigned to the one frame or the frame group, or display time control information indicating the information on which the calculation is based; With reference to the special reproduction control information including at least the frame information including, based on the video position information included in the frame information, to obtain the data of the one frame or the frame group corresponding to each frame information, at least each Based on the display time control information included in the frame information, determine a display time to be assigned to each frame information, the obtained data of one or more frames, the determined display time by the determined display time The special reproduction is performed by performing the reproduction in a predetermined order.

In the embodiment of the present invention, for example, position information of an effective video frame used for display or image data extracted in frame units from an original video is prepared in advance, and the position information of the video frame or the display time of the image data is prepared. Prepare the information separately from the original video. By continuously displaying video frames or image data extracted from the original video based on display information, special reproduction such as double-speed reproduction, trick reproduction, and interlaced continuous reproduction can be performed.

[0307] For example, in double-speed playback for checking the contents at high speed, the display time is increased for places with large movements and the display time is set for places with small movements so that the change of the display screen is as constant as possible. To make it shorter,
The display time may be determined in advance. Alternatively, the same effect can be obtained even if the position information is determined so that a portion of the video frame or image data used for display that has a large amount of movement and a portion of small amount of movement are small. A value that is controlled so as to be a double speed value or a reproduction time specified by the user as a whole may be prepared. A long video can be viewed in a short time with easy-to-view double-speed reproduction, etc., and the content can be grasped in a short time.

For example, by making the display time longer in an important place and shorter in a lower place in accordance with the degree of importance of the display time, it is possible to reproduce the important place without overlooking it.

For example, by not displaying all video frames and partially omitting a part of video, only important portions may be efficiently reproduced.

According to the embodiment of the present invention, as control information for providing special reproduction of video content, a method of obtaining a frame (group) selectively extracted from an original video, and assigning the frame (group) to the frame (group). By arranging and describing a plurality of pieces of frame information including (absolute or relative) display time information or information on which the display time is obtained, an effective trick play based on the control information is performed on the playback side. Becomes possible.

For example, each of the above functions can be implemented as software. The above-described embodiment is a computer-readable recording medium that records a program for causing a computer to execute predetermined means, or for causing a computer to function as predetermined means, or for causing a computer to realize predetermined functions. Can also be implemented.

[0312] The configuration illustrated in each embodiment is an example, and is not intended to exclude other configurations. Some of the illustrated configuration may be replaced with another, or part of the illustrated configuration may be replaced. Another configuration obtained by omitting, adding another function to the illustrated configuration, or combining them is also possible. Another configuration that is logically equivalent to the illustrated configuration, another configuration including a portion that is logically equivalent to the illustrated configuration, another configuration that is logically equivalent to a main part of the illustrated configuration, and the like are also possible. Another configuration that achieves the same or similar purpose as the illustrated configuration, another configuration that achieves the same or similar effect as the illustrated configuration, and the like are also possible. Within each embodiment, various variations on various components can be implemented in appropriate combinations. Each embodiment can be implemented in combination as appropriate. Each embodiment includes various aspects, steps, and concepts, such as an invention as a method of describing information, an invention as described information, an invention as an apparatus or a method corresponding thereto, and an invention as an inside of an apparatus or a method corresponding thereto. Or, the inventions belonging to the category are included or inherent.
Further, the present invention provides a computer-readable recording medium for recording a program for causing a computer to execute predetermined means, for causing a computer to function as predetermined means, or for causing a computer to realize predetermined functions. It can also be implemented as.

Therefore, the present invention can be extracted from the contents disclosed in the embodiments of the present invention without being limited to the exemplified configuration.

[0314]

As described above, according to the present invention, it is possible to provide a frame information description method, a frame information generation device and method, and a video reproduction device and method which enable more effective special reproduction for a user. Can be.

[Brief description of the drawings]

FIG. 1 is a view showing an example of a data structure of trick play control information according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration example of a special reproduction control information generation device.

FIG. 3 is a diagram showing another configuration example of the special reproduction control information generation device.

FIG. 4 is a flowchart illustrating an example of a processing procedure in the case of the configuration of FIG. 2;

FIG. 5 is a flowchart showing an example of a processing procedure in the case of the configuration of FIG. 3;

FIG. 6 is a diagram illustrating a configuration example of a video playback device.

7 is a flowchart showing an example of a processing procedure in the case of the configuration of FIG. 6;

FIG. 8 is a view showing an example of a data structure of special reproduction control information.

FIG. 9 is a view for explaining video position information referring to an original video frame.

FIG. 10 is a view for explaining video position information referring to an image data file.

FIG. 11 is a view for explaining a method of extracting image data according to the movement of a screen.

FIG. 12 is a view for explaining video position information referring to an original video frame.

FIG. 13 is a view for explaining video position information referring to an image data file.

FIG. 14 is a diagram showing a data structure of frame information when the position information of a frame described as original video information has a temporal width.

FIG. 15 is a view for explaining an example in which video position information referring to an original video frame has a temporal width.

FIG. 16 is a view for explaining an example in which video position information referring to an image data file has a temporal width.

FIG. 17 is a diagram illustrating an example in which video position information referring to an original video frame has a temporal width.

FIG. 18 is a view for explaining an example in which an image data file referring to an original video frame has a temporal width.

FIG. 19 is a flowchart for starting reproduction from a frame of an original video corresponding to a frame of a video that is displayed in summary.

FIG. 20 is a view for explaining a method of extracting image data according to the movement of the screen.

FIG. 21 is a view for explaining a method of extracting image data according to the movement of a screen.

FIG. 22 is a flowchart illustrating an example of a processing procedure for obtaining a display time at which the amount of change in the screen is as constant as possible.

FIG. 23 is a flowchart illustrating an example of a processing procedure for obtaining a screen change amount of all frames from an MPEG video.

FIG. 24 is a view for explaining a method of calculating an image change amount from an MPEG stream.

FIG. 25 is a view for explaining a processing method for obtaining a display time in which the amount of change in the screen is as constant as possible.

FIG. 26 is a flowchart illustrating an example of a processing procedure for performing special reproduction based on special reproduction control information.

FIG. 27 is a flowchart illustrating an example of a processing procedure for performing special reproduction based on a display cycle.

FIG. 28 is a diagram illustrating a relationship between a calculated display time and a display cycle.

FIG. 29 is a diagram illustrating a relationship between a calculated display time and a display cycle.

FIG. 30 is a view showing an example of the data structure of special reproduction control information having original video position information.

FIG. 31 is an exemplary view for explaining image position information referring to an original image frame when a plurality of original images are collectively displayed in summary.

FIG. 32 is a view for explaining video position information referring to an image data file when a plurality of original videos are collectively displayed in a summary.

FIG. 33 is a view showing another data structure for describing frame information.

FIG. 34 is a view showing an example of the data structure of special reproduction control information.

FIG. 35 is a view showing an example of the data structure of special reproduction control information.

FIG. 36 is a flowchart showing an example of a processing procedure for obtaining a display time from importance.

FIG. 37 is a view for explaining a method for obtaining a display time from importance.

FIG. 38 is a flowchart showing an example of a processing procedure for calculating importance data with a scene having a high audio level as important.

FIG. 39 is a process for calculating importance data with a scene where many important words appear by voice recognition as important, or a process for calculating importance data with a scene where the number of words spoken per time is important. 5 is a flowchart showing an example of the procedure.

FIG. 40 is a diagram illustrating a process of calculating importance data with a scene in which many important words appear by telop recognition as important or a scene in which the number of words included in a telop appearing per time is significant as important. 9 is a flowchart illustrating an example of a procedure of a calculation process.

FIG. 41 is a flowchart illustrating an example of a procedure of processing for calculating importance data with a scene in which a large character appears as a telop as important.

FIG. 42 is a flowchart illustrating an example of a procedure of processing for calculating importance data with a scene where many human faces appear important or calculating importance data with a scene where a human face appears largely.

FIG. 43 is a flowchart illustrating an example of a procedure of a process of calculating importance data with a scene in which a video similar to a registered important scene appears as important.

FIG. 44 is a view showing an example of the data structure of special reproduction control information.

FIG. 45 is a view showing an example of the data structure of special reproduction control information.

FIG. 46 is a view showing an example of the data structure of special reproduction control information.

FIG. 47 is a view for explaining the relationship between information on whether to reproduce or not reproduce and a reproduced video.

FIG. 48 is a flowchart showing an example of the procedure of a special reproduction process including reproduction / non-reproduction determination.

FIG. 49 is a view showing an example of a data structure when sound information and text information are added.

FIG. 50 is a diagram showing an example of a data structure for describing only sound information separately from frame information.

FIG. 51 is a diagram showing an example of a data structure for describing only text information separately from frame information.

FIG. 52 is an exemplary view for explaining synchronization of media reproduction;

FIG. 53 is a flowchart showing an example of a procedure for determining a sound reproduction start time and a sound reproduction time within a video section.

FIG. 54 is an exemplary flowchart showing an example of the procedure of processing for creating reproduction audio data and correcting the video frame display time.

FIG. 55 is a flowchart showing an example of a procedure of text information acquisition processing by telop recognition.

FIG. 56 is an exemplary flowchart showing an example of the procedure of text information acquisition processing by voice recognition.

FIG. 57 is a flowchart illustrating an example of a procedure of a text information creation process.

FIG. 58 is an exemplary view for explaining a method for displaying text information;

FIG. 59 is a view showing another data structure for describing frame information.

FIG. 60 is a view showing another data structure for describing frame information.

FIG. 61 is an exemplary view for explaining abstract reproduction of music data;

FIG. 62 is an exemplary view for explaining abstract reproduction of a plurality of music data;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Video data processing part 2 ... Video data storage part 3 ... Special reproduction control information storage part 4 ... Image data file storage part 11 ... Video position information processing part 12 ... Display time control information processing part 13 ... Image data file creation part 21 ... Control unit 22 ... Normal reproduction processing unit 23 ... Special reproduction processing unit 24 ... Display unit 25 ... Content storage unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Yuji Mita 1st address, Komukai Toshiba-cho, Saiwai-ku, Kawasaki-shi, Kanagawa Prefecture Inside Toshiba R & D Center (72) Inventor Koji Yamamoto Komukai Toshiba, Sai-ku, Kawasaki-shi, Kanagawa No. 1 in the Toshiba R & D Center, Inc. (72) Inventor Koichi Masukura No. 1, Komukai Toshiba-cho, Koyuki-ku, Kawasaki-shi, Kanagawa F-term (reference) 5C052 AA01 AC01 CC11 DD04 5C053 FA23 GA11 GB37 HA21 JA24 5D044 AB05 AB07 BC03 CC04 FG18 FG23

Claims (24)

[Claims] A step of describing first information for specifying a position in the original video data of a frame extracted from a plurality of frames of the original video data; and a step of describing second information relating to a display time of the extracted frame. A frame information description method, comprising: 2. The frame information according to claim 1, wherein the extracted frame comprises a frame group, and the first information includes information for specifying a position of the extracted frame group in the original video data. Description method. 3. The method according to claim 1, further comprising the step of writing third information relating to the importance of the extracted frame. 4. The frame information description method according to claim 1, wherein the first information includes information for specifying an image data file corresponding to the extracted frame created from the original video data. 5. The method according to claim 5, wherein the extracted frame comprises frames extracted from a plurality of frames in a certain time section of the original video data, and further comprising a step of describing fourth information for specifying the time section. Claim 1.
Described frame information description method. 6. The frame information description method according to claim 5, wherein the first information includes information for specifying an image data file corresponding to the extracted frame created from the original video data. 7. The frame information description method according to claim 1, wherein the second information includes information on a display time such that a screen change amount during reproduction is substantially constant. 8. The frame information description method according to claim 1, further comprising a step of describing fifth information for instructing reproduction or non-reproduction of the extracted frame. 9. The first information is information indicating a position of the extracted frame or image data corresponding to the extracted frame in an image data file generated from the original video data and stored separately from the original video data. 2. The frame information description method according to claim 1, further comprising information indicating a position of the frame information. 10. The method further comprising the step of describing information indicating a position of the media data and information relating to a display time of the media data in media data other than the original video data including the extracted frame. The frame information description method according to claim 1, wherein 11. A computer-readable recording medium for storing frame information related to a frame extracted from a plurality of frames of original video data, wherein the frame information specifies a position of the extracted frame in the original video data. A recording medium comprising: information; and second information relating to a display time of the extracted frame. 12. The recording medium according to claim 11, wherein said extracted frame comprises a frame group, and said first information includes information for specifying a position of said extracted frame group in said original video data. . 13. The recording medium according to claim 11, wherein the frame information further includes third information relating to the importance of the extracted frame. 14. The apparatus according to claim 1, wherein the first information includes information for specifying an image data file corresponding to the extracted frame created from the original video data.
2. The recording medium according to 1. 15. The recording medium according to claim 11, further comprising, together with said frame information, said original video data and an image data file corresponding to said extracted frame created from said original video data. 16. A means for describing first information for specifying a position in the original video data of a frame extracted from a plurality of frames of the original video data, and a means for describing second information relating to a display time of the extracted frame. A frame information description device, comprising: 17. Generating first information for specifying a position in the original video data of a frame extracted from a plurality of frames of the original video data, and generating second information relating to a display time of the extracted frame. A frame information generation method, comprising: 18. A means for referring to first information for specifying a position in the original video data of a frame extracted from a plurality of frames of the original video data, and second information relating to a display time of the extracted frame, Means for acquiring the original video data of the extracted frame based on the first information; means for determining a display time for reproducing the original video data of the extracted frame based on the second information; and Means for reproducing the data for the determined display time. 19. referencing first information specifying a position in the original video data of a frame extracted from a plurality of frames of the original video data, and second information relating to a display time of the extracted frame; Obtaining original video data of the extracted frame based on first information; determining display time for reproducing the original video data of the extracted frame based on the second information; Reproducing the data for the determined display time. 20. A computer-readable recording medium storing a video reproduction program, wherein the video reproduction program causes the computer to specify a position in the original video data of a frame extracted from a plurality of frames of the original video data. A second information and a display time of the extracted frame.
A program code for referring to information; a program code for causing a computer to acquire the original video data of the extracted frame based on the first information; and a computer for reproducing the original video data of the extracted frame based on the second information. A recording medium comprising: a program code for determining a display time to be performed; and a program code for reproducing the display time determined by the computer for the original video data. 21. A step of describing first information for specifying a position in the original sound data of a sound frame extracted from a plurality of frames of the original sound data, and describing second information on a reproduction time of the extracted frame. A frame information description method, comprising: 22. A computer-readable recording medium for storing frame information on sound frames extracted from a plurality of frames of original sound data, wherein the frame information specifies a position of the extracted frame in the original sound data. A recording medium comprising: 1 information; and 2nd information relating to a reproduction time of the extracted frame. 23. A step of describing first information for specifying a position of a text frame extracted from a plurality of frames of original text data, and a step of describing second information relating to a display time of the extracted frame. A frame information description method characterized in that: 24. A computer-readable recording medium for storing frame information related to a text frame extracted from a plurality of frames of original text data, wherein the frame information specifies a position of the extracted frame in the original text data. A recording medium comprising: 1 information; and 2nd information relating to a display time of the extracted frame.