JP2010045486A - Still image extracting device, and still image extracting program - Google Patents

Abstract

The most representative still image is extracted while the same subtitle is displayed.
A still image extraction apparatus that extracts a still image from video data accompanied by caption data in order to create thumbnail data, wherein the caption data is newly displayed from each frame of the video data. Extract the first frame, the second frame when the displayed caption data is erased, and N (N is a natural number of 1 or more) frames between the first frame and the second frame A frame extracting unit that calculates a difference between two temporally adjacent images out of (N + 2) frames extracted by the frame extracting unit, and an image obtained by the difference calculating unit. Identify the two frames with the smallest difference, and extract the video data of the earlier or later frame of the identified two frames as a still image And a still image extracting unit for recording.
[Selection] Figure 1

Description

  The present invention relates to a still image extraction apparatus and a still image extraction program for extracting a still image from a moving image in order to create a thumbnail.

  The data of a program broadcast as a television broadcast includes video data and audio data. In addition, since the broadcast program data is included as video data, there are subtitles (introduction of program titles, casts, etc., or Japanese subtitles in overseas works) that cannot be selected by the user. May be included. In addition, the broadcast program data may include subtitle data that allows the user to select whether or not to display the program data. Such selectable subtitle data is generally called Closed Caption, and it was developed mainly for the hearing impaired, and not only the conversation of performers like Japanese subtitles in overseas works. For example, descriptions of BGM, sound effects, and the like are also included.

  For example, in NTSC (National Television System Committee) analog terrestrial broadcasting, 525 scanning lines are used for video signals. Of these 525 lines, one field is composed of two fields. The first 21 lines are allocated for an interval for starting scanning, which is called VBI (Vertical Blanking Interval). The closed caption is transmitted by multiplexing the character code in the 21st VBI of the VBIs in each field. Each field is used to transmit about 60 characters per second through two types of character sets.

  Regarding the transmission of caption information for digital television broadcasts, terrestrial digital television broadcast operation regulations / technical documents (ARIB TR-B14), BS / broadband CS digital broadcast operation regulations / technical documents (ARIB TR-B15) are domestic regulations. The subtitle information can be transmitted simultaneously with the transmission of the video information using the subtitle information transport stream. Then, in the receiver for digital television broadcasting, the code corresponding to the caption information is decoded, characters and figures constituting the caption are generated, and displayed superimposed on the video. In digital TV broadcasting, the text information transmitted simultaneously with the transmission of video information using a transport stream for subtitle information is stored internally, and the text information stored internally can be displayed and viewed on a TV monitor at any time. Thus, there is a technique in which a user can utilize subtitle information without taking a note of the displayed subtitle (see, for example, Patent Document 1).

  In recent years, television broadcasts can be viewed on terminals that can be carried by users such as mobile phones and PDAs (Personal Digital Assistants). In addition, a program recorded by a recording device at home is recorded on a recording medium mounted on a portable device, and the recorded program is reproduced from the recording medium. Thus, the caption data to be broadcast has begun to be used for various services. In addition, a system is also provided in which television broadcasts themselves are received by various devices and the received programs can be browsed.

  However, a television broadcast is received by various devices, and a program broadcast by the television broadcast is recorded on a predetermined recording medium and viewed on a portable device, but the program is recorded. There is a limit to the capacity of recording media. In addition, when a condition of a recording medium that can be mounted (built in) in a portable device is added, the capacity of the usable recording medium is further limited.

  Therefore, when viewing a program on a portable device, only a short time program can be viewed (only a part of the program can be viewed), and if a high compression ratio is recorded to record a long time program, the video becomes rough and the image quality is improved. There was a problem of being lowered. For this reason, there was a problem that it did not spread. Therefore, there is a demand for a function that allows a portable device to browse a long program or a plurality of programs while preventing the image quality from being deteriorated.

  In order to solve such a problem, even in a portable device, in order to be able to view a plurality of programs and long-time programs, a television broadcast program instructed by a mobile phone is recorded, and this program is recorded. There is known an information processing apparatus capable of grasping the contents of a program by converting the recorded program into thumbnail image data and text data associated with time information and displaying the data (for example, see Patent Document 2). .

  By the way, in order to easily understand the content of video content such as broadcast programs, the content of the content is displayed by displaying subtitle data (text data) attached to the video and still images (thumbnail images) extracted from the video data. Can be browsed or searched. In this case, it is necessary to extract the necessary amount of still images from the video to understand the contents, but it is not practical to extract still images from long-time video data manually. It is necessary to be able to automatically extract still images. In general, when a still image is extracted from video data (moving image data), it is conceivable to extract frames at regular time intervals. As shown in FIG. 12, the frame extraction unit 51 cuts out each frame data from the input video data and refers to the output of the clock T, and converts the frame data into the clock T at a fixed time interval (for example, every 1 minute). Is recorded in the time / still image recording unit 52 together with the time information output by the camera, it is possible to automatically extract and record a predetermined number of still images even for long-time video data. .

However, this method has a problem that if the time interval for cutting out frames is shortened, more still images are extracted unnecessarily, and if the time interval is longer, necessary still images are not extracted. In order to solve such a problem, a television receiver that extracts a still image when new caption information is received is known (see, for example, Patent Document 3). This makes it possible to perform digest recording with an extremely small amount of data by using the subtitle information service to greatly reduce the image data to be recorded.
JP 2003-078889 A JP 2006-253960 A JP 2007-006308 A

  However, in the television receiver shown in Patent Document 3, there are a plurality of scenes even when the same subtitle is displayed, and a still image of a frame at the time when new subtitle information is received is not necessarily obtained. There is a problem that it is not necessarily a representative still image in a scene displaying the caption.

  The present invention has been made in view of such circumstances, and provides a still image extraction device and a still image extraction program capable of extracting the most representative still image among the same subtitles being displayed. With the goal.

  The present invention is a still image extraction device that extracts a still image from video data accompanied by caption data in order to create thumbnail data, and the caption data is newly displayed from each frame of the video data. The first frame at the time, the second frame at the time when the displayed caption data is erased, and N (N is a natural number of 1 or more) between the first frame and the second frame A frame extracting unit for extracting the frames of the frame, a difference calculating unit for obtaining a difference between two temporally adjacent images among (N + 2) frames extracted by the frame extracting unit, and the difference Two frames with the smallest image difference obtained by the calculation means are specified, and the video data of the earlier or later frame of the two specified frames is identified. Characterized in that a still image extracting means for recording by extracting data as a still image.

  The present invention further includes punctuation mark extraction means for extracting punctuation marks from the subtitle data, and the frame extraction means is the (N + 2) only when a punctuation mark or a punctuation mark is detected in the subtitle data by the punctuation mark extraction means. It is characterized by performing frame extraction of a sheet.

  The present invention is a still image extraction program for extracting still images from video data accompanied by subtitle data by a computer in order to create thumbnail data, wherein the subtitle data is newly displayed from each frame of the video data. A first frame at the time when the subtitle data is displayed, a second frame at the time when the displayed caption data is erased, and N between the first frame and the second frame (N is a natural number of 1 or more) ) A frame extraction step for extracting frames, and a difference calculation step for obtaining a difference between two temporally adjacent images among (N + 2) frames extracted by the frame extraction step; The two frames with the smallest image difference obtained by the difference calculating step are specified, and the time of the two specified frames is Characterized in that to perform to earlier or later one frame and a still image extraction step of extracting and recording video data as a still image in the computer.

  The present invention further causes a computer to perform a punctuation extraction step of extracting punctuation marks from the caption data, and the frame extraction step is performed only when a punctuation mark or a punctuation mark is detected in the caption data by the punctuation extraction step. (N + 2) frames are extracted.

  According to the present invention, it is possible to extract a required amount of still images from video data without excess or deficiency in order to understand the content of video data, and in particular, the most representative still image among the same subtitles being displayed. The effect that a picture can be extracted is acquired.

  Hereinafter, a still image extracting apparatus according to an embodiment of the present invention will be described with reference to the drawings. First, a receiving / distributing system to which the still image extracting apparatus according to the present invention is applied will be described. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a distribution device that distributes video, audio, text, and the like. Reference numeral 2 denotes a control unit that performs overall control of processing operations of the distribution apparatus 1. Reference numeral 3 denotes an input / output unit for inputting / outputting various data in addition to input of content data. Reference numeral 4 denotes text / thumbnail distribution / display software 9, video / audio / text distribution / display software 10, content data 11, 12, 13, distribution / display unit software management data 14, content management data 15, user management data 16. The storage unit stores content / user additional data 17 and the like. Reference numeral 5 denotes a text / thumbnail distribution unit that distributes text and thumbnails. Reference numeral 6 denotes a video / audio / text distribution unit that distributes video, audio, and text. Reference numeral 7 denotes a content / user-added data receiving / distributing unit that receives data such as comments and evaluations given to the content by the user and distributes the aggregated data. Reference numeral 8 denotes a communication unit that performs information broadcasting or information communication.

  The text / thumbnail distribution / display software 9 is software for realizing text / thumbnail distribution / display processing. The video / audio / text distribution / display software 10 is software that realizes video / audio / text distribution / display processing. The content data 11 is content data composed of video, audio, text, thumbnails, and other data, and each data is distributed from the distribution device 1 via the communication unit. The content data 11 includes video data (moving image data) 18 including a collection of still images that change over time, audio data 19 that is synchronized with the video over time, video such as data obtained by converting subtitles of a television broadcast into text, or Text data 20 that is temporally linked to audio, thumbnail data 21 that is still image data obtained by reducing video, and overall information data such as the name and outline of the entire content, a scene description of part of the content, BGM, title name, etc. It is roughly divided into individual information data, and is composed of other data 22 that is temporally linked to video or audio.

  The distribution / display software management data 14 is data for managing text / thumbnail distribution / display software and video / audio / text distribution / display software. The content management data 15 is data for managing the content data 11 to 13. The user management data 16 is data for managing user information. The content / user additional data 17 is data such as comments and evaluations given to the content by the user.

  Reference numeral 23 denotes a receiving device that receives and displays video, audio, text, and the like distributed by the distribution device 1, and includes, for example, a mobile phone terminal. Reference numeral 24 denotes a control unit that performs overall control of processing operations of the receiving device 23. Reference numeral 25 denotes an input unit configured by a dial key, a function key, or the like, and performing a man-machine interface with the user. Reference numeral 26 denotes a display device configured with a liquid crystal display or the like in order to display content data or the like to the user. Reference numeral 27 denotes a communication unit that performs information communication with the broadcast reception and distribution device 1. Reference numeral 28 denotes a storage unit that stores text / thumbnail display software 38, video / audio / text display software 39, content data 40, 41, 42, and the like.

  Reference numeral 29 denotes a text / thumbnail display unit that reads text data and thumbnail data in the content data from the storage unit 28 via the data buffer 30 and displays them on the display device 26 using the text / thumbnail display software 38. And comprises a text display unit 31 and a thumbnail display unit 32. The text display unit 31 displays text data in the content data on the display device 26. As the text data, data stored in the storage unit 28 or data acquired via the communication unit 27 is used, and display control such as line feed and page feed is performed in response to an operation input in the input unit 25. The thumbnail display unit 32 displays thumbnail data and other data in the content data. As the thumbnail data and other data, data stored in the storage unit 28 and data acquired via the communication unit 27 are used. Among the thumbnail data and other data, the individual information data is displayed in conjunction with the text data.

  Reference numeral 33 denotes a video / audio / text display software 39 that reads out video data, audio data, text data, and the like in the content data from the storage unit 28 via the data buffer 34 and displays the data on the display device 26. A video / audio / text display unit, which includes a video display unit, an audio output unit, and a text display unit. The video display unit 35 displays video data in the content data. For video data, data stored in the storage unit 28 or data acquired via the communication unit 27 is used, and display control such as playback, stop, fast forward, and rewind is performed in response to an operation input in the input unit 25. Do. The audio output unit 36 displays audio data in the content data. As the audio data, data stored in the storage unit 28 or data acquired via the communication unit 27 is used. The audio data is output in conjunction with the video data, or the audio data is output in advance and displayed in association with the video. The text display unit 37 displays text data and other data in the content data. As text data and other data, data stored in the storage unit 28 and data acquired via the communication unit 27 are used. Among text data and other data, individual information data is displayed in conjunction with video data or audio data.

  The text / thumbnail display software 38 is software that realizes processing for displaying on the display device 26 using text data and thumbnail data in the content data. The text / thumbnail display software 38 is acquired from the distribution device 1 via the communication unit 27 and stored in the storage unit 28. In response to a request from the user, when viewing the content data by the text / thumbnail display unit 29, this software is called and used. The text / thumbnail display software 38 is modified by version upgrade, and the latest software is sequentially acquired from the distribution apparatus 1 via the communication unit 27.

  The video / audio / text display software 39 is software that realizes processing to display on the display device 26 using video data, audio data, text data, etc. in the content data. The video / audio / text display software 39 is acquired from the distribution apparatus 1 via the communication unit 27 and stored in the storage unit 28. In response to a request from the user, when viewing the content data by the video / audio / text display unit 33, this software is called and used. The video / audio / text display unit software 39 is modified by upgrading, and the latest software is sequentially acquired from the distribution apparatus 1 via the communication unit 27.

  Next, the content browsing function in the receiving device 23 shown in FIG. 1 will be described with reference to FIG. FIG. 11 is a diagram showing a content browsing function in the receiving device 23 shown in FIG. The receiving device 23 has three functions: a metadata search function, a reading mode function, and a viewing mode function.

(1) Metadata search function This is a function that allows a user to pinpoint to a desired place in a program by inputting a keyword. Metadata is created using caption data and provided as text. For example, if “deien” is entered as a search keyword, the metadata of all programs can be searched, and “Japanese garden” and scenes in the program can be reached.

(2) Reading mode function This is a function that enables quick browsing of information by displaying text (caption data) and a still image (thumbnail) in association with each other, and can perform scrolling and page turning. Since text and still images are downloaded in advance, it can be viewed “anytime, anywhere” even when radio waves do not reach.

(3) View mode function This is a function capable of playing back a moving image from this point by clicking on a text part in the read mode, and has a fast forward / rewind / search function. The moving image reproduction has a function of reproducing video content stored in advance in the receiving device 23 and a function of reproducing by streaming via communication or broadcasting. In the case of reproduction by streaming, it is possible to reproduce a moving image for a long time, and it is possible to easily resume viewing by storing a portion where viewing is interrupted.

  The reading mode and the viewing mode can be switched in conjunction with each other, and can be used for searching for a moving image in which the reading mode is desired.

  Next, a still image extraction method for generating thumbnail data 21 among the content data 11 stored in the storage unit 4 shown in FIG. 1 will be described with reference to FIG. The thumbnail data may be generated in the distribution device 1, but is generated in a server or the like different from the distribution device 1, input via the input / output unit 3, and stored in the storage unit 4. Also good. Here, description will be made assuming that the text / thumbnail distribution unit 5 generates thumbnail data (extracts a still image) using the video data 18 and the text data (caption data) 20 of the content data 11.

  FIG. 2 is a block diagram illustrating a function of extracting a representative still image while the same subtitle that is temporally linked to the video data 18 is displayed. First, the caption page extraction unit 53 reads the text data 20 of the caption page that is temporally linked to the video data 18. The text data of the subtitle page includes information on the subtitle display start time (elapsed time from the top of the video data) and the erase time of the displayed subtitle (elapsed time from the top of the video data). The subtitle page extraction unit 53 outputs the subtitle display start time and the subtitle deletion time information to the frame extraction unit 51.

  The frame extraction unit 51 extracts a frame that is a candidate for extracting a representative still image while the same caption is displayed from the video data 18. At this time, the frame extraction unit 51 reads the N value stored in advance in the N value storage unit 50. The N value is a value that defines how many candidate frames are extracted while the same subtitle is displayed. If the N value is “2”, the same subtitle is displayed. This means that two frames are extracted. The N value is a value that does not include the subtitle display start time frame and the subtitle erase time frame, and stores a value suitable for still image extraction in advance.

  When extracting a frame that is a candidate for extracting a representative still image while the same subtitle is being displayed, the frame extraction unit 51 is a subtitle display start time frame (first frame in the claims). Then, N frames are extracted so that the displayed subtitle erase time frame (second frame in the claims) and the subtitle are displayed at equal time intervals. That is, if N is 2, four (N + 2) frames are extracted while the same subtitle is displayed.

Here, the processing operation of the frame extraction unit 51 will be described with reference to FIG. FIG. 4 is a diagram illustrating a processing operation of the frame extraction unit 51. In FIG. 4, a frame in which the display section of the nth (n is a natural number) subtitle _n has ended is a frame F _m (an erase frame of subtitle _n ), and a time at which the display of a new subtitle _{n + 1} is started is TC _IN. The time when the subtitle _{n + 1} is erased is TC _OUT . A frame between TC _{IN and} TC _OUT (display section of subtitle _{n + 1} ) is a target frame from which a representative still image is extracted. Here, it is assumed that the N value stored in the N value storage unit 50 is “2”.

First, the frame extraction unit 51 extracts a frame F _{m + 1 at} time TC _IN (display start frame of subtitle _{n + 1} ) and a frame F _{m + z at} time TC _OUT (erasure frame of subtitle _{n + 1} ). Then, the frame extraction unit 51 subtracts TC _IN from TC _OUT (TC _OUT -TC _IN ) to calculate the time Time during which the caption _{n + 1} is displayed, and extracts the frame time based on the N value. The interval (Time / (N + 1)) is calculated. Here, since the N value is “2”, the value is obtained by dividing Time by 3.

Next, the frame extraction unit 51 adds the time (Time / (N + 1)) to the time TC _IN to calculate the frame extraction time TC ₁ . In addition, the frame extraction unit 51 calculates the frame extraction time TC ₂ by adding time (Time / (N + 1)) to the time TC ₁ . Then, the frame extraction unit 51 extracts the frame F _{m + x} to which the time information closest to the time TC ₁ is assigned and the frame F _{m + y} to which the time information closest to the time TC ₂ is assigned. Since the N value is “2”, two frames are extracted. With this processing operation, (N + 2) frames (F _{m + 1} , F _{m + x} , F _{m + y} , F _{m + z} ) are extracted.

Next, returning to FIG. 2, the frame extraction unit 51 outputs the frame data of the extracted (N + 2) frames to the difference calculation unit 54 and the still image extraction unit 59. Here, the output frame data includes time information given to each frame data. In response to this, the difference calculation unit 54 calculates the difference between the images of two temporally adjacent frames from among the (N + 2) frame data. In the example shown in FIG. 4, three difference values (D1) include a difference D1 between the frame F _{m + 1} and the frame F _{m + x} , a difference D2 between the frame F _{m + x} and the frame F _{m + y} , and a difference D3 between the frame F _{m + y} and the frame F _{m + z.} , D2, D3). This is an example when the N value is “2”, and the number of obtained difference values is N + 1.

  The difference calculation process of the two pieces of frame data performed by the difference calculation unit 54 may use a method of simply subtracting each corresponding pixel value to obtain the difference. Also, motion compensation prediction processing used when encoding video and compressing data (for example, refer to Digital Broadcasting Handbook, Ohmsha, Chapter 2, Compression Technology, Chapter 1, Video Coding Technology). Applying a technology to reduce the amount of data using similar properties, detect the amount of motion (motion vector) from the reference frame (temporarily previous frame), and depending on the magnitude of the motion, The difference may be obtained by shifting to obtain the difference.

  By doing so, it is possible to determine that the difference is small when the scene is not changed but only the person in the scene is moving. It is possible to determine whether or not the scene has changed by obtaining the difference using such a method. The calculation of the difference performed by the difference value calculation unit 54 can use a known method capable of detecting that the correlation between the two pieces of frame data is high (the scene has not changed).

Next, the difference value calculation unit 54 outputs the obtained three difference values D1, D2, and D3 to the still image extraction unit 59. The three difference values D1, D2, and D3 are output to the still image extraction unit 59 in the order of early frame time. In response to this, the still image extraction unit 59 compares the absolute values of the three difference values D1, D2, and D3 to obtain the smallest difference value, and identifies two frames having the smallest difference value. To do. Subsequently, the still image extraction unit 59 extracts the earlier frame of the two identified frames. That is, when the time Time shown in FIG. 4 is divided into three sections and there is no difference between the first and last frames of each of the three sections, the scene has not changed or is similar. Since this means that it has continued for a predetermined time, the extraction is performed by regarding this frame as a representative still image at time Time.
Note that the two specified frames are frames in which the scene has not changed or a similar scene has continued for a predetermined time, and therefore, the frame that is later in time among the two specified frames is represented. You may make it extract as a still image.

For example, as shown in FIG. 4, a scene “B” is shown in the frame F _{m + 1} , and a scene “C” is shown in the frame F _{m + x} . Is a large value. Further, since the scene “C” is shown in the frame F _{m + y and} the scene “D” is shown in the frame F _{m + z} , the difference value between the two pieces of frame data is also a large value. However, the frame F _{m + x} and the frame F _{m + y} have different positions, but both have a scene “C”. When the difference is obtained using the motion compensation prediction process described above, the difference value is small. It becomes. Therefore, in the example shown in FIG. 4, the frame F _{m + x} is extracted as a representative still image.

  Next, the still image extraction unit 59 records the extracted frame data and the time information (elapsed time information from the beginning of the video data) attached to this frame in the time / still image recording unit 52. As a result, a representative still image (frame Fm + x in the example shown in FIG. 4) representative of the display section of the caption n + 1 shown in FIG. 4 is extracted and recorded in the time / still image recording unit 52. Then, the frame extraction unit 51 and the caption page extraction unit 53 extract the representative still image in the display section of the caption n + 2 by performing the above-described processing operation again in the display section of the caption n + 2. By repeating this processing operation until the end of the video data, it is possible to extract the most representative still image among the same subtitles being displayed. The representative still image extracted by this processing becomes the thumbnail data 21 stored in the storage unit 4.

  As described above, since the video data 18 is accompanied by the text data (caption), the most representative still image is extracted among the same subtitles displayed. A necessary amount of still images can be extracted from the video data without excess or deficiency. Thereby, the part with the subtitle is a place where there is a speech or explanation by voice, and a still image necessary for understanding the content of the video content such as a broadcast program can be extracted.

  Next, a modification of the functional block shown in FIG. 2 will be described with reference to FIG. The functional block diagram shown in FIG. 3 is different from the functional block diagram shown in FIG. 2 in that a punctuation mark extraction unit 60 is newly provided. The punctuation mark extraction unit 60 extracts the punctuation mark “,” or the punctuation mark “.” Included in the text data 20 of the subtitle page read by the subtitle page extraction unit 53, and when the punctuation mark is included in the text data 20. The frame extraction unit 51 is notified that punctuation marks are included. The frame extraction unit 51 performs the above-described (N + 2) frame extraction processing only when the punctuation mark extraction unit 60 detects the punctuation mark or punctuation mark in the caption text data, and the punctuation mark or punctuation mark is included in the caption text data. Is not detected, the processing proceeds to the next caption data processing without performing the above-described (N + 2) frame extraction processing. Since the processing operation after performing the (N + 2) frame extraction process is the same as the above-described operation, a detailed description of the processing operation is omitted here.

  In this way, when punctuation marks are included in the subtitles, the scene displayed when punctuation marks are included in the displayed subtitles may include a representative still image for understanding the contents. If the displayed subtitles do not contain punctuation marks, the displayed scene contains a representative still image for understanding the content. Since it is considered that the possibility is low and the process of extracting the representative still image is not performed, the representative still image can be extracted efficiently.

  Next, another still image extraction method will be described with reference to FIG. FIG. 5 is a block diagram showing a function of inputting video data and extracting a still image in accordance with the video difference. First, the frame extraction unit 51 reads the video data 18 and cuts out each frame. In parallel with this, the difference calculation unit extraction unit 54 reads the video data 18, cuts out each frame, and calculates the difference between the previous and next frames. When the calculated difference amount is large, the difference calculation unit 54 instructs the frame extraction unit 51 to record the current frame in the time / still image recording unit 52 and An instruction is issued to record the current time (elapsed time from the beginning of the video data) in the time / still image recording unit 52.

  In response to this, the frame extraction unit 51 records the frame at the timing when the instruction is issued in the time / still image recording unit 52. The clock T records the time information of the timing at which the instruction is issued in relation to the frame data (still image) recorded in the time / still image recording unit 52. By repeating this operation until the end of the video data 18, a plurality of thumbnail data in which the extracted still image data and time information are related are recorded in the time / still image recording unit 52. When the processing is completed to the end of the video data 18, the time / still image recording unit 52 stores the recorded thumbnail data in the storage unit 4. The thumbnail data 21 is used for a still image of the reading mode function shown in FIG.

  In this way, an image difference between video frames is calculated, and a still image can be extracted when the image difference from one frame exceeds a certain amount. As a result, even if a still image is extracted when there is no video change, it will not be useful information for content understanding because it is the same content image, but the video change part contains different content, so the video There is a high possibility that video information necessary for understanding the content is included.

  Next, another still image extraction method will be described with reference to FIG. FIG. 6 is a block diagram illustrating a function of inputting video data and extracting a still image in combination with the time interval and the magnitude of the video difference. First, the frame extraction unit 55 reads the video data 18, refers to the time information output by the clock TA, cuts out frames at relatively short fixed time intervals, and cuts out the frames to the frame extraction unit 51 and the difference calculation unit 54. Output the frame. Then, the difference calculation unit 54 calculates the difference between the previous and next frames. When the calculated difference amount is large, the difference calculation unit 54 instructs the frame extraction unit 51 to record the current frame in the time / still image recording unit 52 and An instruction is issued to record the current time (elapsed time from the beginning of the video data) in the time / still image recording unit 52.

  In response to this, the frame extraction unit 51 records the frame at the timing when the instruction is issued in the time / still image recording unit 52. The clock T records the time information of the timing at which the instruction is issued in relation to the frame data (still image) recorded in the time / still image recording unit 52. By repeating this operation until the end of the video data 18, a plurality of thumbnail data in which the extracted still image data and time information are related are recorded in the time / still image recording unit 52. When the processing is completed up to the end of the video data 18, the time / still image recording unit 52 stores the recorded thumbnail data in the storage unit 4. The thumbnail data 21 is used for a still image of the reading mode function shown in FIG.

  As described above, it is possible to extract a still image without extracting a still image with a small change in video by extracting the image at a predetermined time interval. As a result, still images with little change can be omitted because they are redundant.

  Next, another still image extraction method will be described with reference to FIG. FIG. 7 is a block diagram illustrating a function of inputting video data and extracting a still image in combination with the time interval and the magnitude of the video difference. First, the frame extraction unit 56 reads the video data 18, refers to the time information output by the clock TA, cuts out frames at a relatively long fixed time interval, and relates them to the time information output by the time TA. Record in the time / still image recording unit 52. In parallel with this, the frame extraction unit 51 reads the video data 18 and cuts out each frame, and the difference calculation unit 54 reads the video data 18, cuts out each frame, and calculates the difference between the previous and subsequent frames. . When the calculated difference amount is large, the difference calculation unit 54 instructs the frame extraction unit 51 to record the current frame in the time / still image recording unit 52, and Is instructed to record the time (elapsed time from the beginning of the video data) in the time / still image recording unit 52.

  In response to this, the frame extraction unit 51 records the frame at the timing when the instruction is issued in the time / still image recording unit 52. The clock T records the time information of the timing at which the instruction is issued in relation to the frame data (still image) recorded in the time / still image recording unit 52. By repeating this operation until the end of the video data 18, a plurality of thumbnail data in which the extracted still image data and time information are related are recorded in the time / still image recording unit 52. When the processing is completed to the end of the video data 18, the time / still image recording unit 52 stores the recorded thumbnail data in the storage unit 4. The thumbnail data 21 is used for a still image of the reading mode function shown in FIG.

  In this way, still images are extracted at relatively long fixed time intervals, and when it is determined that the difference in video is larger than a certain value, still images can be extracted by extracting and adding still images at this time point. it can. As a result, still images can be obtained at regular time intervals, and important still images can be added where there are video changes.

  Next, another still image extraction method will be described with reference to FIG. FIG. 8 is a block diagram illustrating a function of inputting video data and extracting a still image in combination with a time interval and the number of subtitle pages. First, the frame extraction unit 56 reads the video data 18, refers to the time information output by the clock TA, cuts out frames at a relatively long fixed time interval, and relates them to the time information output by the time TA. Record in the time / still image recording unit 52. In parallel with this, the frame extraction unit 51 reads the video data 18 and cuts out each frame, and the subtitle page number determination unit 57 generates a subtitle page in which the text data is temporally linked to the video data 18. Determine the number of pages (number of subtitle updates). As a result of the determination, if the number of subtitle pages (number of subtitle updates) exceeds a predetermined value, the frame extraction unit 51 is instructed to record the current frame in the time / still image recording unit 52. At the same time, the clock T is instructed to record the current time (elapsed time from the beginning of the video data) in the time / still image recording unit 52.

  In response to this, the frame extraction unit 51 records the frame at the timing when the instruction is issued in the time / still image recording unit 52. The clock T records the time information of the timing at which the instruction is issued in relation to the frame data (still image) recorded in the time / still image recording unit 52. By repeating this operation until the end of the video data 18, a plurality of thumbnail data in which the extracted still image data and time information are related are recorded in the time / still image recording unit 52. When the processing is completed to the end of the video data 18, the time / still image recording unit 52 stores the recorded thumbnail data in the storage unit 4. The thumbnail data 21 is used for a still image of the reading mode function shown in FIG.

  In this way, still images are extracted at relatively long fixed time intervals, and if the number of subtitle pages (the number of times subtitles are updated) exceeds a predetermined value, the still images at this time are extracted and added. Can extract a still image. Thereby, a still image can be added so as to balance the subtitles.

  Next, another still image extraction method will be described with reference to FIG. FIG. 9 is a block diagram illustrating a function of inputting video data and extracting a still image by combining a caption page and a video difference. First, the frame extraction unit 56 reads the video data 18 and cuts out each frame. In parallel with this, the caption page extraction unit 53 extracts a caption page in which text data is temporally linked to the video data 18. Then, the subtitle page extraction unit 53 instructs the frame extraction unit 56 to record the current frame in the time / still image recording unit 52 at the timing when the subtitle page is updated, and to the clock TB. The current time (elapsed time from the beginning of the video data) is instructed to be recorded in the time / still image recording unit 52. In response to this, the frame extraction unit 56 records the frame at the timing when the instruction is issued in the time / still image recording unit 52. In addition, the clock TB records the time information of the timing at which the instruction is issued in relation to the frame data (still image) recorded in the time / still image recording unit 52.

  On the other hand, the frame extraction unit 51 reads the video data 18 and cuts out each frame. In parallel with this, the difference calculation unit 54 reads the video data 18 and, if the time interval determined by the time interval determination unit 58 is a predetermined time, cuts out each frame and calculates the difference between the previous and subsequent frames. When the calculated difference amount is large, the difference calculation unit 54 instructs the frame extraction unit 51 to record the current frame in the time / still image recording unit 52 and also notifies the clock TC of the current time Is instructed to be recorded in the time / still image recording unit 52. In response to this, the frame extraction unit 51 records the frame at the timing when the instruction is issued in the time / still image recording unit 52. The clock TC records the time information of the timing at which the instruction is issued in relation to the frame data (still image) recorded in the time / still image recording unit 52.

  By repeating the above operation until the end of the video data 18, the time / still image recording unit 52 records a plurality of thumbnail data in which the extracted still image data and the time information are associated with each other. When the processing is completed to the end of the video data 18, the time / still image recording unit 52 stores the recorded thumbnail data in the storage unit 4. The thumbnail data 21 is used for a still image of the reading mode function shown in FIG.

  Thus, using the fact that text data (caption) is attached to the video data 18, a still image can be extracted every time the display of the subtitle is updated, and an image difference between video frames is calculated. A still image can be added when the image difference from one frame exceeds a certain amount. Thereby, still images with high importance can be collected.

  The above three types of extraction methods of “extracting at regular time intervals”, “extracting for each subtitle display page”, and “extracting when the video change is large” have the following characteristics. That is, in the method of extracting at a constant time interval, if the time interval is shortened, still images are unnecessarily increased, and if the time interval is long, necessary still images may not be extracted. Also, the method of extracting for each subtitle display page may be that there are too few still images to be extracted if there are very few subtitles, and there may be a shortage of still images of video scenes necessary for understanding. In addition, when extracting a still image when the video change is large, the extraction method when the video change is large increases the probability that a still image with a different content is selected, but the scene where the person is speaking is fixed. In the case of video, there may be extremely few still images compared to the subtitles being spoken. On the other hand, if still images are extracted when the video change is small, there may be too many useless still images.

  With these three extraction methods, it is difficult to extract an appropriate still image with one extraction method, but by combining these extraction methods, the minimum required still image can be automatically extracted. It becomes like this.

  As described above, according to the still image extraction method of the present invention, it is possible to efficiently extract only still images with high importance. By making it possible to browse the extracted still images using the reading mode function, it is possible to quickly browse and understand content including moving images such as television images by using these still images. In addition, it is possible to browse and search for information required in the content using these still images. In addition, after searching with these still images, it can be viewed by switching to a mode function for viewing a necessary moving image scene such as a television image. In particular, since the still image and the information on the time when the still image was extracted are related and stored, the time of the still image and the time of the moving image can be linked as shown in FIG. It is possible to immediately shift to a mode for viewing a necessary moving image scene. On the other hand, when the necessary moving image scene is finished, it is possible to return to the still image near the time when the viewing is finished and return to the reading mode for browsing and searching by the still image.

  A program for realizing the functions of the processing units in FIGS. 2 to 7 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed to execute a still image. An extraction process may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is explanatory drawing which shows the processing operation of the frame extraction part 51 shown in FIG. 2, FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is a block diagram which shows the function of the still image extraction in the text / thumbnail delivery part 5 shown in FIG. It is explanatory drawing which shows the function of the receiver 23 shown in FIG. It is a block diagram which shows the still image extraction function by a prior art.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Distribution apparatus, 23 ... Receiving device, 4 ... Memory | storage part, 5 ... Text / thumbnail distribution part, 50 ... N value memory | storage part, 51, 55, 56 ... Frame extraction , 52 ... Time / still image recording unit, 53 ... Subtitle page extraction unit, 54 ... Difference calculation unit, 57 ... Subtitle page number determination unit, 58 ... Time interval determination unit, 59 ... Still image extraction unit, 60 ... Punctuation mark extraction unit, T, TA, TB, TC ... Clock

Claims (4)

A still image extraction device for extracting still images from video data accompanied by caption data in order to create thumbnail data,
From each frame of the video data, a first frame when the caption data is newly displayed, a second frame when the displayed caption data is erased, the first frame, and the frame Frame extracting means for extracting N (N is a natural number of 1 or more) frames between the second frames;
A difference calculating means for respectively obtaining a difference between two temporally adjacent images out of (N + 2) frames extracted by the frame extracting means;
The two frames having the smallest image difference obtained by the difference calculating means are identified, and the video data of the earlier or later frame of the identified two frames is extracted and recorded as a still image. A still image extracting device. Further comprising punctuation mark extraction means for extracting punctuation marks from the caption data;
2. The still image according to claim 1, wherein the frame extraction unit extracts the (N + 2) frames only when a punctuation mark or a punctuation point is detected in the caption data by the punctuation mark extraction unit. Extraction device. A still image extraction program for extracting still images from video data accompanied by caption data by a computer in order to create thumbnail data,
From each frame of the video data, a first frame when the caption data is newly displayed, a second frame when the displayed caption data is erased, the first frame, and the frame A frame extraction step of extracting N (N is a natural number of 1 or more) frames between the second frames;
A difference calculating step for obtaining a difference between two temporally adjacent images among (N + 2) frames extracted by the frame extracting step;
The two frames having the smallest image difference obtained by the difference calculating step are identified, and the video data of the earlier or later frame of the identified two frames is extracted and recorded as a still image. A still image extraction program for causing a computer to perform a still image extraction step. Further causing the computer to perform a punctuation extraction step of extracting punctuation from the caption data;
The still image according to claim 3, wherein the frame extracting step extracts the (N + 2) frames only when a punctuation mark or a punctuation mark is detected in the caption data by the punctuation mark extraction step. Extraction program.

Images