KR100711948B1 - Personalized Video Classification and Retrieval System - Google Patents

Abstract

Video retrieval systems allow users to quickly and easily select and receive stories of interest from a video stream. The video retrieval system categorizes the stories and delivers samples of selected stories that match each user's current preferences. The user's preference may include a specific broadcasting network, a person, a story topic, a keyword, and the like. The key frames of each selected story are displayed in succession, and when the user views the frame of interest, the user selects the story associated with the key frame to view in more detail. The present invention is particularly suitable for targeted news retrieval. In a preferred embodiment, the news story is stored and the selection of a new story to view in detail based on the associated key frame performs playback of the selected news story. The principles of the present invention also allow the user to perform a direct search of other forms of broadcast. For example, a user may initiate an automated scan that presents a sample of a broadcast that matches the user's current preferences similar to direct channel surfing.

Story segment identifier, classification system, user preference, key frame

Description

Personalized video classification and retrieval system             

The present invention relates to the field of communications and information processing, and more particularly to the field of video classification and retrieval.

Customers are provided with an increasing supply of information and entertainment choices. Hundreds of television channels are available to customers through broadcast, cable and satellite communication systems. As the supply of information increases, it becomes more difficult for customers to efficiently select information sources that provide particular information or special interests. For example, suppose that a customer randomly searches (“channel surfs”) a topic of interest among the 12 television channels. If the topic of particular interest to the customer is not a popular topic, only one or two broadcasters will broadcast a story dealing with the topic for a short period of time. If the customer is not advised in advance, the interested customer will not be able to turn to the particular broadcaster's channel when the story of interest is broadcast. Conversely, if the stock price of interest is very popular, many broadcasters will broadcast stories that cover the topic, and channel surfers will be overfilled with excess information.

Automated scanning is generally used in radio broadcasts and somewhat less commonly in television broadcasts. Traditionally, these scans provide short term samples of each broadcast channel. When the user selects a channel, the tuner is turned to remain on that channel, otherwise the scan proceeds to the next found channel. However, the scan is not directed or selected. For example, there is no aid provided for the user to specifically scan a news station on the radio or a sports show on the television. Each found channel can be sampled and presented to the user regardless of the user's current interest.

The continued integration of computers and televisions gives customers the opportunity to provide information of particular interest. For example, many websites provide news summaries with links to hearing-visual and multimedia segments corresponding to current news stories. The classification and presentation of the news summary can be optimized for each customer. For example, one customer may want to see weather first, then world news, then local news, while another customer may want to see only sports stories and investment articles. The advantages of such a system are the optimization of the news presented to the user, the need for someone to prepare a summary, and the need for the customer to read the summary to determine whether the story is worth seeing. .

MITRE's Broadcast News Editor (BNE) and Broadcast News Navigator (BNN) (Andrew Merlino, Daryl Morey, and Mark Maybury, MITRE Corporation, Berford MA, Broadcast News Navigation using Story Segmentation, ACM Multimedia Conference Proceedings, 1997, pp. 381- As indicated by 389, there has been continuous progress in the field of automated story partitioning and identification. Using BNE, the newscast is automatically split into each story segment, and the first line of subtitle text associated with the segment is used as the summary of each story. Subtitled text or audio keywords are determined for each story segment. The BNN allows the customer to enter a search word, which classifies the story segment by the number of keywords in each story segment that matched the search word. Based on the occurrence frequency of the matching keyword, the user selects a story of interest. Similar search and search techniques are technically common. For example, conventional text search techniques could be applied to computer based television guidance to allow a person to search for a particular show title, a particular performer, a particular type of show, and the like.

A disadvantage of existing search and search techniques is that they require a corresponding search and clear search operation among others based on clear search. However, often the user does not have a clear search topic in mind. In existing channel-surfing scenarios, the user does not have a clear search topic. Channel surfing users randomly sample various channels on any number of topics of interest, rather than searching for a particular topic. That is, for example, the user may initiate random sampling without a particular subject in mind, and select one of the many channels sampled based on the subject presented to the channel at the sampling time. In another scenario, the user may monitor the television in "background" mode while performing other tasks such as reading or cooking. When a topic of interest is presented, the user reassigns his focus on television and redirects his attention to other work when a topic of less interest is presented.

It is an object of the present invention to provide a news retrieval system that allows a user to quickly and easily select and receive stories of interest. It is another object of the present invention to identify broadcasts of potential interest to the user and to provide a user with a random or systematic sampling of the broadcasts for subsequent selection.

These and other objects are achieved by providing a system that delivers samples of selected news stories that characterize new stories and match each user's current preferences. The user's preferences may include a particular broadcast network, anchors, story topics, keywords, and the like. The key frames of each selected story are displayed in succession, and when the user views the frame of interest, the user selects the story associated with the key frame to view in more detail. In a preferred embodiment, the news story is stored and the selection of a new story to view in detail performs playback of the selected news story.

Although the present invention is particularly suitable for targeted news retrieval, the principles of the present invention also allow the user to perform a direct search of other forms of broadcast. For example, a user may initiate an automated scan that presents a sample of a broadcast that matches the user's current preferences similar to direct channel surfing.

1 is a block diagram illustrating an example of a personalized video search system in accordance with the present invention.

2A shows a video stream 200 of an example of a news broadcast.

FIG. 2B illustrates extraction of key frames from a story segment of a video stream in accordance with the present invention. FIG.

3 illustrates an example user interface for a video retrieval system in accordance with the present invention.

4 is a block diagram illustrating an example of a customer product 400 in accordance with the present invention.

1 illustrates a block diagram of an example of a personalized video search system in accordance with the present invention. The video retrieval system comprises a classification system 100 for classifying each segment of a video stream and a retrieval system 150 for selecting and displaying segments that match one or more user preferences. The video retrieval system receives the video stream 101 from a broadcast channel selector 105, for example a television tuner or satellite receiver. The video stream may be in digital or analog form, and the broadcast may be in any form or medium used to communicate the video stream, including point-to-point communication. While it may be understood by those skilled in the art that the principles presented herein can be extended to other video search applications, the example video search system presented herein provides user preferences for clarity and ease of understanding. Presented as a search system for news stories according to the set of.

The example classification system 100 of FIG. 1 includes a story segment identifier 110, a classifier 120, and a visual characterizer 130. Story segment identifier 110 processes video stream 101 and identifies a separate segment 111 of video stream 101. In the example content, video stream 101 corresponds to a news broadcast and includes multiple news stories that are spread out commercially or commercially. Story segment identifier 110 copies each distributed story segment 111 from video stream 101 to storage device 115, or the beginning and end of each distributed story segment 111 on a copy of video stream 101. The video stream 101 is divided into a new story segment 111 by forming a set of positional parameters that identify. As illustrated by dashed line 106, in a preferred embodiment, video stream 101 is segmented based on the location of segment 111 on a medium such as a video tape recorder, laser disk, DVD, DVR, computer file system, or the like. 111 is stored in the storage device 115 to allow playback. For ease of understanding, the present invention is presented with a story segment 111 stored on storage device 115. As will be apparent to those skilled in the art, this is equivalent to recording the entire video stream 101 and indexing each story segment 111 to the video stream 101.

Story segment 111 is identified using various techniques. Existing news broadcasts have a common format that is particularly suited for story segmentation. 2A illustrates an example video stream 200 of a news broadcast. After the preface 201, a news reporter, or anchor, appears 211 to introduce the first news story segment 221. After the first news story segment 221 is completed, the anchor reappears 222 to introduce the next story segment 222. After the story segment 222 is complete, it is cut 218 for the commercial 228. After the commercial 228, the anchor reappears 213 to introduce the next story segment 223. The sequence of anchor stories distributed as broadcast advertisements is repeated until the end of the news broadcast.

In general, repeated occurrences of anchors 211-214 at the same staged place serve to clearly identify the beginning of a new segment and the end of an existing news segment or commercial. The technique is generally used to identify a commercial in a video stream, such as in a device that mutes the sound when the commercial is presented. Commercial 228 can also occur within story segment 222. The cut 218 for the commercial 228 may also include a repeated appearance of the anchor, but the occurrence of the commercial 228 serves to identify the appearance according to the cut 218 rather than the introduction of a new news segment. do. Anchors appear during the broadcast of story segments 221-224, but most broadcasters use one staged place for story introduction and another staged appearance for post-commercial conversation shots or repeated appearances. . For example, an anchor is shown sitting at a news desk for a story introduction, and the next video of the newscaster is close-up without the news desk in the video. Or, an anchor is presented on the full screen to introduce the story, and on the split screen when talking to the outdoor reporter. Or, anchor shots are taken full face to introduce the story and sided within the story. Once the characteristic story introduction image is identified, technically generalized image matching techniques can be used to automate the story segmentation process. In situations where there is no story segmentation brake that provides automated story segmentation, manual or semi-automatic techniques may also be used. In addition, as standards such as MPEG are developed for optimizable video compression and splicing, video streams may include specified markers that identify the beginning and end of independent segments within the stream.

In addition, a video stream and an audio stream 230 are associated, and in most cases, the subtitle stream 240 corresponds to the audio stream 230. Each story segment 221-224 of FIG. 2A has an associated audio segment 231-234 and possible subtitle text 241-244. The audio segments 231-234 occur simultaneously with the video segments and can be included in each story segment 221-224. Because of the different number of transmissions of audio and text, subtitle segments 241-244 do not necessarily consume the same time interval as each audio segment 221-224. Story segment identifier 110 also includes a speech recognition device that generates text segments 241-244 corresponding to respective audio segments 231-234.

In addition to copying the audio segment, text segments 241-244 also include text from other sources. For example, in non-news broadcasting, the television guide can provide a list of each story, a list of characters, an audience rating, and the like. In a news broadcast, the on-line guide may provide a list of headlines, a list of newscasters, and a list of companies or people who participated in the broadcast. In addition, each broadcast and each story segment may be associated with a text annotation indicating the broadcast channel monitored by the broadcast channel selector 105, such as "ABC", "NBC", "CNN", as well as each anchor introducing each story. Can be. The name of the anchor may be determined automatically or manually based on image recognition technology. Other annotations may include broadcast time, location of each story, and the like. In a preferred embodiment of the present invention, each of the text formatted information segments may be associated with its corresponding story segment. Teletext formatted data may also be included in text segments 241-244.

The story segments 221-224, the audio segments 231-234, and the text segments 241-244 of FIG. 2A include the story segments 111, audio from the story segment identifier 110 of FIG. 1. Corresponding to segments 112, and text segments 113, and video 228, audio 238, and text 248 segments correspond to commercials.

2B illustrates extraction of key frames from story segments of a video stream in accordance with an aspect of the present invention. Story segment 221 includes a number of scenes 251-253. For example, the first scene 251 of the story segment 221 corresponds to the image 211 of the anchor introducing the story segment 221. The next scene 252 may be an image from a remote camera reporting a story or the like. Each scene consists of frames. The first frames 261, 271, 281 of each scene 251, 252, 253 form a set of key frames 291, 292, 293 associated with the story 221, the key frames being the story segment 221. Form a figure summary. The key frames 291, 292, 293 of FIG. 2B correspond to the key frames 114 from the story segment identifier of FIG. 1.

The first frame of each scene can be identified based on the difference between the frames. If the anchor moves during the introduction of the story, only the visual difference may be displayed for each frame. The area of the image corresponding to the news desk or news room background does not substantially change from frame to frame. For example, when a scene change occurs by switching to a remote camera, substantially the entire image changes. Multiple image compression or transform structures provide the ability to store or transmit a sequence of images as a sequence of other frames. If there are many differences, the news frame is generally encoded directly as a sequence of reference frames, and the next frame is encoded as a difference from the reference frame. 2B illustrates the structure with the relative size of each frame F in each scene 251-253. The first frames 261, 271, 281 of each scene 251, 252, 253 are encoded in a reference frame that contains a significant amount of information, or in another frame that contains a significant amount of difference from their previous frame. do. After the change of scene, the next frame is small if it reflects the entire scene with a minimum change due to the movement of an object in the frame or a change to camera angle or magnification. The amount of information contained in each frame is directly related to the change from one frame to the next. In the MPEG compression scheme, for example, the image is transformed using Discrete Cosine Transform (DCT), which produces an encoding of each frame having a size closely related to the amount of random change from one frame to the next. That is, for example, frames 262, 263, and 264 are shown substantially smaller than frame 261 because they contain less information than frame 261, which is a frame corresponding to scene changes. Thus, in a preferred embodiment of the present invention, key frames 291, 292, 293 correspond to frames that include maximum information 261, 271, 281 in segment 221. Other techniques for selecting key frames are apparent to those skilled in the art. For example, one technique selects a frame at the center of each scene, or selects the frame with the minimum difference from all other frames in the scene, such as using least squares determination. In the case of story segmentation, manual and semi-automatic techniques are also used to select key frames, and the composites form a pictorial summary of each story segment. Further, as in the case of story segmentation, subsequent encoding standards may include a direct representation of the key frame within each story segment.

Classifier 120 characterizes each story segment 111 of FIG. 1. In a preferred embodiment, classifier 120 automatically characterizes although manual or semi-automatic techniques may also be used. The main means of the feature in the preferred embodiment is based on the text segment 113 from the segment identifier 110. If the text segments 113 include annotations, such as the name of the broadcast channel and anchor, the annotations are used to identify story segments corresponding to the "broadcaster" and "anchor" categories. If the text segments 113 are a copy or summary of the story segment, keywords such as "victim", "police", "crime" and "defendant" are used to characterize the new story under the theme of "crime". Keywords such as "democratic", "republican", "house", "congress", "minister" are used to characterize new stories under the theme of "politics." Auxiliary categorizations are also defined such that "home run" characterizes the story as a subcategory "baseball" under category "sports", while "touch down" characterizes the story as a subcategory "american football" under the same category "sports" Make up. Likewise, certain names such as "Clinton", "Bill Gates" and "John Wayne" are used as category stories such as "politics", "computers" and "entertainment" respectively. Story segments can have multiple categories, for example "Bill Gates" can be used to categorize stories as both "computer" and "finance." Similarly, if there is a "defendant" and a "democracy" in the same story, the story is categorized as both "crime" and "politics." In the same way, audio segments 112 are used for categorization. In an indirect manner, the audio segment 112 switches to text and categories that apply to the text. In a direct manner, audio segment 112 is analyzed with sounds such as laughter, anger, bombardment, and cheer to determine appropriate characterizations such as "comedy", "violence", "festival".

Optionally, visual characterizer 130 characterizes story segment 111 based on its visual content. The visual characterizer 130 is used to identify a person presented in a story segment based on visual recognition techniques, or to identify a subject based on analysis of image background information. For example, visual characterizer 130 includes a library of people worth commenting. The visual characterizer 130 identifies images that contain a single or dominant picture, which are compared with the images in the library. The visual characterizer 130 may also include a library of content scenes and topic categories associated with them. For example, an image including a person on a map with isobars may uniquely identify the subject as "weather". Similarly, image processing techniques may be used to characterize an "indoor" or "outdoor", "city", "provincial" or "sea" location, and the like. The visual characterizations 131 are provided to the classifier 120 to add, change or supply categorization formed of text 113 and audio 112 associated with each story segment 111. For example, the appearance of smoke in story segment 111 may be used to revise the characteristics of the siren sound in audio segment 112 as "fire" rather than "police".

Visual feature 130 may also be used to prioritize key frames. A newscast can have dozens or hundreds of key frames based on the selection of each new scene. In a preferred embodiment, the number of key frames is reduced by selecting an image that can have more information than others. Specific image content is a representation of an image having important content. For example, a person's name is often displayed below the person's video when the person is first introduced during the newscast. This composition of the person and the text generally conveys important information about the story segment 111. Similarly, close-ups of people or small groups of people generally provide more information than distant scenes or scenes of large groups of people. Many image analysis techniques are commonly used to recognize photographs, flesh colors, text, and other discriminant features in an image. In a preferred embodiment, key frames are prioritized by the video content analysis along with other clues, such as the age of the scene. In general, more important scenes are displayed faster in story segment 111 than more important scenes. The prioritization of the key frames is also used to create a visual table of the contents for the story segments 111 along with a visual table of the content for the video stream 101 by selecting a given number of frames in order of priority.

The classification system 100 receives a set of characterizations, that is, a classification 121 of each story segment 111 from the classifier 120, and a set of key frames 114 for each story segment 111. 110 to the retrieval system 150. The classifier 121 may be provided in various forms. Predefined categories such as "broadcaster", "anchor", "time", "place" and "topic" are provided in predefined categories such as "place" and "topic" allowing multiple entries in a preferred embodiment. . Another classification method used in association with a predefined category is a histogram of a list or selection keyword of the person or organization mentioned in the story segment 111. The classification 121 used in the classification system 100 need not be identical to the filtering system used in the filter 160 of the search system 150 but must be consistent or compatible. As will be apparent to those skilled in the art, the classification translator may convert the classification 121 or a portion of the classification 121 into a form compatible with the filtering system used in the filter 160. It may be added between the 100 and the search system 150. The translation can be automatic, manual or semi-automatic. For ease of understanding, it is assumed herein that the classification 121 of each story segment 111 by the classification system 100 is compatible with the filter 160 of the search system 150.

The filter 160 of the search system 150 identifies the story segments 111 that match the set of user preferences 191 based on the classification 121 of each of the story segments 111. In a preferred embodiment of the present invention, the user is provided with a profiler 190 that encodes a set of user inputs with a preference 191 that is compatible with the filtering system of filter 160 and compatible with classification 121. For example, classification 121 includes identification of a broadcast channel or anchor, and profiler 190 provides the user with the option to specify a particular channel or anchor for inclusion or exclusion by filter 160. In a preferred embodiment, the profiler 190 includes both "constant" preferences as well as "momentary" preferences, allowing the user to easily change the preferences according to the current state of the user's mind while maintaining the overall set of preferences. You can do that. In a temporary set, there may be a selection of subjects such as "sports" and "weather". In some sets, for example, there may be a list of anchors to exclude regardless of whether the anchor addresses the current subject of interest. Similarly, a certain set may include topics such as "baseball" or "stock market" that can be included regardless of instantaneous choices. Like general techniques used for search, profiler 190 allows for combinations of criteria using combinations, separations, and the like. For example, a user assigns continued interest to a "stock market" story that includes one or more words that match a specified list of company names.

Filter 160 identifies each story segment 111 having a classification 121 that matches the user preferences 191. The degree of matching, that is, the density of the filter, can be controlled by the user. Extremely, a user may request all story segments 111 that match any of the user's preferences 191, and as another extreme, the user may request all story segments 111 that match all of the user's preferences 191. May be required. The user may request all story segments 111 that match at least two of the three subject areas, and may also include at least one set of keywords or the like. The user may also have a negative preference 191, such as the subject or keyword that the user does not want, for example, "spot" but "Hockey." The filter 160 identifies each of the story segments 111 identifying the user's preferences 191 as filtered segments 161. In a preferred embodiment, each story is ranked according to the degree of matching between the classification 121 and the user preferences 191 using, for example, a count of the number of keywords of each subject in each classification 121 of the story segment 111. Include a classifier. To facilitate understanding, the rankings herein are proposed as unidirectional scalar quantities, although multidimensional or vector rankings are technically generalized. In the case of the same story being reported on multiple broadcast channels, ranking 162 may also be greatly weighted by the anchor desired by the user or by the broadcast channel desired by the user, and ranking 162 may also be the most recent story. It can be weighted by the time of each newscast at the preference of. In a preferred embodiment, the user may have a choice to adjust the weighting factor. For example, a user can create an absolute negative choice, and if a segment contains negative subjects or keywords, they are assigned the lowest rank, regardless of other matching preferences. Any number of general techniques can be used to implement the priorities, including the use of artificial intelligence techniques such as knowledge based systems, fuzzy logic systems, expert systems, learning systems. The filter 160 selects the story segment 111 based on the ranking 162 and assigns the ranking 162 of each of the selected or filtered segments 161 to the presenter 170 of the search system 150. to provide.

In another embodiment of the present invention, filter 160 also identifies occurrences of the same story in multiple story segments to identify popular stories, commonly referred to as "top stories." The identification is determined by the similarity of the classifications among the story segments 111 independent of the user's preferences 191. The similarity measure is based on the same subject classification applied to different story segments 111 according to the correlation between histograms of keywords. Based on the number of occurrences of a similar story, the filter 160 identifies the most popular current story among the story segments 111 that are independent of the user's preferences 191. In addition, filter 160 identifies the most popular stories that have at least some identity with preferences 191. From the most popular current story, the filter selects one or more story segments 111 for presentation by the presenter 170 based on the user's preferences 191 for the broadcast channel, anchor.

In accordance with the present invention, the presenter 170 presents the key frame 114 of the filtered story segment 161 on the display 175. As described above, the set of key frames associated with each story segment 161 provides a pictorial summary of each story segment 111. Thus, in accordance with the present invention, presenter 170 presents a pictorial summary 171 of said story segment 161 corresponding to user preference 191. In a preferred embodiment, the number of key frames displayed in each story segment 161 is determined by the prioritization structure described above based on the image content, dates, texts associated with it, and the like. Optionally, the representation of the picture summary may be accompanied by playing back the audio segment portion associated with the story segment 111. For example, the audio segment portion may be the first audio segment of each story segment corresponding to the introduction of the story segment by the anchor. In the same way, the summary of the text segment is also displayed simultaneously with the display of the picture summary 171. The pictorial summary 171 of the particular filtered story segment raises the user's attention, and the user selects the filtered story segment by the player 180 in the search system 150 for full playback. In a general technique, a user may make a selection by, for example, using a mouse to indicate a displayed key frame of a story of interest, or by voice command, gesture, keyboard input, or the like. Upon receipt of user selection 1876, player 180 displays selected story segment 181 on display 175.

3 illustrates an example user interface for search system 150. Display 175 has panes 310 for displaying filtered story segment key frame 171. As illustrated in FIG. 3, the display 175 includes four windows 310a, 310b, 310c, 310d but fewer or more windows may be selected via the presentation controller 350. The presenter presents each of the key frames 171 in windows 310 in succession. In a preferred embodiment, each of the key frames 171 corresponding to one story segment 161 is presented in succession in one of the windows 310a, 310b, 310c, 310d. That is, in FIG. 3, the key frames of the four story segments 161 are displayed simultaneously, each window providing a picture summary to each of the story segments 161. The user can select the key frames 171 from the story segment 161 for the duration of each key frame 171 and for a given period of time before the set of key frames 171 from the other story segment 161 is presented in the window. ) Has the choice of determining whether or not to repeat. After all key frames 114 of all filtered story segments 161 are presented, the cycle is repeated by providing a continuous slide show of key frames of story segments that match the user's preferences. Other display methods may be used. For example, four segments from story segment 161 may be displayed simultaneously in all four windows 310a-310d. Similarly, one window may be defined as the main window configured to contain the high priority scene of the story segment 161 while the other window continuously displays the low priority scene. The above and other techniques for video presentation are apparent to those skilled in the art. In a preferred embodiment, the representation controller 350 is provided for the selection of the key frame 171 and for the optimization of the representation.

Once filter 160 provides a ranking 162 associated with each filtered story segment 161, the presenter 170 ranks 162 to determine the frequency or duration of each presented set of key frames 171. Can be used. That is, for example, the presenter 170 may present a key frame 114 of the filtered segment 161 at a repetition rate proportional to the degree of agreement between the filtered segment 161 and the user preferences 191. Similarly, if a large number of filtered segments 161 are provided to the filter 160, the presenter 170 will display the key frame 114 of the segment 161 with a high degree of agreement with the user preferences 191 every cycle. While presenting, it is possible to present a key frame 114 of a segment having a lower match with user preference 191 less than every cycle.

The presentation controller 350 also allows the user to control the interaction between the presenter 170 and the player 180. In a preferred embodiment, the user can simultaneously view the selected story segment 181 in one window 310, while key frames 171 from the other story segment continue to be displayed in the other window. Alternatively, the selected story segment 181 may be displayed in the entire area of the display 175. These and other choices for visual display are apparent to those skilled in the art. The user is also provided with playback control functionality within 350 for existing playback functions such as volume control, repeat, fast forward, reverse progress. Since the story segment 111 is divided into scenes in the story segment identifier, the playback functions 350 may include options such as next scene, previous scene, and the like.

A user interface with profiler 190 is also provided via display 175. In the example interface of FIG. 3, button 320 is provided to allow a user to set preferences 191 in a selection category. "Media" button 320a provides user options regarding time settings such as how much time later in filter 160 considers the story segment. "Time" button 320b provides user options regarding time settings such as how far behind filter 160 should consider story segments. The "topic" button 320c allows the user to select from topics such as sports, art, finance, crime. "Place" button 320d allows the user to specify filter parameters that apply to the above-described identification of popular story segments. A "top stories" button 320e allows the user to specify filter parameters that apply to pre-identified identification of popular story segments. "Keyword" button 320f allows the user to specify a particular keyword of interest. Other categories and options may also be provided as would be apparent to those skilled in the art.

The user interface of FIG. 3 may also make selection of the presentation 330 and player 340 modes. The presenter 170 may be set to present key frames of story segments or key frames of "top" story segments selected by the user's preference setting. The player 180 may be set to operate in a browse mode corresponding to the above-described operation, where the user browses a key frame, selects a story segment of interest, or in the playback mode, the player 180 continuously Each filtered story segment 161 is presented, and in scan mode, the player 180 presents each filtered first scene in succession.

Other means for presenting key frames and associated materials are provided. The representation may be multidimensional, for example the correlation of the segment 111 to the user's interface 191 identifies the depth, and the key frame is multidimensional using the depth to determine how far the keyframe appears from the user. Presented in the visual view. Similarly, different categories 320 of user preferences may be associated with different sides of the view, with the key frame of each segment having a strong correlation with the user preferences within each category being displayed on each corresponding side. These and other presentation techniques are apparent to those skilled in the art in view of the present invention.

Although the present invention has been described primarily in the context of a news retrieval system, those skilled in the art will recognize that the principles presented herein are applicable to other retrieval tasks as well. For example, the principles of the present invention presented herein can be used for direct channel surfing. Generally, a channel surfing user searches for a program of interest by randomly or systematically sampling a number of broadcast channels until one of the broadcast programs suits the user's interest. By using the classification system 100 and the retrieval system 150 in the on-line mode, a more efficient search for the program of interest may be performed even with some processing delay. In the on-line mode, story segment identifier 110 provides text segment 113, audio segment 112, and key frame 114 corresponding to the current non-commercial portion of the broadcast channel. Classifier 120 classifies the portions using the techniques presented above. The filter 160 identifies portions that match the user's preferences 191, and the presenter 170 presents a set of key frames 171 from each of the filtered portions 161. When the user selects a particular set of key frames 171, the broadcast channel selector 105 is turned to the channel corresponding to the selected key frame 171, the story segment identifier 110, the storage device 115 and the player ( 180 is located in bypass mode to present the video stream 101 of the selected channel.

As will be apparent to those skilled in the art, the principles and techniques presented herein may comprise a variety of embodiments. 4 illustrates an example customer product 400 in accordance with the present invention. The product 400 may be a home computer or television and may be a video recording device such as a VCR, CD-R / W or DVR device. Example product 400 records a story segment 111 that is potentially of interest for presentation and selection by a user. Story segments 111 may be extracted or inserted from video stream 101 by classification system 100 as described above with reference to FIG. 1. Video stream 101 is selected from multichannel input 401, such as cable or antenna input, via selector 420 and tuner 410.

In one embodiment of FIG. 4, the selector 420 is a programmable multi-event channel selector as identified in conventional VCR devices. The user programs the selector 420 to tune the tuner 410 to the particular channel of interest at each particular event time for a specified period of time. For example, a user may program the time and duration of morning news on one channel, evening news on another channel, and last night's news on another channel. As each channel is substantially selected by the selector 420, the stories 111 are also segmented and stored on the recorder 430 through a classification system 100 that classifies each segment 111, as described above. Similarly, the relevant key frame 171 is extracted for display on the input / output device 440. In a preferred embodiment, the recorder 430 automatically removes the oldest segment 111 as it records each of the most recent segment 111 so that it is provided in succession to several recent segments 111 as the recording medium allows. It is a continuous survey recorder or a continuous circular buffer recorder. The user accesses the system via input / output device 440 and is presented in a key frame of the most recent segment 111 that matches the user's preferences. The user then selects segment 181 for display based on the presented key frame 171.

Many optional functions are also illustrated in FIG. 4. In order to optimize the use of available recording media, recording system 150 may be configured to provide selectable removal through 451 rather than the oldest recordable structure described above. If the new segment 111 requires allocation of the recording medium, then the retrieval system 150 identifies the segment 111 on the recording medium with the minimum correlation with the user's preference. Instead of replacing the oldest segment with the most recent segment, the least potentially interested segment to the user may be replaced with the most recent segment. The search system 150 also sends a record of the most recent segment based on the classification of the most recent segment by the classification system 100 if the most recent segment is determined to be uninteresting by the user based on user preferences. do.

As illustrated by dashed lines 191, 402, product 400 is provided for selection of a channel by selector 420 through line filter 425. The line filter 425 filters the segment 111 by controlling the selection of the channel 401 through the selector 420 and the tuner 410. As indicated above, additional text information is generally available to describe the program to be presented to each of the channel inputs 401 of the multi-channel 401. As illustrated by the dashed lines, the additional information, ie, the program guide, may be part of the multi-channel input unit 401 via the independent program guide connection 402. Using techniques similar to filter 160 described above, line filter 425 identifies a program in program guide 402 that has a strong correlation with user preference 191 and records, sorts, and searches as described above. Program a selector 420 to select the program for.

As will be apparent to those skilled in the art, the functions and parameters of the present invention may be adjusted according to the functions of each particular embodiment. For example, product 400 may be a palmtop viewing device for a customer who does not have time to watch a live newscast. The commuter connects the product to the source of the multi-channel input 401 the previous night to record the story 111 of potential interest. At this time, the commuter (to passengers) uses product 400 to retrieve the story 181 of interest from the recorded story 111. In this embodiment, resources are limited and the parameters of each component are adjusted accordingly. For example, the number of key frames 114 associated with each segment 111 can be substantially reduced, and the prefilter 425 or filter 160 can be substantially more selective. Similarly, the classifier 100 and search system 150 of FIG. 1 may be provided as standalone devices that dynamically adjust their parameters based on the components to which they are added. For example, the classification system 100 may be a very large and versatile system used to classify story segments to various users, with different models of retrieval system 150 each having different levels of complexity and cost. It is provided to the user to retrieve the selected story segment.

The foregoing merely illustrates the principles of the invention. Thus, those skilled in the art may implement the principles of the present invention and apply various devices within its spirit and scope even if not explicitly described or shown herein. For example, key frame 114 is represented as a single image in this specification even if the key frame is equivalent to a series of images such as a short video chip, and the representation of the key frame may be a representation of each of the video chips. The components of classification system 100 and retrieval system 150 may be implemented in hardware, software, or a combination thereof. Components can include tools and techniques common to classification and retrieval techniques, including expert systems, knowledge-based systems, and so on. Fuzzy logic, neural networks, various regression analysis, non-monotonic rationalization, semantic processing, and other technically general tools and techniques can be used to implement the functions and components presented herein. The processor 170 and the filter 160 have a high correlation with the user preference 191 having a key frame 114 of a randomly selected segment, regardless of their correlation with the preference 191. It may include a randomization element that augments the representation of the frame 114. The source of the video stream 101 may be digital or analog and the story segment 111 may be stored in digital or analog form regardless of the source of the video stream 101. Although the present invention has been presented in the context of television broadcasting, the techniques presented herein may also be used for the classification, retrieval and presentation of video information from sources such as public and private networks, including the Internet and the World Wide Web. For example, the relationship between the set of key frames 114 and the story segment 111 may be through embedded HTML commands, including a website address, and the search of the selected story segment 181 may be a selection of the corresponding website. It may be through.

As will be apparent to those skilled in the art, the separation of groups presented herein is presented for purposes of illustration only. For example, the broadcast channel selector 105 is a necessary part of the story segment identifier 110, or a classification and retrieval system is used to retrieve story segments from a single source video stream or previously recorded video stream 101. May be absent. Similarly, story segment identifier 110 may process multiple broadcast channels using parallel processors simultaneously. Filter 160 and profiler 190 may be integrated as a single selection device. The key frame 114 may be stored in or indexed from the recorder 115 and the presenter 170 functionally provided by the player 180. In the same way, extraction of key frames 114 from story segments 111 may be selected in story segment identifier 110 or presenter 170. The above and other partitioning and optimization techniques within the spirit and scope of the present invention are apparent to those skilled in the art.

Claims (10)

As video classification system 100, One or more key frames 114 for processing video stream 101, dividing the video stream 101 into a plurality of story segments 111, and associated with each story segment of the plurality of story segments 111. A story segment identifier (110) for generating a); And In order to facilitate selection among the plurality of story segments 111 based on one or more classifications 121, the one or more classifications 121 are assigned to each story segment of the plurality of story segments 111. And a classifier (120) operatively associated with the story segment identifier (110). The method of claim 1, The video stream 101 comprises an associated text stream 240, The story segment identifier 110 converts the text stream 240 into at least one text segment 241-244 corresponding to at least one story segment 221-224 of the plurality of story segments 111. Dividing, The classifier 120 associates the one or more classifications 121 with the at least one story segment 221-244 based on the at least one text segment 241-244. ). The method of claim 1, The video stream 101 comprises an associated audio stream 230, The story segment identifier 110 converts the audio stream 230 into at least one audio segment 231-234 corresponding to at least one story segment 221-224 of the plurality of story segments 111. Dividing, The classifier 120 associates the one or more classifications 121 with the at least one story segment 221-244 based on the at least one audio segment 231-234. ). The method of claim 3, wherein The classifier 120 includes a converter for converting the at least one audio segment 231-234 into at least one text segment 241-244, wherein the classifier 120 includes the at least one text segment 241. -244 to associate the one or more classifications (121) with the at least one respective story segment (221-224) based on. The method of claim 1, The story segment identifier (110) divides the video stream (101) based on at least one recognized photo, recognized scene, video cut, and detected commercial. The method of claim 1, The one or more key frames (114) are determined based on a variation in the encoding of each story segment of the plurality of story segments (111). The method of claim 1, And a storage device (115) for storing said plurality of story segments (111). A search system 150 for retrieving story segments of the plurality of story segments 111 based on one or more classifications 121 associated with each story segment of the plurality of story segments 111, A filter (160) for identifying one or more filtered story segments (161) of the plurality of story segments (111) based on the one or more classifications (121) associated with each story segment; And A presenter operatively coupled to the filter 160 that sequentially presents one or more key frames 114 associated with the one or more filtered story segments 161 on the display 175 ( Search system 150, including 170). As a video device, Classification to classify the plurality of segments 111 of the video stream 101 by generating the classification 121 based on at least one of text, audio, or visual information associated with each segment of the plurality of segments 111. Device 100; And Matching the classification 121 of at least one respective segment 181 of the plurality of segments 111 with at least one user preference 191 and at least one of the plurality of segments 111. A search that facilitates the selection of each segment 181 of at least one of the plurality of segments 111 by presenting on display 175 at least one key frame 171 of one segment 181. Video device comprising device 150. A user interface device for retrieving a selected segment 181 of a plurality of segments 111 of a video stream 101, Means (170) for rendering one or more key frames (114) associated with one or more segments (161) of the plurality of segments (111); And Means (178) for selecting the selected segment (181) based on the rendering of the one or more key frames (114).

Images