JP2006217046A - Video index image generator and generation program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video index image generator for generating and redisplaying the main scene of a video image effectively, and to provide a program for generating a video index image. <P>SOLUTION: A plurality of frames belonging to each segment of a video image segmented, based on the feature of a frame are stored in a database (1)16, and a frame satisfying predetermined conditions is extracted among the plurality of frames as a representative image for that segment at analysis in a segment/main frame-extracting section 24 in a processing unit 10. The representative image thus extracted is displayed on a display 12 and then a new representative image for each segment is searched at a subjectivity/taste-inferring section 26, based on the taste of a user among the plurality of frames stored in the database (1)16. The representative image thus searched is displayed on the display 12. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an index image generating apparatus that generates an index image from video composed of a plurality of continuous frames, and a program that generates an index image.

  In recent years, digital information has been used in various situations. Such information is used in various fields not only for text information but also for enormous data such as images and videos. In the future, as the capacity of storage devices increases, the importance of technology for searching and displaying data satisfying user requirements from a large number of images and videos will inevitably increase.

  By the way, when searching for an image, many similar image search systems have been proposed that search for images having similar features by calculating various features from the images and comparing these features regardless of keywords. . This technology is not limited to still images, but is also applied to digest image generation technology by searching for similar scenes from moving images and extracting feature images.

  When generating a personal digest video, refer to the event information to extract the semantic section of the video, perform the semantic feature calculation of that section, select a video scene with a large feature, and generate the digest Techniques for performing are known. Furthermore, at the time of calculating the feature value, the preference information of the category associated with each attribute is referred to at the same time, and the preference value of the corresponding preference information is used to increase or decrease the feature value to reflect the user's preference. The generated digest is generated (see, for example, Patent Document 1).

In addition, when multiple viewers individually play and watch the same program, a cluster is analyzed based on the scenes in which the viewers have sequentially changed the playback order and the operation history of the timing of those operations. When an analysis result is input, a representative point of the cluster based on the operation history on the time axis in the cluster is generated as an index image, or when a cut point based on the cluster analysis result and video features is input A technique for generating an index by determining a difference from a representative point based on a threshold is known (see, for example, Patent Document 2).
JP 2002-232828 A JP 2004-157786 A

  However, in the technique described in Patent Document 1, it is necessary to specify a category and an item name when inputting preference information, and a user sets a correspondence between a displayed image and text information such as a category. There is a need. For this reason, annoyance or inconvenience that the user wants to set a category does not exist. In the technique of Patent Document 1, an image having a large feature is extracted as a digest scene. However, an image having a large feature is not always appropriate as a digest, and a change in the feature of successive images is digested. It must be understood as a feature for extraction.

  In the technique described in Patent Document 2, since an average operation history of a plurality of viewers is used, the generated index is also an average index of an unspecified number of viewers. Therefore, index generation that considers trials for each individual is not performed.

  Therefore, the present invention has been made in view of the above problems, and an index image generation apparatus and a program for generating an index image that can accurately and easily generate an index image suitable for the user's preference are provided. The purpose is to provide.

  That is, according to the first aspect of the present invention, in an index image generating apparatus for generating an index image from a video composed of a plurality of continuous frames, each video segment divided into video segments based on the characteristics of the frame is provided. Image storage means for storing a plurality of frames belonging thereto, extraction means for extracting a frame satisfying a predetermined condition from the plurality of frames stored in the image storage means as an index image for the video segment, and the index Display means for displaying an image; and search means for searching based on a user's preference to extract a new index image for the video segment from the plurality of frames stored in the image storage means. Then, the display means displays a new input searched by the search means. And displaying the box image.

  According to a second aspect of the present invention, in the first aspect of the invention, the search unit accepts a selection by the user from the index images displayed on the display unit, and the selected index image On the basis of the feature, a new index image related to the index image is searched for extraction from the video segment.

  According to a third aspect of the present invention, in the invention according to the second aspect, the search means accepts a selection made by the user from the index images displayed on the display means, and selects a video segment to be searched. A search is performed so as to extract a new index image for the search target video segment from the search target video segment based on the characteristics of the index image to which the index image belongs and the characteristics of the selected index image. And

  The invention according to claim 4 is the invention according to claim 2, wherein the search means accepts selection of a plurality of index images from the user for the index image displayed on the display means, and the selection is performed. A first search condition is obtained based on the characteristics of the plurality of index images, and a video segment to which the part of the index images belongs is determined based on the characteristics of the part of the selected index images. A plurality of selected images are selected from among the plurality of selected images, a second search condition is obtained from the characteristics of the plurality of selected images, and a new search is performed from the video segment based on the first search condition and the second search condition. Searching is performed to extract an index image.

  The invention according to claim 5 is the invention according to any one of claims 2 to 4, wherein the feature of the index image is represented by a plurality of types of feature elements, and the search means By comparing the feature element of the feature with the feature element of the feature of the selected index image, setting the feature importance indicating the importance for each type of the feature element, and based on the feature importance, The search is performed so as to extract a new index image.

  The invention according to claim 6 is the invention according to any one of claims 2 to 4, wherein the feature of the index image is represented by a plurality of types of feature elements, and the search means A feature importance level indicating importance is set by the user, and a search is performed so as to extract the new index image based on the feature importance level set by the user.

  According to a seventh aspect of the present invention, in the invention according to any one of the second to fourth aspects, the setting means for setting the image importance of the index image for the selected index image. Further, the search means is characterized in that a search is performed so as to extract a new index image related to the index image based on the feature and the image importance.

  The invention described in claim 8 is the invention described in claim 7, wherein the setting means accepts the setting of the image importance by the user.

  The invention according to claim 9 is the invention according to claim 1, further comprising browsing recording means for storing a browsing status of the frame by a user, wherein the search means is displayed on the display means. Performing a selection so as to extract a new index image related to the index image based on the characteristics of the index image selected based on the browsing status from the index image. Features.

  The invention according to claim 10 is the invention according to claim 1, further comprising browsing recording means for storing a browsing status of the frame by a user, wherein the search means is displayed on the display means. Based on the characteristics of the index image belonging to the video segment to be searched and the characteristics of the index image selected based on the browsing status, the selection target video is selected from the index images based on the browsing status. The search is performed such that a new index image for the video segment to be searched is extracted from the segments.

  The invention according to claim 11 is the invention according to claim 2, further comprising browsing recording means for storing a browsing status of the frame by a user, wherein the search means is displayed on the display means. A plurality of index images are selected from the index images based on the browsing status, a first search condition is obtained based on the characteristics of the index images selected based on the browsing status, and the selected plurality of index images are selected. Based on the characteristics of a part of the index images, a plurality of selected images are selected from the video segments to which the part of the index images belong, and a second search is performed from the characteristics of the plurality of selected images. A search is performed so as to extract a new index image from the video segment based on the first search condition and the second search condition. It is characterized in.

  According to a twelfth aspect of the present invention, in the invention according to any one of the ninth to eleventh aspects, the image storage means includes an image other than the frame. It also includes browsing status of images other than frames.

  A thirteenth aspect of the invention is the invention according to any one of the ninth to twelfth aspects, wherein the features of the index image are represented by a plurality of types of feature elements, and the search means is adapted to the browsing situation. Based on the feature importance, the feature importance indicating the importance of the feature element is set, and a search is performed to extract the new index image based on the feature importance.

  A fourteenth aspect of the invention is the invention according to any one of the ninth to thirteenth aspects, further comprising registration means for registering a user and recording the browsing status for each user, and the search means. Is characterized by performing a search based on the browsing status of each user.

  A fifteenth aspect of the present invention is the information processing apparatus according to the first aspect, further comprising: a search condition storage unit that stores a plurality of registered search conditions based on user preferences. A user's selection is received from the registration search condition, and a new index image is searched from the video segment based on the selected registration search condition.

  A sixteenth aspect of the present invention is the information processing apparatus according to the first aspect, further comprising search condition storage means for storing a plurality of registered search conditions based on user preferences, wherein the search means includes the display means. The index image displayed on the screen receives a selection from the user, obtains a first search condition based on the characteristics of the selected index image, receives a user's selection from the plurality of registered search conditions, Searching to extract a new index image from the video segment based on the first search condition and the second search condition, with the selected registered search condition as a second search condition To do.

  According to a seventeenth aspect of the present invention, in the invention according to any one of the first to sixteenth aspects, the image storage means stores an image other than the frame, and the search means stores the image. Searching is performed so as to extract a new index image from the frame and the image other than the frame stored in the storage unit.

  The invention according to claim 18 is a program for generating an index image from a video composed of a plurality of continuous frames, and a plurality of frames belonging to each video segment of a video divided into video segments based on the characteristics of the frame. A second procedure for extracting a frame satisfying a predetermined condition from the plurality of frames stored by the first procedure as an index image for the video segment, and the index A third procedure for displaying an image, and a fourth search for extracting a new index image for the video segment from the plurality of frames stored by the first procedure. A fifth procedure for displaying a procedure and a new index image retrieved by the fourth procedure. Characterized by comprising the, the.

  ADVANTAGE OF THE INVENTION According to this invention, the program which produces | generates the image index production | generation apparatus and index image which can produce | generate the index image suitable for a user's liking accurately and easily for a user can be provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of the entire system of the image information display device 1 according to the first embodiment of the present invention. The image information display device 1 corresponds to an index image generation device of the present invention.

  In FIG. 1, this image information display device 1 includes an arithmetic processing device 10, a display device 12, an input device 13, an instruction device 14, a database 1 (hereinafter referred to as database (1)) 16, and a database. 2 (hereinafter referred to as database (2)) 17.

  The display device 12 is for displaying data processing results acquired from the database (1) 16 and the database (2) 17. The input device 13 is for performing an input operation of the image information display device, and is composed of a device such as a keyboard. Similarly, the instruction device 14 is used for performing an instruction operation on the image information display device, and includes a pointing device such as a mouse.

  In the database (1), a plurality of videos / images / features are stored to be displayed on the display device 12. Similarly, the database (2) 17 stores user's subjectivity / preference / history and the like to be displayed on the display device 12.

  The arithmetic processing unit 10 includes a video capturing unit 21, an interframe comparison / analysis / index generation unit 22, a video segmentation unit 23, an intra-segment analysis / main frame extraction unit 24, a display image designation unit 25, A preference estimation unit 26 and a re-search unit 27 are included. The arithmetic processing device 10 is realized by a CPU or the like, and performs an index generation process for generating an index image by executing a program stored in a storage unit (not shown). FIG. 2 shows a flow of index generation processing executed by the arithmetic processing device 10.

  The video capturing unit 21 reads video data from the video / image database 11 (step S1) and develops it into successive frames. It should be noted that the frames developed here are not limited to continuous ones, and may be, for example, every other frame as long as they are related in time series.

  The inter-frame comparison / analysis / index generation unit 22 calculates the feature of the frame extracted by the video reading unit 21 and the change of the feature between the frames, and calculates the feature of the frame to extract the index image. The statistical quantity and the statistical quantity of feature change are calculated (step S2). Further, the inter-frame comparison / analysis / index generation unit 22 selects a frame that matches a predetermined condition from the video based on the characteristics of each frame in the video, and indicates a scene change frame indicating the position of the change in the video scene. (Step S3).

  Note that the features calculated by the inter-frame comparison / analysis unit 22 and the statistic of the feature change can be mapped on a memory (not shown) of the arithmetic processing unit 10, but the capacity is limited. Since it may be enormous, it can be stored in a hard disk (not shown).

  Here, the characteristics of the frame are a color histogram, a resolution-converted image (for example, an image reduced to a resolution of 16 × 16), an edge distribution histogram (after performing edge calculation by image processing, the pixel value is a histogram) ) And histograms with respect to the direction of the edge (calculated as a histogram of the direction having the maximum number of pixels for each pixel). it can. Furthermore, regarding color histograms, edge distribution histograms, and edge direction histograms, it is also possible to specify an area for calculating a histogram within a frame and use the calculated histogram for the specified area as a feature element of the frame. It is done. Furthermore, the feature of each frame can be expressed by various feature elements obtained from the image data of the frame as the feature of the frame, such as a covariance matrix or a moment as the texture feature.

  Since the feature of a frame is represented by a plurality of feature elements in this way, each feature can be expressed as a vector, and a change in feature between frames can be defined as a vector difference between a plurality of frames. .

  Scene change frames can be extracted in various ways. For example, it is possible to compare the features between two frames preceding and following in time series, and extract a frame at the time when the features change greatly as a scene change frame. In addition, for each feature element of multiple frames in the vicinity of a certain target frame, statistics such as the average and standard deviation of each feature element, or statistics such as the average and standard deviation of each feature element difference are displayed. Based on this statistic, it can be determined whether or not the feature has changed significantly in the target frame, and the frame at the time when the feature has changed greatly can be extracted as a scene change frame. Further, a plurality of frames in the vicinity of the target frame can be extracted as a scene change frame as a plurality of frames at intervals of several frames instead of a plurality of consecutive frames.

  The video segmentation unit 23 performs video segmentation by dividing the video at the position of the scene change frame after the inter-frame comparison / analysis / index generation unit 22 extracts the scene change frame. That is, the video segmentation unit 23 generates a video segment obtained by dividing the video at the position of the scene change frame (step S4). Here, the video segmentation unit 23 considers the interval between the scene change frames and adds a process of integrating the generated video segments in step S4 when the time interval between the scene change frames is too narrow. You may do it.

  Next, the intra-video segment analysis / main frame extraction unit 24 uses the various features and statistics calculated by the above-described inter-frame comparison / analysis unit 22 to use successive frames in the video segment generated by the video segmentation unit 23. The main frame is extracted from each video segment according to the extraction condition of the main frame (step S5). The main frame corresponds to the index image of the present invention.

  As shown in FIG. 3, the arithmetic processing unit 10 includes a scene change frame 31 extracted by the inter-frame comparison / analysis / index generation unit 22 and a main segment extracted from each video segment by the intra-segment analysis / main frame extraction unit 24. The frame 32 is displayed on the display device 12 (step S6). The displayed scene change frame and main frame can be associated with each video segment, and in particular, the relationship with the video segment may be displayed on the array without setting. Note that conditions such as the number of main frames to be displayed on the display device 12 can be specified by the instruction device 14 or the like.

  The display image designating unit 25 selects at least one image selected by the user based on subjectivity or preference from the images such as the main frame extracted by the intra-segment analysis / main frame extracting unit 24 as the designated image (step S7). ).

  The subjectivity / preference estimation unit 26 sets search conditions for re-searching the main frame by estimating the subjectivity and preference for each user from the image specified by the display image specifying unit 25 (step S8). The conditions set here are registered in the database (2) 17 handling the user's subjectivity / preference / history for each user, and can be arbitrarily referred to when the corresponding user uses it. It can be done.

  The re-search unit 27 performs the main frame re-search according to the conditions set by the subjectivity / preference estimation unit 26, and displays the new main frame extracted by the re-search on the display device 12 (step S9).

  Next, with reference to the flowchart of FIG. 4, the process which the arithmetic processing unit 10 performs about the above-mentioned step S4-step S9 is further demonstrated.

  After extracting the scene change frame in step S3, first, in step S1, a video segment is generated. Step S11 is a process corresponding to the process of step S4, and the video segmentation unit 23 generates a video segment by dividing the video at the position of the scene change frame.

  In the subsequent step S12, main frames are extracted from each video segment. Step S12 is a process corresponding to the process of step S5, in which the intra-segment analysis / main frame extraction unit 24 extracts the main frame in accordance with the extraction conditions of the main frame. In the process of step S12, various methods can be considered as a method of detecting the main frame. In this embodiment, frames are extracted from each video segment under conditions different from the conditions for extracting scene change frames.

  For example, the characteristics of a plurality of frames in the video segment generated by the video segmentation unit 24 are analyzed using the various features and statistics calculated by the inter-frame comparison / analysis unit 22, and the main frame is extracted from the analysis result. Set conditions.

  The setting of main frame extraction conditions by the intra-segment analysis / main frame extraction unit 24 is performed as follows, for example.

  First, frames in each video segment are sequentially set as target frames, and feature quantities between the frames are calculated for a plurality of types of feature elements in a plurality of frames near the target frame.

  Here, the feature amount indicates a change state of the feature of each frame in a plurality of frames. For example, the amount of change of a plurality of types of feature elements between frames obtained from the plurality of frames. , Or feature statistic between frames obtained from multiple frames, or feature change statistics between frames obtained from multiple frames, or feature statistic between frames obtained from multiple frames The amount of change in the quantity. The statistic is obtained by an average, standard deviation, variance or the like.

  Then, using the relationship between the calculated feature value and the threshold value set for extracting the main frame as the extraction condition of the main frame, the feature value and the threshold value are compared, and a frame at a position where the feature value is smaller than the threshold value is determined. Detect and extract as main frame. For example, a frame with a small change in the feature between frames is extracted as a main frame, or the standard deviation of multiple types of features obtained from multiple frames in the vicinity of the target frame, or the feature quantity obtained from variance is a threshold If it is smaller than the target frame, the target frame is extracted as a main frame.

  Alternatively, the in-segment analysis unit 25 determines the main subject state in each frame, and the main frame extraction unit 26 uses the main frame extraction condition as a main frame extraction condition to determine whether or not the main subject state is in a predetermined state. A frame may be extracted. That is, the frame that the main subject is determined to be in the center, the frame that is determined that the size of the main subject is a predetermined size, the frame that is determined to be in focus with the main subject, and the main subject that are in front Intra-segment analysis of various states, such as a frame that is determined to be facing the scene, a frame that is determined to be a scene with a small movement of the main subject, and a frame that is determined to be a scene that contains a characteristic object as the main subject The main frame extraction unit 26 can extract the main frame as determined by the unit 25.

  As described above, the scene change frame is often a frame at a position where the frame characteristics greatly change between frames. A frame in which the feature amount indicating the degree is larger than a predetermined threshold is extracted as a scene change frame. On the other hand, as a main frame, a frame at a position where a change in the frame feature between frames is relatively small is often appropriate. For this reason, in this embodiment, when extracting a main frame, unlike the case of extracting a scene change frame, a frame in which the feature amount indicating the degree of change in the feature between frames is smaller than a predetermined threshold. Are extracted as main frames.

Note that threshold values for extracting main frames are set for a plurality of types of feature elements. Further, the number of extracted main frames is not particularly limited. For example, in a video divided into m video segments, the number of main frames extracted for each video segment so that a main frame is extracted for each k _i for video segment i (i = 1 to m). Is set in advance. This setting may be performed by user selection.

  In step S13, the main frame extracted in step S12 is displayed. Step S13 is a process corresponding to the process of step S6, and is a process in which the arithmetic processing unit 10 displays the main frame 32 as shown in FIG. Here, the newly extracted frames for each video segment are displayed after being associated with the video segment. That is, since segmentation is performed in step S11 and at least one or more main frames are searched from each video segment in step S12, a matrix-like image sequence as shown in FIG. 3 can be generated.

  Next, in step S14, at least one or more images selected by the user according to subjectivity or preference from the main frames are selected as designated images. Step S14 is a process corresponding to the process of step S7, in which the display image designating unit 25 selects at least one or more images selected by the user according to subjectivity or preference from the images such as the main frames as the designated image. is there. That is, the user who has viewed the main frame displayed in step S13 selects one image from the displayed main frames according to the user's subjectivity and preference. The image selected by the user is stored as a designated image, and the display image designation unit 25 selects this designated image.

  In steps S15 to S18, a search condition for re-searching the main frame is set, and a new main frame is extracted and the extracted new main frame is updated. Steps S15 to S18 are processes corresponding to the processes of steps S8 to S9.

First, in step S15, an integrated search condition is set from the designated image and the main frame extracted from the target video segment. That is, as shown in FIG. 5, the main frame frame () extracted from the designated image (1) to the designated image (p) and the video segment (i) that is the target when the iterative processing is performed on the video segment. For the frame group T composed of 1) to frame (m _i ), the integrated search condition Q _pref (i) is set based on the characteristics of these frames.

The integrated search condition Q _pref (i) is a condition for extracting a frame similar to the frame of the frame group T from the video segment (i). In the present embodiment, in the integrated search condition Q _pref (i), a weighting factor for each feature element and a weighting factor for each designated image are considered. The weighting factor for the feature element corresponds to the feature importance of the present invention. The weight coefficient for the designated image corresponds to the image importance of the present invention.

  When considering the weight coefficient for each feature element, if the value of the feature element is close between the frames constituting the frame group T, the feature element is an important feature element, and the feature element has a high weight. Set the coefficient. On the other hand, when the difference in the value of the feature element between the frames constituting the frame group T is large, the feature element is not important, so a low weighting factor is set for the feature element.

  When considering the weight coefficient for the designated image, the weight coefficient is set for each frame constituting the frame group T based on the feature (or feature amount) of the frame. Among the features of a frame, if the value of a feature (or feature amount) for a given feature element is higher than a given threshold, that frame is an important frame and a high weighting factor is set for that frame To do. On the other hand, if the value of the feature (or feature value) for a given feature element is lower than a given threshold among the features of the frame, the frame is not important, so a low weighting factor is specified for that frame. Set for images.

  The weighting factor for the feature element and the weighting factor for the designated image can be set by various methods. For example, the weighting factor can be set by the input device 13 or the pointing device 14 receiving the setting of the weighting factor by the user.

In addition, when setting a weighting factor for a feature element, the standard deviation of each feature element calculated for all frames in the video segment (i) is set as σ ₀ and extracted as a designated image and a main frame. There is a method of setting the ratio σ ₀ / σ _pref as a weighting factor when the standard deviation of each feature element calculated for the entire frame is σ _pref . By using the ratio to the standard deviation of all the frames in the video segment, it is possible to take into account the tendency of the entire frame group in the video segment. The standard deviation σ ₀ does not necessarily have to be calculated for all frames in the video segment (i), and may be calculated for some frames. The weighting factor may be set by another method that is statistically calculated from the characteristics of each frame.

When setting the weighting coefficient for the designated image, the standard deviation of each feature element calculated for all the frames in the frame group T is set as σ ₀ , and the value of each feature element in each designated image is set as σ _pref In this case, the ratio σ ₀ / σ _pref is set as a weighting factor. By using the ratio to the standard deviation of all the frames in the frame group T, it is possible to take into account the tendency of the entire frame group in the frame group T. Note that the standard deviation σ ₀ does not necessarily have to be calculated for all frames in the frame group T, and may be calculated for some frames. The weighting factor may be set by another method that is statistically calculated from the characteristics of each frame.

As described above, the integrated search condition Q _pref (i) of the present embodiment compares the feature amount obtained from the frame group T with the feature amount of each frame in the video segment (i), and determines from the frame group T. A condition for extracting a frame having a feature quantity that approximates the obtained feature quantity as a frame similar to the frame of the frame group T, and a condition for preferentially extracting a frame that approximates the feature quantity of a feature element having a high degree of importance It has become.

By using such an integrated search condition Q _pref (i), main frames can be extracted by effective search for each user in consideration of the main image in the video segment and the user's subjectivity / preference.

Next, in step S16, a search is performed within the video segment (i) that is the target under the generated integrated search condition Q _pref (i).

  In step S17, it is determined whether or not the iterative process for all video segments has been completed. Steps S15 to S16 are sequential repetition processes (i = 1 to m) for each of the video segments divided into m pieces. If the repetition is not yet completed in step S17, the process proceeds to step S15. Steps S15 to S16 are repeated.

  On the other hand, if the repetitive processing is completed in step S17, the process proceeds to step S18. In step S18, the main frame is updated with the new main frame re-searched from each video segment, and the new main frame is displayed.

  In this result display as well, the structure is the same as the form displayed first, and a matrix-like structure as shown in FIG. 3 is obtained.

  Here, a method of re-searching the main frame will be described. In the present embodiment, a vector having each feature element as a component is used as a feature vector, and the main frame is re-searched based on the definition of the following similarity calculation formula using this feature vector.

The feature quantity matrix F is represented by the following equation (1), assuming that there are _M feature elements f _{1 to} f _M and that there are _N frames I _{1 to} I _N in the video segment to be searched. Then, the feature quantity vector of the q-th object data I _q is expressed as the following equation (2). In the following equation (2), each feature term is normalized by multiplying by κ _j .

Further, when the following equation (3) holds and the feature vector is given by the above equation (2), the similarity of the object I _{p to} the object I _q can be expressed by the following equation (4), and Sim _pq is small. The similarity is higher.

By calculating this value for all images in the database, the order of similarity can be determined.

  As described above, according to the present embodiment, a new index image extracted by a search based on the user's subjectivity and preference is displayed for a plurality of index images displayed to represent the outline from the video. ing. Therefore, it is possible to generate and display an index image that suits the user's preference. At this time, since a new index image for the video segment is extracted, the user can grasp the outline of the video segment without viewing the entire content of the video.

  In step S14, at least one image selected by the user according to subjectivity or preference is selected as the designated image from the main frames displayed in step S13. Since the preference can be specified based on a visual sense, the burden on the user for generating a new index image is small, and the index image can be generated with a simple user operation. Further, in steps S15 to S16 described above, the integrated search condition is automatically set from the designated image and the main frame extracted from the target video segment to generate a new index image. Therefore, it is possible to generate an index image while reflecting the user's subjectivity and preference, and to generate and display a better index image for each user.

  In addition, a user assigns or automatically sets a weighting factor for a feature element that the user wants to focus attention on among the frame features, and generates a new index image in consideration of this weighting factor. Therefore, an index image appropriate for the user can be generated.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.

  In the second embodiment, the configuration and basic operation of the image information display apparatus are the same as the configuration and operation of the image information display apparatus of the first embodiment shown in FIGS. Since these configurations and operations are the same, the same parts are denoted by the same reference numerals, and the illustration and description thereof will be omitted, and only different configurations and operations will be described.

  FIG. 6 further explains the processing performed by the arithmetic processing device 10 for the above-described steps S4 to S9.

  In the flowchart of FIG. 6, the processes of steps S21 to S24 are the same as the processes of steps S11 to S14 of the flowchart of the first embodiment shown in FIG. Therefore, the processing operations of steps S21 to S24 refer to steps S11 to S14 in the flowchart of FIG.

  In steps S25 to S31, a search condition for re-searching the main frame is set, and a new main frame is extracted and the extracted new main frame is updated. Steps S25 to S31 are processes corresponding to the processes of steps S8 to S9.

In step S25, the preference integrated search condition _Qpref is set from p designated images designated by the user from the main frames generated from the video, as in the first embodiment described above. The preference integrated search condition Q _pref here is used to determine whether or not the frame has a feature amount approximate to the feature amount of the p designated images, and corresponds to the first search condition of the present invention. To do.

  On the other hand, along with the processing operation of step S25, iterative processing is performed on p designated images (i) selected by the user after step S26. That is, a search is performed to extract r frames related to the designated image (i) as selected images from the video segment to which the designated image (i) belongs. The r sheets may be set in advance in the arithmetic processing device 10, can be automatically set by the arithmetic processing device 10, or set by the user operating the input device 13 or the instruction device 14. You can also.

  The search for extracting a frame related to the designated image (i) from the video segment to which the designated image (i) belongs is, for example, a search for extracting a frame similar to the designated image (i). In this extraction, the feature (or feature amount) of the designated image (i) is compared with the feature (or feature amount) of each frame in the video segment to which the designated image (i) belongs, and both are approximated. The frame is extracted by extracting from the video segment to which the designated image (i) belongs. At this time, the concept of the weighting factor described in the above embodiment may be applied to preferentially extract a frame in which a feature (or feature amount) approximates a feature factor having a high weighting factor.

Next, in step S27, a _partial integrated search condition Q _partial (i) is set based on the features of the r frames extracted in step S26. The _partial integrated search condition Q _partial (i) here is used to determine whether or not the frame has a feature quantity approximate to the feature quantity of r frames, and the second search condition of the present invention. It corresponds to.

Next, in step S28, the integrated search condition Q _total (i) is set from the preference search condition Q _pref and the partial integrated search condition Q _partial (i). The integrated search condition Q _total (i) can be set by the logical sum or logical product of the preference search condition Q _pref and the partial integrated search condition Q _partial (i).

  It should be noted that the feature values of the r frames searched in step S26 and the feature values of the p designated images are integrated, and it is determined whether or not the frame has a feature value approximate to the integrated feature value. What is to be done can also be used as an integrated search condition.

Further, at step S29, by using the set integrated search condition Q _total (i), from the video segment belongs specified image (i), and extracts a frame that satisfies the integrated search condition Q _total (i). In the search in step S29, the set n _i frames are extracted. The number n _i may be set in advance in the arithmetic processing device 10, can be automatically set by the arithmetic processing device 10, or is set by the user operating the input device 13 or the instruction device 14. You can also.

  Then, in step S30, it is determined whether or not the iterative process for all designated images has been completed. Steps S26 to S29 are sequential repetitive processing (i = 1 to p) for p designated images. If the repetition is not yet completed in step S30, the process proceeds to step S26 and steps S26 to S26. The repetition process of step S29 is performed.

  On the other hand, if the repetitive processing is completed in step S30, the process proceeds to step S31. In step S31, the main frame designated by the new main frame re-searched from the video segment to which the designated image belongs is updated, and the new main frame is displayed as shown in FIG.

That is, in the second embodiment, as shown in FIG. 7, the preference search condition Q _pref is set from the designated image (1) to the designated image (p). In addition, the video to which the designated image (i) belongs based on the feature (or feature amount) of the designated image (i) that is a target when the repeated processing is performed on the designated image (1) to the designated image (p). Extract r frames related to the designated image (i) from the segment. Then, when r search result images corresponding to the set number are extracted, the partial integrated search condition Q _partial (i) is set from the r frames. Then, the integrated search condition Q _total (i) is set by an arbitrary logical method such as taking the logical sum or logical product of the search conditions from the preference search condition Q _pref and the partial integrated search condition Q _partial (i). and, using the integrated search condition Q _total (i), performs a search from video segment specified image (i) belongs, are extracted n _i frames that have been set.

There is also a method of setting the integrated search condition Q _total (i) by integrating the feature amounts of the designated image (1) to the designated image (p) and the feature amounts of r frames. In addition, a new main frame can be extracted by reflecting past search conditions. For example, a search condition for extracting a new main frame performed in the past is stored as history information in the database (2) 17, and the preference search condition _Qpref or the partial integrated search condition Q _partial (i) Instead, a search condition stored as history information can be used.

In the flowchart of FIG. 6, the processing operation of step S25 and the processing operations of steps S26 and S26 are performed simultaneously, but the present invention is not limited to this. That is, it is sufficient so integrated search condition _total (i) is set because the preference search condition Q _pref and partial integration search _partials (i) and at step S28, the preference search condition Q _pref and partial integrated search The condition _partial (i) may be obtained sequentially.

(Third embodiment)
Next, a third embodiment of the present invention will be described.

  In the third embodiment, instead of performing an operation in which the user instructs the main frame displayed in the initial stage according to subjectivity or preference, the history information of past user operations is referred to from the history database for each user. Thus, a preference search condition is generated using this history information.

  In the third embodiment, the configuration and basic operation of the image information display device are the same as the configuration and operation of the image information display device of the first embodiment shown in FIGS. Since these configurations and operations are the same, the same parts are denoted by the same reference numerals, and the illustration and description thereof will be omitted, and only different configurations and operations will be described.

  FIG. 8 further explains the processing performed by the arithmetic processing device 10 for the above-described steps S4 to S9.

  In the flowchart of FIG. 8, steps S41 to S43 and steps S45 to S48 are the same as steps S11 to S13 and steps S15 to S18 of the flowchart of the first embodiment shown in FIG. Is done. Accordingly, the processing operations of steps S41 to S43 and steps S45 to S48 refer to steps S11 to S13 and steps S15 to S18 in the flowchart of FIG.

  When the main frame is displayed in step S43, history information is read from the database (2) 17 in the subsequent step S44. Then, for the extracted main frame, as in step S15 in the flowchart of FIG. 1 described above, the integrated search condition is generated and the search in the target video segment is performed. The main image is re-searched.

  Here, by referring to the user history database (database (2) 17), for the images in the database (1) 16, a set number of p images are extracted and used in descending order of viewing frequency. become. The image in the database (1) 16 may be a video frame, a still image other than a video, or the like. The browsing history information of the images in the database (2) 17 includes, for each image browsed by the user, the number of times of browsing, the browsing date, printing of the browsed image, the history of image processing, etc. Preference information is included. Therefore, the preference for each user can be estimated from the history information, and a plurality of images for setting the integrated search condition are selected using the information.

  If there is a lot of usage history, the importance of the image is increased, or an image highly relevant to the shooting date and time of the image in the history is extracted, etc. Thus, it is possible to set a search condition by setting a weighting factor.

  Further, the database (2) 17 is a database for each user, and it can be considered that the user who uses the database has means for inputting user information. Further, a method of authenticating a user when the user logs in, or an apparatus having a configuration that automatically recognizes the user may be used. In addition, since there are many types of history information to be used, each user has a means for selecting whether or not to use these history information and a means for setting importance for each history information. It is possible to display an index by generating an index that reflects customized subjectivity and preferences.

  In addition, when specifying an image according to the user's preference or referring to the user's history, the user's subjectivity / preference is not limited to only the image in the target video, but other users' videos and images By referring to the history in the database, it is possible to further extract information reflecting the user's subjectivity and preference, and to perform appropriate index extraction and display.

  Furthermore, each time a user uses the system, different search conditions are set. However, if such information is also registered in the history database, the user can refer to past search condition information in the past. It is possible to reproduce the execution or to reproduce the work by partially changing the condition with reference to the past condition and performing the search. In this way, it is possible to provide a more convenient system for the user.

  According to this embodiment, in addition to the first embodiment described above, the following effects are provided.

  That is, according to the present embodiment, the user's preference is estimated based on the history information of the image that the user has previously referred to, and a new index image is generated based on the history information. It is possible to generate an index image suitable for each user's preference.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described.

  In the fourth embodiment, instead of performing an operation in which the user designates the index displayed in the initial stage according to subjectivity or preference, the history information of past user operations is referred to from the history database for each user. The preference search condition is generated using this history information.

  In the fourth embodiment, the configuration and basic operation of the image information display device are the same as those of the image information display device of the first embodiment shown in FIGS. Therefore, with respect to these configurations and operations, the same reference numerals are given to the same portions, and illustration and description thereof will be omitted, and only different configurations and operations will be described.

  FIG. 9 further explains the processing performed by the arithmetic processing unit 10 for the above-described steps S4 to S9.

  In the flowchart of FIG. 9, the processes of steps S51 to S54 are the same as the processes of steps S41 to S44 of the flowchart of the third embodiment shown in FIG. 8, and the processes of steps S55 to S61 are The same processing as steps S25 to S31 in the flowchart of the second embodiment described above shown in FIG. 6 is performed. Accordingly, the processing operations of steps S51 to S54 refer to steps S41 to S44 in the flowchart of FIG. 8, and the processes of steps S55 to S61 refer to steps S25 to S31 of the flowchart of FIG. .

  In other words, the fourth embodiment can be considered as a combination of the second embodiment and the third embodiment described above. Therefore, the index display in step S53 is in a matrix form as shown in FIG.

  Also, the difference between the fourth embodiment and the second embodiment is that the user's image history is used in the operation process of step S54, as in step S44 in the flowchart of the third embodiment. That is. Here, in step S57, the setting of the partial integrated search condition from the search result obtained by the search for each selected image and the integrated search condition with the preference search condition from the user-specified image are used. A search will be performed.

  In the first to fourth embodiments described above, the generation (extraction) of the main frame has been described with reference to an image including the main frame. However, the present invention is limited to the case where the image includes the main frame. Not necessary. For example, a main frame generated from another video registered in the database may be used. Furthermore, if all the data in the database registered without distinguishing video and still images or means for specifying the search range is set in the device, the data other than the target video It is also possible to generate a main frame from

  For these, the same concept can be applied to the first to fourth embodiments described above. That is, in step S16 of the flowchart of FIG. 4, step S29 of the flowchart of FIG. 6, step S46 of the flowchart of FIG. 8, and step S59 of the flowchart of FIG. It is also possible to provide a database with a wider range than the video to be used, or to provide the apparatus with means for allowing the user to set the range of the search database. By searching such a broad database, the user's subjectivity and preference can be expressed more effectively as a display result.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

1 is a block diagram showing a configuration of an entire system of an image information display device according to a first embodiment of the present invention. It is a flowchart which shows the flow of an index production | generation process. It is the figure which showed the example of the index displayed on the matrix form. It is a flowchart explaining the detection operation of the scene change of the image information display apparatus by the 1st Embodiment of this invention. It is a figure explaining the detail of the detection operation of the scene change by 1st Embodiment. It is a flowchart explaining the detection operation of the scene change of the image information display apparatus in the 2nd Embodiment of this invention. It is a figure explaining the detail of the detection operation of the scene change by 2nd Embodiment. It is a flowchart explaining the detection operation of the scene change of the image information display apparatus in the 3rd Embodiment of this invention. It is a flowchart explaining the detection operation of the scene change of the image information display apparatus in the 4th Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Arithmetic processing device, 12 ... Display device, 13 ... Input device (keyboard), 14 ... Instruction device (mouse), 16 ... Database 1 (video / image / feature), 17 ... Database 2 (user's subjectivity / preference)・ History), 21 ... Video capture unit, 22 ... Interframe comparison / analysis / index generation unit, 23 ... Video segmentation unit, 24 ... Intra-segment analysis / Main frame extraction unit, 25 ... Display image designation unit, 26 ... Subject / Preference estimation unit, 27 ... re-search unit, 31 ... scene change image, 32 ... main image.

Claims (18)

In an index image generating device for generating an index image from a video composed of a plurality of continuous frames,
Image storage means for storing a plurality of frames belonging to each video segment of the video divided into video segments based on the characteristics of the frame;
Extraction means for extracting a frame satisfying a predetermined condition from the plurality of frames stored in the image storage means as an index image for the video segment;
Display means for displaying the index image;
Search means for searching based on user preference to extract a new index image for the video segment from the plurality of frames stored in the image storage means;
Comprising
The video index image generating apparatus, wherein the display means displays a new index image searched by the search means.   The search means accepts selection by the user from the index images displayed on the display means, and based on the characteristics of the selected index image, a new index image related to the index image is extracted from the video segment. The video index image generation apparatus according to claim 1, wherein a search is performed so as to extract.   The search means accepts selection by the user from the index images displayed on the display means, and based on the characteristics of the index image belonging to the video segment to be searched and the characteristics of the selected index image, The video index image generation device according to claim 2, wherein a search is performed so as to extract a new index image for the video segment to be searched from the video segment to be searched.   The search means receives a selection of a plurality of index images from the user for the index image displayed on the display means, obtains a first search condition based on features of the selected index images, Based on the characteristics of some of the selected index images, a plurality of selected images are selected from the video segments to which the some index images belong. The search condition of 2 is calculated | required, It searches so that a new index image may be extracted from the said video segment based on the said 1st search condition and the said 2nd search condition. Video index image generator. The features of the index image are represented by multiple types of feature elements,
The search means compares the feature element of the feature of the index image with the feature element of the feature of the selected index image, sets the feature importance indicating the importance for each type of the feature element, 5. The video index image generation apparatus according to claim 2, wherein a search is performed so as to extract the new index image based on the feature importance. The features of the index image are represented by multiple types of feature elements,
The search means accepts a setting by the user of the feature importance indicating the importance of the feature element, and performs a search so as to extract the new index image based on the feature importance set by the user. The video index image generation device according to claim 2, wherein the video index image generation device is a video index image generation device. For the selected index image, further comprises setting means for setting the image importance of the index image,
5. The search unit according to claim 2, wherein the search unit performs a search so as to extract a new index image related to the index image based on the feature and the importance of the image. Video index image generator.   8. The video index image generation apparatus according to claim 7, wherein the setting unit accepts a user setting of the image importance level. Comprising browsing recording means for storing the browsing status of the frame by the user;
The search means performs selection based on the browsing situation from the index images displayed on the display means, and relates to the index image based on the characteristics of the index image selected based on the browsing situation. The video index image generation apparatus according to claim 1, wherein a search is performed so as to extract a new index image. Comprising browsing recording means for storing the browsing status of the frame by the user;
The search means performs selection based on the browsing situation from the index images displayed on the display means, and is selected based on the characteristics of the index image belonging to the video segment to be searched and the browsing situation. 2. The video index image generation according to claim 1, wherein a search is performed so as to extract a new index image for the video segment to be searched from among the video segments to be searched based on characteristics of the index image. apparatus. Comprising browsing recording means for storing the browsing status of the frame by the user;
The search means selects a plurality of index images based on the browsing situation from the index images displayed on the display means, and based on the characteristics of the plurality of index images selected based on the browsing situation. The first search condition is obtained, and based on the characteristics of some of the selected index images, a plurality of selected images are selected from video segments to which the some index images belong. Obtaining a second search condition from the characteristics of the plurality of selected images, and performing a search so as to extract a new index image from the video segment based on the first search condition and the second search condition. The video index image generation device according to claim 2, wherein:   The image storage means includes an image other than the frame, and the browsing status includes a browsing status of an image other than the frame by a user. Video index image generator. The features of the index image are represented by multiple types of feature elements,
The search means sets feature importance indicating the importance of the feature element based on the browsing status, and searches to extract the new index image based on the feature importance. The video index image generation device according to any one of claims 9 to 12. Along with registering users, the registration means for recording the browsing status for each user,
The video index image generation apparatus according to any one of claims 9 to 13, wherein the search means performs a search based on a browsing situation of each user. Comprising search condition storage means for storing a plurality of registered search conditions based on user preferences,
The said search means receives a user's selection from these registration search conditions, and searches a new index image from the said video segment based on the selected registration search conditions. Video index image generator. Comprising search condition storage means for storing a plurality of registered search conditions based on user preferences,
The search means accepts a selection from the user for the index image displayed on the display means, obtains a first search condition based on the characteristics of the selected index image, and from the plurality of registered search conditions Accepting the user's selection, using the selected registration search condition as the second search condition, and extracting a new index image from the video segment based on the first search condition and the second search condition The video index image generation device according to claim 1, wherein the video index image generation device is searched as described above.   The image storage means stores an image other than the frame, and the search means extracts a new index image from the frame and the image other than the frame stored in the image storage means. The video index image generation device according to claim 1, wherein the video index image generation device is searched. A program for generating an index image from a video composed of a plurality of continuous frames,
A first procedure for storing a plurality of frames belonging to each video segment of the video divided into video segments based on the characteristics of the frame;
A second procedure for extracting a frame satisfying a predetermined condition from the plurality of frames stored by the first procedure as an index image for the video segment;
A third procedure for displaying the index image;
A fourth procedure for searching based on a user's preference to extract a new index image for the video segment from the plurality of frames stored by the first procedure;
A fifth procedure for displaying a new index image retrieved by the fourth procedure;
A program for generating an index image, comprising:

Images