JP2007174260A - Device for producing digest information - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem that a starting point and a terminating point of each extracted high lighted portion makes a user uncomfortable because the starting point and the terminating point of each high lighted portion is partitioned by before and after, and the time of an important event. <P>SOLUTION: Image characteristics acquiring means 11 is designed to acquire characteristics of image content from an image before digest to be inputted from former image holding means 2. Shot change detecting means 12 is designed to detect all of shot changes in the image before digest, and to decide a shot change of a starting shot as a scene starting candidate from among detected shot changes. Scene starting point/terminating point deciding means 13 is designed to decide a starting point/terminating point of the high lighted scene including the event from among shot change points detected by the shot change detecting means 12 sequentially from a highly important event from important event information holding means 3 based on a condition acquired by the image characteristics acquiring means 11. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a digest information generation apparatus, and more particularly to a digest information generation apparatus that automatically extracts each highlight scene of video content as digest information.

  Currently, with the spread of hard disk recorders and the like, the amount of video content accumulated by viewers is increasing at an accelerated rate. However, even if the accumulation amount of video content increases, it does not mean that the viewer has increased time to view it. For this reason, as the amount of video content accumulated increases, it becomes increasingly difficult for viewers to view all content completely. Accordingly, the viewer becomes more selective with respect to the video content and selects the video content to view. However, video content not selected here will be deleted without being viewed. Therefore, a technique has been developed that provides digest information that is a summary of video content that shows the video content to a certain extent.

  For example, when a recording medium on which a sports television relay program is recorded is reproduced and viewed on a reproducing apparatus, there are many portions that are considered to be useless and worthless for the viewer. For example, taking baseball as an example, the player switches from hitting to defensive, the manager speaks to the pitcher, the referee cleans the home plate, the batter swings the bat, and the batter waits for the pitcher. It is a time when the audience in the outfield seats are reflected and the commentator is talking. The video content summary provides a game summary by omitting these parts.

  Similarly in other sports, a summary of the game can be provided by omitting video that is not necessary for the viewer. In addition to omitting the wasteful parts mentioned above, the focus is on particularly important events among the videos that the viewer is likely to be interested in (such as a big cheer for sports programs). By extracting only the highlight portion consisting of the front and back and collecting them, a digest video is created, so that the viewer can be provided with the same level of excitement as the original game with a shorter video.

  For example, as a method of automatically creating a digest video, there is a conventionally known method of automatically calculating the importance in the highlight portion of video content, extracting events according to this importance, and creating a summary video. (For example, refer to Patent Document 1).

Japanese Patent Laid-Open No. 2004-126811

  However, in the conventional digest information generation method described in Patent Document 1, although the highlight portion can be detected, the start point and the end point of each highlight portion are separated by time, such as from several seconds before the important event to several seconds later. The start point and end point of each extracted part are uncomfortable for the viewer.

  However, when a highlight is extracted from a sports program to create a summary, each highlight should be extracted as one scene determined by the rules of the sport. A “scene” here refers to a section that is contextually continuous in a series of images. For example, it is a series of movements until a score is given to either one from the serve in the case of tennis, until a result of a strike, a hit, a home run, etc. is known in the case of baseball. If the extracted video of each highlighted portion is shorter than this, the viewer cannot obtain sufficient information about the game by the video. On the other hand, if the extracted video is too long, the viewer will see a boring scene.

  The present invention has been made in view of the above points, and can generate digest information that can automatically extract each highlight portion of summary information as a single scene with a sense of incongruity in viewing according to the content. An object is to provide an apparatus.

  In order to achieve the above object, the present invention is a digest information generation device that generates digest information from a series of video data including time information, and includes video feature acquisition means for acquiring video content features from video data, From the video data, multiple shot changes, which are locations where the video is physically separated, are detected along with the time information at that time, and for each detected shot change, the shot changes are contextually continuous in the video data. A shot change detecting means for determining whether the scene is a start point of a scene that is a section to be processed, and storing the scene start point in association with a time when a shot change has occurred as a scene start candidate, and a feature obtained by the video feature acquisition means And the time and scene of the shot change obtained by the shot change detection means A scene that determines the start point and the end point of each highlight scene in the video data based on at least the information that associates the time when the shot change occurred and the scene start candidate among the information that associates the start candidate And digest information including at least a start point and an end point of each highlight scene determined by the scene start point / end point determination unit.

  In the present invention, the feature of the video data and the time at which the shot change occurred in the video data is associated with the shot change at the start point of the scene as the scene start candidate among the shot changes detected from the video data. Since the start point and the end point of each highlight scene in the video data are determined based on at least the information that associates the time when the shot change occurs and the scene start candidate, it is contextual in the video data. A series of scenes that are continuous sections can be generated as digest information.

  In order to achieve the above object, according to the present invention, the above-described shot change detection means performs color information on a predetermined image area of an image selected from the shot after the shot change in each detected shot change. Based on the above, it is determined whether or not it is a scene start candidate, and the determined scene start candidate is stored in association with the time when a shot change occurs. In the present invention, since the scene start candidate is determined based on the color information of a predetermined image area of the image selected from the shot after the shot change, the scene start candidate is automatically and accurately determined. Can be distinguished.

  In order to achieve the above object, the present invention further includes important event information holding means for pre-stored important event information including an event occurrence time in video data and its importance, and a scene start point The end point determination means includes information obtained by associating the characteristics obtained by the video feature acquisition means, the time when the shot change obtained by the shot change detection means occurs and the scene start candidate, and the important event information holding means. Based on the important event information and the important event information, the scene that includes the event in the important event information is based on the information that associates at least the time when the shot change occurred and the scene start candidate, and the important event information. A starting point and an ending point of the light scene are determined respectively.

  In the present invention, the scene including the event indicated by the important event information is set as the highlight scene, and the start point and the end point thereof are determined. Therefore, only the highly important scene in the video data can be generated as the digest information. .

  According to the present invention, at least the start point and the end point of each highlight scene in the video data are determined based on the information associating at least the time when the shot change occurs and the scene start candidate. Since a series of scenes that are contextually continuous sections are generated as digest information, each highlight scene can be extracted and viewed as a scene that does not feel uncomfortable at the start and end points.

  In addition, according to the present invention, a scene including an event indicated by important event information is set as a highlight scene, and a start point and an end point are determined, so that only a highly important scene in the video data is generated as digest information. As a result, it is possible to generate digest information that matches the viewing mode of the viewer, such as viewing only a highly important scene in a short time.

  Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a digest information generating apparatus according to the present invention. As shown in the figure, the digest information generating apparatus of the present embodiment includes a highlight scene detecting unit 1 that detects a highlight scene, an original video holding unit 2 that holds an original video (pre-summary video), It comprises event information holding means 3, highlight scene information holding means 4 for holding highlight scene information, reproduction control means 5 for generating and outputting summary video, and input instruction means 6.

  The original video holding means 2 holds in advance the original video (pre-summary video) of a program for which digest information is generated, together with time information indicating the playback time of the video, on a recording medium, a storage element, or the like. The highlight scene detection means 1 includes the pre-summary video held by the original video holding means 2 and the important event information held by the important event information holding means 3 (important event information is shown in FIG. Highlight scene containing each event is extracted from the occurrence time of each important event and its importance), and highlight scene information (digest information) consisting of the start and end times of the scene and the importance of the scene Is output. Here, a scene is a section that is contextually continuous in a series of videos, and is divided for each play when the program of the original video (pre-summary video) is a sports program.

  That is, the highlight scene detection unit 1 includes a video feature acquisition unit 11, a shot change detection unit 12, and a scene start point / end point determination unit 13. First, the video feature acquisition unit 11 starts from the original video holding unit 2. The feature of the video content is acquired from the input pre-summary video. The feature of the video content is information such as under what circumstances the sport is being played when the pre-summary video program is a sports program (for example, in the case of baseball, whether the infield is soil or lawn) For example, what kind of tennis courts are used for tennis?) The feature acquisition work by the video feature acquisition unit 11 includes a necessary program and an unnecessary program as will be described later, and the feature acquisition operation by the video feature acquisition unit 11 is skipped when unnecessary.

  Subsequently, the shot change detection means 12 detects all shot changes in the pre-summary video together with the time information at that time. A shot change refers to a break point when a video is physically separated due to camera switching or the like. Further, the shot change detection means 12 determines whether each detected shot change is a shot change of a start shot of a scene that is a section that is contextually continuous. Then, it is determined as a scene start candidate, and shot change information associating the shot change time with information on whether or not it is a scene start candidate as shown in FIG. 3 is generated and stored.

  In the scene start point / end point determination means 13, the shot change detection means 12 detects the start point / end point of the highlight scene in order from the most important event based on the conditions obtained by the video feature acquisition means 11. Decide from the points.

  Here, the operation of the scene start point / end point determination means 13 will be described in detail with reference to the flowchart of FIG. First, the scene start point / end point determination means 13 has the highest importance at the important point where the start point / end point of the highlight scene is not determined from the important event information shown in FIG. 2 from the important event information holding means 3. A location (event) is selected (step S11 in FIG. 4). If there are a plurality of events having the same importance, the event with the later occurrence time is selected with priority. In FIG. 2, since the importance level “12” has the highest importance level, the important event occurrence time 125374 ms is substituted for the variable i, and the start point of the scene including this important event is traced back from the image of the important event occurrence time. Search is performed (step S12 in FIG. 4).

  Subsequently, it is compared whether or not the time indicated by the variable i matches the start candidate time of the scene including the important event (step S13 in FIG. 4). If not, the variable i is subtracted by 1 ms (FIG. 4). 4 step S14), it is compared again whether the variable i matches the scene start candidate time (step S13 in FIG. 4). The scene start time candidate is determined based on whether the hue, brightness, and saturation of one representative image detected from each shot match the scene start shot for each sport.

  When the above operation is repeated and a scene start candidate time is found going back from the important event occurrence time, that time becomes the start time of the scene and is assigned to the variable i (steps S13 to S15 in FIG. 4). In the shot change information in FIG. 3, the highlight scene start time is the shot change time 119750 ms immediately before the important event start time 125374 ms.

  In a sumo program, a scene start shot may be detected during a long effort. In order for the highlight scene not to start from such an active scene start shot, if there is another scene start shot within 15 seconds before the start of the detected scene start shot, the detected scene start shot is You may perform the process of disregarding.

  If the scene start shot cannot be detected even after some significant event occurrence time, in order to prevent the scene from becoming too long, the scene start point is the shot start point at a point that is a predetermined time after the important event occurrence time. It is good. For example, if the start point of the scene start shot cannot be detected 90 seconds before the important event start time, the start point of the shot 90 seconds before is set as the scene start point. In the case of a baseball program, since there are not so many highlights before an important event, it is also preferable to change the time to go back depending on the genre of the program, such as going back only 30 seconds before the other genres.

  Subsequently, the end point of the highlight scene is determined. Here, after initializing the variable k to “0” (step S16 in FIG. 4), the most important important event occurrence time (125374 ms in the case of FIG. 2) is substituted into the variable j (FIG. 4). In step S17), the process proceeds by 1 ms until the time of the variable j coincides with the shot change time (steps S18 and S22 in FIG. 4). When the time indicated by the variable j coincides with the shot change time, the variable k is incremented by 1 (step S19 in FIG. 4), it is determined whether k is 3 (step S20 in FIG. 4), and k is less than 3 At this time, it is determined whether or not the variable j is a scene start time candidate (step S21 in FIG. 4).

  When the time indicated by the variable j is not a scene start time candidate, when the variable k indicating the number of times of coincidence with the shot change time becomes “3”, that is, when there is a third shot change, it is highlighted. The end point of the scene, that is, the scene end time is set (steps S18 to S23 in FIG. 4). In the shot change information shown in FIG. 3, the highlight scene end point is 132847 ms, which is the third shot change time in the direction in which the time advances from the important event start time 125374 ms.

  Even if the shot change point is not the third shot change, if the shot change point is a scene start candidate point, that point is set as the end point of the scene (steps S21 and S23 in FIG. 4). For example, the end point of the scene having the importance level “9” and the event occurrence time 29350 ms in FIG. 2 is not 94215 ms in which the third shot change occurs as shown in FIG. 3, but is 40769 ms, which is the previous scene start candidate point. .

  In addition to this method, there is also a method in which the end point of a shot after a certain time has elapsed from the occurrence time of an important event is used as the end point of the scene. For example, the end point of the shot 5 seconds after the important event start time may be set as the end point of the scene. In the case of a baseball program, since there are many highlights after an important event, it is also preferable to change the elapsed time depending on the genre of the program, such as making the end point of the shot after 10 seconds longer than other genres the end point of the scene. is there.

  In this way, the start point and end point of the scene including the most important important event are determined. Subsequently, it is determined whether or not the start points and end points of all highlight scenes (important points) have been extracted (step S24 in FIG. 4), and when the start points and end points of all highlight scenes have not been extracted, It is determined whether the total time of the summary video extracted so far and the number of scenes satisfy a preset condition (step S25 in FIG. 4). If it is determined in step S25 that the condition is not satisfied, the process returns to the first step S11, and the processes after step S11 are repeated again.

  If it is determined in step S24 that all the highlight scene start points and end points have been extracted, or if all the highlight scene start points and end points have not been extracted, the predetermined condition is set in step S25. If it is determined that the condition is satisfied, the process ends there.

  As described above, the scene start point / end point determination means 13 can extract all highlight scenes based on the important event information in order from the most important event, but the highlight scene to be extracted according to the viewer's preference. You can also determine the number and its total time. Accordingly, it is possible to provide a summary video that matches the viewing mode of the viewer, such as viewing only a relatively high importance scene in a short time. The viewer can input the number of highlight scenes and the total time by using the input instruction unit 6.

  Returning to FIG. 1 again, the highlight scene information holding means 4 includes highlight scene information including the start and end points of the highlight scene determined by the highlight scene detection means 1 and the importance of the scene. Records. Further, the playback control means 5 provides an instruction given by the viewer using the input instruction means 6, so that the original video held in the original video holding means 2 and the highlight scene information held in the highlight scene information holding means 4 are displayed. From the light scene information, digest information that is a summary video is generated and output.

  The digest information, which is a summary video that is automatically cut out and generated in this way, is a video of a highlight scene that matches the content, so that the viewer views it as a scene that does not feel uncomfortable. be able to.

  Next, Example 1 of the present invention will be described with reference to the drawings. This embodiment is an embodiment in which the highlight scene detecting means 1 is applied to a baseball program, and its operation will be described with reference to the flowchart of FIG. First, the highlight scene detection means 1 in FIG. 1 reads one frame of the baseball program image from the original video holding means 2 (step S31 in FIG. 5). When 300 ms or more have passed since the color information was detected last time, the video feature acquisition unit 11 detects the color information of the image. (Step S32 in FIG. 5).

  There are two main types of baseball fields. Uchino is for soil and turf. In these two, the characteristics of the image of the shot in which the scene where the pitcher as shown in FIG. 6 throws the ball is photographed from behind the pitcher by the center camera are greatly different. In the Uchino stadium, most of the lower half of the shots from the center camera are colored in soil (Fig. 6 (a)), whereas in the turf stadium, the colors of soil and turf are similar. Exists (FIG. 6B).

  The video feature acquisition means 11 detects which of the two types of the stadium where the game is played by the following method. Specifically, the upper 40% of the image is scanned, and it is detected whether or not turf-colored pixels are not included in the upper 40%. If it is not included, the image is determined to be an image taken from the center camera, and the characteristics of this image are acquired. At this time, the lower 25% of the image is scanned to detect whether 50% or more of the earth-colored pixels are present. Similarly, it is detected whether there are 20% or more of earth-colored pixels and grass-colored pixels. If 50% or more of the soil-colored pixels are present, it is determined that the image is taken by Uchino shooting a soil stadium. Also, if there are 20% or more of the earth-colored pixels and the turf-colored pixels, it is determined that Uchino has taken the turf stadium. (Step S33 in FIG. 5).

  Subsequently, the shot change detection means 12 in FIG. 1 detects all shot changes (step S34 in FIG. 5), and determines whether or not the detected shot change has occurred (step S35 in FIG. 5). There are various methods for detecting a shot change. Here, it is determined whether or not a shot change has occurred by comparison with an image 30 ms before. A histogram of the hue and brightness of the current image and the image 30 ms before is created and the correlation is obtained. Here, if either one of the correlations is equal to or less than the threshold value, it is set as a shot change candidate point.

  Subsequently, images are selected frame by frame before and after the shot change candidate points, and the correlation is obtained. If this correlation is less than or equal to the threshold value, it is determined that a shot change has occurred at this shot change candidate point, and it is determined that the shot change candidate point is a shot change point (scene start shot) (steps S35 and S36 in FIG. 5). ). On the other hand, if the correlation is greater than or equal to the threshold, it is considered that there is a temporary decrease in correlation due to camera movement or people passing in front of the camera, and there is no shot change at this shot change candidate point. (Step S35 in FIG. 5).

  In the case of the baseball program of this embodiment, when a shot change is detected in step S24, it is determined whether or not it is a scene start candidate point under the conditions of the soil and turf stadiums (step S35 in FIG. 5). In the case of baseball, a scene from when the pitcher throws the ball to when the batter hits the ball or fails to hit the ball and the result of strike, hit, etc. is determined as one scene. Therefore, the scene start shot is a shot in which the center camera shoots the scene where the pitcher throws the ball as shown in FIG. 6 from behind the pitcher. At each detected shot change point, an image is selected from the shot after the shot change, and using this representative image, it is determined whether or not this shot is a scene start shot.

  Depending on whether the lower 25% of the image contains 50% or more of the earth-colored pixels and whether the upper 40% of the image contains more than 30% of the turf-colored pixels, the image is the center camera of the soil stadium. To determine whether the image was taken. Similarly, the lower 25% of the image scans the upper 40% by checking if there are 20% or more of earth-colored pixels and turf-colored pixels, and includes 30% or more of the turf-colored pixels. Whether or not the image is an image taken by the center camera of the turf stadium is determined. If it is determined that the image is taken by the center camera, the shots including the images are assumed to be the scene start shots of the footage of the game on the ground and the turf stadium, respectively, and the shot change point to this shot is set to the ground. , The scene start candidate point of each video of the turf (step S36 in FIG. 5).

  Subsequently, it is checked whether or not detection has been completed for all images (step S37 in FIG. 5). If detection has not been completed for all images, the process returns to step S31 in FIG. 5 to read the next image. When the video ends, the process proceeds to the important scene start / end determination process by the scene start / end determination means 13 (step S38 in FIG. 5).

  The scene start point / end point determination means 13 determines the start point / end point of the highlight scene from the information obtained by the video feature acquisition means 11 and the shot change detection means 12 and the important event information obtained from the important event information holding means 3. To do. First, the image feature acquisition means 11 compares the number of frames of the image in which the infield is a stadium stadium and the image in which the infield is a turf stadium, and the more features of the stadium where this video game is played. And

  Thereby, it is selected which one of the scene start candidate point information when the infield obtained by the shot change detection means 12 is soil and the scene start candidate point information when the infield is turf is used. The start point and end point of each highlight scene are determined from the scene start candidate point information. The start and end points of the highlight scene are determined in order from the most important events having the highest importance and the later time.

  The starting point of the highlight scene is a scene start candidate point immediately before the important event. In order to improve the viewer's understanding of the highlight scene, the shot change point immediately before the scene start candidate point immediately before the important event may be set as the start point of the highlight scene. Further, when the scene start candidate point is not detected 30 seconds before the important event, the start point of the shot 30 seconds before may be set as the scene start point. The end point of the highlight scene is a scene start candidate point at the time when the third shot change has occurred from the important event or immediately after the important event (step S38 in FIG. 5). There is also a method in which the end point of the scene 10 seconds after the important event is set as the end point of the scene.

  The numerical values such as the ratios described above are merely examples, and detection may be performed using other numerical values. In addition to baseball, it is expected that a highlight scene of a sports program having a configuration similar to baseball, such as softball, can be extracted by the same method.

  Next, a second embodiment of the present invention will be described with reference to the drawings. This embodiment is an embodiment in which the highlight scene detection means 1 is applied to a tennis program, and the operation of this embodiment will be described with reference to the flowchart of FIG. First, the highlight scene detection means 1 reads one frame of an image (step S31 in FIG. 5). When 300 ms or more have passed since the color information was detected last time, the image feature acquisition unit 11 detects the color information of the image (step S32 in FIG. 5).

  As shown in FIGS. 7 (a) and 7 (b), tennis courts include turf, soil, and other various colors. Therefore, the video feature acquisition means 11 detects the color of the tennis court where the game is being played by the following method. First, the hue is divided into five types of hues ranging from yellow to green, cyan, blue, magenta, and red. In the image area excluding 20% each of the left and right sides of the image, 50% or more of the above colors are used. Determine if there is a color. Subsequently, if there is a color that is used by 50% or more, it is assumed that the image is an entire view of the tennis court of that color, and a feature of this image, that is, a color that is used by 50% or more is acquired.

  Subsequently, the shot change detection means 12 in FIG. 1 detects all shot changes (step S34 in FIG. 5), and determines whether or not the detected shot change has occurred (step S35 in FIG. 5). Although there are various methods for detecting a shot change, in the present embodiment as well as in the first embodiment, it is determined whether or not a shot change has occurred by comparison with an image 30 ms before. A histogram of the hue and brightness of the current image and the image 30 ms before is created and the correlation is obtained. Here, if either one of the correlations is equal to or less than the threshold value, it is set as a shot change candidate point.

  Subsequently, images are selected frame by frame before and after the shot change candidate points, and the correlation is obtained. If this correlation is less than or equal to the threshold value, it is determined that a shot change has occurred at this shot change candidate point, and the shot change candidate point is determined to be a shot change point (scene start shot) (steps S35 and S26 in FIG. 5). ). On the other hand, if the correlation is greater than or equal to the threshold, it is considered that there is a temporary decrease in correlation due to camera movement or people passing in front of the camera, and there is no shot change at this shot change candidate point. (Step S35 in FIG. 5).

  In the case of the tennis program of this embodiment, when a shot change is detected, it is determined whether or not it is a scene start candidate point on each color tennis court (step S35 in FIG. 5). In the case of tennis, one scene is from when the ball is served until the rally ends. Therefore, the scene start shot is assumed to be a shot in which the entire court view as shown in FIGS. At each detected shot change point, an image is selected from the shot after the shot change, and using this representative image, it is determined whether or not this shot is a scene start shot.

  Pixels in which the color of the area excluding the left and right 20% each of the representative image is 50% or more in the area is included in one of the hues in the range from yellow to green, hues divided into five of cyan, blue, magenta, and red If so, the representative image is assumed to be the entire view of the tennis court of that color, and this shot is assumed to be a scene start shot. A shot change point to this shot is set as a scene start candidate point in that color (step S36 in FIG. 5).

  Subsequently, it is checked whether or not detection has been completed for all images (step S37 in FIG. 5). If detection has not been completed for all images, the process returns to step S31 in FIG. 5 to read the next image. When the video ends, the process proceeds to the important scene start / end determination process by the scene start / end determination means 13 (step S38 in FIG. 5).

  The scene start point / end point determination means 13 in FIG. 1 uses the information obtained by the video feature acquisition means 11 and the shot change detection means 12 and the important event information obtained from the important event information holding means 3 to determine the start point / highlight of the highlight scene. Determine the end point. First, the color most frequently selected by the video feature acquisition unit 11 is set as the color of the tennis court where the tennis game of this video is being performed. As a result, it is selected which shot change point information for each tennis court color obtained by the shot change detection means 12 is used, and the start point and end point of each important event are determined from the selected shot change point information.

  The start and end points of the highlight scene are determined in order from the most important event with the highest importance and the later time. The starting point of the highlight scene is a scene start candidate point immediately before the important event. In order to improve the viewer's understanding of the highlight scene, the shot change point immediately before the scene start candidate point immediately before the important event may be set as the start point of the highlight scene. In addition, when the scene start candidate point is not detected 90 seconds before the important event, the start point of the shot 90 seconds before may be set as the scene start point. The end point of the highlight scene is a scene start candidate point at the time when the third shot change has occurred from the important event or immediately after the important event (step S38 in FIG. 5). There is also a method in which the end point of the scene after 5 seconds from the important event is set as the end point of the scene.

  The numerical values such as the ratios described above are merely examples, and detection may be performed using other numerical values. In addition to tennis, it can be expected that a highlight scene of a sports program having a structure similar to tennis, such as volleyball, will be extracted in the same manner.

  Next, a third embodiment of the present invention will be described with reference to the drawings. The present embodiment is an embodiment in which the highlight scene detection means 1 is applied to a soccer program, and the operation of this embodiment will be described with reference to the flowchart of FIG. First, the highlight scene detection means 1 reads one frame of an image (step S31 in FIG. 5). In the case of soccer, most of the matches are played on the grass field, and there is almost no difference between programs, so the video feature acquisition means 11 is not used.

  Subsequently, the shot change detection means 12 in FIG. 1 detects all shot changes (step S34 in FIG. 5), and determines whether or not the detected shot change has occurred (step S35 in FIG. 5). There are various methods for detecting a shot change. In this embodiment, as in the first and second embodiments, whether or not a shot change has occurred is determined by comparison with an image 30 ms before. A histogram of the hue and brightness of the current image and the image 30 ms before is created and the correlation is obtained. Here, if either one of the correlations is equal to or less than the threshold value, it is set as a shot change candidate point.

  Subsequently, images are selected frame by frame before and after the shot change candidate points, and the correlation is obtained. If this correlation is less than or equal to the threshold value, it is determined that a shot change has occurred at this shot change candidate point, and the shot change candidate point is determined to be a shot change point (scene start shot) (steps S35 and S26 in FIG. 5). ). On the other hand, if the correlation is greater than or equal to the threshold, it is considered that there is a temporary decrease in correlation due to camera movement or people passing in front of the camera, and there is no shot change at this shot change candidate point. (Step S35 in FIG. 5).

  In the case of soccer, the ball is turned on the field by passing, dribbling, etc., and reaching the shoot. In most cases, the scene of turning the ball in a pass or dribbling is a composition that shows a distant view of the field. Therefore, the scene start shot is a shot in which a field distant view as shown in FIG. 8 is shown. At each detected shot change point, an image is selected from the shot after the shot change, and using this representative image, it is determined whether or not this shot is a scene start shot. If 60% or more of the pixels of the representative image have a turf color, this shot is assumed to be a scene start shot, and a shot change point to this shot is set as a scene start candidate point (step S36 in FIG. 5).

  Subsequently, it is checked whether or not detection has been completed for all images (step S37 in FIG. 5). If detection has not been completed for all images, the process returns to step S31 in FIG. 5 to read the next image. When the video ends, the process proceeds to the important scene start / end determination process by the scene start / end determination means 13 (step S38 in FIG. 5).

  The scene start point / end point determination means 13 in FIG. 1 determines the start point / end point of the highlight scene from the information obtained by the shot change detection means 12 and the important event information obtained from the important event information holding means 3. The start and end points of the highlight scene are determined in order from the most important event with the highest importance and the later time. The starting point of the highlight scene is a scene start candidate point immediately before the important event.

  In order to improve the viewer's understanding of the highlight scene, the shot change point immediately before the scene start candidate point immediately before the important event may be set as the start point of the highlight scene. In addition, when the scene start candidate point is not detected 90 seconds before the important event, the start point of the shot 90 seconds before may be set as the scene start point. The end point of the highlight scene is a scene start candidate point at the time when the third shot change has occurred from the important event or immediately after the important event (step S38 in FIG. 5). There is also a method in which the end point of the scene after 5 seconds from the important event is set as the end point of the scene.

  The numerical values such as the ratios described above are merely examples, and detection may be performed using other numerical values. In addition to soccer, we can expect to extract highlight scenes of sports programs that have a similar structure to soccer, such as rugby and American football.

  Next, a fourth embodiment of the present invention will be described with reference to the drawings. This embodiment is an embodiment in which the highlight scene detection means 1 is applied to a sumo program, and the operation of this embodiment will be described with reference to the flowchart of FIG. First, the highlight scene detection means 1 reads one frame of an image (step S31 in FIG. 5). In the case of sumo, since all the efforts are performed on earthen soil and there is almost no difference between programs, the video feature acquisition means 11 is not used as in the third embodiment.

  Subsequently, the shot change detection means 12 in FIG. 1 detects all shot changes (step S34 in FIG. 5), and determines whether or not the detected shot change has occurred (step S35 in FIG. 5). Although there are various methods for detecting a shot change, in this embodiment as well, as in the first, second, and third embodiments, it is determined whether or not a shot change has occurred by comparison with an image 30 ms before. A histogram of the hue and brightness of the current image and the image 30 ms before is created and the correlation is obtained. Here, if either one of the correlations is equal to or less than the threshold value, it is set as a shot change candidate point.

  Subsequently, images are selected frame by frame before and after the shot change candidate points, and the correlation is obtained. If this correlation is less than or equal to the threshold value, it is determined that a shot change has occurred at this shot change candidate point, and the shot change candidate point is determined to be a shot change point (scene start shot) (steps S35 and S26 in FIG. 5). ). On the other hand, if the correlation is greater than or equal to the threshold, it is considered that there is a temporary decrease in correlation due to camera movement or people passing in front of the camera, and there is no shot change at this shot change candidate point. (Step S35 in FIG. 5).

  In the case of the sumo program of the present embodiment, one approach is regarded as one scene. This scene often starts with a shot that shows the entire ring. Therefore, the scene start shot of the sumo program is assumed to be a shot showing the entire ring as shown in FIG. At each detected shot change point, an image is selected from the shot after the shot change, and using this representative image, it is determined whether or not this shot is a scene start shot. If 60% or more of the pixels in the lower 30% of the representative image have a soil color, this shot is determined to be a scene start shot, and a shot change point to this shot is set as a scene start candidate point (FIG. 5). Step S36).

  The scene start point / end point determination means 13 determines the start point / end point of the highlight scene from the information obtained by the shot change detection means 12 and the important event information obtained from the important event information holding means 3. The start and end points of the highlight scene are determined in order from the most important event with the highest importance and the later time. The starting point of the highlight scene is a scene start candidate point immediately before the important event.

  In order to improve the viewer's understanding of the highlight scene, the shot change point immediately before the scene start candidate point immediately before the important event may be set as the start point of the highlight scene. In addition, when the scene start candidate point is not detected 90 seconds before the important event, the start point of the shot 90 seconds before may be set as the scene start point. The end point of the highlight scene is a scene start candidate point at the time when the third shot change has occurred from the important event or immediately after the important event (step S38 in FIG. 5). There is also a method in which the end point of the scene after 5 seconds from the important event is set as the end point of the scene.

  The numerical values such as the ratios described above are merely examples, and detection may be performed using other numerical values. In addition to sumo, it can be expected to extract the highlight scenes of sports programs that have a structure similar to sumo, such as boxing martial arts.

  The present invention is not limited to the above-described embodiments and examples. For example, the conditions for determining the start and end points of a scene are automatically changed using the input instruction means 6 according to the contents of input video data. It is also possible to adopt a configuration in which the input instruction means 6 can change the characteristics and parameters for determining the shot change.

  Further, the present invention includes a computer program for causing a computer to execute each means of the block diagram of FIG. 1 and each process of the flowchart of FIG. In this case, the computer program is recorded on a recording medium, may be taken into the computer via the recording medium, may be distributed via a network and downloaded to the computer, or may be stored in the computer. It may be incorporated in advance in a memory or the like.

It is a block diagram of one embodiment of the present invention. It is a figure which shows an example of the important event information hold | maintained by the important event information holding means in FIG. It is a figure which shows an example of the shot change data calculated | required by the shot change detection means in FIG. 2 is a flowchart for explaining detection processing by highlight scene detection means in FIG. 1. 2 is a flowchart for explaining processing by a scene start point / end point determination unit in FIG. 1; It is a figure which shows the characteristic acquisition image and scene start shot in Example 1 of this invention. It is a figure which shows the feature acquisition image and scene start shot in Example 2 of this invention. It is a figure which shows the scene start shot in Example 3 of this invention. It is a figure which shows the scene start shot in Example 4 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Highlight scene detection means 2 Original image holding means 3 Important event information holding means 4 Highlight scene information holding means 5 Playback control means 6 Input instruction means 11 Video feature acquisition means 12 Shot change detection means 13 Scene start point / end point determination means

Claims (3)

A digest information generating device that generates digest information from a series of video data including time information,
Video feature acquisition means for acquiring features of video content from the video data;
A plurality of shot changes, which are locations where the video is physically separated, are detected from the video data together with time information at that time, and for each detected shot change, the shot change is contextual in the video data. A shot change detection means for determining whether or not the scene start point is a continuous section, and holding the scene start point as a scene start candidate in association with the time when the shot change occurred,
Of the information obtained by associating the feature obtained by the video feature acquisition unit, the time when the shot change obtained by the shot change detection unit is associated with the scene start candidate, at least the time when the shot change occurs And scene start point / end point determination means for determining the start point and end point of each highlight scene in the video data based on the information associating the scene start candidate with the scene start candidate. A digest information generating apparatus that generates digest information including at least a start point and an end point of each highlight scene determined by the means. The shot change detection means includes
In each of the detected shot changes, it is determined whether or not the scene is a start candidate based on the color information of a predetermined image area of an image selected from the shot after the shot change, and the determined scene 2. The digest information generation apparatus according to claim 1, wherein a start candidate is held in association with a time when the shot change occurs. An important event information holding means for storing in advance important event information including the time when the event occurred in the video data and its importance;
The scene start point / end point determination means, the information obtained by associating the feature obtained by the video feature acquisition means, the time when the shot change obtained by the shot change detection means and the scene start candidate, Of the important event information from the important event information holding means, based on the important event information and at least the information that associates the time when the shot change occurs and the scene start candidate, The digest information generating apparatus according to claim 1 or 2, wherein a scene including the event is set as the highlight scene, and a start point and an end point of the highlight scene are respectively determined.