JP2018007134A - Scene extraction device and its program - Google Patents

Abstract

Kind Code: A1 A scene extracting device for extracting a highly realistic scene from content is provided. Kind Code: A1 A scene extracting apparatus 1 analyzes video and audio feature amounts of content for each time interval, and extracts a sense of presence from presence learning data in which the degree of presence for the feature amounts of signals constituting the content is learned in advance. Presence estimation means 10 for calculating an estimated value; Determination means 50 for determining whether the sense of presence is high for each time interval based on a preset threshold; and extracting means 60 for extracting from the content a scene corresponding to . [Selection drawing] Fig. 1

Description

  The present invention relates to a scene extraction apparatus that extracts a scene from video / audio content based on a subjective impression of a viewer, and a program thereof.

Conventionally, there is a method for extracting a certain subject such as a summary video, a digest video, and the like from video / audio contents.
As a conventional method, a method of extracting a specific scene from video / audio content using a motion vector that is a feature amount of video motion, a method of extracting a scene showing a specific person using a face recognition technology For example, there is a technique for extracting a scene based on a video feature amount (see, for example, Patent Documents 1 and 2).
Further, as a conventional method, there is a method of detecting an utterance from an acoustic feature of speech and extracting an utterance scene (see, for example, Patent Document 3).
As described above, a method for partially extracting a scene such as a summary video or a digest video from video / audio content is generally a method for objectively extracting a specific subject from a feature amount of video or audio. .

JP 2012-10265 A JP 2008-287594 A JP, 2014-33417, A

Since the conventional method extracts a specific subject only from objective features from video and audio content, there is a problem that a scene with high realistic sensation that is an abstract impression cannot be extracted.
Here, the sense of presence means a sense of being in the place. For example, in "Inoue, Evaluation of the sense of reality that people feel in ultra-realistic communication, The Institute of Electronics, Information and Communication Engineers, IEICE Technical Report, CQ 2008-47, pp. 7-12, 2008", It is composed of spatial elements, time elements such as simultaneous feelings, and body elements such as emotions, and is felt by humans under the influence of external factors such as vision and hearing, and internal factors such as past experiences.
In other words, a scene with a high sense of realism cannot be extracted with objective features of video and audio content, but needs to be extracted based on a person's subjective impression.

  The present invention has been made in view of such a problem, and provides a scene extraction apparatus and a program thereof capable of extracting a scene with high presence on the basis of a subjective impression from video and audio content. The task is to do.

  In order to solve the above-described problem, a scene extraction device according to the present invention is a scene extraction device that extracts a highly realistic scene from content based on a sense of presence defined by learning a predetermined feature amount of the content. The sensation learning data storage unit, the setting information storage unit, the presence sensation estimation unit, the presence sensation estimated value storage unit, the determination unit, the determination result storage unit, and the extraction unit are provided.

  In such a configuration, the scene extraction apparatus stores, in the presence learning data storage unit, the presence learning data in which the degree of presence with respect to the feature amount of a signal such as a video signal or an audio signal that constitutes the content is previously learned. . This presence learning data is data learned by, for example, neural network, machine learning, or the like. In addition, the scene extraction device stores in advance in the setting information storage means, as setting information, a time interval and a time width of the signal for analyzing the presence, and a threshold value as a criterion for determining whether or not the presence is high. Remember.

Then, the scene extraction device analyzes the feature amount of the content by the presence estimator, and calculates the degree of presence as a presence estimation value from the presence learning data. At this time, the realistic sensation estimation means calculates the realistic sensation estimated value of the content for each time interval having a predetermined time width that includes the overlapping interval and is shifted by a predetermined time interval, and associates the realistic sensation with the time interval. The value is stored in the realistic sensation estimated value storage means. As described above, by providing the overlapping section, the presence sensation estimation means can secure a sufficient time for calculating the presence sensation estimation value when estimating the presence sensation at a predetermined time interval.
As a result, the scene extraction apparatus can obtain a realistic feeling state that changes with the passage of time of the content that a person feels as a subjective impression as a time-series realistic feeling estimated value for each time interval.

Then, the scene extraction device determines whether or not the presence of the scene in the time interval associated with the estimated presence value is high based on a preset threshold by the determination unit, and determines the determination result. Store in the result storage means. Note that the threshold value may be the realistic sensation estimation value itself (absolute threshold) that serves as a criterion for determining the sensation of realism, or a relative value that indicates how much the scene is extracted from the one with the large realistic sensation estimation value. It may be a threshold value.
Thereby, the scene extraction apparatus can determine whether or not the presence is high in the time series of time sections.

Then, the scene extraction apparatus extracts, from the content, a scene corresponding to the time interval determined by the extraction unit as having a high sense of presence stored in the determination result storage unit. That is, the extraction unit extracts, from the content, a scene corresponding to the time from the beginning time of the first time interval to the end time of the last time interval, among the continuous time intervals determined to have high presence. .
As a result, the scene extraction device can extract a scene with high presence from the content based on the subjective impression of the viewer.

  Note that the scene extraction apparatus can operate a computer with a scene extraction program for causing a computer to function as each of the above-described means.

The present invention has the following excellent effects.
According to the present invention, it is possible to extract a highly realistic scene from content based on a subjective impression felt by a person with respect to the content instead of an objective feature of the content as in the past.
In addition, according to the present invention, since the realistic sensation estimated value is estimated from the signals constituting the content using the pre-learned realistic sensation learning data, a scene with high sensation of realism is automatically obtained from the content. Can be extracted.

It is a block block diagram which shows the structure of the scene extraction apparatus which concerns on 1st Embodiment of this invention. It is a block block diagram which shows the structure of the auditory presence sense estimation means of FIG. It is a block block diagram which shows the structure of the visual presence estimation means of FIG. It is a block block diagram which shows the structure of the sense presence realistic estimation means of FIG. It is explanatory drawing for demonstrating the time interval which estimates presence. FIG. 2 is a data structure diagram showing an example of the structure of data stored in a realistic sensation estimated value / determination result storage unit of FIG. It is a flowchart which shows the setting operation | movement by the setting apparatus to the scene extraction apparatus which concerns on 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the scene extraction apparatus which concerns on 1st Embodiment of this invention. It is a flowchart which shows the detail of realistic presence estimation operation | movement of FIG. It is a flowchart which shows the detail of the presence determination operation | movement of FIG. It is a block block diagram which shows the structure of the scene extraction apparatus which concerns on 2nd Embodiment of this invention. FIG. 12 is a data structure diagram illustrating a structure example of data stored in the realistic sensation estimated value / determination result storage unit of FIG. 11.

Embodiments of the present invention will be described below with reference to the drawings.
<< First Embodiment >>
[Configuration of scene extraction device]
First, the configuration of the scene extraction device 1 according to the first embodiment of the present invention will be described with reference to FIG.

The scene extraction apparatus 1 extracts a highly realistic scene from video / audio content (hereinafter, content) C. The scene extraction device 1 outputs the extracted scene to the display device 2.
Note that the content C may include a time-series sensory signal in addition to the video signal and the audio signal. This sensory signal can be, for example, a vibration signal that represents, in amplitude, the magnitude of vibration for reproducing vibration given to a chair of a home theater, a movie theater, or the like. Alternatively, the sensory signal may be a time-series angle signal representing the tilt of the chair.
Here, it is assumed that the scene extraction device 1 is connected to a setting device 3 such as a PC (personal computer) to the outside, and various setting information is set in advance.
As shown in FIG. 1, the scene extraction apparatus 1 includes a realistic sensation estimation unit 10, a realistic sensation learning data storage unit 20, a realistic sensation estimated value / determination result storage unit 30, a setting information storage unit 40, and a determination unit. 50 and extraction means 60.

The realistic sensation estimation means 10 reads the time interval and the time width from the setting information storage means 40, and calculates the realistic sensation estimation value indicating the degree of realism from the content C in units of time. In other words, the realistic sensation estimation means 10 analyzes the feature amount of the content C for each time interval having a predetermined time width that is shifted by a predetermined time interval including the overlapping interval, and is stored in the realistic sensation learning data storage means 20. From the sense of presence learning data, the degree of presence is calculated as an estimated presence value.
Here, the realistic sensation estimation means 10 includes an auditory realistic sensation estimation means 11, a visual presence sensation estimation means 12, a sensory presence sensation estimation means 13, and a presence sensation identification means 14.

The auditory realistic sensation estimation unit 11 calculates an estimated realistic sensation value from the sound signal of the content C. This auditory presence estimation means 11 can be configured as shown in FIG.
As shown in FIG. 2, the auditory presence estimation unit 11 includes an acoustic signal analysis unit 110 and an auditory presence calculation unit 111.

The acoustic signal analyzing unit 110 analyzes the acoustic signal of the content C and extracts an acoustic feature amount with a predetermined time width at predetermined time intervals. The time interval here refers to the interval between the first time in the time interval and the first time in the next time interval in the time-series time interval from which the acoustic feature value is extracted. In addition, the time width here refers to the length of the time interval in which the acoustic feature value is extracted.
For example, as shown in FIG. 5, the acoustic signal analyzing unit 110 sequentially analyzes acoustic signals having a predetermined time interval of 0.5 seconds and a predetermined time width of 3 seconds.

  Further, the acoustic signal analysis unit 110 may include, for example, loudness, sharpness, roughness, dynamic range (relative level of 5% time rate sound pressure level with respect to 95% time rate sound pressure level), sound image motion (motion) as acoustic feature amounts. (Represented by changes in elevation angle / horizontal angle). Further, if the acoustic signal has two left and right channels, the acoustic signal analyzing unit 110 may use the interaural cross-correlation, the interaural level difference, and the interaural phase difference as the acoustic feature amount.

Note that these acoustic analysis methods are general and will not be described. Further, it goes without saying that the acoustic feature amount obtained by the acoustic signal analyzing means 110 is not limited to the exemplified one as long as acoustic analysis is possible.
However, the acoustic feature amount obtained by the acoustic signal analyzing unit 110 is the same as the acoustic feature amount used when learning the realistic sense learning data stored in the realistic sense learning data storage unit 20 described later. The acoustic signal analyzing unit 110 sequentially outputs the acoustic feature amounts obtained at a predetermined time interval and a predetermined time width to the auditory presence sense calculating unit 111.

The auditory realistic sensation calculating means 111 is based on the realistic sensation learning data stored in the realistic sensation learning data storage means 20 and is based on the acoustic feature amount analyzed by the acoustic signal analyzing means 110 (individual realistic sensation). Estimated value) is calculated.
Here, the auditory realistic sensation calculating means 111 uses a neural network function that calculates an auditory realistic sensation estimated value from a plurality of acoustic feature quantities previously learned as realistic sensation learning data. An auditory realistic sensation estimate is calculated from the acoustic feature value. Note that the auditory presence sense value is a normalized value (for example, a range of 0 to 1).
The auditory realistic sensation calculating unit 111 outputs the calculated auditory realistic sensation estimated value to the realistic sensation specifying unit 14.
Returning to FIG. 1, the description of the configuration of the scene extraction device 1 will be continued.

The visual presence estimation means 12 calculates a visual presence estimation value from the video signal of the content C. The visual presence estimation means 12 can be configured as shown in FIG.
As shown in FIG. 3, the visual presence estimation unit 12 includes a video signal analysis unit 120 and a visual presence calculation unit 121.

The video signal analyzing unit 120 analyzes the video signal of the content C at a predetermined time interval and extracts a video feature amount at predetermined time intervals.
The time interval for analyzing the video signal is the same as the time for the acoustic signal analyzing unit 110 to analyze the acoustic signal.
The video signal analysis unit 120 uses, for example, luminance features (luminance average value, standard deviation, skewness) and chroma features (saturation average value, standard deviation) in a frame section of a predetermined time width as video feature amounts. , Skewness), hue characteristics (average number of pixels for each predetermined hue value such as red, yellow, and blue), moving object characteristics (average value of the number of pixels occupied by the moving object, 95th percentile value of the number of pixels occupied by the moving object) And the fifth percentile value).

Note that these video analysis techniques are general and will not be described. Needless to say, the video feature amount required by the video signal analyzing unit 120 is not limited to the exemplified feature as long as video analysis is possible.
However, the video feature amount required by the video signal analysis unit 120 is the same as the video feature amount used when learning the realistic learning data stored in the realistic learning data storage unit 20 described later.
The video signal analysis unit 120 sequentially outputs video feature amounts obtained at a predetermined time interval and a predetermined time width to the visual presence calculation unit 121.

Based on the presence learning data stored in the presence learning data storage unit 20, the visual presence calculation unit 121 calculates a visual presence estimate (individual presence sense) from the video feature amount analyzed by the video signal analysis unit 120. Estimated value) is calculated.
Here, the visual presence calculation means 121 uses a function of a neural network that calculates a visual presence estimation value from a plurality of video feature quantities previously learned as the presence learning data, and inputs a plurality of predetermined time widths. The visual presence estimation value is calculated from the video feature amount. Note that the visual presence estimation value is a normalized value (for example, a range of 0 to 1).
The visual presence calculation means 121 outputs the calculated visual presence estimation value to the presence determination means 14.
Returning to FIG. 1, the description of the configuration of the scene extraction device 1 will be continued.

The sensory realistic sensation estimation means 13 calculates a sensory realistic sensation estimated value from the sensory signal of the content C. This sensory presence estimation means 13 can be configured as shown in FIG.
As shown in FIG. 4, the sensory presence estimation unit 13 includes a sensory signal analysis unit 130 and a sensory presence calculation unit 131.

The sensory signal analyzing unit 130 analyzes the sensory signal of the content C at a predetermined time width and extracts a sensory feature amount at predetermined time intervals.
The time interval for analyzing the sensory signal is the same as the time for the acoustic signal analyzing unit 110 to analyze the acoustic signal.
When the sensory signal analyzing unit 130 uses a vibration signal as the sensory signal, for example, amplitude characteristics (average amplitude, standard deviation, difference between maximum amplitude and minimum amplitude) of predetermined time width, period information (period Average value, standard deviation, difference between the maximum period and the minimum period).
In addition, when the angle signal is used as the sensory signal, the sensory signal analysis unit 130 obtains, for example, an average value of the angle from the horizontal, a standard deviation, a maximum angular displacement, and the like as the angle displacement feature having a predetermined time width.

In addition, since these analysis methods are general, description is abbreviate | omitted. Needless to say, the sensory feature amount required by the sensory signal analysis unit 130 is not limited to the exemplified one as long as it can be analyzed as a sensory sense.
However, the sensory feature value calculated by the sensory signal analysis unit 130 is the same as the sensory feature value used when learning the realistic sense learning data stored in the realistic sense learning data storage unit 20 described later.
The sensory signal analyzing unit 130 sequentially outputs the sensory feature amounts obtained at predetermined time intervals and predetermined time widths to the sensory presence calculating unit 131.

The sensory realistic sensation calculating means 131 is based on the realistic sensation learning data stored in the realistic sensation learning data storage means 20 and is based on the sensory feature amount analyzed by the sensory signal analyzing means 130 (individual presence sensation). Estimated value) is calculated.
Here, the sensory realistic sensation calculating means 131 uses a neural network function that calculates a sensory realistic sensation estimated value from a plurality of sensory feature quantities previously learned as realistic sensation learning data. An estimated sensory presence value is calculated from the sensory feature amount. Note that the sense realistic sensation estimated value is a normalized value (for example, a range of 0 to 1).
The sensory realistic sensation calculating means 131 outputs the calculated sensory realistic sensation estimated value to the realistic sensation specifying means 14.

The realistic sensation specifying means 14 is a representative representative of the time interval from the individual realistic sensation estimated values of the time intervals estimated by the auditory realistic sensation estimation means 11, the visual presence sensation estimation means 12, and the sensory presence sensation feeling estimation means 13, respectively. This is to identify the estimated feeling value (representative value).
Here, the realistic sensation specifying means 14 sets the maximum value among the individual realistic sensation estimated values (auditory realistic sensation estimated value, visual realistic sensation estimated value, sensory realistic sensation estimated value) as the realistic sensation estimated value in the time interval. . However, the presence feeling specifying unit 14 does not necessarily need to specify the presence feeling estimated value with the maximum value. For example, the presence feeling estimated value may be specified by a statistic such as an average value or weighted addition.

When the realistic sensation estimated value is specified by weighted addition, the realistic sensation specifying unit 14 stores the individual realistic sensation estimated values (auditory realistic sensation estimated value, visual realistic sensation estimated value) stored in the setting information storage unit 40 in advance. ), And the weight coefficient of the sense of realistic sensation) is read. Alternatively, information that classifies the contents of the content in advance as a tag may be added as a tag, and the presence specifying unit 14 may use a value that is predetermined for each classification indicated by the tag as the weighting coefficient. For example, if the category of the content C is “music program”, the weight of the auditory presence sense value is increased, and if the category of the content C is “travel program”, the weight of the visual presence estimate value is increased. It is.
The presence feeling specifying unit 14 sequentially writes and stores the specified presence feeling estimated value in association with the elapsed time from the top of the content C in the presence feeling estimated value / determination result storage unit 30.

The sense of presence learning data storage means 20 stores learning data (presence learning data) obtained by learning in advance the degree of sense of presence with respect to the feature quantities of signals (acoustic signals, video signals, and sensory signals) constituting the content C. is there. The presence learning data storage means 20 can be configured by a general recording medium such as a hard disk.
The sense of presence learning data is obtained by learning in advance the results of subjective evaluation by a plurality of subjects in a plurality of contents, and is defined by a neural network or the like as a model for estimating the sense of presence from a plurality of feature amounts.

  For example, in the learning of realistic sense learning data, an audio signal, a video signal, and a sensory signal included in the evaluation content are presented to the subject, and for each signal, the realistic feeling is “not felt at all” to “very felt” Are evaluated over time at multiple levels (eg, 7 levels). And the feature value for each predetermined time width (for example, 0.5 seconds) of each of the audio signal, the video signal, and the sensory signal, and the average value of the level of presence felt by all subjects in the interval of the time width To learn a neural network. Here, the output of the neural network is a normalized value (for example, a range of 0 to 1).

Note that the feature amount of the acoustic signal in the learning of the realistic sense learning data is the same as the acoustic feature amount analyzed by the acoustic signal analysis unit 110 (see FIG. 2). Further, the feature amount of the video signal in the learning of the realistic sense learning data is the same as the video feature amount analyzed by the video signal analysis unit 120 (see FIG. 3). In addition, the sensory signal feature amount in the learning of the realistic sense learning data is the same as the sensory feature amount analyzed by the sensory signal analysis unit 130 (see FIG. 4).
As described above, by learning the realistic sense learning data with the neural network, it is possible to construct a model for estimating the realistic sense from a plurality of feature amounts. Of course, the realistic sense learning data is not limited to the neural network, and a model for estimating the realistic sense from a plurality of feature amounts can be constructed by general machine learning.

The realistic sensation estimated value / determination result storage means (the realistic sensation estimated value storage means, the determination result storage means) 30 calculates the realistic sensation estimated value for each time interval of the content C estimated by the realistic sensation estimation means 10. This is stored in association with the elapsed time from the beginning.
Further, the realistic sensation estimated value / determination result storage means 30 indicates the result (for example, flag) determined by the determination means 50 as to whether or not the realistic sensation estimated value in each time interval gives the viewer a sense of realism. It has an area to record.

  For example, as shown in FIG. 6, the realistic sensation estimated value / determination result storage means 30 records time (elapsed time from the beginning), realistic sensation estimated value, and determination result (overall determination result, threshold determination result, rank). It has the area to do. The time and the realistic sensation estimation value are written by the realistic sensation estimation means 10, and the determination result is written by the determination means 50. The overall determination result, the threshold determination result, and the rank in the determination result will be described in detail in the description of the determination unit 50.

  The realistic sensation estimated value / determination result storage means 30 can be composed of a general recording medium such as a semiconductor memory. Note that the realistic sensation estimated value / determination result storage means 30 is an individual sensation estimated value storage means for storing the realistic sensation estimated value for each time interval, and a determination result storage means for storing the determination result for each time interval. You may comprise as a means.

  The setting information storage means 40 stores various information set by the setting device 3 and can be configured by a general recording medium such as a semiconductor memory. Here, the setting information storage means 40 is written by the setting device 3 with the threshold value, the extraction condition, and the paste margin time.

The threshold value is a value serving as a criterion for determining whether or not the presence is high. Here, the setting device 3 inputs a threshold value via the setting screen displayed on the display device 2 and sets it in the scene extraction device 1.
This threshold value may be an absolute threshold value that is a fixed value, or may be a relative threshold value that varies depending on the time length in the content C.
For example, when the threshold value is an absolute threshold value, the setting device 3 sets the threshold value within the range of realistic sensation estimation values (for example, a range of 0 to 1). That is, the setting device 3 sets a value (for example, 0.8) set by the viewer via the setting screen displayed on the screen of the display device 2 as a threshold value.

  For example, when the threshold value is a relative threshold value, the setting device 3 sets a time length or a ratio for extracting a scene from the higher content C estimated value in the entire content C. That is, the setting device 3 calculates the threshold value for the time length (for example, 5 minutes) or the ratio (for example, 80%) set by the viewer via the setting screen displayed on the screen of the display device 2. Set as information. This time length or ratio is used when the relative threshold is calculated in the determination means 50 described later.

The extraction condition is a condition for extracting a scene. The extraction conditions are, for example, the shortest scene extraction time, the maximum number of scene extractions, the display order, and the like.
The shortest scene extraction time is at least the shortest time that the realistic sensation continues as a highly realistic scene. In other words, the shortest scene extraction time is a time limit that prevents extraction even if a scene is determined to have a high sense of realism if the time is short.
The maximum number of scene extractions is the maximum number of scenes to be extracted. That is, the maximum number of scene extractions indicates the upper limit of the number of scenes to be extracted when the content C includes many scenes with a high presence.
The display order indicates the display order of scenes that are extracted and displayed on the display device 2 as highly realistic scenes. For example, the display order may be an order in accordance with the playback time of the content C, an order with a higher presence estimate value, or the like.

  Here, the setting device 3 uses the setting screen displayed on the display device 2 to identify the shortest scene extraction time, the maximum number of scene extractions, the identification order for identifying the display order, and the specific values (extraction time of each information). , The number of extractions, etc.) are input and set in the scene extraction apparatus 1. In addition, the display order is determined in advance in a specific order such as the order of reproduction time, the order of higher realistic estimation values, and the like.

  The paste margin time is a paste margin time that is added extra before and after the scene extracted as a highly realistic scene. This glue allowance time is a time for extracting not only a scene with high presence, but also including the lingering sound before and after. The pasting time may be set to the same time before and after the scene, or may be set to different times. Here, the setting device 3 inputs the paste allowance time via the setting screen displayed on the display device 2 and sets it in the scene extraction device 1.

Based on the setting information stored in the setting information storage unit 40, the determination unit 50 determines whether or not the presence estimation unit 10 should extract a scene in a time interval associated with the presence estimation value. Is.
Here, as shown in FIG. 6, the determination unit 50 writes the threshold determination result, the rank, and the overall determination result in the realistic sensation estimated value / determination result storage unit 30 as the determination result.

  The “threshold determination result” indicates a determination result as to whether or not the time interval is an estimated presence value equal to or greater than a set threshold. Specifically, the determination unit 50 compares the threshold value stored in the setting information storage unit 40 with the realistic sense estimated value / determined result storage unit 30 to store the realistic sense estimate. If the value is greater than or equal to the threshold value, it is determined that the time interval is highly realistic and a flag is set. On the other hand, if the estimated presence value is less than the threshold value, the determination unit 50 determines that the time interval is not high and resets the flag. For example, in the example of FIG. 6, a state in which “0.7” is set as the threshold and the flag (“1”) is set for the time interval (t1, t2, t10) where the realistic sensation estimation value is equal to or greater than the threshold. Show.

  When a relative threshold value is used as the threshold value, the determination unit 50 determines the realistic feeling estimated value that reaches the time length or ratio set in the setting information storage unit 40 from the higher realistic feeling estimated value in the entire content C. Is a threshold value. That is, the determination unit 50 sets the time length (for example, 5 minutes) or the ratio (for example, 80%) from the higher level of the realistic presence estimated value / determination result storage unit 30 stored in the presence feeling estimated value. The estimated realistic sensation reached is set as a threshold value.

“Rank” ranks higher on the basis of the highest realistic sensation estimation value set for each consecutive time interval (flag is set) determined to be high in the “threshold determination result”. The numbers are assigned in order. Specifically, the determination unit 50 determines that the interval in which the flag continues in the threshold determination result stored in the realistic sensation estimated value / determination result storage unit 30 is the shortest scene extraction time stored in the setting information storage unit 40. Whether or not it is a continuous time interval as described above is searched, and the continuous time intervals are ranked in the order of the high presence based on the highest estimated realistic value set for each continuous time interval.
When the shortest scene extraction time is not set, the determination unit 50 sets all time intervals in which the flag is continuous as a threshold determination result as a target for ranking. Of course, a predetermined default value may be provided as the shortest scene extraction time.

  For example, in the example of FIG. 6, the rankings “from the top of the realistic feeling estimated value to the time interval (t1, t2) and the time interval (t10) in which the estimated realistic feeling value equal to or greater than the threshold value continues for the shortest scene extraction time or longer, respectively. In this example, 1 ”and ranking“ 10 ”are set. Of course, the intermediate ranks are not shown here.

The “total determination result” indicates a determination result as to whether or not the scenes in the time interval ranked in the order of the realistic sensation estimation value by the “rank” are the scenes having the set maximum number of extracted scenes. Specifically, the determination unit 50 sets up to the maximum number of scene extractions stored in the setting information storage unit 40 in the order of the ranked scenes, and uses the time interval corresponding to the scene as the extraction target scene. Set flags and reset flags for other time intervals.
Note that when the maximum number of scene extractions is not set, the determination unit 50 does not limit the number of extractions. Of course, a predetermined default value may be provided as the maximum number of scene extractions.
For example, in the example of FIG. 6, when the maximum number of scene extractions is “5”, the flag of the comprehensive determination result is set in the time interval (t1, t2) of the rank “1”, and the time of the rank “10” The flag is not set in the section (t10).

  Based on the comprehensive determination result, the extraction unit 60 can specify the scene to be extracted. Here, when the margin time is set as the setting information in the setting information storage unit 40, the determination unit 50 sets a flag in the time interval corresponding to the margin time before and after the scene in which the flag of the comprehensive determination result is set. It will be set. As a result, the pasting time is expanded before and after the scene time extracted by the extracting means 60.

  When the display order is set as the setting information in the setting information storage unit 40, the determination unit 50 sets the value indicating the display order with the flag of the overall determination result in the realistic sensation estimated value / determination result storage unit 30. A display order number is added to each scene (not shown). When the display order is not set, the determination unit 50 adds display order numbers in a predetermined default display order, for example, in the order of higher realistic sensation estimation values. This display order may be used by overwriting the “order” (see FIG. 6) of the realistic sensation estimated value / determination result storage means 30.

The extraction unit 60 extracts a scene with a high presence from the content C. The extracting unit 60 extracts scenes determined as high realistic scenes in the content C by the realistic presence estimated value / determination result storage unit 30.
Specifically, the extracting unit 60 is set with a scene in which a flag is set in the determination result (specifically, the comprehensive determination result) stored in the realistic sensation estimated value / determination result storage unit 30. In the display order, the content C is sequentially read and reproduced for each section corresponding to the time from the beginning time of the first time section to the end time of the last time section, and output to the display device 2.

At this time, when the determination unit 50 sets a flag for a continuous time interval equal to or longer than the shortest scene extraction time, the extraction unit 60 can extract only a scene equal to or longer than the shortest scene extraction time. Further, the determination unit 50 sets a flag for the continuous time interval with the maximum number of scene extractions as an upper limit, so that the extraction unit 60 can extract the scenes by limiting the number of scenes to be extracted.
When a sensation signal such as a vibration signal is added to the content C, the extraction unit 60 applies a sensation signal corresponding to the extraction time of the content C to a reproduction device (for example, a vibration reproduction device) (not shown). Output.
Thereby, the extraction means 60 can extract a highly realistic scene from the content C by the extraction method set by the viewer.

  By configuring the scene extraction device 1 as described above, the scene extraction device 1 extracts a highly realistic scene based on the subjective sense of the viewer, not the objective feature amount of the content C. can do.

[Operation of the scene extraction device]
Next, the operation of the scene extraction apparatus 1 according to the first embodiment of the present invention will be described with reference to FIGS. It is assumed that the realistic sense learning data storage means 20 stores learning data (real sense learning data) in which the realistic sense with respect to the feature quantities of the sound signal, the video signal, and the sensory signal is learned in advance. In the setting information storage unit 40, various setting information is set in advance by the setting device 3. This setting operation will be described with reference to FIG.

As illustrated in FIG. 7, the setting device 3 sets a threshold value serving as a criterion for determining whether or not presence is high in the scene extraction device 1 (step S100). That is, in step S100, the setting device 3 displays a setting screen on the display device 2, and accepts the setting of the threshold by the viewer. This threshold value is stored in the setting information storage unit 40.
When a relative threshold value is used as the threshold value, the setting device 3 accepts a time length or a ratio for extracting a scene from the one with the higher realistic sensation value in the entire content C by the viewer, and the setting information storage unit 40 To remember.

  Further, the setting device 3 sets, in the scene extraction device 1, a scene extraction condition to be displayed on the display device 2 as a highly realistic scene (step S101). That is, in step S101, the setting device 3 displays a setting screen on the display device 2, and accepts the setting of extraction conditions by the viewer. The extraction conditions are, for example, the shortest scene extraction time, the maximum number of scene extractions, the display order, and the like. This extraction condition is stored in the setting information storage unit 40.

Further, the setting device 3 sets, in the scene extraction device 1, an extra margin time to be added before and after the scene extracted as a highly realistic scene (step S102). That is, in step S102, the setting device 3 displays a setting screen on the display device 2 and accepts the setting of the glue allowance time. This paste allowance time is stored in the setting information storage means 40.
Note that the operations from step S100 to step S102 described above may be performed in any order.

Next, the overall operation of the scene extraction apparatus 1 will be described with reference to FIG. 8 (see FIG. 1 as appropriate).
The scene extraction apparatus 1 first reads various setting information from the setting information storage means 40 (step S1).
After reading the various settings in step S1, the scene extraction apparatus 1 estimates the presence in a predetermined time width for each predetermined time interval of the content C by the presence estimation means 10 (step S2).
The operation of step S2 will be described in more detail with reference to FIG.

As shown in FIG. 9, the realistic sensation estimation means 10 uses the auditory presence sensation estimation means 11, the visual presence sensation estimation means 12, and the sensory presence sensation estimation means 13 for each of a plurality of signals (video signal, sound signal, sensory signal). In addition, the realistic sensation estimated value is calculated at a predetermined time interval and a predetermined time width (step S20).
That is, the auditory realistic sensation estimation unit 11 calculates an auditory realistic sensation estimated value from the acoustic signal of the content C based on the realistic sensation learning data stored in the realistic sensation learning data storage unit 20. The visual presence estimation means 12 calculates a visual presence estimation value from the video signal of the content C based on the presence learning data. Further, the sensory realistic sensation estimation means 13 calculates a sensory realistic sensation estimated value from the sensory signal of the content C based on the realistic sensation learning data.

  Then, the scene extraction apparatus 1 uses the presence sensation specifying unit 14 of the presence sensation estimation unit 10 to calculate a plurality of presence sensation estimation values (audience presence sensation estimation values, visual presence sensation estimation values, sensory presence sensation estimations) calculated in step S20. The maximum value is specified as the realistic presence estimated value (representative value) in the time interval (step S21). In step S21, instead of the maximum value, a statistical amount such as an average value or weighted addition may be specified as the realistic sensation estimated value in the time interval.

  Then, the realistic sensation specifying means 14 writes and stores the realistic sensation estimated value (representative value) for each time interval specified in step S21 in association with the time interval in the realistic sensation estimated value / determination result storage means 30. (Step S22).

Here, the scene extracting apparatus 1 determines whether or not the content C is read by the presence estimation means 10 until the end (step S23). If the data of the content C still remains (No in step S23), the scene extraction device 1 returns to step S20 and estimates the presence in the next time interval.
On the other hand, when the reading of the data of the content C is completed (Yes in step S23), the scene extraction device 1 ends the realistic feeling estimation operation.
Through the operations from step S20 to step S23, the realistic sense estimated value is stored in the realistic sense estimated value / determination result storage means 30 in time series.
Returning to FIG. 8, the overall operation of the scene extraction apparatus 1 will be described.

After the estimation of the presence in step S2, the scene extraction apparatus 1 determines whether the scene gives the viewer a sense of presence for each time interval by the determination unit 50, and records the result. (Step S3).
The operation in step S3 will be described in more detail with reference to FIG.

As shown in FIG. 10, the determination unit 50 has a threshold value in a time interval that is equal to or greater than the threshold value stored in the setting information storage unit 40. A flag of the determination result (see FIG. 6) is set (step S30). At this time, the flag is reset for a time interval in which the realistic sense estimated value is less than the threshold.
When a relative threshold value is used as the threshold value, the determination unit 50 determines the realistic feeling estimated value that reaches the time length or ratio set in the setting information storage unit 40 from the higher realistic feeling estimated value in the entire content C. Is a threshold value.

Then, the determination unit 50 continues for the minimum scene extraction time stored in the setting information storage unit 40 in the time interval in which the flag is set in step S30 stored in the realistic sensation estimated value / determination result storage unit 30. A continuous time interval is searched for (step S31).
Thereafter, the determination unit 50 ranks the continuous time intervals searched in step S31 in order from the highest maximum realistic sensation estimation value in the continuous time interval (see FIG. 6) (step S32).

Then, the determination unit 50 performs a comprehensive determination result for each time interval corresponding to the maximum number of scene extraction scenes (continuous time intervals) stored in the setting information storage unit 40 in the continuous time intervals ranked in step S32. The flag (see FIG. 6) is set (step S33).
Further, when the margin time is set as the setting information in the setting information storage unit 40, the determination unit 50 determines the total determination result in the time interval corresponding to the margin time before and after the scene in which the flag of the total determination result is set. Is set (step S34).
Through the operations from step S30 to step S34, the determination result as to whether or not the presence is high in the time interval in the content C is recorded in the presence estimation value / determination result storage means 30.
Returning to FIG. 8, the overall operation of the scene extraction apparatus 1 will be described.

After determining the presence in units of time in step S3, the scene extracting apparatus 1 extracts a scene with high presence from the content C by the extracting means 60 (step S4).
That is, the extracting unit 60 refers to the determination result (specifically, the comprehensive determination result) of the estimated presence value / determination result storage unit 30 and determines that the presence is high (a flag is set). A scene in the time interval (continuous time interval) is extracted from the content C and output to the display device 2.
Through the operation described above, the scene extraction device 1 can extract a scene with high presence based on the subjective impression of the viewer, not the objective feature amount of the content C.

<< Second Embodiment >>
Next, the configuration of the scene extraction device 1B according to the second embodiment of the present invention will be described with reference to FIG.

The scene extraction device 1B extracts a highly realistic scene from the content C. The scene extraction device 1B outputs the extracted scene to the display device 2.
The scene extracting apparatus 1 described with reference to FIG. 1 specifies (calculates) one realistic feeling estimated value from a plurality of realistic feeling estimated values and holds one realistic feeling estimated value in a certain time interval. On the other hand, the scene extraction apparatus 1B holds a plurality of realistic sensation estimation values in a certain time interval, and by setting a weight from the outside, a scene with a higher sensation of realism is extracted when a highly realistic scene is extracted. It is possible to set whether to do.

  The scene extracting apparatus 1B includes a realistic sensation estimation unit 10B, a realistic sensation learning data storage unit 20, a realistic sensation estimated value / determination result storage unit 30B, a setting information storage unit 40B, a determination unit 50B, and an extraction unit 60. . The presence learning data storage unit 20 and the extraction unit 60 have the same configuration as that of the scene extraction device 1 (see FIG. 1), and thus description thereof is omitted.

The realistic sensation estimation means 10B calculates a realistic sensation estimated value indicating the degree of realistic sensation in time units from the content C. Here, the realistic sensation estimation means 10B is obtained by removing the realistic sensation specifying means 14 from the realistic sensation estimation means 10 (see FIG. 1).
This realistic sensation estimation means 10B is an auditory realistic sensation estimation means 11, a visual presence sensation estimation means 12, and a sensory presence sensation estimation means 13 that are estimated by the individual presence sensation estimation values (audience realistic sensation estimation values, visual presence sensation estimation values). , Sensory presence sense value) is stored in the presence sense value / determination result storage means 30B.

The realistic sensation estimated value / determination result storage means 30B stores the individual realistic sensation estimated value for each time interval of the content C estimated by the realistic sensation estimation means 10B in association with the elapsed time from the beginning of the content C. It is.
Further, the realistic presence estimated value / determination result storage unit 30B determines whether or not the individual realistic presence estimated value in each time interval gives the viewer a sense of realism by the determining unit 50B (for example, a flag). Has an area for recording.
For example, as shown in FIG. 12, the realistic sensation estimated value / determination result storage unit 30B records time (elapsed time from the beginning), realistic sensation estimated value, and determination result (overall determination result, threshold determination result, rank). Have a region to The time and the realistic sensation estimated value are written by the realistic sensation estimation means 10B, and the determination result is written by the determination means 50B. Note that the auditory sense of reality, the visual sense of reality estimated value, and the sense of sense of presence realistic value are individually stored as the sense of presence estimated value.
The realistic sensation estimated value / determination result storage means 30B can be configured by a general recording medium such as a semiconductor memory.

  The setting information storage means 40B stores various information set by the setting device 3, and can be configured by a general recording medium such as a semiconductor memory. Here, in the setting information storage unit 40B, the threshold value, the extraction condition, the paste margin time, and the weight are written by the setting device 3.

  The weight is a weighting coefficient for a content signal (acoustic signal, video signal, sensory signal) whose presence is estimated by the presence estimation means 10B. This weight is set to a value such that the sum of the weights is “1” with respect to a content signal (sound signal, video signal, sensory signal) whose presence is estimated by the presence estimation means 10B. Here, the setting device 3 inputs weights via the setting screen displayed on the display device 2 and sets them in the scene extraction device 1.

  Based on the setting information stored in the setting information storage unit 40B, the determination unit 50B determines whether or not to extract a scene in the time interval associated with the realistic sensation estimation value in the realistic sensation estimation unit 10B. Is. The function of the determination unit 50B is basically the same as that of the determination unit 50 (see FIG. 1). However, the determination unit 50B determines the individual sense of presence (audience realistic sense estimate, visual sense of real sense, value based on the weight stored in the setting information storage means 40B in order to determine the sense of realism for each time interval. The difference is that a threshold value is compared with a value obtained by weighted addition of (sensory realistic sense estimated value).

  In other words, the determination unit 50B is an individual presence estimation value for each time interval stored in the realistic presence estimated value / determination result storage unit 30B (auditory presence estimation value, visual presence estimation value, sensory presence estimation value). Is weighted and added with the weight set by the setting device 3 as the estimated realistic sensation value for the time interval. Then, the determination unit 50B uses the weighted addition value as the realistic presence estimated value as a determination result in the time interval determined to be high by the same determination method as the determination unit 50 (see FIG. 1). The result flag is set, and the threshold determination result flag is reset in the time interval in which it is determined that the presence is not high (see FIG. 12).

  By configuring the scene extraction device 1B as described above, the scene extraction device 1B extracts a scene with high presence based on the subjective impression of the viewer, not the objective feature amount of the content C. can do. Furthermore, since the scene extraction device 1B can change the weights of a plurality of realistic sensations after estimating the realistic sensations, the viewer can extract a desired scene from the content C by appropriately changing the weights. .

Although the embodiment of the present invention has been described above, the scene extraction apparatuses 1 and 1B can be operated by a scene extraction program that causes a computer to function as each of the above-described means.
Further, the present invention is not limited to these embodiments. Hereinafter, modifications of the present invention will be described.

<< Modification 1 >>
In the scene extraction devices 1 and 1B, the setting device 3 sets one threshold value that is a criterion for determining whether or not the presence is high. However, a plurality of threshold values may be set by providing a high level of presence. For example, a plurality of threshold values are set according to levels such as “very high sense of reality” and “highly realistic”.

In this case, when the determination means 50 and 50B set the determination result of the realistic sensation in the realistic sensation estimated value / determination result storage means 30, the determination result (the comprehensive determination result, the threshold value) according to the level of the realistic sensation level. (Determination result) may be stored. For example, for a time interval in which an estimated value of realistic sensation equal to or greater than the threshold value indicating “very high sense of reality” is used, the determination result is “2”, and the value is less than the threshold value indicating “very high sense of reality”. The determination result is distinguished as “1” for the time interval in which the realistic sensation estimated value equal to or greater than the threshold indicating “slightly high” is estimated.
Then, when the level is designated by the setting device 3, the extraction unit 60 extracts a scene corresponding to the time interval in which the level is set.
Thus, the viewer can categorize the magnitude of the presence and view a scene with a different presence.

<< Modification 2 >>
The scene extraction devices 1 and 1B display the extracted scene on the display device 2 by the extraction means 60. However, the extraction means 60 may once display the first frame of each scene to be extracted as a thumbnail image on the display device 2 and extract and display only the scene selected by the viewer from the content C. Further, the order in which the thumbnail images are displayed may be in the order of time or in the order of the higher realistic estimate value.
Thus, the viewer can select and view a scene from highly realistic scenes.

<< Modification 3 >>
In the scene extraction devices 1 and 1B, the auditory realistic sensation estimation unit 11 calculates the realistic sensation estimated value from the sound signal of the content C. However, the auditory realistic sensation estimation means 11 determines whether or not the acoustic signal is mainly human speech from the acoustic feature quantity, and the auditory realistic sensation estimated value is determined when the human speech level is high by the setting. Alternatively, the auditory realistic sensation estimated value may be lowered when the degree of human voice is low.
As a result, the scene extraction devices 1 and 1B can extract the scene by distinguishing whether to extract a scene that has a high sense of realism due to interpersonal activities or to extract a scene that has a high sense of realism due to an environment other than interpersonal activities. it can.

<< Modification 4 >>
The scene extraction devices 1 and 1B store the realistic sensation estimated value in the realistic sensation estimated value / determination result storage unit 30 by the realistic sensation estimating unit 10B.
However, the realistic sensation estimation means 10B may further estimate the degree of emotion other than the realistic sensation. In this case, the realistic learning data storage means 20 is preliminarily trained for each of the sound signal, video signal, and sensory signal, the degree of the exciting scene, the degree of the scene to be gene, the degree of the comfortable scene, Learn the degree of a scene. Accordingly, the realistic sensation estimation means 10B estimates the degree of emotion (emotion estimated value) in addition to the realism in each of the estimation means 11, 12, and 13.
Then, the setting device 3 sets a weight for the estimated value of the feeling level in addition to the realistic weight, so that the determination unit 50B weights and adds the estimated presence value and the estimated emotion value.
As a result, the scene extraction apparatus 1B can extract a scene in which the emotional height increases in addition to the presence.

<< Modification 5 >>
The scene extraction devices 1 and 1B individually estimated auditory presence, visual presence, and sensory presence in the presence estimation means 10 and 10B.
However, the realistic sensation estimation means 10 and 10B may estimate the realistic sensation represented at a time instead of estimating the realistic sensation individually. In this case, in the realistic sense learning data storage unit 20, a model for estimating one realistic sense from feature quantities of a plurality of signals (sound signals, video signals, and sensory signals) is previously learned using a neural network or the like. Then, the realistic sensation estimation means 10 and 10B may estimate the realistic sensation representing each signal from the feature amounts of a plurality of signals (acoustic signal, video signal, and sensory signal) constituting the content C using a neural network.

DESCRIPTION OF SYMBOLS 1 Scene extraction device 10 Realistic feeling estimation means 11 Auditory realistic feeling estimation means 12 Visual realistic feeling estimation means 13 Sensory realistic feeling estimation means 14 Realistic feeling specification means 20 Realistic feeling learning data storage means 30 Realistic feeling estimated value / judgment result storage means ( (Presence sense storage means, judgment result storage means)
40 setting information storage means 50 determination means 60 extraction means 2 display device 3 setting device

Claims (10)

A scene extraction device that extracts a highly realistic scene from content by a sense of presence defined by learning a predetermined feature amount of the content,
Presence learning data storage means for storing presence learning data in which the degree of presence with respect to the feature amount of the signal constituting the content is previously learned;
A setting information storage means for storing, as setting information, a time interval and a time width of the signal for analyzing at least the presence, and a threshold value that is a criterion for determining whether or not the presence is high;
The feature amount of the content is analyzed for each time interval of the predetermined time width that includes an overlapping interval and is shifted by the predetermined time interval, and the degree of presence is estimated from the presence learning data Presence estimation means to calculate as a value;
A sense of presence estimated value storing means for storing the sense of presence estimated value in association with the time interval;
Determining means for determining whether or not the presence of the scene in the time interval associated with the estimated presence value is high based on the preset threshold;
Determination result storage means for storing the determination result of the determination means;
Extraction means for extracting a scene corresponding to a time interval determined to have high presence stored in the determination result storage means from the content;
A scene extraction apparatus comprising:   The presence estimation means calculates an individual presence estimation value for each signal constituting the content, and uses the maximum value, average value, or weighted addition value as the presence estimation value. Item 2. The scene extraction device according to Item 1. The realistic sensation estimation means calculates an individual realistic sensation estimated value for each signal constituting the content and stores it in the realistic sensation estimated value storage means,
The determination means calculates a maximum value, an average value, or a weighted addition value of the plurality of individual realistic sensation estimated values for each time interval as the realistic sensation estimated value, and based on the preset threshold value The scene extraction device according to claim 1, wherein it is determined whether or not the value is high. As the setting information, a time length or a ratio of a scene determined to be high in the content is stored in the setting information storage unit in advance.
The said determination means uses the realistic feeling estimated value which reaches the said time length or the said ratio in an order from the time interval with the said high realistic feeling estimated value as the said threshold value, The any one of Claim 1 to 3 characterized by the above-mentioned. The scene extraction device according to item. The threshold value is a plurality of values determined in advance according to the height of the realistic sensation,
The determination means sets the level of the level of presence in the time interval by comparing the estimated value of presence in the time interval and a plurality of the threshold values in the time interval,
4. The scene according to claim 1, wherein the extraction unit extracts a scene corresponding to the time interval in which a level designated from the outside is set from the content. 5. Extraction device. As the setting information, the shortest scene extraction time is previously stored in the setting information storage means,
The scene extraction apparatus according to any one of claims 1 to 5, wherein the extraction unit extracts only scenes having a time longer than the shortest scene extraction time as the highly realistic scenes. As the setting information, the maximum scene extraction number is stored in advance in the setting information storage means,
The scene extraction apparatus according to any one of claims 1 to 6, wherein the extraction unit extracts a scene with the maximum number of extracted scenes as an upper limit as the scene with high presence. As the setting information, paste margin time is previously stored in the setting information storage means,
8. The extraction unit according to claim 1, wherein the extraction unit extracts, from the content, a scene corresponding to a time interval in which the pasting time is added to a time interval in which the presence is determined to be high. The scene extraction device according to any one of claims. The realistic sensation estimation means calculates an emotion estimation value for each of the time intervals from the content by learning data in which the feature amount of the content and the degree of emotion are learned in advance.
The said determination means determines whether the said presence is high by the weighted addition value of the said presence feeling estimated value and the said emotion estimated value. The scene extraction device described in 1.   A scene extraction program for causing a computer to function as each unit of the scene extraction device according to any one of claims 1 to 9.

Images