JP2014110020A - Image processor, image processing method and image processing program - Google Patents

Abstract

A scene change is detected with higher accuracy and higher speed based on the characteristics of temporal and spatial changes of an image.
An image processing apparatus according to the present invention includes a first acquisition unit that acquires learning video data, a first feature extraction unit that extracts features of temporal and temporal changes of the video from the learning video data, and Using the first feature, the learning unit 10 that learns whether or not the scene is switched, the second obtaining unit 11 ′ that obtains the display video data, and the temporal and spatial change characteristics of the video from the display video data. A second feature extracting unit 13 ′ for extracting, and an estimating unit 12 ′ for estimating presence / absence of a scene change based on the learning result obtained by the second feature and the learning unit 10; Two features are the co-occurrence probability of the difference value of luminance or color between frame images and the temporal change rate of the difference value, the difference value of luminance or color between adjacent blocks in the frame image, and the temporal change of the difference value. Rate co-occurrence and Even without including any.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  In recent years, there is a demand for detecting a scene change (also referred to as a scene change) between frame images in a video including a plurality of continuous frame images such as a television video. For example, in a video including a TV commercial, by detecting a scene change before and after the TV commercial, it is possible to skip the TV commercial portion and display a video other than the TV commercial.

  As a technology related to this, in a video representing the movement of an object, a scene based on an error value for each pixel between a frame image including the object before moving and a frame image including the object after moving. There is one that detects a change (for example, Patent Document 1). Specifically, in this technique, the planar distance from the position in the frame image of the object before the movement to the position in the frame image after the movement of the object after the movement Detect as a vector. Then, a part of the frame image ahead of the motion vector in the later frame image is cut out as a block, and the sum of error values for each pixel between the cut out block and the block including the object before moving is cut out. And a scene change is detected based on the calculated value.

  In addition, in order to detect a scene change, there is a technique that uses a SIFT (Scale-Invariant Feature Transform) algorithm that calculates the number of matching of feature points in a frame image as a feature amount, or a technique that uses FFT (Fast Fourier Transform) (for example, Non-patent document 1 and Non-patent document 2).

Japanese Patent Laid-Open No. 08-102938

Li, J .; , Ding, Y. Shi, Y .; Li, W .; : "A Divide-And-Rule Scheme For Shot Boundary Detection Based on SIFT"; JDCTA (2010) pp. 202-214 A. Mine, Th. Hermes, G.M. T.A. Ioannidis, O.I. Herzog: “Automatic Shot Boundary Detection Using Adaptive Thresholds”, [searched on September 12, 2012], Internet <http://www-nlpir.nist.gov/projects/tvpubs/tvpapers03/ubremen.paper.pdf

  However, in the technique described in Patent Document 1, when detecting a scene change, it is necessary to detect a motion vector or to extract a block from a frame image using the motion vector. It takes extra time. Further, in the techniques described in Non-Patent Document 1 and Non-Patent Document 2, since a SIFT algorithm or FFT with a large amount of processing is used, processing time is required.

  The present invention has been made in view of the above problems, and provides an image processing apparatus, an image processing method, and an image processing program capable of detecting a scene change with higher accuracy and higher speed than conventional ones.

  In order to achieve the above object, an image processing apparatus according to the present invention includes a first acquisition unit for acquiring learning video data including data on a plurality of continuous frame images, and a time-space of video from the learning video data. A first feature extraction unit that extracts a change feature as a first feature; a learning unit that learns whether scenes are switched between successive frame images using the first feature; and a plurality of consecutive frame images A second acquisition unit that acquires display video data including data about, a second feature extraction unit that extracts, from the display video data, a feature of temporal and temporal changes of the video as a second feature, and the second feature And an estimation unit that estimates the presence or absence of scene switching between successive frame images based on the learning result obtained by the learning unit, and the first feature and the second feature are Co-occurrence of luminance or color difference value between frame images and temporal change rate of the difference value, and co-occurrence of luminance or color difference value and temporal change rate of the difference value between adjacent blocks in the frame image And a co-occurrence probability of at least one of the probabilities.

  In addition, an image processing method according to the present invention for achieving the above object includes a step (a) of acquiring learning video data including data on a plurality of continuous frame images, and the learning video data from the learning video data. A step (b) for extracting a feature of spatio-temporal change as a first feature, a step (c) for learning whether or not scenes are switched between successive frame images using the first feature, and a plurality of successive features (D) obtaining display video data including data on the frame image, and (e) extracting a feature of temporal and temporal changes of the video from the display video data as the second feature. (F) estimating the presence / absence of scene switching between successive frame images based on the characteristics and the learning result obtained in step (c), The feature and the second feature are: a co-occurrence probability of a luminance or color difference value between frame images and a temporal change rate of the difference value, a luminance or color difference value between adjacent blocks in the frame image, and the difference It includes the co-occurrence probability of the time change rate of the value and at least one of the co-occurrence probabilities.

  That is, the present invention provides a co-occurrence probability of a luminance or color difference value between frame images and a temporal change rate of the difference value, or a luminance or color difference value between adjacent blocks in the frame image and the difference value. The scene change is detected based on the co-occurrence probability of the time change rate.

  According to the present invention, it is possible to detect a scene change with higher accuracy and higher speed than before.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a block diagram which shows the structure of a spatiotemporal feature extraction part. It is a figure where it uses for description of the feature-value of a spatiotemporal change. It is a flowchart which shows the procedure of the process for learning the presence or absence of the scene change based on this embodiment. It is a figure which illustrates the co-occurrence histogram of a frame image and a spatiotemporal change feature-value based on this embodiment. It is a flowchart which shows the procedure of the process for estimating the presence or absence of a scene change based on this embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a configuration of an image processing apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 1 according to the present embodiment includes a learning unit 10 and an estimation unit 10 ′, and an estimation result regarding the presence or absence of a scene change in video data including data on a plurality of consecutive frame images. Is output. The learning unit 10 includes an acquisition unit 11, a first feature extraction unit 12, and a scene change learning unit 17, and outputs a learning result for detecting a scene change learned from the input learning video data. The first feature extraction unit 12 includes a spatio-temporal feature extraction unit 13, a distance calculation unit 14, an inter-frame correlation value calculation unit 15, and a screen luminance average difference calculation unit 16. From each frame image included in the learning video, A plurality of types of features (feature quantities or feature vectors) are extracted.

  The estimation unit 10 ′ includes an acquisition unit 11 ′, a second feature extraction unit 12 ′, and a scene change estimation unit 17 ′. Based on the learning result by the scene change learning unit 17, the scene change is displayed from the display video data. Presence / absence is estimated and the estimation result is output. Similar to the first feature extraction unit 12, the second feature extraction unit 12 ′ is a spatio-temporal feature extraction unit 13 ′, a distance calculation unit 14 ′, an inter-frame correlation value calculation unit 15 ′, and a screen luminance average difference calculation unit 16 ′. Multiple types of features (features or feature vectors) are extracted from each frame image included in the display video.

  Hereinafter, each configuration of the image processing apparatus 1 will be described in detail.

  The acquisition unit 11 acquires learning video data prepared for the learning unit 10 to learn whether there is a scene change from the outside. The acquisition unit 11 functions as a first acquisition unit.

  The first feature extraction unit 12 extracts a plurality of types of features from each frame image included in the learning video acquired by the acquisition unit 11 and outputs the extracted features to the scene change learning unit 17. The multiple types of features will be described later.

  A spatiotemporal feature extraction unit (hereinafter referred to as an extraction unit) 13 compares each frame image included as image data in the acquired learning video data with each other, and compares the spatiotemporal change in luminance with a feature amount (first feature). Amount) and output to the scene change learning unit 17. Details will be described later with reference to FIG.

  The distance calculation unit 14 creates a color histogram for each frame image, and calculates a distance (similarity) between the frame images based on the created color histogram. The method for calculating the distance between the frame images includes a method using a correlation value or a Manhattan distance. In this embodiment, for example, an Intersection distance expressed by the following Equation 1 is used.

  Here, D is a distance between frame images, Y, Cr, and Cb are each color in the frame image, N is the number of bins of the color histogram H, and H (b, t) is a bin b of the color histogram of the frame image at time t. , Min indicates the smaller value of the bin values of the color histogram of the frame image at times t and t−1. Thereby, the distance calculation unit 14 calculates the similarity D between the frame images, and outputs this to the scene change learning unit 17 as a feature amount (second feature amount).

  An inter-frame correlation value calculation unit (hereinafter referred to as a correlation value calculation unit) 15 calculates a correlation value (correlation) indicating the degree of correlation between frame images. Specifically, the correlation value calculation unit 15 first calculates, for example, a luminance difference S between adjacent blocks (hereinafter referred to as adjacent blocks) in one frame image. For example, when the horizontal coordinate in the frame image is x and the vertical coordinate is y, the difference in luminance I between adjacent blocks (blocks in [x, y] and [x + 1, y]) is expressed by the following equation: 2 is calculated.

  Then, the correlation value calculation unit 15 linearly converts the calculated numerical value S into the numerical value E so that the numerical value E from 0 to 1 is obtained. For example, the numerical value E of the frame image at time t is calculated by the following mathematical formula 3.

  Then, the correlation value calculation unit 15 calculates a correlation value (Correlation) between the frame images based on the absolute value of the difference of the numerical value E with respect to the frame images at different times t and t−1 according to the following mathematical formula 4.

  The correlation value calculation unit 15 outputs the correlation value (Correlation) calculated in this way to the scene change learning unit 17 as a feature amount (third feature amount) regarding the degree of correlation between frame images.

  A screen luminance average difference calculation unit (hereinafter referred to as an average difference calculation unit) 16 calculates an average luminance value for each frame image, and calculates an average luminance difference between the frame images. The average difference calculation unit 16 outputs the difference thus calculated to the scene change learning unit 17 as a feature amount (fourth feature amount).

  The scene change learning unit 17 includes a spatio-temporal change feature amount (first feature amount) from the extraction unit 13, an inter-image distance (second feature amount) from the distance calculation unit 14, and a correlation value from the correlation value calculation unit 15 (second feature amount). The scene change is learned based on the third feature amount) and the difference of the average luminance (fourth feature amount) from the average difference calculation unit 16. Specifically, the scene change learning unit 17 performs learning that associates a plurality of types of feature amounts (first to fourth feature amounts) obtained from the first feature extraction unit 12 with correct data. That is, the scene change learning unit 17 generates a learning model (discriminator) for determining the presence / absence of a scene change from a plurality of types of feature amounts obtained from the first feature extraction unit 12. For example, the scene change learning unit 17 adopts a mathematical model having at least one parameter (variable) as a learning model, and when each feature quantity (first to fourth feature quantities) is input, The parameter value of the mathematical model is determined so as to output the value (presence / absence of scene change). The parameter value determined here is stored as a learning result (learning parameter).

  The estimation unit 10 'will be described later.

  Next, with reference to FIG. 2 and FIG. 3, the detailed structure and each function of the extraction part 13 are demonstrated in detail.

  FIG. 2 is a block diagram showing the configuration of the spatiotemporal feature extraction unit, and FIG. 3 is a diagram for explaining the feature quantity of the spatiotemporal change.

  As shown in FIG. 2, the extraction unit 13 includes a block division unit 13a, a temporal difference calculation unit 13b, a ratio calculation unit 13c, a co-occurrence histogram creation unit 13d, and a one-dimensionalization processing unit 13e. The extraction unit 13 includes a spatial difference calculation unit 13f, a difference calculation unit 13g, a ratio calculation unit 13h, a co-occurrence histogram creation unit 13i, and a one-dimensional processing unit 13j. Furthermore, the extraction unit 13 includes a combining unit 13k and a buffer for temporarily storing a part of the frame image, numerical values, and the like.

  The block dividing unit 13a divides the input frame image into blocks having a predetermined size. The divided block images are passed to the temporal difference calculation unit 13b and the spatial difference calculation unit 13f.

  The temporal difference calculation unit 13b calculates a luminance difference T between corresponding blocks in temporally continuous frame images. For example, as illustrated in FIG. 3, the temporal difference calculation unit 13b, between the frame images at time t and t−1, the luminance I difference T (x, y, t) between the blocks at the position [x, y]. ) (= I (x, y, t) −I (x, y, t−1)). T is calculated for all blocks in the frame image. Note that a buffer is appropriately used to temporarily store the luminance of the block in the temporally preceding frame image.

Ratio calculating unit 13c calculates a parameter L _T representing the time rate of change of T calculated for different time of the frame image. Specifically, the ratio calculation unit 13c sets T (x, y, t-1) calculated at time t-1 immediately before time t for the block located at [x, y] at time t. By dividing by the calculated T (x, y, t), L _T (= T (x, y, t−1) / T (x, y, t)) is calculated. Here, the value of L _T, for example, are quantized to a number up to the first decimal place. Also, if the denominator is 0 (zero) or very small values, L _T may be output as an upper limit value is appropriately set. In addition, a buffer is appropriately used to store T calculated in advance in time.

The co-occurrence histogram creation unit 13d creates a co-occurrence histogram based on the calculated value. In the present embodiment, the co-occurrence histogram is, for example, a two-dimensional histogram in which the total number (number of votes) of the same combination (co-occurs) among the combinations of two variables is the bin value. For example, co-occurrence histogram creation unit 13d, and classifies each block in accordance with a combination of the values of T and L _T that is calculated for each block by the time difference calculating portion 13b and the ratio calculation unit 13c, the same classification A co-occurrence histogram is created by accumulating (voting) the number of blocks and using the number of votes as a bin value. Details of the co-occurrence histogram will be described later.

The one-dimensionalization processing unit 13e makes one-dimensional information representing the generated co-occurrence histogram. Specifically, one-dimensional processing unit 13e, for example, the co-occurrence histograms and variables T and L _T, T, by converting the individual values of L _T and co-occurrence probability enumerated information in a predetermined order Convert to one-dimensional information.

  The spatial difference calculation unit 13f calculates a luminance difference between adjacent blocks in one frame image. For example, as shown in FIG. 3, the spatial difference calculation unit 13f, as shown in FIG. 3, the difference S (x, y, y) of the luminance I between blocks in [x, y] and [x + 1, y] in the frame image at time t. t) (= I (x + 1, y, t) −I (x, y, t)) is calculated. S is calculated between all adjacent blocks. A buffer is used as appropriate to temporarily store the brightness of the block.

  The difference calculation unit 13g calculates the difference of S calculated by the spatial difference calculation unit 13f. Specifically, the difference calculation unit 13g calculates the same position at time t-1 from the luminance difference S (x, y, t) between the blocks at position [x + 1, y] and [x, y] at time t. The luminance difference S (x, y, t−1) between the blocks is subtracted to obtain U (x, y, t) (= S (x, y, t) −S (x, y, t−1)). Is calculated. Note that a buffer is appropriately used to temporarily store S calculated in the preceding time.

The ratio calculation unit 13h calculates a parameter L _U that represents the time change rate of U calculated for different times. Specifically, the ratio calculation unit 13h calculates the difference U (x, y, t−) between the times t−2 and t−1 of the luminance difference S between the blocks [x, y] and [x + 1, y]. 1) is divided by the difference U (x, y, t) between the times t−1 and t of the luminance difference S between the same blocks, and L _U (= U (x, y, t−1) / U (x, y, t)) is calculated. Here, the value of L _U, like the above-mentioned L _T, may be quantized to a number up to the first decimal. Also, if the denominator is 0 (zero) or very small values, L _U may be output as an upper limit value is appropriately set. In addition, a buffer is appropriately used to temporarily store U calculated in the preceding time.

The co-occurrence histogram creation unit 13i creates a co-occurrence histogram similarly to the above-described co-occurrence histogram creation unit 13d. For example, co-occurrence histogram creation section 13i in accordance with the combination of the values of U is calculated for each neighboring block and L _U by the difference calculation section 13g and the ratio calculation unit 13h, classifies each adjacent block, the same classification and The number of adjacent blocks is accumulated (voted), and a two-dimensional histogram is created with the number of votes as a bin value. Details of the co-occurrence histogram will be described later.

  The one-dimensionalization processing unit 13j converts the created co-occurrence histogram into one-dimensional information in the same manner as the one-dimensionalization processing unit 13e described above.

  The combination processing unit 13k combines the information one-dimensionalized by the one-dimensionalization processing unit 13e and the one-dimensionalization processing unit 13j, and outputs the combined information as a spatiotemporal change feature amount (first feature amount).

  Next, the estimation unit 10 ′ (see FIG. 1) will be described in detail.

  The estimation unit 10 'acquires display video data, estimates a scene change in the video data, and outputs an estimation result. Of the acquisition unit 11 ′, the second feature extraction unit 12 ′, the spatio-temporal feature extraction unit 13 ′, the distance calculation unit 14 ′, the inter-frame correlation value calculation unit 15 ′, and the screen luminance average difference calculation unit 16 ′ of the estimation unit 10 ′. Since the functions are the same as the functions of the corresponding units of the learning unit 10 described above, description thereof is omitted to avoid duplication.

  The scene change estimator 17 ′ is for display video data calculated by the spatiotemporal feature extractor 13 ′, the distance calculator 14 ′, the inter-frame correlation value calculator 15 ′, and the screen luminance average difference calculator 16 ′. Based on the feature amount and the learning result (learning model) acquired by the learning unit 10, the presence / absence of a scene change in the display video data is estimated.

  The scene change estimation unit 17 ′ can identify the type of scene change based on the feature amounts S and T calculated by the spatiotemporal feature extraction unit 13 ′. For example, it is possible to detect a hard cut that represents a sudden change in the scene and a dissolve that gradually changes the scene. Specifically, the scene change estimator 17 ′ calculates T (x, y, t) ≠ T (x, y, t−1) and S (x, y, t) ≠ S (x, y, t−). In the case of 1), it is determined that it is a hard cut, and T (x, y, t) ≈T (x, y, t−1) and S (x, y, t) ≈S (x, y, t− In the case of 1), it is determined to be a dissolve. At this time, the scene change learning unit 17 is configured to learn the type of scene change.

  The above-described units of the image processing apparatus 1 are controlled by a control unit (not shown). For example, the control unit is realized by a CPU (Central Processing Unit) reading a program installed in a storage into a memory and executing the program.

  Next, with reference to FIG. 4 and FIG. 5, a process procedure for learning the presence / absence of a scene change by the image processing apparatus 1 according to the present embodiment will be described in detail.

  FIG. 4 is a flowchart showing a processing procedure for learning the presence / absence of a scene change according to the present embodiment, and FIG. 5 is a diagram illustrating a co-occurrence histogram of a frame image and a spatiotemporal variation feature amount according to the present embodiment. It is.

  As shown in FIG. 4, first, learning video data is acquired (step S1). In this step, the acquisition unit 11 of the learning unit 10 acquires video data for learning.

Subsequently, the feature (the first feature amount) of the temporal and spatial change of the learning video is extracted (step S2). In this step, the spatiotemporal feature extraction unit 13 extracts a spatiotemporal change feature amount for the learning video data acquired in step S1. Specifically, as described above, the spatio-temporal feature extraction unit 13 calculates T and S for each frame image included in the learning video, calculates U based on S, and based on T and U L _T and L _U are calculated. Then, to create a co-occurrence histogram based on the T and _{L T} and U and _{L U.}

Examples of temporally continuous frame images included in the learning video are as shown in FIGS. 5 (A) to 5 (C). A frame image (for example, an image at time t) shown in FIG. 5A and a frame image (not shown) that precedes in time in FIG. 5A (for example, images at time t-1 and time t-2). Are used to calculate parameters T, S, U, L _T and L _{U at} time t. Then, a two-dimensional histogram is created for the co-occurrence probabilities of the combination of T and L _{T and} the combination of U and L _U. For example, indicating the number of votes of the combination of T and L _T, co-occurrence histogram is generated as shown in FIG. 5 (D). The co-occurrence histogram indicates that there are approximately 1400 blocks with respect to the frame image in FIG. 5A, where (0 ≦ T <1, 0 ≦ L _T <0.25). This is brightness difference T between the frame images and a small than 1, the amount of change T is increased As the elapsed time (L _T is low) because.

Similarly, co-occurrence histogram shown in FIG. 5 (E), the combination of values of the calculated T and L _T by using the frame image shown in FIG. 5 (B), and a frame image temporally preceding thereto Indicates the co-occurrence probability. In the co-occurrence histogram, with respect to the frame image of FIG. 5B, the number of blocks satisfying 0 ≦ T <4 is increased, and a block having a large difference value T between temporally consecutive frame images is illustrated in FIG. It can be seen that there is a tendency to be larger than the case of the frame image of A). This is at the same time as the brightness difference T is relatively large and two or more, the amount of change T is increased As the elapsed time (L _T is low) because.

Co-occurrence histogram of FIG. 5 (F) also show a frame image shown in FIG. 5 (C), the co-occurrence probabilities of the combination of the values of the calculated T and L _T by using the frame image which precedes it Yes. In the co-occurrence histogram, the value of the co-occurrence histogram is wide in the range of (0 ≦ T <1, 0 ≦ L _T <1.0) with respect to the frame image in FIG. It can be seen that the luminance difference T increases in a range where the temporal change rate LT of _T is relatively large. This is because there are blocks that change variously in a range where the luminance difference T is 0 to 9 or less, but the change amount of T is uniform over time on the entire screen (L _T is 1). It is close to).

The spatio-temporal feature extraction unit 13 similarly creates a co-occurrence histogram (not shown) indicating the number of votes of a combination of U and L _U for each frame image. Then, as described above, the spatiotemporal feature extraction unit 13 unifies the information representing the co-occurrence histogram and outputs the integrated information as the spatiotemporal change feature amount.

  Returning to FIG. 4, after step S <b> 2, a feature amount (the above-described second feature amount) for the similarity is extracted (step S <b> 3). In this step, the distance calculation unit 14 described above calculates the similarity between temporally consecutive frame images in the learning video acquired in step S1, and outputs the similarity as a feature amount.

  Subsequently, a feature amount (the above-described third feature amount) regarding the degree of correlation is extracted (step S4). In this step, the correlation value calculation unit 15 described above calculates the degree of correlation between temporally continuous frame images in the learning video acquired in step S1, and outputs it as a feature amount for the degree of correlation.

  Subsequently, a feature amount (the above-described fourth feature amount) regarding the difference in average screen luminance is extracted (step S5). In this step, the above-described average difference calculation unit 16 calculates a difference in average luminance between temporally continuous frame images in the learning video acquired in step S1, and outputs the difference as a feature amount for the luminance average difference. .

  Subsequently, a learning parameter for determining the presence or absence of a scene change is generated from a plurality of types of feature amounts (first to fourth feature amounts) extracted from the first feature extraction unit 12 (step S6). In this step, the scene change learning unit 17 learns whether or not there is a scene change based on the feature amounts for all the frame images in the learning video acquired in step S1 and the correct data acquired separately. Then, the parameter value of the learning model is determined so that the correct data value (presence / absence of scene change) is output when each feature value (first to fourth feature values) is input to the learning model. The parameter value determined here is output and stored as a learning result (learning parameter).

  Thereafter, the processing procedure for learning the presence / absence of a scene change ends.

  Next, with reference to FIG. 6, a procedure of processing for estimating a scene change by the image processing apparatus 1 according to the present embodiment will be described in detail.

  FIG. 6 is a flowchart illustrating a procedure of processing for estimating the presence / absence of a scene change according to the present embodiment.

  As shown in FIG. 6, first, display video data is acquired (step S11). In this step, the acquisition unit 11 ′ of the estimation unit 10 ′ acquires display video data.

  Subsequently, the feature (the first feature amount) of the temporal and spatial change of the display video is extracted (step S12). In this step, the spatiotemporal feature extraction unit 13 'extracts a spatiotemporal change feature amount for the display video data acquired in step S11. This step is the same as step S2 executed by the spatiotemporal feature extraction unit 13.

  Subsequently, a feature amount (the above-described second feature amount) regarding the similarity is extracted (step S13). In this step, the distance calculation unit 14 ′ calculates the similarity between the frame images included in the display video acquired in step S <b> 11 and outputs it as a feature amount for the similarity. This step is the same as step S3 executed by the distance calculation unit 14.

  Subsequently, the feature amount (the above-described third feature amount) regarding the degree of correlation is extracted (step S14). In this step, the correlation value calculation unit 15 ′ calculates the degree of correlation between the frame images included in the display video acquired in step S <b> 11 and outputs it as a feature amount for the degree of correlation. This step is the same as step S4 executed by the correlation value calculation unit 15.

  Subsequently, a feature amount (the above-described fourth feature amount) regarding the difference in average screen luminance is extracted (step S15). In this step, the above-described average difference calculation unit 16 ′ calculates a difference in average luminance between temporally continuous frame images in the display video acquired in step S11, and outputs it as a feature amount for the luminance average difference. To do. This step is the same as step S5 executed by the average difference calculation unit 16.

  Subsequently, the presence / absence of a scene change is estimated (step S16). In this step, the feature amount (first to fourth extraction amounts) extracted from the frame image in the display video acquired in step S11 is input to the learning model based on the learning result generated in step S6. Thus, the scene change estimation unit 17 ′ estimates the presence / absence of a scene change in the display video data and outputs the estimation result.

  Thereafter, the processing procedure for estimating the presence / absence of a scene change ends.

As described above, according to the present embodiment, the scenes are determined based on the co-occurrence probabilities for the parameters S T, U and T based on the luminance difference values between the blocks in the frame image, and the parameters L _T and L _U based on their ratio. Changes can be detected with high accuracy and at the same time. Furthermore, since these parameters can be calculated relatively easily, the presence / absence and type of a scene change can be detected at a higher speed.

  Although the embodiment to which the present invention is applied has been described above, it goes without saying that various modifications are possible within the scope of the technical idea described in the claims of the present application.

  For example, the size of the block is not limited to the illustrated size. Can be set arbitrarily. You may calculate a parameter for every pixel instead of a block. Alternatively, the frame image may be reduced, that is, the data amount of the frame image may be reduced, and the luminance may be calculated based on the data to extract various feature amounts.

  In the flowchart of FIG. 4, the mode of extracting the feature quantity regarding the difference between the spatio-temporal change feature quantity, the similarity degree, the correlation degree, and the screen luminance average in steps S2 to S5 and S12 to S15 is described. Not. Other feature quantities for detecting a scene change may be additionally extracted.

  Further, in the co-occurrence histograms of FIGS. 5D to 5F, only values of 0 (zero) or more are shown for convenience, but the present invention is not limited to this. May contain negative values.

  Moreover, in the said embodiment, although parameters, such as S and T, were calculated based on the difference of a brightness | luminance, it is not limited to this. Each parameter may be calculated based on a value such as brightness or saturation of each color in the frame image or a difference in color temperature.

  The means and method for performing various processes in the image processing apparatus according to the present embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided by a computer-readable recording medium such as a memory stick and a CD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. The program may be provided as a single application software, or may be incorporated in the software of the apparatus as one function of the image forming system.

1 image processing device,
10 Learning Department,
11 Acquisition Department,
12 1st feature extraction part,
13 Spatio-temporal feature extraction unit,
13a Block division unit,
13b Temporal difference calculation unit,
13c, 13h ratio calculation unit,
13d, 13i co-occurrence histogram generator,
13e, 13j one-dimensional processing unit,
13f spatial difference calculation unit,
13g difference calculation unit,
13k joint processing unit,
14 Distance calculator,
15 inter-frame correlation value calculator,
16 screen brightness average difference calculation unit,
17 Scene Change Learning Department,
10 'estimator,
11 'acquisition unit,
12 '2nd feature extraction part,
13 'spatiotemporal feature extraction unit,
14 'distance calculation part,
15 'interframe correlation value calculator,
16 'screen brightness average difference calculation section,
17 'Scene change estimation part.

Claims (15)

A first acquisition unit that acquires learning video data including data on a plurality of consecutive frame images;
A first feature extraction unit for extracting, from the learning video data, a feature of temporal and spatial changes of the video as a first feature;
A learning unit that learns whether or not scenes are switched between successive frame images using the first feature;
A second acquisition unit that acquires display video data including data on a plurality of consecutive frame images;
A second feature extraction unit for extracting, as a second feature, a feature of temporal and spatial changes of the video from the display video data;
Based on the learning result obtained by the second feature and the learning unit, an estimation unit that estimates the presence or absence of scene switching between successive frame images;
Have
The first feature and the second feature are: a luminance or color difference value between frame images and a co-occurrence probability of a temporal change rate of the difference value, and a luminance or color difference value between adjacent blocks in the frame image. And an image processing apparatus including at least one of the co-occurrence probabilities of the time change rate of the difference value. The first feature extraction unit extracts a feature based on the similarity between frame images from the learning video data as a third feature amount,
The image processing apparatus according to claim 1, wherein the learning unit learns whether or not scenes are switched between consecutive frame images by further using the third feature amount. The first feature extraction unit extracts a feature based on a correlation degree between frame images from the learning video data as a fourth feature amount,
The image processing apparatus according to claim 1, wherein the learning unit learns whether or not scenes are switched between consecutive frame images by further using the fourth feature amount.   The image processing apparatus according to claim 1, wherein the learning unit identifies a type of scene switching for learning video data having the first feature. The first feature extraction unit reduces the learning video data, extracts a temporal and spatial change feature of the video from the reduced learning video data as a first feature,
The said 2nd feature extraction part reduces the said video data for a display, and extracts the characteristic of the spatio-temporal change of a video as a 2nd feature from the reduced video data for a display. An image processing apparatus according to 1. Acquiring learning video data including data on a plurality of consecutive frame images;
(B) extracting a feature of temporal and temporal changes of the video from the learning video data as a first feature;
Learning whether or not there is a scene change between successive frame images using the first feature;
Obtaining video data for display including data on a plurality of continuous frame images (d);
(E) extracting from the display video data a feature of temporal and spatial changes of the video as a second feature;
Estimating the presence or absence of scene switching between successive frame images based on the second feature and the learning result obtained in step (c);
Have
The first feature and the second feature are: a luminance or color difference value between frame images and a co-occurrence probability of a temporal change rate of the difference value, and a luminance or color difference value between adjacent blocks in the frame image. And an image processing method including at least one of the co-occurrence probabilities of the time change rate of the difference value. A step (g) of extracting a feature based on a similarity between frame images as a third feature amount from the learning video data;
The image processing method according to claim 6, wherein in step (c), the presence or absence of scene switching between successive frame images is learned by further using the third feature amount. A step (h) of extracting a feature based on the degree of correlation between the frame images from the learning video data as a fourth feature amount;
The image processing method according to claim 6 or 7, wherein in the step (c), the presence or absence of scene switching between successive frame images is learned by further using the fourth feature amount.   The image processing method according to any one of claims 6 to 8, further comprising a step (i) of identifying a type of scene change for the learning video data having the first feature. Before the step (b), there is a step of reducing the learning video data. In the step (b), a feature of temporal and temporal changes of the video is extracted from the reduced learning video data as a first feature. ,
The display video data is reduced before the step (e), and a feature of temporal and spatial changes of the video is extracted as a second feature from the reduced display video data in the step (e). The image processing method as described in any one of -9. A procedure (a) for acquiring video data for learning including data on a plurality of continuous frame images;
A procedure (b) for extracting, from the learning video data, a feature of temporal and spatial changes of the video as a first feature;
A procedure (c) for learning whether or not there is a scene change between successive frame images using the first feature;
A procedure (d) of acquiring display video data including data on a plurality of consecutive frame images;
A procedure (e) for extracting, from the display video data, a feature of temporal and spatial changes of the video as a second feature;
A procedure (f) for estimating the presence or absence of a scene change between successive frame images based on the learning result obtained in the second feature and the procedure (c);
In an image processing program for causing a computer to execute a procedure including:
The first feature and the second feature are: a luminance or color difference value between frame images and a co-occurrence probability of a temporal change rate of the difference value, and a luminance or color difference value between adjacent blocks in the frame image. And an image processing program including at least one of the co-occurrence probabilities of the time change rate of the difference value. A step (g) of extracting a feature based on the similarity between frame images as a third feature amount from the learning video data;
The image processing program according to claim 11, wherein in the step (c), the presence or absence of a scene change between consecutive frame images is learned by further using the third feature amount. A step (h) of extracting a feature based on the degree of correlation between the frame images from the learning video data as a fourth feature amount;
The image processing program according to claim 11 or 12, wherein in the step (c), the presence or absence of a scene change between successive frame images is learned by further using the fourth feature amount.   The image processing program according to any one of claims 11 to 13, further comprising a procedure (i) for identifying a type of scene switching for the learning video data having the first feature. Before the step (b), there is a step of reducing the learning video data. In the step (b), a feature of temporal and temporal changes of the video is extracted from the reduced learning video data as a first feature. ,
12. The display video data is reduced before the procedure (e), and the temporal and spatial change characteristics of the video are extracted as second characteristics from the reduced display video data in the procedure (e). The image processing program as described in any one of -14.