RU2526049C2 - Method and apparatus for detecting game incidents in field sports in video sequences - Google Patents

Abstract

FIELD: physics, video.SUBSTANCE: invention relates to video processing techniques and specifically to automation of the process of classifying video content by genre and real-time content. The result is achieved by calculating brightness and saturation of each frame pixel described by values in red, green and blue colour channels; calculating the absolute value of brightness gradient for each frame pixel; performing colour classification of each frame pixel; calculating statistics based on results of colour classification for the whole frame; calculating statistics based on results of colour classification for green frame regions; determining if said frame is a game incident in field sports based exclusively on characteristics of the current frame; determining if the current frame belongs to the same scene as the previous frame; using the detection result obtained for the previous frame as a refined detection result if the current frame belongs to the same scene as the previous frame of the video sequence; or using the detection result obtained for the current frame based exclusively on characteristics of the current frame as the refined detection result if the current frame does not belong to the same scene as the previous frame of the video sequence.EFFECT: automatic adjustment of image settings.10 cl, 29 dwg

Description

The present invention relates to video processing technologies, namely, to automate the process of classifying video content by genre and content in real time in order to optimally select display device parameters.

The prior art various approaches to the automatic determination of the genre of a video sequence. Theoretical aspects of developments in this area are described in a number of publications, such as N. Watcharapinchai, S. Aramvith, S. Siddhichai, S. Marukatat, S., A discriminant approach to sports video classification, proc. of the ISCIT '07. International Symposium on Communications and Information Technologies, pp.557-561, 2007 [1], J. Park, S. Han, Y. An, Heuristic Features for Color Correlogram for Image Retrieval, proc. of the ICCSA'08. International Conference on Computational Sciences and Its Applications, pp. 9-13, 2008 [2], Automatic feature construction and a simple rule induction algorithm for skin detection, G. Gomez, E. Morales Proceedings of Workshop on Machine Learning in Computer Vision, 2002, pp.31-38 [3].

Practical developments in the field of automatic determination of the genre of video sequences are reflected in a number of patent documents.

So, in US patent 7831112 [4] a method for dividing a video stream into time segments is proposed depending on the cries of the audience present at a sporting event. This invention relates to the determination of a genre of multimedia content and consists of a feature detector calculating predetermined features of multimedia content and a classification unit.

US patent application 20070113248 [5] describes a method and apparatus for defining a genre of multimedia content. The device includes a feature detector and a genre detector, while for solving the problem, both features highlighted based on the audio stream only and features highlighted based on the video stream are used.

Closest to the claimed invention is a solution known from US patent application 20080186413 [6] and describing the video processing pipeline in a television receiver, customizable depending on the genre of the broadcast being watched. The authors propose using various settings of contrast, gain level of details, spatio-temporal noise reduction, gamma correction, overdrive and backlight depending on the genre of the displayed video image. The following genres are considered as examples in the prototype solution: sports, music, studio broadcasts, films, television films, cartoons. The classification block in the prototype [6] consists of a histogram generator, a maximum gray gradation detector, a medium gray gradation detector, a minimum gray gradation detector, and a medium brightness detector.

When choosing the optimal approach to solving the problem of automatically determining the genre of video content, it is necessary to take into account the following features of this problem.

If the genre is determined solely on the basis of features extracted from the audio stream, the solution to the problem cannot be obtained with an accuracy of one frame, which is an important requirement when using the genre detection algorithm to control the settings of the display device.

In the case of fast movement within the video frame, the histogram can change so dramatically that for neighboring frames of the same video sequence, different classification results can be obtained.

Video images are usually subjected to different types of preprocessing, such as improved contrast, which can change the histogram, which may result in different classification results for the same frames of the video sequence subjected to different types of preprocessing. Video images can be processed by compression algorithms of various types and with various quality settings, which can also significantly affect image characteristics.

Methods for detecting a genre based on dynamic features (such as interframe brightness variation) or on a statistical analysis of low-level features within a scene allow you to obtain a classification result only after receiving the last frame of this scene. In the case of adaptive control of the settings of the image device, this type of algorithm would cause a large (and depending on the particular scene of the image) delay, which is unacceptable for an application in a television receiver.

Thus, methods for determining the genre using features extracted from the audio stream cannot be used in applications for managing collections of multimedia content in which audio and video data are stored separately.

Artificial intelligence methods require retraining of the algorithm when new erroneously classified samples appear and are added to the training set.

The problem to which the claimed invention is directed is to develop an improved method for detecting episodes associated with the demonstration of game sports in real-time video sequences with the aim of automatically adjusting image settings in a television receiver. Moreover, such a method should be free from the main disadvantages inherent in solutions known from the prior art.

The technical result is achieved by developing a method for detecting game episodes in field sports in real time, which consists in performing the following operations:

- calculate the brightness and saturation of each pixel in the frame described by the values in the red, green and blue color channels;

- calculate the absolute value of the brightness gradient for each pixel of the frame;

- perform classification by color of each pixel of the frame;

- calculate statistics according to the results of the classification by color for the entire frame;

- calculate statistics according to the results of the classification by color for the green areas of the frame;

- determine whether a given frame is a game episode in field sports based solely on the characteristics of the current frame;

- determine whether the current frame belongs to the same scene as the previous frame;

- use as a refined detection result the detection result obtained for the previous frame, if the current frame belongs to the same scene as the previous frame of the video sequence; or

- use as a refined detection result the detection result obtained for the current frame based solely on the characteristics of the current frame, if the current frame does not belong to the same scene as the previous frame of the video sequence.

The inventive method allows to obtain a classification result for each frame of a video sequence, while the result is constant throughout the scene. The method works in real time (the delay time corresponds to the reception time of one frame and ranges from 40 to 67 ms). The method is resistant both to fast movement in the frame and to preliminary processing (for example, gamma correction or local improvement in contrast). In addition, the classifier is a set of simple rules that are understood by man, so when erroneously classified samples appear, retraining of the existing classifier is not required, but adding a new rule is enough. The classifier is a superposition of one-dimensional and two-dimensional threshold functions, a linear classifier and logical functions, and uses very simple statistical features that allow hardware implementation using shift blocks, adders and comparators.

Compared with existing methods, the claimed invention has six main differences:

- detection is performed for each frame of the video sequence separately, but at the same time the temporal smoothness of the detection result within the same scene is maintained;

- detection is based on signs empirically understood by a person;

- Four new types of color detectors have been proposed that detect colors in a manner similar to human perception: yellow, green, white, bright and saturated color;

- four types of low-level statistical features used in the classification are proposed: the average value of the brightness gradient in green areas, the compactness of the histogram of the green color channel in green areas, the average brightness in green areas, the average value of the blue color channel in green areas;

- the classifier has the form of a directed acyclic graph, in the nodes of which are one-dimensional and two-dimensional threshold functions, linear classifiers and logical functions;

- a new type of scene change detector is proposed, based on the k-means segmentation algorithm.

For a better understanding of the claimed invention the following is a detailed description with the corresponding drawings.

Figure 1. Graph diagram of a method for detecting game episodes.

Figure 2. Examples of game episodes in field sports with varying amounts of green pixels.

Figure 3. Saturation of green pixels as a sign of field play.

Figure 4. The image from Figure 3, view 3.3, in which the average values of the red, green and blue color channels in the green areas are set equal to the average values of these values in the green areas in Figure 3, view 3.4.

Figure 5. The image from Figure 3, view 3.4, in which the average value of the blue color channel in the green areas is set equal to the average value of this value in the green areas in Figure 3, view 3.3.

6. The histogram of the green color channel in the green areas of the image for the images in Figure 3, view 3.3, and Figure 3, view 3.4; the brightness gradient in green areas for the images in Figure 3, view 3.3 and Figure 3, view 3.4.

7. An episode of a field game with a zero proportion of green pixels.

Fig. 8. Additional classification according to the proportion of green pixels and the saturation of green pixels.

Fig.9. A scene in a room that is not a game episode in a field game: view 9.1 is an example of a frame, view 9.2 is a feature graph and classification result for a scene consisting of 300 consecutive frames.

Figure 10. A scene on the street that is not a game episode of a field game: view 10.1 - frame example, view 10.2 - feature chart and classification result for a scene consisting of 350 consecutive frames

11. A scene that is not a game episode of a field game containing a large proportion of green pixels: view 11.1 is an example of a frame, view 11.2 is a feature graph and classification result for a scene consisting of 300 consecutive frames.

Fig. 12. A game episode of a field game containing a long-range plan: view 12.1 is an example of a frame, view 12.2 is a feature graph and classification result for a scene consisting of 350 consecutive frames.

Fig.13. A close-up gaming episode of a field game; view 13.1 is an example of a frame, view 13.2 is a graph of features and a classification result for a scene consisting of 350 consecutive frames.

Fig.14. Game episodes alternating with non-game episodes: view 14.1 is an example of a frame that is not a game episode, view 14.2 is a feature graph and classification result for a video sequence consisting of 850 consecutive frames (3 scenes).

Fig.15. Block diagram of a game episode detector for field games.

Fig.16. Block diagram of a low level feature detector.

Fig.17. A way to implement the color classifier of pixels.

Fig. 18. The area corresponding to white in the RGB color space.

Fig.19. An example of white pixel detection results.

Fig.20. The area corresponding to skin tones in the RGB color space.

Fig.21. The area corresponding to yellow in the RGB color space.

Fig.22. An example of yellow pixel detection results.

Fig.23. The area corresponding to green in the RGB color space.

Fig.24. Example of green pixel detection results.

Fig.25. A measure of the compactness of a histogram.

Fig.26. Block diagram of the feature analyzer.

Fig.27. A block diagram of a scene change detector.

Fig.28. Block diagram of a dynamic game episode detector.

Fig.29. Possible application in the field of television.

A method for detecting game episodes of field sports consists of the following steps shown in FIG. 1:

- for each pixel described by the values in the red, green and blue color channels, brightness and saturation are calculated (step 101);

- calculate the absolute value of the brightness gradient (step 102)

- classify each pixel by color, namely: (step 103)

- is the pixel white?

- is the pixel bright and saturated?

- is the pixel yellow?

- is the pixel green?

- is the pixel color a skin tone?

- calculate color statistics for the entire frame (step 104);

- the proportion of white pixels in the frame

- the proportion of bright and saturated pixels in the frame

- the proportion of green pixels in the frame

- the proportion of pixels in the skin tone in the frame

- calculate color statistics for green areas (step 105); namely, for all pixels that were classified as green in step 103, the following characteristics are calculated:

- average brightness

- average saturation

- average value of the blue color channel

- average value of the absolute value of the brightness gradient

- for pixels classified as green, a histogram of the green color channel is calculated (step 106)

- to determine whether the current frame is a game episode of a field game, the following rules are used (step 107):

- if the proportion of green pixels is zero or very low, then the detection result is negative;

- if the average brightness of green pixels is very low, then the detection result is negative;

- if the average saturation of green pixels is very low, then the detection result is negative;

- if the average saturation of green pixels is low and the average brightness of green pixels is low, then the detection result is negative;

- if the average saturation of green pixels is average and the histogram of the green channel in the green areas is wide, then the detection result is negative;

- if the average saturation of green pixels is very large and the histogram of the green channel in the green areas is very narrow, then the detection result is negative;

- if the average saturation of green pixels is low or high and the average value of the absolute value of the brightness gradient in green areas is very low, then the detection result is negative;

- if the proportion of green pixels is medium or small and the proportion of bright and saturated pixels is medium or high, and the number of pixels of a human skin tone above zero is small or medium, then the detection result is positive;

- if the proportion of green pixels is large and the proportion of bright and saturated pixels is small, but greater than zero, then the detection result is positive;

- to determine whether the current frame belongs to the same scene as the previous one, a scene change is detected (step 108),

- if the scene change has not occurred, the detection result is set equal to the detection result for the previous frame (step 109).

The following explains the details of the proposed method.

The principle of detecting game episodes of field games is based on the observation that, in comparison with other types of scenes, the proportion of green pixels in these frames is relatively large.

The detector proposed on the basis of the method makes it possible to distinguish game episodes even if this proportion varies greatly (Figure 2: view 2.1 - 15%, view 2.2 - 24%, view 2.3 - 46%, view 2.4 - 67%, view 2.5 - 82%, species 2.6 - 97%).

It is assumed that usually in the game episodes of field games the green saturation S _{GR is} relatively high (Figure 3, view 3.1): S _GR = 93%), and in other types of scenes it may be lower (Figure 3, view 3.2): S _GR = 15%). However, in some cases, due to different lighting conditions of the scene or the features of the compression algorithm used, the saturation of green pixels can be quite low (Figure 3, view 3.3); S _GR = 31%). Moreover, sometimes the saturation of green pixels can be even lower than that of scenes of a different type (Figure 3, view 3.4): S _GR = 35%).

A person can easily notice the difference between shades of green in these two images, which can be easily verified by normalizing these two images by the average values of blue, green and red channels in green areas (Figure 4). It can be seen that the blue color component plays a key role and replacing its average value with the average value of the blue component in the green areas of the football field gives almost the same color (Figure 5). Thus, in order to distinguish between these two scenes, it makes sense to use the average value of the blue channel in the green areas of the image. It makes sense to use the average brightness value in the present invention as a classification feature in order to distinguish between very dark scenes and classify them as scenes that are not game scenes, because sports competitions are usually held in well-lit places.

6 (views 6.1 and 6.2) shows histograms of the green color channel for green areas in the images of Figure 3 (views 3.3 and 3.4, respectively). These figures show that if the histogram is more compact, it is more likely that the frame is a game episode than when this histogram is blurred. It is proposed to use the proportion of pixels near the maximum of the histogram as a measure of compactness. If this histogram is smeared, we can assume that the frame is not a game episode.

6 (views 6.3 and 6.4) shows the gradient of the luminance channel in the green areas for the frames in Figure 3 (views 3.3 and 3.4, respectively). It can be seen that very low gradient values may correspond to frames other than game scenes.

If the average brightness of green pixels is very low, then we can conclude that the frame is not a game episode.

Obviously, in some game episodes, the proportion of green pixels may be zero (Fig. 7), but even a person can hardly classify such frames without reading the accompanying text (which may be absent). For frames with a zero proportion of green pixels, we can assume that the frame is not a game episode.

An effective classifier should separately consider frames with a different proportion of green pixels and their different average saturation (Fig. 8). Frames with a very low proportion of green pixels and low saturation of green pixels are classified as non-game scenes.

In each cell in the table of FIG. 8, additional rules apply, depending on the expected appearance of the scene. If the number of green pixels is very large, then it is assumed that a distant view is presented. If the number of bright and saturated pixels (which may correspond to the shape of the players) is low or medium, and the number of white pixels is small, but greater than zero, then the frame is classified as a game episode.

If the number of green pixels is small or medium, then a close-up is assumed. If the number of bright and saturated pixels is medium or large, the number of white pixels is average, and the number of pixels of a human skin tone is greater than zero and small or medium, then the frame is classified as a game episode.

If there was no scene change and the previous frame was classified as a game episode, then the current frame is also classified as a game episode.

Examples of frames and graphs of the most important features along with the classification results for game episodes, non-game episodes, and mixed frame sequences are shown in FIGS. 9-14. In this case, Fig. 9 shows the results for a sequence of frames shot indoors and which are not a game episode. Figure 10 shows a graph of signs and detection results for a sequence of frames taken on the street and not a game episode. 11 shows a graph of signs and detection results for a sequence of frames that are not a game episode and contain a large proportion of green pixels. 12 shows a graph of signs and detection results for a game episode containing a long-range plan. Fig. 13 shows a graph of features and detection results for a game episode containing a close-up. Fig.14 shows a graph of signs and detection results for a video sequence consisting of two game episodes (as in Fig.12 (view 12.1), 13 (view 13.1)), between which there is one non-game episode (as in Fig.14 (view 14.3 )).

The block diagram of FIG. 15 contains the structure of a video content analysis system. The input video stream 1500 is stored in a frame buffer 1501, from which frames, represented as sequences of pixels in the RGB color space, are supplied to a low-level feature calculator 1503 and a scene change detector 1505. The output of the low-level feature calculator 1503 is connected to the input of the feature analyzer 1504, which processes the low-level features and makes a preliminary conclusion about whether the current frame refers to game episodes of field sports. The scene change detector 1505 analyzes the video stream and concludes whether the current frame belongs to the same scene as the previous one. The outputs of the feature analyzer 1504 and scene change detector 1505 are connected to the inputs of the content type dynamic detector 1506, which clarifies the preliminary conclusion of the feature analyzer. In particular, a dynamic content type detector distributes the solution obtained for the previous frame in the event that a scene change has not occurred.

A block diagram of a low level feature calculator is shown in FIG. 16. A sequence of 1600 pixels in the RGB color space is supplied to the input of block 1601, which calculates the saturation 5 for each pixel, and to the input of block 1602, which calculates the brightness Y for each pixel, and also to the first input of the color classifier 1603. The pixel saturation value from block 1601 is supplied to the second input of the color classifier 1603. The brightness value from block 1602 is supplied to the third input of the color classifier 1603 and to the input of the gradient calculation block 1604. The output of the color classifier 1603 and the output of the gradient calculation unit 1604 are connected to the first and second inputs of the statistical analysis unit 1605, respectively. The saturation of the pixel S is calculated in block 1601 for each pixel in the frame based on its values in the color channels R, G, and B according to the formula $$S=\{\begin{array}{l}\mathrm{one}-\frac{M0}{M\mathrm{one}},M\mathrm{one}<0\\ 0\mathrm{and}n\mathrm{but}he\end{array}$$

where M0 is the minimum of the values in the color channels M0 = MIN (R, G, B) and M1 is the maximum of the values in the color channels M1 = MAX (R, G, B).

The brightness of the pixel Y is calculated in block 1602 for each pixel in the frame based on its values in the color channels R, G, and B according to the formula

$$Y=\frac{306R}{1024}+\frac{601G}{1024}+\frac{58B}{512}$$

  An example implementation of the classifier 1603 colors shown in Fig.3. The three inputs of this block are supplied with values in the color channels R, G, and B (1701), saturation (1702) and brightness (1703). The white pixel detector 1704 analyzes the values of R, G and B for each pixel and calculates a logical value, meaning that this pixel will look white for a person. Detector 1705 bright and saturated pixels analyzes the values of R, G and B for each pixel and calculates a logical value, meaning that this pixel will look bright and saturated for a person. The human skin tone detector 1706 pixels analyzes R, G, and B values for each pixel and calculates a logical value, meaning that this pixel will look like a human skin in color. The yellow pixel detector 1707 analyzes the R, G, and B values for each pixel along with its saturation (1702) and brightness (1703) and calculates a logical value, which means that this pixel will look yellow for a person. The green pixel detector 1708 analyzes the R, G, and B values for each pixel along with its brightness (1703) and the result of detecting the yellow pixel, and calculates a logical value, which means that this pixel will look green for a person. Multiplexer 1709 combines the outputs of a white pixel detector 1704, a bright and saturated pixel detector 1705, a human skin tone pixel detector 1706 and a green pixel detector 1708 for all pixels and is represented in four channels.

. The white pixel detector calculates the expression W = S _RGB > 384 ^ M1-M0 <30, where S _RGB is the sum of the values in the color channels S _RGB = R + G + BB The region described by this formula in the RGB color space is shown in Fig. 18 . Examples of the application of this detector are shown in Fig. 19. The detector 1705 bright and saturated pixels calculates the expression B _S = M1> 150 & M1-M0≥M1 / 2. A human skin tone detector 1706 pixels computes the expression S _k = (G ≠ 0) ⋀ $$B\le G+\frac{G}{2}\wedge S_{RGB}>\frac{267R}{2^7}\wedge B\le \frac{83\cdot S_{RGB}}{2^8}\wedge G\le \frac{83\cdot S_{RGB}}{2^8}$$

.

The region described by this formula in the RGB color space is shown in FIG.

The yellow pixel detector 1707 calculates the expression Y _e = B <G⋀B <R⋀9 · (M1 _RG -M0 _RG ) <M0 _RG -B⋀S>0.2⋀Y> 110, where the maximum value between the value in the green channel G in the red channel R, and M0 _RG is the minimum of these two numbers. The region described by this formula in the RGB color space is shown in FIG. Examples of the application of this detector are shown in FIG.

, где M1_RB - максимальное значение между величиной в красном канале R и синем канале В. Область, описываемая этой формулой в цветовом пространстве RGB, показана на Фиг.23. Сравнение результатов детектирования для простого детектора зеленых пикселей и предложенного детектора приведено на Фиг.24.Green pixel detector 1708 calculates an expression $$G_r=G>M{\mathrm{one}}_{RB}\wedge S_{RGB}>80\wedge (R+B<\frac{3}{2}G\vee R+B<255\vee R-B<35)\wedge Y>80\wedge Y_e$$

where M1 _RB is the maximum value between the value in the red channel R and the blue channel B. The region described by this formula in the RGB color space is shown in FIG. 23. A comparison of the detection results for a simple detector of green pixels and the proposed detector is shown in Fig.24.

.Gradient calculator 1604 calculates the brightness derivative D _Y by convolution of the luminance channel Y with a linear core $$K_{grad}=[\begin{array}{ccccccc}0& 0& 0& \mathrm{one}& 0& 0& -\mathrm{one}]\end{array}$$

.

The statistical analysis unit 1605 calculates for each frame the following low-level features: the normalized number of green pixels F ₁ , the normalized number of pixels of human skin color F ₂ , the average brightness of all pixels F ₃ , the average value of the gradient module for green pixels F ₄ , the normalized number of bright and saturated F pixels _5, the average saturation of green pixels F _6, the normalized number of white pixels F _7, the average brightness of green pixel F _8, the average value of the blue channel for the green pixels F _9, compactness histo Ranma green color channel for green pixels F _10.

, где w - ширина кадра в пикселях, h - высота кадра в пикселях, i, j - координаты пикселя, а δ - функция, преобразующая логический тип в вещественный по формуле $$\delta \left(x\right)=\{\begin{array}{l}\mathrm{0,}\neg x\\ \mathrm{1,}x\end{array}$$

.The normalized number of green pixels is calculated according to the expression $$F_{\mathrm{one}}=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta \left(G_r\left(i,j\right)\right)}}{w\cdot h}$$

, where w is the width of the frame in pixels, h is the height of the frame in pixels, i, j are the coordinates of the pixel, and δ is a function that converts the logical type to real using the formula $$\delta \left(x\right)=\{\begin{array}{l}0\neg x\\ \mathrm{one,}x\end{array}$$

.

.The normalized number of pixels of the color of human skin is calculated according to the expression $$F_2=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta \left(S_k\left(i,j\right)\right)}}{w\cdot h}$$

.

.The average brightness of all pixels is calculated according to the expression $$F_3=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}Y\left(i,j\right)}}{w\cdot h}$$

.

.The average value of the gradient modulus for green pixels is calculated according to the expression $$F_{\mathrm{four}}=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}|\; |\; |D_y\left(i,j\right)|\; |\; |\cdot \delta \left(G_r\left(i,j\right)\right)}}{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta \left(G_r\left(i,j\right)\right)}}$$

.

.The normalized number of bright and saturated pixels is calculated according to the expression $$F_5=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta (B_s\left(i,j\right))}}{w\cdot h}$$

.

.The average saturation of green pixels is calculated according to the expression $$F_6=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}S\left(i,j\right)\cdot \delta \left(G_r\left(i,j\right)\right)}}{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta \left(G_r\left(i,j\right)\right)}}$$

.

.The normalized number of white pixels is calculated according to the expression $$F_7=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta (W\left(i,j\right))}}{w\cdot h}$$

.

.The average brightness of green pixels is calculated according to the expression $$F_{}$$$${}_8=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}Y(i,j)\cdot \delta (G_r\left(i,j\right))}}{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta (G_r\left(i,j\right))}}$$

.

.The average value of the blue channel for green pixels is calculated according to the expression $$F_9=\frac{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}B(i,j)\cdot \delta (G_r\left(i,j\right))}}{{\displaystyle \sum _{i=\mathrm{one}...w,j=\mathrm{one}...h}\delta (G_r\left(i,j\right))}}$$

.

The compactness of the histogram of the green color channel for green pixels E _{10 is} calculated in the following three steps:

1) A histogram H _{YGr is plotted} for the values of the green color channel G for pixels classified by color as green (Fig. 25);

2) Histogram width D It is calculated as the difference between the rightmost (MHY1) and the leftmost (MHY0) nonzero histogram elements;

3) The sign F _{10 is} calculated as the proportion of the histogram elements lying at a distance of no more than an eighth of the width of the histogram from the maximum of the histogram:

. $$F_{10}=\frac{{\displaystyle \sum _{i=P_8-D/8}^{{\mathrm{<figure-callout\; id="8"\; label="P"\; filenames="00000066.png"\; state="\{\{state\}\}">P</figure-callout>}}_8+D/8}{H}_{YGr}(i)}}{{\displaystyle \sum _{i=0}^{255}{H}_{YGr}(i)}}$$

.

The output of statistical analysis block 1605 is a 10-component feature vector (F ₁ , F ₂ , F ₃ , F ₄ , F ₅ , F ₆ , F ₇ , F ₈ , F ₉ , F ₁₀ ).

A block diagram of a feature analyzer 1504 is shown in FIG. Sign 2601 "average green pixel saturation" F ₆ and sign 2602 "normalized number of green pixels" F ₁ are fed to the first and second inputs of the block 2620 two-dimensional threshold conversion, which calculates the vector of logical quantities

(y ₁₁ , y ₁₂ , y ₁₃ , y ₁₄ , y ₂₁ , y ₂₂ , y ₂₃ , y ₂₄ , y ₃₁ , y ₃₂ , y ₃₃ , y ₃₄ , y ₄₁ , y ₄₂ , y ₄₃ , y ₄₄ ),

, где $$T_0^1$$

, $$T_1^1$$

, $$T_2^1$$

, $$T_3^1$$

, $$T_4^1$$

, $$T_0^2$$

, $$T_1^2$$

, $$T_2^2$$

, $$T_3^2$$

, $$T_4^2$$

- предопределенные пороговые величины, удовлетворяющие условию $$T_0^1=0$$

, $$T_4^1=1$$

, $$T_0^2=0$$

, $$T_4^2=0$$

,where each of these quantities y _{ij is} calculated by the formula $$y_{ij}=F_{\mathrm{one}}\ge T_{i-\mathrm{one}}^{\mathrm{one}}\wedge F_{\mathrm{one}}\le T_i^{\mathrm{one}}\wedge F_6\ge T_{j-\mathrm{one}}^2\wedge F_6\le T_j^2$$

where $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^{\mathrm{one}}$$

, $$T_{\mathrm{one}}^{\mathrm{one}}$$

, $$T_2^{\mathrm{one}}$$

, $$T_3^{\mathrm{one}}$$

, $$T_{\mathrm{four}}^{\mathrm{one}}$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^2$$

, $$T_{\mathrm{one}}^2$$

, $$T_2^2$$

, $$T_3^2$$

, $$T_{\mathrm{four}}^2$$

- predefined threshold values that satisfy the condition $$T_{}^{}$$$${}_0^{\mathrm{one}}=0$$

, $$T_{\mathrm{four}}^{\mathrm{one}}=\mathrm{one}$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^2=0$$

, $$T_{\mathrm{four}}^2=0$$

,

, $$T_0^2<T_1^2<T_2^2<T_3^2<T_4^2$$

. $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^{\mathrm{one}}<T_{\mathrm{one}}^{\mathrm{one}}<T_2^{\mathrm{one}}<T_3^{\mathrm{one}}<T_{\mathrm{four}}^{\mathrm{one}}$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^2<T_{\mathrm{one}}^2<T_2^2<T_3^2<T_{\mathrm{four}}^2$$

.

Sign 2602 "normalized number of green pixels" is also fed to the input of block 2611 threshold conversion, calculating the logical value N ₁ = F ₁ <T ³ , where T ³ is a predetermined threshold value that satisfies the condition 0 <T ³ <1.

Sign 2603 "normalized number of pixels of the color of human skin" is fed to the input of a threshold transform block 2612 that calculates a logical quantity N ₂ = F ₂ <T ⁴ , where T ⁴ is a predetermined threshold value that satisfies the condition 0 <T ⁴ <1.

Sign 2604 "the average value of the modulus of the gradient for green pixels" is fed to the input of block 2613 of the threshold transformation, which calculates the logical value N ₃ = F ₄ <T ⁵ , where T ⁵ is a predetermined threshold value satisfying the condition 0 <T ⁵ <1.

Sign 2605 "normalized number of white pixels" is fed to the input of the threshold transform block 2614, which calculates the logical value N ₄ = F ₇ > T ⁶ , where T ⁶ is a predefined threshold value that satisfies the condition 0 <T ⁶ <1.

Sign 2603 "normalized number of pixels of the color of human skin" and sign 2607 "normalized number of bright and saturated pixels" are fed to the first and second inputs of block 2621 two-dimensional threshold transformation, which calculates a vector of logical quantities

(z ₁₁ , z ₁₂ , z ₂₁ , z ₂₂ ), where each of these quantities z _ij i is calculated by the formula

, где $$T_0^7$$

, $$T_1^7$$

, $$T_2^7$$

, $$T_0^8$$

, $$T_1^8$$

, $$T_2^8$$

- предопределенные пороговые величины, удовлетворяющие условию $$T_0^7=0$$

, $$T_2^7=1$$

, $$T_0^8=0$$

, $$T_2^8=1$$

, $$T_0^7<T_1^7<T_2^7$$

, $$T_0^8<T_1^8<T_2^8$$

. $$z_{ij}=F_2\ge T_{i-\mathrm{one}}^7\wedge F_2\le T_i^7\wedge F_5\ge T_{j-\mathrm{<figure-callout\; id="8"\; label="one"\; filenames="00000066.png"\; state="\{\{state\}\}">one</figure-callout>}}^8\wedge F_5\le {\mathrm{<figure-callout\; id="8"\; label="T\; j"\; filenames="00000066.png"\; state="\{\{state\}\}">T\; </figure-callout>}}_j^{}$$

where $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^7$$

, $$T_{\mathrm{one}}^7$$

, $$T_2^7$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^8$$

, $$T_{\mathrm{one}}^8$$

, $$T_2^8$$

- predefined threshold values that satisfy the condition $$T_{}^{}$$$${}_0^7=0$$

, $$T_2^7=\mathrm{one}$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^8=0$$

, $$T_2^8=\mathrm{one}$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^7<T_{\mathrm{one}}^7<T_2^7$$

, $${\mathrm{<figure-callout\; id="0"\; label="T"\; filenames="00000062.png,00000074.png"\; state="\{\{state\}\}">T</figure-callout>}}_0^8<T_{\mathrm{one}}^8<T_2^8$$

.

Sign 2606 "average brightness" F ₃ and sign 2607 "normalized number of bright and saturated pixels" F ₅ are fed to the first and second inputs of linear classification block 2622, which calculates the logical value Q ₁ = K ₁ · F ₃ + K ₂ · F ₅ + B> 0, where K ₁ , K ₂ and B are predefined constants.

Sign 2607 "normalized number of bright and saturated pixels" F ₅ is fed to the input of a threshold transform block 2615 that calculates a logical quantity Q ₂ = F ₅ > T ⁹ _, where T ⁹ is a predetermined threshold value satisfying the condition 0 <T ⁹ <1.

Sign 2608 "average brightness of green pixels" is fed to the input of block 2616 of the threshold transform that calculates the logical value P ₁ = F ₈ > T ¹⁰ , where T ¹⁰ is a predefined threshold value that satisfies the condition 0 <T ¹⁰ <1.

Sign 2608 "average brightness of green pixels" is fed to the input of block 2617 of the threshold transformation that calculates the logical value P ₂ = F ₈ > T ¹¹ , where T ¹¹ is a predefined threshold value satisfying the condition 0 <T ¹¹ <1, T ¹¹ ≠ T ¹⁰ .

Sign 2609 "the average value of the blue channel for green pixels" is supplied to the input of block 2618 threshold conversion,

calculating a logical quantity P ₃ = F ₉ <T ¹² , where T ¹² is a predetermined threshold value satisfying the condition 0 <T ¹² <1.

The sign 2610 "compactness of the histogram of the green color channel for green pixels" is supplied to the input of the threshold transform unit 2619, which calculates the logical value P ₄ = F ₁₀ <T ¹³ , where T ¹³ is a predefined threshold value satisfying the condition 0 <T ¹³ <1.

The outputs of the threshold transform blocks 2611, 2612, 2613, 2614, 2615, 2616, 2617, 2618, 2619, along with the outputs of the two-dimensional threshold transform blocks 2620 and 2621 and the output of the linear classification block 2622 are connected to the inputs of the logical analysis block 2623 that calculates the result of the detection of the game episode a field sport for the current frame solely based on the pixels of that frame.

,The logical analysis unit implements the logical formula $$R=\left(\neg V_{\mathrm{one}}\right)\wedge P_{\mathrm{one}}\wedge P_{\mathrm{four}}\wedge V_2\wedge \left(\neg P_2\wedge P_3)\right)$$

,

where V ₁ = N ₁ ⋁ N ₂ ⋁ N ₃ ⋁ N ₄ ⋁z ₁₁ and

. $$V_2=\left(y_{22}\wedge Q_2\right)\vee y_{23}\vee \left(y_{24}\wedge Q_{\mathrm{one}}\right)\vee y_{32}\vee y_{33}\vee y_{34}\vee y_{42}\vee y_{43}\vee y_{44}$$

.

A block diagram of a scene change detector 1505 is shown in FIG. The values 2701 in the color channels R, G, and B of the current frame and the centers of the segmentation clusters from the previous frame from the output of the delay unit 2703 are supplied to the input of the clustering unit 2702. The centers of the segmentation clusters a: (are described by a matrix of size N _K for 3 elements, where N _K is the number of clusters used in segmentation:

$$K_C=\left(\begin{array}{ccccc}{R}_{\mathrm{one}}^C& R_2^C& R_3^C& ...& R_{N_K}^C\\ G_{\mathrm{one}}^C& G_2^C& G_3^C& ...& G_{N_K}^C\\ B_{\mathrm{one}}^C& B_2^C& B_3^C& \cdots & B_{N_K}^C\end{array}\right)$$

, где $$Ck=(R_k^C{G}_k^C{B}_k^C)$$

- центр k-го кластера, а ∥·∥ - некоторая векторная норма. При этом суммарная ошибка кластеризации вычисляется как $$E={\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\Vert P(i,j)-C_{K(i,j)}\Vert }$$

. Обновленные центры кластеров вычисляются по формуле $${\tilde{K}}_C=\left(\begin{array}{ccccc}{\tilde{R}}_1^C& {\tilde{R}}_2^C& {\tilde{R}}_3^C& \dots & {\tilde{R}}_{N_K}^C\\ {\tilde{G}}_1^C& {\tilde{G}}_2^C& {\tilde{G}}_3^C& \dots & G_{N_K}^C\\ {\tilde{B}}_1^C& {\tilde{B}}_2^C& {\tilde{B}}_3^C& \cdots & {\tilde{B}}_{N_K}^C\end{array}\right)$$

, где $${\tilde{R}}_k^C=\frac{{\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\delta (K(i,j)=k)\cdot R\left(i,j\right)}}{{\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\delta (K(i,j)=k)}}$$

, $${\tilde{G}}_k^C=\frac{{\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\delta (K(i,j)=k)\cdot G\left(i,j\right)}}{{\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\delta (K(i,j)=k)}}$$

, $${\tilde{B}}_k^C=\frac{{\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\delta (K(i,j)=k)\cdot B\left(i,j\right)}}{{\displaystyle \sum _{\begin{array}{l}i=1\dots w\\ j=1\dots h\end{array}}\delta (K(i,j)=k)}}$$

.Clustering block 2702 performs one iteration of the K-means algorithm, i.e. each pixel P (i, j) with coordinates i, j and described by three values in the color channels P (i, j) = (R (i, j) G (i, j) B (i, j)) is assigned to the cluster $$K(i,j)=\arg \underset{k=\mathrm{one}...N_k}{\min }\Vert P(i,j)-C_k\Vert$$

where $$Ck=(R_k^C{G}_k^C{B}_k^C)$$

is the center of the kth cluster, and ∥ · ∥ is some vector norm. In this case, the total clustering error is calculated as $$E={\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\Vert P(i,j)-C_{K(i,j)}\Vert }$$

. Updated cluster centers are calculated by the formula $${\tilde{K}}_C=\left(\begin{array}{ccccc}{\tilde{R}}_{\mathrm{one}}^C& {\tilde{R}}_2^C& {\tilde{R}}_3^C& ...& {\tilde{R}}_{N_K}^C\\ {\tilde{G}}_{\mathrm{one}}^C& {\tilde{G}}_2^C& {\tilde{G}}_3^C& ...& G_{N_K}^C\\ {\tilde{B}}_{\mathrm{one}}^C& {\tilde{B}}_2^C& {\tilde{B}}_3^C& \cdots & {\tilde{B}}_{N_K}^C\end{array}\right)$$

where $${\tilde{R}}_k^C=\frac{{\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\delta (K(i,j)=k)\cdot R\left(i,j\right)}}{{\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\delta (K(i,j)=k)}}$$

, $${\tilde{G}}_k^C=\frac{{\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\delta (K(i,j)=k)\cdot G\left(i,j\right)}}{{\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\delta (K(i,j)=k)}}$$

, $${\tilde{B}}_k^C=\frac{{\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\delta (K(i,j)=k)\cdot B\left(i,j\right)}}{{\displaystyle \sum _{\begin{array}{l}i=\mathrm{one}...w\\ j=\mathrm{one}...h\end{array}}\delta (K(i,j)=k)}}$$

.

  Updated cluster centers from the output of block 2702 clustering are fed to the input of block 2703 delay. The total clustering error from the output of block 2702 clustering is fed to the input of block 2705 delay. The total clustering error from the output of the clustering block 2702 and the output of the delay block 2705 are fed to the input of block 2706, which calculates the minimum of these two values, and to the input of block 2707, which calculates the maximum of these two values. The output of block 2706 is fed to the input of gain block 2708, which multiplies the input quantity by a predetermined constant greater than one. The output of block 2708 and the output of block 2707 are connected to the first and second inputs of the subtraction block 2709, the output of which is connected to the threshold conversion input 2710. The threshold transform output 2710 is a logic value that takes a true value if the current frame does not belong to the same scene as the previous frame.

A block diagram of a content type dynamic detector 1506 is shown in FIG. 28. The output of the statistical analysis block 1605 is connected to the first data input of the switch 2802 and the first input of the logical disjunction unit 2803, the output of the scene change detector 2801 is connected to the control input of the switch 2802. The output of the logical disjunction block 2803 is connected to the second data input of the switch 2802. To the output of the switch 2802 a signal is transmitted from the first data input if the control input is set to logic one, and a signal from the second data input if the control input is set to logic zero. The output of switch 2802 is connected to the input of delay line 2804. The output of the delay line 2804 is connected to the second logical disjunction unit 2803 and is the output of a content type dynamic detector 1506.

One possible application of the present invention in a television receiver is shown in FIG. The video signal 2900 is received by the receiver 2901 and stored in the frame buffer 2902. The output of the receiver is connected to the device 2905 for detecting game episodes in field sports in video sequences. The output of the frame buffer 2902 is connected to the video image enhancement block 2903, which performs video image processing, in particular, noise reduction, contrast enhancement, sharpening, and the like. The output of the device 2905 for detecting game episodes in field sports in video sequences is connected to the input of the adaptation block 2906, which transmits the settings of the video processing algorithms required for this type of image to the video image enhancement block 2903. The output of video image enhancement unit 2903, consisting of the processed video stream, is supplied to a display device 2904.

The invention can also be used for automatic indexing of archives of video sequences.

Claims (10)

1. A method for detecting game episodes in field sports in real time, which consists in performing the following operations:
- calculate the brightness and saturation of each pixel in the frame described by the values in the red, green and blue color channels;
- calculate the absolute value of the brightness gradient for each pixel of the frame;
- perform classification by color of each pixel of the frame;
- calculate statistics according to the results of the classification by color for the entire frame;
- calculate statistics according to the results of the classification by color for the green areas of the frame;
- determine whether a given frame is a game episode in field sports based solely on the characteristics of the current frame;
- determine whether the current frame belongs to the same scene as the previous frame;
- use the detection result obtained for the previous frame as an updated detection result if the current frame belongs to the same scene as the previous frame of the video sequence, or
- use the detection result obtained for the current frame based solely on the characteristics of the current frame, as an updated detection result if the current frame does not belong to the same scene as the previous frame of the video sequence. 2. The method according to claim 1, characterized in that the procedure for performing the classification of pixels by color consists of the following operations:
- determine whether the pixel is white;
- determine whether the pixel is bright and saturated;
- determine whether the pixel is yellow;
- determine whether the pixel is green;
- determine whether the pixel is similar in color to human skin. 3. The method according to claim 1, characterized in that the procedure for calculating statistics according to the results of color classification for the entire frame consists in performing the following operations:
- calculate the proportion of white pixels in the frame;
- calculate the proportion of bright and saturated pixels in the frame;
- calculate the proportion of green pixels in the frame;
- calculate the proportion of pixels similar in color to human skin in the frame. 4. The method according to claim 1, characterized in that the procedure for calculating statistics according to the results of color classification for green areas of the frame consists in performing the following operations:
- calculate the average brightness in the green areas of the frame;
- calculate the average saturation in the green areas of the frame;
- calculate the average value in the blue color channel in the green areas of the frame;
- calculate the average absolute value of the brightness gradient in the green areas of the frame;
- calculate the histogram of the green color channel in the green areas of the frame. 5. A device for detecting game episodes in field sports in real time, consisting of a frame buffer, a low level feature detector, a scene change detector, a feature analysis unit and a dynamic content type detector, where the frame buffer is connected to a low level feature detector, line input delays and with the first input of the scene change detector; the output of the low level feature detector is connected to the feature analysis unit, the delay line output is connected to the second input of the scene change detector; a feature analysis unit is connected to a first input of a content type dynamic detector; the output of the scene change detector is connected to a second input of a content type dynamic detector; the output of a dynamic content type detector is the output of the device. 6. The device according to claim 5, characterized in that the scene change detector is configured to determine whether the current frame belongs to the same scene as the previous frame. 7. The device according to claim 5, characterized in that the dynamic content type detector is configured to transmit, as an updated result, the detection result obtained by detecting the previous frame if the current frame belongs to the same scene as the previous frame, and the detection result obtained of the current frame based solely on the characteristics of the current frame, if the current frame is the first frame of the video sequence or if the current frame does not belong to the same scene as and the previous frame. 8. A low-level feature detector in a device for detecting game episodes in field sports in real time, consisting of a pixel conversion unit configured to calculate saturation and brightness for each pixel and coupled to a pixel color classifier, which is configured to determine pixel color in accordance with how it is perceived by man; and a gradient calculation unit, which is configured to calculate a gradient in the luminance channel and whose output is connected, along with the output of the pixel classifier, with the input of the statistical analysis unit. 9. The low-level feature detector of claim 8, characterized in that it is configured to classify the pixel color by computing a vector consisting of the following logical values:
- whether the pixel is white;
- whether the pixel is bright and saturated;
- whether the pixel is yellow;
- whether the pixel is green;
- Is the pixel similar in color to human skin. 10. The low level feature detector of claim 8, characterized in that the statistical analysis unit contained in the detector is configured to calculate a vector consisting of the following frame characteristics:
- the normalized number of green pixels in the frame;
- the normalized number of bright and saturated pixels in the frame;
- the normalized number of pixels similar in color to human skin in a frame;
- the average brightness of the green areas of the frame;
- average saturation of green areas of the frame;
the average value of the blue color channel in the green areas of the frame;
- the average absolute value of the brightness gradient in the green areas of the frame;
- compactness of the histogram of the green color channel in the green areas of the frame.

Images