JP2009129003A - Trimming processing apparatus and trimming processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a trimming processor for trimming a region under consideration even in the case of a video image in which the movement of a whole screen is almost constant. <P>SOLUTION: A trimming processor 1 includes: an image reduction/division means 3 for reducing a frame image, and for dividing it into blocks; an image feature extraction means 4 for extracting image features in the blocks; a shot classification means 5 for classifying shots based on whether or not the shots are long shots based on the degree of complication of the shape of a field block group configured of blocks similar to the predetermined color information of the field; a region under consideration setting means 6 for, when the shots are classified as the long shots, setting a region including the block under consideration whose luminance distribution value is higher than a predetermined luminance distribution value as a region under consideration; and a trimming video image generation means 7 for extracting the region including the region under consideration corresponding to the aspect ratio of the display device as a trimming region from the frame image, and outputting the extracted region. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a trimming processing apparatus and a trimming processing program for trimming a region of interest in a video.

  In recent years, with the spread of mobile devices such as mobile phones and portable information terminals (PDA: Personal Digital Assistants), the viewing environment for viewing content (moving content) on a low-resolution display is increasing. This content is distributed and distributed in various formats such as terrestrial digital broadcasting, streaming, and moving image files.

  However, there are few contents produced in consideration of the viewing environment on mobile devices, and most contents are contents assuming a high-resolution display such as a high-definition television. For this reason, there is a problem that such high-resolution content is too small to be visually recognized on a low-resolution display when the character is captured in a long shot.

Therefore, conventionally, trimming technology for improving the visibility of content by trimming and enlarging the video of the content has been disclosed (see, for example, Patent Documents 1 and 2). In the trimming process disclosed in this patent document, an area (person or the like) that should be watched by a viewer is determined based on a motion vector in a video of content, and the area is determined as a trimming area.
JP 2002-281506 A JP 2007-101867 A

  In the conventional technique described above, the trimming region is determined by the motion vector, and therefore, the region of the object such as a person shown in the video is tracked separately from other regions by comparing the motion vectors, and the attention region (trimming region) ) Can be extracted. However, the conventional technique has a problem in that it is difficult to extract a region of interest because there is almost no motion vector motion amount in the case of an image in which the motion of the entire screen is almost constant.

  Further, in the conventional technique, because the attention area is extracted only by the motion vector, for example, when an area of a person (player) is extracted as an attention area from an image of sports performed in the field, When an unexpected video such as an electronic bulletin board is shot, a region of interest is extracted from the video. As described above, the conventional technique has a problem that it cannot be applied to a broadcast video for broadcasting a sports video because an insignificant attention area is trimmed by an unexpected shot.

  Further, in the conventional technique, when the video format of the content is a format other than MPEG (Moving Picture Expert Group) that can extract a motion vector as motion compensation data, a means for obtaining a motion vector from the video is required. However, since a large amount of calculation is required to obtain the motion vector, it is difficult to determine the trimming region by obtaining the motion vector in real time on a terminal having a low calculation capability such as a mobile phone or an information portable terminal. There is a problem that.

  The present invention has been made in order to solve the above-described problems. In a video of sports performed in a field, even if the motion of the entire screen is almost constant, Trimming processing apparatus and trimming processing program capable of trimming a region of interest focused only on a person (player) appearing in the field even when an image other than the field is shot at an unexpected timing The purpose is to provide.

  The present invention has been developed to achieve the above object. First, the trimming processing device according to claim 1 trims a region of interest of an image of sports performed in a field and displays the image. And a trimming processing apparatus for outputting to the image processing apparatus, including image segmentation means, image feature extraction means, shot classification means, attention area setting means, and trimming video generation means.

  In such a configuration, the trimming apparatus divides the frame image constituting the video into blocks of a predetermined size by the image dividing unit. Then, the trimming processing apparatus extracts the color information and the luminance information in the block divided by the image dividing unit by the image feature extracting unit as the image feature of the block. As a result, the features in the frame image are extracted in units of blocks.

  Then, the trimming processing device has a complicated shape of the entire field block formed by a field block that is a block in which the color information extracted by the image feature extraction unit and the color information of the predetermined field are similar by the shot classification unit. Based on whether or not the degree exceeds a predetermined reference value, the shot in which the video is captured is classified as a long shot. For example, in the case of a long shot, since the person (player) who is the subject is photographed small, the shape of the entire field block is a shape close to a rectangle. For non-long shots (non-long shots), blocks such as people (players) erode the shape of the entire field block, separating the shape of the entire field block, increasing irregularities, and making holes. It becomes a complicated shape with respect to a rectangle such as a shape. Therefore, the shot classification means can determine whether or not the shot is a long shot based on the complexity of the shape.

  Further, the trimming processing device is configured such that when the shot classification unit classifies the shot as a long shot by the attention area setting unit, the luminance variance value is greater than a predetermined luminance variance value based on the block luminance information. A region including a target block that is a high block is set as a target region. As a result, a region including a person (player) or the like that is a region having a different luminance dispersion value in the field is extracted.

  Then, the trimming processing device extracts the region including the region of interest set by the region-of-interest setting unit by the trimming video generation unit and extracts the region according to the aspect ratio of the display device from the frame image as the trimming region and outputs the trimmed region. As a result, the attention area is enlarged and displayed on the screen of the display device.

  According to a second aspect of the present invention, the trimming processing apparatus according to the first aspect further includes an image reducing unit.

  In such a configuration, the trimming apparatus reduces the frame image constituting the video by the image reduction unit. Then, the trimming apparatus divides the frame image reduced by the image reduction unit into blocks by the image division unit.

  Furthermore, a trimming processing apparatus according to a third aspect is the trimming processing apparatus according to the first or second aspect, wherein the shot classification means includes a shape complexity calculation means.

  In such a configuration, the trimming apparatus uses the shape complexity calculation unit of the shot classification unit to calculate the complexity of the shape of the entire field block as a ratio of the perimeter of the shape of the entire field block to the number of blocks forming the entire field block. Calculate as In addition, since the number of unevenness | corrugation will increase if the shape becomes complicated, the shape of the whole field block will show that it is a complicated shape, so that perimeter length becomes long. The peripheral length is a length including not only the outer periphery of the entire field block but also the inner periphery when a hole shape is generated. If the field blocks are formed separately, the length is the sum of their perimeters.

  According to a fourth aspect of the present invention, in the trimming processing device according to the third aspect, the shot classification unit further includes an occupation ratio calculation unit.

  In such a configuration, the trimming apparatus calculates the ratio of the number of field blocks to the total number of blocks in the lower area as the field occupation ratio in the lower area of the frame image by the occupation ratio calculation means of the shot classification means. The shot classification means classifies the shot as a long shot when the complexity of the shape of the entire field block is larger than a predetermined value and the field occupation ratio is smaller than a predetermined value.

  Furthermore, the trimming processing device according to claim 5 is the trimming processing device according to any one of claims 1 to 4, wherein the attention area setting means includes attention block extraction means, clustering means, class And a selection unit.

  In such a configuration, the trimming processing device extracts, as the target block, the block whose luminance variance value is higher than the predetermined luminance variance value based on the luminance information of the block by the target block extracting unit. Then, the trimming apparatus clusters the blocks of interest extracted by the block of interest extracting unit with the adjacent blocks that are not more than a predetermined distance by the clustering unit. As a result, one or more regions including the target block are clustered as candidates for the target region. Then, the trimming apparatus selects one class as a region of interest based on a predetermined selection criterion when the block of interest is clustered into a plurality of classes by the clustering unit by the class selection unit.

  In addition, the trimming processing program according to claim 6, a computer is provided with an image dividing unit, an image feature extracting unit, an image segmenting unit, an image feature extracting unit, It is configured to function as shot classification means, attention area setting means, and trimmed video generation means.

  In such a configuration, the trimming processing program divides the frame image constituting the video into blocks of a predetermined size by the image dividing means. Then, the trimming processing program extracts the color information and the luminance information in the block divided by the image dividing unit by the image feature extracting unit as the image feature of the block. Then, the trimming program stores the complexity of the shape of the entire field block formed by the field classification block which is similar to the color information extracted by the image feature extraction unit and the color information of the predetermined field by the shot classification unit. Based on whether or not the degree exceeds a predetermined reference value, the shot in which the video is captured is classified as a long shot.

  Further, the trimming processing program is configured such that when the shot classification unit classifies the shot as a long shot by the attention area setting unit, the luminance variance value is greater than a predetermined luminance variance value based on the block luminance information. A region including a target block that is a high block is set as a target region. Then, the trimming processing program extracts from the frame image an area corresponding to the aspect ratio of the display device, including the attention area set by the attention area setting means by the trimming video generation means, and outputs the trimming area.

The present invention has the following excellent effects.
According to the first and sixth aspects of the present invention, it is possible to trim and display only a region including a region of interest from a long shot video in a video shot of sports performed in a field. As a result, the present invention can enlarge and display the region of interest, and can improve visibility in a low-resolution display device (display).

  According to the first and sixth aspects of the invention, it is possible to perform trimming by setting a region of interest based on luminance information without using a motion vector. As a result, the present invention can set a region of interest with a small amount of calculation compared to the case of using a motion vector, and can perform trimming even if the motion of the entire screen is substantially constant. Further, according to the first and sixth aspects of the present invention, since it is determined whether or not the shot is a long shot according to the shape of the field block extracted based on the color information, an unexpected shot such as a spectator seat is used. It is possible to prevent the attention area from being trimmed. Accordingly, the present invention can perform trimming processing on a broadcast video for broadcasting a sports video.

  According to the second aspect of the present invention, since the frame image is reduced, it is possible to reduce the amount of calculation in processing such as image feature extraction and shot determination. As a result, the present invention can realize the trimming process at high speed.

  According to the third aspect of the invention, since the complexity of the shape of the entire field block is obtained by the ratio of the perimeter of the shape of the entire field block to the number of blocks of the entire field block, the complexity is obtained quantitatively. be able to. Thus, according to the present invention, since it is possible to determine a long shot based on a certain standard, it is possible to stably determine a long shot.

  According to the invention described in claim 4, it is possible to determine the long shot based on the ratio of the field blocks at the bottom of the screen. Thus, according to the present invention, it is possible to accurately determine a long shot even for a video in which a spectator seat or the like is captured at the top of the screen.

  According to the fifth aspect of the present invention, since a plurality of blocks of interest are extracted as one region of interest by clustering, a scene to be noticed is not subdivided more than necessary. Thus, according to the present invention, by cropping an image including the extracted attention area, the attention area having the optimum size in the scene can be enlarged and displayed.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of trimming apparatus]
First, the configuration of a trimming processing apparatus according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing the overall configuration of a trimming apparatus according to an embodiment of the present invention.

  The trimming processing device 1 trims a region of interest of a video shot of a sport (such as soccer or rugby) performed in a field. The trimming apparatus 1 is incorporated in a mobile phone or an information portable terminal, and trims the attention area of the input video in accordance with the resolution and aspect ratio of a display device (display) such as a mobile phone. Here, the trimming processing apparatus 1 includes a video input unit 2, an image reduction / division unit 3, an image feature extraction unit 4, a shot classification unit 5, an attention area setting unit 6, and a trimmed video generation unit 7. Prepare.

  The video input means 2 inputs video from outside. The video is, for example, MPEG2-TS used in digital broadcasting or the like, an AVI (audio video interleave) file that can be played back on a personal computer, and the video format and resolution are not particularly limited. Here, the video input means 2 inputs a video for each frame image, writes it in a memory (not shown), and notifies the image reduction / division means 3.

  The image reduction / division means 3 reduces each frame image of the video input by the video input means 2 and divides it into blocks of a predetermined size. Here, with reference to FIG. 2 (refer to FIG. 1 as appropriate), a detailed configuration of the image reduction / division means 3 will be described. FIG. 2 is a block diagram showing the configuration of the image reduction / division means. Here, the image reduction / division means 3 includes an image reduction means 31 and an image division means 32.

  The image reducing means 31 reduces the frame image. For example, when the video is a high-definition video, the image reduction unit 31 reduces a frame image having a resolution of 1920 × 1080 pixels (horizontal × vertical) to a resolution of about 240 × 135 pixels (horizontal × vertical). Generate. For the reduction process in the image reduction unit 31, a general process can be used. For example, an average pixel method, a bicubic method, or the like can be used.

  In this way, by reducing the frame image, the trimming apparatus 1 can perform subsequent processing at high speed. The reduction ratio in the image reduction means 31 is not particularly limited. For example, when the processing performance of the CPU of the trimming apparatus 1 is low, the degree of the reduction ratio is increased.

  The image dividing unit 32 divides the reduced frame image generated by the image reducing unit 31 into blocks having a predetermined size. For example, the image dividing unit 32 divides the reduced frame image into 16 × 16 pixel macroblocks.

  The blocks (macroblocks) reduced by the image reduction unit 31 and divided by the image division unit 32 are output to the image feature extraction unit 4. Note that the position information in the reduced frame image of the block divided by the image dividing unit 32 is output to the image feature extracting unit 4 together with the block data.

  Here, the image dividing unit 32 divides the reduced frame image generated by the image reducing unit 31 into blocks, but the resolution of the input video is low (for example, the same as the resolution of the display device). In this case, or when the processing performance of the CPU of the trimming processing apparatus 1 is high, it is not always necessary to divide the reduced frame image into blocks, and the frame image constituting the input video may be directly divided into blocks. . In this case, the image reduction means 31 can be omitted from the configuration.

  Here, referring to FIG. 7 (refer to FIG. 2 as appropriate), the image reduction / division processing performed by the image reduction / division means 3 will be described using a specific image example. 7A and 7B are explanatory diagrams for explaining the contents of the image reduction / division processing, where FIG. 7A is a frame image constituting the input video, FIG. 7B is a reduced frame image obtained by reducing the frame image, c) shows a state in which the reduced frame image is divided into macro blocks.

  As shown in FIG. 7, the image reduction / division means 3 uses the image reduction means 31 to convert a frame image H of 1920 × 1080 pixels, which is a frame image constituting a video of a soccer game, into an image of 240 × 135 pixels. Reduce to (reduced frame image L). Then, the image reduction / division means 3 divides the reduced frame image L into 16 × 16 pixel macroblocks (blocks B, B...) By the image division means 32. These blocks B, B... Are processed by the image feature extraction means 4 (FIG. 1). Returning to FIG. 1, the description of the configuration of the trimming apparatus 1 will be continued.

  The image feature extraction unit 4 extracts (calculates) color information and luminance information as image features of each block divided by the image division unit 32 of the image reduction / division unit 3. Here, the image feature extraction unit 4 uses an average value (color average value) of colors in a block as color information, and uses luminance variance (luminance variance value) as luminance information.

  For example, when the block size is m pixels vertically and n pixels horizontally, and the RGB values of the pixels constituting the block are R (red), G (green), and B (blue), the image feature extracting unit 4 Is calculated as an average value of R, G, and B as shown in the following equation (1). Note that k indicates the number of pixels in the block. For example, when m = 16 and n = 16, k = 256.

Further, when the luminance of the pixels constituting the block is L, the image feature extraction unit 4 calculates the luminance information in the block as a luminance value variance (luminance variance value S ² ) as shown in the following equation (2). To do.

  Here, the average value of the RGB values is used as the color information. However, the color value is not limited to the RGB values as long as it is an index representing the color. For example, HSV (H: Hue, S: Saturation, V: Lightness) may be used.

  The shot classification means 5 is a shape complexity (shape complexity) of the entire field block formed by field blocks that are blocks whose color information extracted by the image feature extraction means 4 is similar to color information of a predetermined field. ) Is classified as a long shot based on whether a predetermined reference value is exceeded or not. Here, the detailed configuration of the shot classification means 5 will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of the shot classification means. Here, the shot classification unit 5 includes a block determination unit 51, a shape complexity calculation unit 52, an occupation ratio calculation unit 53, and a shot determination unit 54.

  Based on the color information extracted by the image feature extraction unit 4, the block determination unit 51 determines whether or not the block divided by the image reduction / division unit 3 is a block (field block) in which a field portion is shown. Is determined. For example, if the input video is a sports video played on a lawn such as soccer, a block containing a lot of green grass can be determined as a field block. In addition, for example, when the input video is a sports video performed on soil such as rugby, a block containing a lot of soil brown can be determined as a field block.

  For example, when determining a field block in a soccer video, the block determination unit 51 uses an average value of RGB, which is color information, to determine whether the block is a field depending on whether the following equation (3) is satisfied. It is determined whether it is a block. In equation (3), α and β are constants. For example, when RGB has 256 gradations, a field block of lawn can be determined by setting α = 5 and β = 25.

  Here, the field color information is used as a reference for determining whether or not the block is a field block. However, a plurality of the field color information is stored in advance in a memory (not shown). To do. Thus, for example, the color information selection means (not shown) allows the viewer to select the type of sports video to be viewed on the screen in the menu format, thereby changing the color information of the field to be used or digital broadcasting. Change the color information of the field appropriately for multiple sports videos, such as discriminating the type of sports video to be viewed from the program guide (EPG: Electric Program Guide) and changing the color information of the field to be used. be able to.

  The shape complexity calculation unit 52 calculates the complexity (shape complexity) of the shape of the entire block (field block) determined by the block determination unit 51 as a field block. The shape complexity calculated by the shape complexity calculation unit 52 is output to the shot determination unit 54.

  Usually, in a video shot with a long shot, a person (player) or the like as a subject is shot small, so that the shape of the entire field block is a shape close to a rectangle. On the other hand, in the video shot other than the long shot, the person who is the subject is shot large, so the frame image is eroded by non-field blocks (blocks other than field blocks), and the shape of the entire field block is The shape becomes more complicated than the rectangle. Therefore, here, the shape complexity is used as an index for classifying whether or not the frame image is a long shot image.

  Since it can be said that the shape becomes more complex as the unevenness of the shape of the entire field block increases, the shape complexity can be obtained by the ratio of the perimeter of the shape of the entire field block to the number of blocks of the entire field block. For example, when the perimeter of the shape of the entire field block is “D” and the area is “A”, the shape complexity calculation means 52 calculates the shape complexity “C” by the following equation (4). Here, the peripheral length “D” of the shape of the entire field block can be the number of field blocks in contact with a block other than the field block, and the area “A” can be the number of field blocks.

  Even when the shape of the entire field block is separated, the shape complexity can be calculated by the equation (4). Further, when the shape of the entire field block is a perforated shape, the peripheral length “D” is a length including the outer periphery of the entire field block and the inner periphery of the hole shape.

  The occupation ratio calculation means 53 calculates the ratio of the number of field blocks to the total number of blocks (field occupation ratio) in the reduced frame image. The field occupation ratio calculated by the occupation ratio calculation unit 53 is output to the shot determination unit 54.

  Usually, in a video shot with a long shot, a person (player) or the like as a subject is shot small, so that the proportion of field blocks in the screen (reduced frame image) increases. Therefore, here, the field occupation ratio is used as an index for classifying whether or not the frame image is a long shot image.

  In general, an image of sports performed in a field may include an image other than a field such as a spectator seat at the top of the screen. Therefore, the occupation ratio calculating means 53 calculates the ratio of the number of field blocks to the total number of blocks in the lower area of the screen (reduced frame image) as the field occupation ratio.

The field block occupancy ratio is calculated by assuming that the number of blocks in the lower area (specifically, the lower half) of the screen is γ and the number of field blocks in the area is N _d , The field occupation ratio _Rd is calculated from the equation.

  For example, if the reduced frame image has 240 × 135 pixels and is divided into 16 × 16 pixel blocks (macroblocks), the total number of blocks is “120”, so the value of γ is “60”. Is used.

  The shot determination means 54 is a frame image (reduced frame image) constituting a video based on the shape complexity calculated by the shape complexity calculation means 52 and the field block occupancy ratio calculated by the occupation ratio calculation means 53. Is a long shot image or not. Here, the shot determination means 54 determines “long shot” when the shape complexity is smaller than a predetermined threshold and the field block occupation ratio is larger than the predetermined threshold, and otherwise Then, it is determined as “non-long shot”. The type of the frame image classified by this determination (“long shot” or “non-long shot”) is output to the attention area setting unit 6 and the trimmed video generation unit 7.

  Here, the shot determination unit 54 performs the long shot determination using the two indexes of the shape complexity and the field block occupation ratio, but may determine either one. However, in order to increase the accuracy of the determination, it is desirable to use these two indexes. Returning to FIG. 1, the description of the configuration of the trimming apparatus 1 will be continued.

  The attention area setting unit 6 determines a luminance dispersion value based on the luminance information of the block extracted by the image feature extraction unit 4 when the frame image is classified as a long shot image by the shot classification unit 5. An area including an attention block that is a block higher than the luminance dispersion value is set as the attention area. Here, the detailed configuration of the attention area setting means 6 will be described with reference to FIG. 4 (refer to FIG. 1 as appropriate). FIG. 4 is a block diagram showing the configuration of the attention area setting means. Here, the attention area setting means 6 includes attention block extraction means 61, clustering means 62, and class selection means 63.

  The attention block extraction means 61 extracts attention blocks including players and the like from the blocks (macroblocks) divided by the image reduction / division means 3 based on the luminance information extracted by the image feature extraction means 4. is there. Usually, in a field with uniform luminance, the luminance dispersion value is high in an area where a player or the like is shown. Therefore, the block-of-interest extracting unit 61 extracts a block having a block whose luminance variance value is higher than a predetermined luminance variance value as the block of interest.

  The clustering means 62 clusters the attention blocks extracted by the attention block extraction means 61. Here, the clustering means 62 clusters a plurality of blocks of interest in close proximity to each other with a predetermined distance or less as the same class. As described above, the clustering unit 62 performs clustering by using adjacent blocks of interest as one class, thereby forming one or more candidate regions of interest. Note that when only one class is clustered as a clustering result, that class becomes a region of interest. As a result, even if the blocks of interest exist separately, blocks of a certain distance or less can be made into one class, so the scene to be noticed can be subdivided more than necessary. Absent.

  The class selection unit 63 selects any one class as a region of interest based on a predetermined selection criterion when the block of interest is clustered into a plurality of classes by the clustering unit 62. This selection criterion can be based on the position and size of the candidate for the region of interest. For example, when the position of the candidate for the attention area is used as a reference, the class selection unit 63 selects the class closest to the center position of the reduced frame image among the classes (candidate areas for attention) as the attention area. For example, when the size of the attention area candidate is used as a reference, the class selection unit 63 selects the class having the largest occupied area among the classes (notification area candidates) as the attention area.

Here, with reference to FIG. 8 (refer to FIG. 4 as appropriate), the attention area setting processing performed by the attention area setting means 6 will be described using a specific image example. FIG. 8 is an explanatory diagram for explaining the contents of the attention area setting processing, where (a) shows a state in which attention blocks are extracted from the reduced frame image, and (b) shows attention areas based on the attention blocks. Each state is shown.
As illustrated in FIG. 8, the attention area setting unit 6 uses the attention block extraction unit 61 to extract, as a target block, a block in which the luminance dispersion value of the block is higher than a predetermined luminance dispersion value. Thus, by extracting a block having a high luminance dispersion value, it is possible to extract a region where a player or the like is shown. FIG. 8A shows an example in which six blocks of interest (B _{N1 to} B _N6 ) are extracted.

Then, the attention area setting unit 6 uses the clustering unit 62 to cluster adjacent blocks as the same class. Here, the blocks having the distance between the blocks of interest within two blocks are set as the same class. As a result, the attention blocks B _{N1 to} B _N3 are clustered as one class, and the attention blocks B _{N4 to} B _N6 are clustered as one class. As shown in FIG. 8B, the attention area candidates C ₁ and C ₂ are clustered. Will be selected. In the case to select the nearest class to the center position as the region of interest, or, if you choose the most occupied area is wide class as the region of interest, the class selecting unit 63 selects a candidate C ₁ region of interest as a region of interest It will be. Returning to FIG. 1, the description of the configuration of the trimming apparatus 1 will be continued.

  The trimming video generation means 7 extracts and outputs a region corresponding to the aspect ratio of a display device (not shown) as a display target from the frame image as a trimming region. Here, the detailed configuration of the trimmed video generation means 7 will be described with reference to FIG. 5 (refer to FIG. 1 as appropriate). FIG. 5 is a block diagram showing the configuration of the trimmed video generation means. Here, the trimmed video generation means 7 includes a trimming area setting means 71 and a trimming area extraction means 72.

  The trimming area setting unit 71 sets an area corresponding to the aspect ratio of the display device as a trimming area in the frame image. Here, when the shot classification means 5 classifies the frame image as a long shot image, the trimming area setting means 71 includes at least the attention area set by the attention area setting means 6, and the display device An area corresponding to the aspect ratio is set as a trimming area. Usually, the attention area set by the attention area setting means 6 is not the same as the aspect ratio of the display device. Therefore, the trimming area setting means 71 sets the trimming area by extending the area vertically and horizontally with respect to the attention area. Alternatively, the trimming area setting unit 71 may set the trimming area by extending the area to the top, bottom, left, or right of the attention area.

  For example, the aspect ratio and aspect ratio of the attention area are compared, and when the vertical ratio of the attention area is large, the trimming area setting unit 71 fixes the vertical width of the attention area so that the aspect ratio becomes the aspect ratio. The width (left and right area) is expanded to make a trimming area. Further, the aspect ratio of the attention area is compared with the aspect ratio, and when the horizontal ratio of the attention area is large, the trimming area setting means 71 fixes the width of the attention area and the aspect ratio becomes the aspect ratio. The vertical width (upper and lower area) is expanded to be a trimming area.

  On the other hand, when the shot classification unit 5 classifies the frame image as a non-long shot image, the trimming region setting unit 71 sets the center of the frame image (the center of the screen) as the center of the trimming region and the aspect of the display device. The maximum area for the ratio is set as the trimming area.

  The trimming area extraction unit 72 extracts the trimming area set by the trimming area setting unit 71 from the frame image, and generates an image enlarged or reduced according to the resolution of the display device. The image extracted and enlarged or reduced by the trimming area extraction unit 72 is sequentially output to the display device as a video frame image. As a result, the image of the trimmed area is displayed on the display device.

  Here, referring to FIG. 9 (refer to FIG. 2 as appropriate), a technique for extracting a trimming area performed by the trimming video generation means 7 will be described with a specific image example. FIGS. 9A and 9B are explanatory diagrams for explaining the contents of the trimming area extraction process. FIG. 9A shows the respective trimming areas at the time of “long shot” shooting, and FIG. 9B shows the respective trimming areas at the time of “non-long shot” shooting. ing.

As shown in FIG. 9A, when the frame image is a long shot image, the trimming video generation means 7 extends the area vertically and horizontally with respect to the attention area C ₁ by the trimming area setting means 71. in sets trimming area T ₁ corresponding to the aspect ratio of the display device. Note that the trimming area T ₂ are, when the target area is large vertical ratio as C _2, the longitudinal width of the region of interest C ₂ is fixed, so that the aspect ratio is the aspect ratio, an example of extending the width Show.

Further, as shown in FIG. 9B, when the frame image is a non-long shot image, the trimming video generation unit 7 uses the trimming region setting unit 71 to set the center of the frame image as the center of the trimming region. setting the maximum area ratio partial trimming area T _3. For example, if the original frame image is a 16: 9 image having a high-definition aspect ratio and the display device has an aspect ratio of 4: 3 QVGA, as shown in FIG. image so that the trimming area T ₃ that is cut is set.

  By configuring the trimming processing apparatus 1 as described above, the trimming processing apparatus 1 determines whether or not the shot is a long shot when displaying an image of sports performed in the field. In such a case, it is possible to enlarge and display the attention area where the player or the like is present, as compared with the case where the entire frame image is displayed. As a result, the trimming apparatus 1 can display a video with improved visibility of a region of interest on a low-resolution display, even for a video that assumes a high-resolution display (high-vision video).

  Note that the trimming apparatus 1 can be operated by a trimming processing program that causes a general computer to function as each of the above-described units. Further, this trimming processing program may be stored in a non-volatile memory (not shown) or the like and operated, or may be acquired and operated via a broadcast wave or a communication line, for example. .

[Operation of trimming device]
Next, the operation of the trimming apparatus according to the embodiment of the present invention will be described with reference to FIG. 6 (refer to FIGS. 1 to 5 for the configuration as appropriate). FIG. 6 is a flowchart showing the operation of the trimming processing apparatus according to the embodiment of the present invention.

(Video input)
First, the trimming apparatus 1 inputs a video from the outside by the video input unit 2 (step S1).

(Image reduction / division)
Then, the trimming apparatus 1 performs image reduction for each frame image of the video by the image reduction / division means 3 and divides it into blocks of a predetermined size (step S2). Specifically, the trimming apparatus 1 performs image reduction for each frame image by the image reduction unit 31 of the image reduction / division unit 3, for example, to convert a frame image having a resolution of 1920 × 1080 pixels to 240. A reduced frame image reduced to × 135 pixels is generated. Then, the trimming apparatus 1 divides the reduced frame image into blocks of a predetermined size (for example, 16 × 16 pixel macroblocks) by the image dividing unit 32.

(Image feature extraction)
Then, for each block (macroblock) divided in step S2 by the image feature extraction unit 4, the trimming processing device 1 obtains color information (color average value) and luminance information (luminance variance value) for the image feature of the block. (Step S3).

(Shot classification)
Thereafter, the trimming apparatus 1 uses the shot classification unit 5 to complicate the shape of the entire field block of a specific color (for example, the color of the lawn) based on the color information (color average value) for each block calculated in step S3. The degree and the field occupation ratio are calculated (step S4), and whether the shot is a long shot or other (non-long shot) is classified (step S5).

  Specifically, the trimming apparatus 1 uses the block determination unit 51 of the shot classification unit 5 to determine whether the color information of each block is similar to the color information of a specific color (eg, lawn color). Thus, it is determined whether or not each block is a block (field block) in which the field portion is reflected.

  Then, the trimming processing device 1 calculates the shape complexity indicating the degree of complexity of the shape of the entire block (field block) determined as the field block by the shape complexity calculating unit 52 of the shot classification unit 5. Further, the trimming apparatus 1 calculates the ratio of the number of field blocks to the total number of blocks (field occupation ratio) by the occupation ratio calculation means 53 of the shot classification means 5 (step S4).

  Thereafter, the trimming apparatus 1 uses the shot determination unit 54 of the shot classification unit 5 to generate a frame image when the shape complexity is smaller than a predetermined threshold value and the field block occupation ratio is larger than a predetermined threshold value. Is a “long shot” image, otherwise it is determined to be a “non-long shot” image (step S5).

(Attention area setting)
When it is determined in step S5 that the frame image is a “long shot” image, the trimming apparatus 1 uses the attention area setting unit 6 to add the luminance information (luminance variance value) for each block calculated in step S3. Based on this, a block to be noticed is extracted as a noticed block (step S6), and an area including the noticed block is set as a noticed area (step S7).

  Specifically, the trimming processing apparatus 1 extracts, as a target block, a block in which the luminance variance value of each block is higher than a predetermined luminance variance value by the target block extracting unit 61 of the target area setting unit 6 (step S6). ).

  Then, the trimming apparatus 1 uses the clustering unit 62 of the attention area setting unit 6 to cluster a plurality of blocks of interest close to each other with a predetermined distance or less as the same class. As a result, one or more candidate regions of interest are formed. Then, the trimming processing apparatus 1 selects any one class as the attention area by the class selection means 63 of the attention area setting means 6 according to a predetermined selection criterion (for example, the class closest to the screen center position). (Step S7).

(Trimming area setting)
Then, the trimming processing apparatus 1 sets an area corresponding to the aspect ratio of the display device (not shown) to be displayed as a trimming area by the trimming video generation means 7 (steps S8 and S9), and sets the trimming area as a frame. Extracted from the image and displayed (step S10).

  Specifically, the trimming processing device 1 includes at least a region of interest set in step S7 by the trimming region setting unit 71 of the trimming video generation unit 7, and a region corresponding to the aspect ratio of the display device as a trimming region. (Step S8). If it is determined in step S5 that the frame image is a “non-long shot” image, the trimming area setting unit 71 uses the center of the frame image (the center of the screen) as the center of the trimming area to match the aspect ratio of the display device. Is set as a trimming area (step S9).

  Then, the trimming processing device 1 extracts the trimming region set in step S8 or step S9 from the frame image by the trimming region extraction unit 72 of the trimmed video generation unit 7, and enlarges or reduces it according to the resolution of the display device. The generated image is generated and displayed on the display device (step S10).

  With the above operation, the trimming apparatus 1 determines whether or not the shot is a long shot when displaying an image of sports performed in the field, and displays the entire frame image when the shot is a long shot. Compared with the case where it does, the attention area where a player etc. exist can be expanded and displayed.

It is a block diagram which shows the whole structure of the trimming processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the image reduction / division | segmentation means of the trimming processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the shot classification means of the trimming processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the attention area setting means of the trimming processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the trimming image | video production | generation means of the trimming processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the trimming processing apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the content of the image reduction / division process. It is explanatory drawing for demonstrating the content of an attention area setting process. It is explanatory drawing for demonstrating the content of the trimming area | region extraction process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Trimming processing apparatus 2 Image | video input means 3 Image reduction / division means 31 Image reduction means 32 Image division means 4 Image feature extraction means 5 Shot classification means 51 Block determination means 52 Shape complexity calculation means 53 Occupancy ratio calculation means 54 Shot determination means 6 attention area setting means 61 attention block extraction means 62 clustering means 63 class selection means 7 trimming video generation means 71 trimming area setting means 72 trimming area extraction means

Claims (6)

A trimming processing device for trimming a region of interest of an image of sports performed in a field and outputting the image to a display device,
Image dividing means for dividing a frame image constituting the video into blocks of a predetermined size;
Image feature extraction means for extracting color information and luminance information in the block divided by the image dividing means as image features of the block;
The complexity of the shape of the entire field block formed by the field block, which is a block in which the color information extracted by the image feature extraction unit and the predetermined color information of the field are similar, exceeds a predetermined reference value. Shot classification means for classifying the shot of the video based on whether or not to be a long shot,
When the shot classification means classifies the shot as a long shot, based on the luminance information of the block, an area including a block of interest that is a block whose luminance variance value is higher than a predetermined luminance variance value. Attention area setting means for setting as the attention area;
Trimming video generation means including the attention area set by the attention area setting means, and extracting and outputting the area according to the aspect ratio of the display device from the frame image as a trimming area;
A trimming processing apparatus comprising: Image reduction means for reducing the frame image;
2. The trimming apparatus according to claim 1, wherein the image dividing unit divides the frame image reduced by the image reducing unit. The shot classification means includes
3. The shape complexity calculating means for calculating the complexity of the shape as a ratio of the perimeter of the shape of the entire field block to the number of blocks forming the entire field block. The trimming processing apparatus described in 1. The shot classification means includes
In the lower area of the frame image, further comprises an occupation ratio calculating means for calculating a ratio of the number of blocks of the field block to the total number of blocks in the lower area as a field occupation ratio, and the complexity is larger than a predetermined value, 4. The trimming apparatus according to claim 3, wherein the shot is classified as a long shot when the field occupation ratio is smaller than a predetermined value. The attention area setting means includes:
Attention block extraction means for extracting, as the attention block, a block having a luminance dispersion value higher than a predetermined luminance dispersion value based on the luminance information of the block;
Clustering means for clustering the blocks of interest extracted by the block of interest extracting means in close proximity to each other within a predetermined distance;
Class selection means for selecting one class as the attention area according to a predetermined selection criterion when the attention block is clustered into a plurality of classes by the clustering means;
The trimming apparatus according to any one of claims 1 to 4, further comprising: In order to crop and output the attention area of the video of sports performed in the field to the display device,
Image dividing means for dividing the frame image constituting the video into blocks of a predetermined size;
Image feature extraction means for extracting color information and luminance information in the block divided by the image dividing means as image features of the block;
The complexity of the shape of the entire field block formed by the field block, which is a block in which the color information extracted by the image feature extraction unit and the predetermined color information of the field are similar, exceeds a predetermined reference value. Shot classification means for classifying the shot of the video as a long shot based on whether or not
When the shot classification means classifies the shot as a long shot, based on the luminance information of the block, an area including a block of interest that is a block whose luminance variance value is higher than a predetermined luminance variance value. Attention area setting means for setting as the attention area;
Trimming video generation means that includes the attention area set by the attention area setting means, and extracts and outputs an area corresponding to the aspect ratio of the display device from the frame image as a trimming area;
A trimming processing program characterized by functioning as

Images