JP2013033321A - Movie presentation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically track an arbitrary tracking object on a distorted wide-angle image.SOLUTION: A distorted wide-angle image S obtained by fisheye-photographing is input by frame units in a memory 130. An image in an m size is segmented from the position of a point P of the distorted image S in a direction shown by φ on the basis of segmentation conditions in a storage part 170, and converted into a regular image T by a conversion part 150, and stored in a memory 140. When a tracking start instruction including a tracking start point Q is given, a flag is turned on by a flag setting part 220 so that the point Q can be updated as a new point P. The neighboring color of the new point P is extracted as a reference color α by a reference color extraction part 200, and stored in a reference color storage part 190. The neighboring block of the point P is extracted by a neighboring block extraction part 160, and the block in which the number of pixels approximating to the reference color α is made maximum is selected by a block selection part 180. The central point of a candidate area having a color which fits the most the reference color α in the selected block is updated as the new point P by an automatic changing part 210.

Description

  The present invention relates to a moving image presentation device, and in particular, a part of a distorted wide-angle image constituting each frame of a moving image obtained by wide-angle shooting is cut out and converted into a planar regular image, which is sequentially output to present the moving image. The present invention relates to a moving picture presentation apparatus.

  When photographing is performed using a wide-angle photographing device equipped with an optical system such as a fisheye lens, an image having an extremely wide field of view can be obtained. However, since an image obtained by such wide-angle imaging is originally an image that should be projected onto a curved surface, it becomes a distorted image when displayed on a flat display screen. Therefore, usually, a method is adopted in which a part of a distorted wide-angle image obtained by photographing using a fisheye lens is cut out, converted into a planar regular image with little distortion, and displayed on a display screen. .

  For example, the following Patent Documents 1 to 3 disclose a technique for converting a part of a distorted wide-angle image captured using a fisheye lens into a planar regular image using a computer. Patent Document 4 discloses a technique for converting a part of a distorted wide-angle image photographed using an omnidirectional mirror into a planar regular image, and Patent Document 5 photographed using a super-wide-angle camera mounted on an automobile. A technique for performing distortion correction on a distorted wide-angle image in real time is disclosed.

  Recently, in order to ensure security, an image taken by a security camera installed on the ceiling or wall of a building is displayed on a monitor screen to help monitor the surroundings. A camera for such a surveillance system often uses a fisheye lens. If the above-described conversion technique is used, a moving image composed of a distorted wide-angle image captured by a video camera equipped with a fisheye lens is converted into a moving image composed of a planar regular image. It becomes possible to observe in real time, and a monitoring system having an angle of view of 180 ° can be constructed.

  In the industrial field of such a monitoring system, a moving body tracking technique has been studied for a long time. This is a technique for analyzing a captured image obtained in time series in units of frames by a computer, and automatically recognizing and tracking a specific tracking target existing in the field of view of the camera. Attempts have also been made to introduce such moving body tracking technology into fisheye monitoring systems. For example, in Patent Documents 6 and 7 below, a distorted wide-angle image captured by a fisheye lens is analyzed to recognize a moving object, and a portion including the moving object is cut out and converted into a planar regular image and displayed on a monitor screen. Technology is disclosed.

  On the other hand, a captured image using a wide-angle imaging device has an advantage that the line-of-sight direction can be arbitrarily set at the time of reproduction. Therefore, a viewing device that takes advantage of such advantages has been proposed. For example, Patent Document 8 below discloses a technique for presenting a planar regular image in an arbitrary line-of-sight direction designated by a viewer on a head-up display based on a distorted wide-angle image obtained by a wide-angle imaging device. Yes. Taking advantage of the above advantages, it is expected that in the future, video content that allows viewers to freely change the viewing direction will be widespread by shooting a wide range of scenes and indoor situations using fisheye lenses. .

Japanese Patent No. 3012142 Japanese Patent No. 3051173 JP 2009-176273 A JP 2010-140292 A JP 2006-240383 A JP 2002-329207 A JP 2011-061511 A JP-A-8-336128

  As described above, there have been proposals for introducing a moving body tracking technique on a distorted wide-angle image obtained by a wide-angle imaging device. However, the methods proposed heretofore have practical problems, such as being unable to perform sufficiently accurate tracking and requiring a complicated process.

  The fundamental cause is that the tracking target is distorted on the distorted wide-angle image. On the planar regular image obtained by a normal photographing apparatus, the position and size of the tracking object may change due to movement, but the overall shape does not change greatly. In contrast, a distorted wide-angle image differs in the degree of image distortion from part to part. Therefore, when the tracking target moves, not only the position and size change, but also the overall shape changes greatly. . For this reason, it is difficult to apply a conventional general moving object recognition technique to a distorted wide-angle image as it is.

  As mentioned above, technologies for automatically tracking arbitrary tracking objects will be in demand in various fields in the future, such as surveillance systems using fisheye cameras and video content playback systems that can freely change the direction of the line of sight. This is a growing technology.

  In view of this, the present invention provides a simpler process for accurately tracking an arbitrary tracking object when a part of a moving image composed of a distorted wide-angle image obtained by wide-angle shooting is cut out and reproduced. It is an object of the present invention to provide a moving picture presentation apparatus that can be executed in a simple manner.

(1) In the first aspect of the present invention, a part of a distorted wide-angle image constituting each frame of a moving image obtained by wide-angle shooting is cut out and converted into a planar regular image, which is sequentially output to present a moving image. In the video presentation device that performs
A distorted wide-angle image memory for storing a distorted wide-angle image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY coordinate system;
An image input unit for sequentially storing a distorted wide-angle image of an i-th frame (i is an integer that sequentially increases in time series) sequentially given as time-series data in units of frames;
A planar regular image memory for storing a planar regular image composed of a collection of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system;
An image output unit that reads and outputs a planar regular image stored in the planar regular image memory;
As a condition for cutting out a planar regular image from a part of a distorted wide-angle image, a cutting center point P that is one point on the distorted wide-angle image, a parameter φ indicating the cutting direction of the image, and a predetermined magnification m are: A cutting condition storage unit for storing cutting conditions including,
Based on the cutting conditions stored in the cutting condition storage unit, parameters are determined from the cutting position indicated by the cutting center point P of the distorted wide-angle image of the i-th frame stored in the distorted wide-angle image memory. Cut out an image with a cut-out size indicated by the magnification m in the cut-out direction indicated by φ, convert it to a flat regular image, and store it in the planar regular image memory as a planar regular image of the i-th frame A conversion unit;
An instruction input unit for inputting instructions from the viewer;
When the instruction input unit inputs a cutting condition change instruction to change the cutting condition, a cutting condition manual changing unit that changes the cutting condition stored in the cutting condition storage unit based on the instruction When,
When the instruction input unit inputs a tracking start instruction including the position of the tracking start point Q on a specific planar regular image, the automatic tracking flag is turned ON, and the two-dimensional XY coordinate system corresponding to the tracking start point Q The automatic tracking start process for updating the cutting center point P stored in the cutting condition storage unit is executed with the upper point as the new cutting center point P, and the instruction input unit inputs the tracking end instruction An automatic tracking flag setting unit for executing an automatic tracking end process for switching the automatic tracking flag to OFF,
When the automatic tracking flag is ON, based on the plane regular image of the i-th frame stored in the plane regular image memory or the distorted wide-angle image of the i-th frame stored in the distorted wide-angle image memory. Then, i updates the reference color α of the i-th frame representing the color of the tracking object located at the extraction center point P for the i-th frame stored in the extraction condition storage unit. A reference color extraction unit that sequentially extracts each time,
A reference color storage unit that stores the latest reference color extracted by the reference color extraction unit;
When the automatic tracking flag is ON, the distorted wide-angle image of the (i + 1) th frame stored in the distorted wide-angle image memory is divided into a plurality of blocks, and the i-th stored in the extraction condition storage unit. A neighboring block extraction unit that extracts a plurality of blocks in the vicinity of the cut-out center point P for the th frame as neighboring blocks;
When the automatic tracking flag is ON, for each of a plurality of neighboring blocks, a predetermined value is set for the reference color α of the i-th frame stored in the reference color storage unit among the individual pixels included in the block. A block selection unit for obtaining an approximate color pixel number indicating the number of pixels having a color falling within the approximate range, and selecting a block having the maximum approximate color pixel number;
When the automatic tracking flag is ON, a plurality of candidate regions whose positions are different from each other are defined in the block selected by the block selection unit, and the i th stored in the reference color storage unit is selected from the plurality of candidate regions. A candidate area having a color that most closely matches the reference color α is selected as an optimal candidate area, and the center point of the selected optimal candidate area is set as a cut center point P for the (i + 1) th frame. A cutting condition automatic changing unit for updating information indicating the cutting center point P stored in the storage unit;
Is provided.

(2) According to a second aspect of the present invention, in the video presentation device according to the first aspect described above,
The reference color extraction unit extracts the color of one pixel including the cut center point P as the reference color α.

(3) According to a third aspect of the present invention, in the video presentation device according to the first aspect described above,
The reference color extraction unit defines a neighborhood area composed of the center pixel including the cut-out center point P and surrounding pixels around it, and extracts the average value of the colors of the pixels in the neighborhood area as the reference color α. Is.

(4) According to a fourth aspect of the present invention, in the video presentation device according to the first aspect described above,
The reference color extraction unit defines a neighborhood area composed of the center pixel including the cut-out center point P and surrounding pixels around the center pixel, and out of the pixels in the neighborhood area, the reference color extraction unit has a predetermined approximate range with respect to the color of the center pixel. A pixel having an input color is used as a reference pixel, and an average value of colors of the reference pixel is extracted as a standard color α.

(5) According to a fifth aspect of the present invention, in the video presentation device according to the third or fourth aspect described above,
The reference color extraction unit defines a square pixel array of n rows and n columns centering on a central pixel including the cut-out center point P as a neighboring region.

(6) According to a sixth aspect of the present invention, in the video presentation device according to the first to fifth aspects described above,
The neighboring block extraction unit divides the distorted wide-angle image into a plurality of rectangular blocks arranged in vertical and horizontal directions, and uses the block including the cut-out center point P as a central block, and the central block and its upper, lower, left, and right A total of nine blocks including adjacent blocks located in four diagonal directions (excluding blocks that do not exist in the corresponding position) are extracted as neighboring blocks.

(7) According to a seventh aspect of the present invention, in the video presentation device according to the first to sixth aspects described above,
When the cutting condition automatic changing unit defines a pixel array of a predetermined size and arranges the frame of the pixel array at a predetermined position in the block, the area in the frame is set as a candidate area, and the arrangement of the frame of the pixel array is determined. A plurality of candidate areas whose positions are different from each other are defined by shifting each pixel vertically and horizontally.

(8) According to an eighth aspect of the present invention, in the video presentation device according to the seventh aspect described above,
The automatic cutting condition changing unit is configured to select a candidate area whose pixel color average value in the candidate area is closest to the reference color α as the optimum candidate area.

(9) According to a ninth aspect of the present invention, in the video presentation device according to the seventh aspect described above,
The cutting condition automatic changing unit uses, as a reference pixel, a pixel having a color that falls within a predetermined approximate range with respect to the reference color α among the pixels in the candidate region, and the average value of the color of the reference pixel is set as the reference color α. The closest candidate area is selected as the optimum candidate area.

(10) According to a tenth aspect of the present invention, in the video presentation device according to the first to ninth aspects described above,
If the block selection unit sets the minimum reference value of the approximate color pixel number in advance and there is no block with the approximate color pixel number equal to or greater than the minimum reference value, the block selection unit does not select the block and the automatic tracking flag setting unit Give an automatic tracking failure signal to
The automatic tracking flag setting unit switches the automatic tracking flag to OFF when an automatic tracking failure signal is given.

(11) According to an eleventh aspect of the present invention, in the video presentation device according to the first to tenth aspects described above,
The cutting condition automatic changing unit sets a minimum matching criterion for the reference color α in advance, and when there is no candidate region whose degree of matching is equal to or greater than the minimum matching criterion, selection of the optimal candidate region and the cutting center point P An automatic tracking failure signal is given to the automatic tracking flag setting unit without updating the information indicating
The automatic tracking flag setting unit switches the automatic tracking flag to OFF when an automatic tracking failure signal is given.

(12) According to a twelfth aspect of the present invention, in the video presentation device according to the first, fourth, and ninth aspects described above,
As a condition for the color of the second pixel to fall within a predetermined approximate range of the color of the first pixel, the pixel value of each pixel is expressed in the HSV color system, and the value of hue H is an angle of 0 to 360 °. The condition that “the angle difference between the angle representing the hue H of the color of the first pixel and the angle representing the hue H of the color of the second pixel is equal to or less than a predetermined allowable angle θ” It is intended to be used.

(13) According to a thirteenth aspect of the present invention, in the video presentation device according to the first, fourth, and ninth aspects described above,
As a condition for the color of the second pixel to fall within a predetermined approximate range of the color of the first pixel, the pixel value of each pixel is expressed in the RGB color system, and the color of each pixel is represented in the three-dimensional RGB coordinate system. When plotting as coordinate points on the color space indicated by the above, the condition that “the Euclidean distance between the coordinate points of two pixels is equal to or less than a predetermined value” is used.

(14) According to a fourteenth aspect of the present invention, in the video presentation device according to the third, fourth, eighth, and ninth aspects described above,
When calculating the average value of the colors of a plurality of pixels, the pixel value of each pixel is expressed in the RGB color system, the average value Rav of the pixel value R, the average value Gav of the pixel value G, and the average of the pixel value B The value Bav is obtained, and the color indicated by these pixel values (Rav, Gav, Bav) is set as the average value of the colors of the plurality of pixels.

(15) According to a fifteenth aspect of the present invention, in the video presentation device according to the first to fourteenth aspects described above,
The instruction input unit
A controller device having four cursor movement buttons arranged vertically and horizontally and a tracking start instruction button;
Cursor position indicating means for instructing the image output unit of the position of the cursor that moves up, down, left, and right based on the operation of the four cursor movement buttons;
Have
The image output unit outputs an image in which the cursor is superimposed at the position indicated by the cursor position instruction means on the planar regular image,
When the tracking start instruction button is pressed, the automatic tracking flag setting unit executes the automatic tracking start process on the assumption that a tracking start instruction having the tracking start point Q as the position of the cursor at that time is input. It is a thing.

(16) According to a sixteenth aspect of the present invention, in the video presentation device according to the first to fourteenth aspects described above,
The instruction input unit includes a controller device having four screen movement buttons arranged vertically and horizontally, and a tracking start instruction button,
The extraction condition manual change unit stores the extraction center stored in the extraction condition storage unit so that the extraction position on the planar regular image moves up, down, left, and right based on the operation of the four screen movement buttons. A function to change the point P;
When the tracking start instruction button is pressed, the automatic tracking flag setting unit assumes that the tracking start instruction with the position of the cut-out center point P at that time as the tracking start point Q is input, and performs automatic tracking start processing. It is something to be executed.

(17) According to a seventeenth aspect of the present invention, in the video presentation device according to the fifteenth or sixteenth aspect described above,
When the tracking end instruction button provided in the controller device is pressed by the automatic tracking flag setting unit, or when the tracking start instruction button is pressed when the automatic tracking flag is ON, the tracking end instruction is input. As an example, an automatic tracking end process is executed.

(18) According to an eighteenth aspect of the present invention, in the video presentation device according to the fifteenth to seventeenth aspects described above,
The controller device further includes a magnification change button for increasing / decreasing the magnification m, and a direction changing button for increasing / decreasing a parameter φ indicating the image cutting direction,
When the cutting condition manual changing unit has pressed the magnification changing button or the direction changing button, it is assumed that the cutting condition changing instruction is input, and the magnification m or parameter φ stored in the cutting condition storing unit is set. It is something to change.

(19) According to a nineteenth aspect of the present invention, in the video presentation device according to the first to eighteenth aspects described above,
The image input unit inputs a circular distorted wide-angle image obtained by photographing using a fisheye lens, stores this in a distorted wide-angle image memory,
The neighboring block extraction unit divides a circular distorted wide-angle image into a plurality of rectangular blocks, and does not extract a block that partially protrudes from the outline of the circular distorted wide-angle image as a neighboring block. It is what I did.

  (20) In a twentieth aspect of the present invention, the moving picture presentation apparatus according to the first to nineteenth aspects described above is configured by incorporating a program into a computer.

  In the moving image presentation apparatus according to the present invention, the reference color α representing the tracking object is extracted from the image of the previous frame, and the position of the tracking object is determined on the image of the next frame using the reference color α as a clue. The estimated tracking method is taken. This eliminates the need for analysis regarding the shape of the tracking object, and enables accurate tracking even when the shape of the tracking object changes on the distorted wide-angle image. Also, when estimating the position of the tracking object on the image of the next frame, the image of the next frame is divided into a plurality of blocks, and blocks near the position of the tracking object in the previous frame are adjacent. The block that is extracted as a block, the block with the maximum number of pixels approximated to the reference color α is selected, and the position of the candidate area having the color that best matches the reference color α in the selected block is the position of the tracking target Therefore, it is possible to prevent erroneous detection of a region with similar colors in a distant block as a tracking target, and it is possible to perform accurate tracking processing by a relatively simple process with less computational burden. become.

It is a perspective view which shows the basic model which forms the distortion wide angle image S by imaging | photography using the fisheye lens of an orthogonal projection system. It is a top view which shows an example of the distortion wide-angle image S obtained by imaging | photography using a fisheye lens (It shows the general image of the distortion wide-angle image S, and does not show an exact image.). It is a top view which shows the example which cuts out the image of the cutting size shown by the magnification m from the cutting position shown by the cutting center point P of the distortion wide angle image S to the cutting direction shown by the parameter (phi). FIG. 3 is a plan view showing an example of a planar regular image T obtained on a two-dimensional UV orthogonal coordinate system by cutting out a part of the distorted wide-angle image S shown in FIG. 2. It is a top view which shows another example of the planar regular image T obtained on the two-dimensional UV orthogonal coordinate system by cutting out a part of the distortion wide-angle image S shown in FIG. It is a side view which shows the state which image | photographs outdoor scenery with the video camera 40 with a fisheye lens. It is a top view which shows an example of the distortion wide angle image S obtained by imaging | photography using the video camera 40 with a fisheye lens shown in FIG. FIG. 8 is a plan view showing an example in which two types of cutout regions E1 and E2 are set on the distorted wide-angle image S shown in FIG. It is a top view which shows the plane regular image T1, T2 obtained corresponding to two kinds of cutting area E1, E2 shown in FIG. It is a top view which shows an example of the tracking target object 60 which exists in the distortion wide angle image S obtained by imaging | photography using the video camera 40 with a fisheye lens shown in FIG. It is a top view which shows the state which the tracking target object 60 moved on the distortion wide angle image S shown in FIG. 12 is a plan view showing a planar regular image obtained by tracking a tracking target object 60 with respect to the distorted wide-angle image S shown in FIGS. 10 and 11. FIG. It is a block diagram which shows the structure of the moving image presentation apparatus which concerns on fundamental embodiment of this invention. It is a front view of the controller apparatus used as the hardware component of the instruction | indication input part 240 shown in FIG. It is a top view which shows the image cutting-out process about the i-th flame | frame by the image cutting-out conversion part 150 shown in FIG. FIG. 16 is a plan view illustrating an example of a planar regular image obtained by the image cutting process illustrated in FIG. 15. It is a top view which shows the state which displayed the cross cursor C on the plane regular image shown in FIG. FIG. 18 is a plan view showing a state in which the cross cursor C shown in FIG. As shown in FIG. 18, a plane showing a planar regular image T (i + 1) of the (i + 1) th frame obtained when a tracking start instruction is given in a state where the cross cursor C is moved onto the tracking target 60. FIG. It is a top view which shows the relationship between the distortion wide-angle image S (i) of the i-th frame when the crosshair cursor C is moved on the tracking target object 60, and the tracking start instruction | indication is given, and the planar regular image T (i). is there. FIG. 21 is a plan view showing a relationship between a distorted wide-angle image S (i + 1) and a planar regular image T (i + 1) of the (i + 1) -th frame when a tracking start instruction is given in the state shown in FIG. FIG. 21 is a plan view showing an example of reference color extraction processing when a tracking start instruction is given in the state shown in FIG. 20. It is an enlarged view of the periphery of the vicinity area | region N of FIG. It is a figure which shows the color system conversion process about the pixel in the vicinity area | region N shown in FIG. It is a figure which shows the example of a setting of the approximate condition of the hue H in a HSV color system. It is a top view which shows an example of the neighboring block extraction process performed by the neighboring block extraction part 160 shown in FIG. It is a figure which shows the process which selects one block from the neighboring blocks B1-B9 shown in FIG. It is a top view which shows an example of the selection process of the optimal candidate area | region performed by the cutting condition automatic change part 210 shown in FIG. It is a figure which shows the process sequence which investigates the approximation degree with reference | standard color (alpha) about the kth candidate area | region A (k) shown in FIG. It is a top view which shows the state from which a neighboring block is extracted from the distortion wide angle image S (i + 2) of the (i + 2) th frame. It is a top view which shows the actual example of the reference | standard color extraction process performed on the planar regular image T (i + 1) of the (i + 1) th frame. It is a perspective view which shows a general RGB color space. It is a figure which shows the computing equation which calculates | requires the Euclidean distance d shown in FIG.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Relationship between distorted wide-angle image and planar regular image >>>
The present invention relates to a moving picture presentation apparatus that cuts out a part of a distorted wide-angle image constituting each frame of a moving picture obtained by wide-angle shooting, converts it into a planar regular image, and sequentially outputs this to present a moving picture. It is. Here, first, the relationship between a distorted wide-angle image obtained by wide-angle imaging and a planar regular image obtained by converting the image will be briefly described. The most popular device as a wide-angle photographing device is a video camera equipped with a fisheye lens. Therefore, the relationship with a planar regular image will be described below by taking a distorted wide-angle image obtained by photographing using a fisheye lens as an example.

  By using a fisheye lens, it is possible to take a scene of the outside world with a hemispherical field of view. However, since the obtained captured image is a distorted circular wide-angle image, in order to display it on a normal display screen, it is necessary to perform processing for converting the distorted wide-angle image into a planar regular image with less distortion. . Hereinafter, a general characteristic of a distorted wide-angle image obtained by photographing using a fisheye lens and a basic principle of processing for cutting out a part of the image and converting it into a planar regular image will be described.

  FIG. 1 is a perspective view showing a basic model for forming a distorted wide-angle image S by photographing using an orthographic fisheye lens. In general, fish-eye lenses are classified into a plurality of types according to the projection method, but the model shown in FIG. 1 is for a fish-eye lens of an orthographic projection method. FIG. 1 shows an example in which a distorted wide-angle image S is formed on an XY plane in a three-dimensional XYZ orthogonal coordinate system. Here, as shown in the figure, an example is shown in which the Z axis is taken upward in the figure and a dome-like virtual spherical surface H (hemisphere) is defined on the positive region side of the Z axis.

  The distorted wide-angle image S formed on the XY plane is an image that forms a circle with a radius r centered on the origin O of the coordinate system, and is an area having an angle of view of 180 ° on the positive area side of the Z axis. Is equivalent to the image recorded with distortion. FIG. 2 is a plan view showing an example of a distorted wide-angle image S obtained by photographing using a fisheye lens. As described above, in the distorted wide-angle image S, all the images existing on the positive region side of the Z axis are recorded, but the scale scale of the image is different between the central portion and the peripheral portion. The shape of the recorded image is distorted. Note that the distorted wide-angle image S shown in FIG. 2 shows a general image of a distorted wide-angle image obtained by shooting using a fisheye lens, and does not show an accurate image obtained using an actual fisheye lens. Absent.

  An actual fisheye lens is constituted by an optical system that combines a plurality of convex lenses and concave lenses, and its optical characteristics can be modeled by a virtual spherical surface H as shown in FIG. That is, considering a model in which a dome-shaped virtual spherical surface H (hemisphere) having a radius r is arranged on the upper surface of the distorted wide-angle image S, the optical characteristics of the orthographic fish-eye lens are arbitrary on the virtual spherical surface H. The incident light beam L1 incident from the normal direction to the point H (x, y, z) behaves toward the point S (x, y) on the XY plane as an incident light beam L2 parallel to the Z axis. You may think. In other words, the pixel located at the point S (x, y) on the distorted wide-angle image S in FIG. 2 indicates one point on the object existing on the extension line of the incident light beam L1 shown in FIG. become. In other words, the distorted wide-angle image S can be said to be a projection image obtained by orthogonally projecting an image drawn on the inner surface of the hemispherical dome H onto the XY plane.

  Of course, an optical phenomenon occurring in an actual fisheye lens is that a specific point of an object to be imaged is connected to a specific point S (x, y) on the XY plane due to refraction by a plurality of convex lenses and concave lenses. Although this is an image phenomenon, there is no problem in performing an image conversion process or the like even if the argument is replaced with a model using a virtual spherical surface H as shown in FIG. Therefore, even in the image conversion processing disclosed in the above-mentioned patent document, a technique based on such a model is shown. In the following description of the present invention, description based on such a model is also given. To do.

  In the apparatus according to the present invention, it is necessary to cut out a part of the distorted wide-angle image S and convert it into a planar regular image. For example, suppose that a viewer who has viewed the distorted wide-angle image S shown in FIG. 2 wants to observe the image of the woman drawn in the lower left of the image with a correct image without distortion. In such a case, the viewer needs to specify which part of the distorted wide-angle image S should be cut out and converted. FIG. 3 is a plan view showing an example of processing for cutting out a part of the distorted wide-angle image S shown in FIG. In this example, a fan-shaped cutout area E is drawn, and a part of the distorted wide-angle image S is cut out from the cutout area E and converted into a planar regular image.

  FIG. 4 is a plan view showing a planar regular image T obtained by performing conversion processing on the partial image in the cutout region E of the distorted wide-angle image S shown in FIG. Here, the distorted wide-angle image S is a circular image defined on a two-dimensional XY orthogonal coordinate system as shown in FIG. 3, and the planar regular image T is a two-dimensional UV orthogonal coordinate system as shown in FIG. Assume that it is a rectangular image defined above. As shown in the live-action example in FIG. 2, the female image on the distorted wide-angle image S is distorted. However, as shown in the real-shot example in FIG. So the distortion has been corrected. Therefore, if the planar regular image T is displayed on the display screen, it is possible to present an image with no discomfort to the viewer.

  Of course, it is possible to convert any part of the distorted wide-angle image S shown in FIG. 2 into a planar regular image T according to the viewer's request, and instead of a female image, an image of a tree or an image of a building It is also possible to obtain a planar regular image T for. Eventually, in order to obtain a specific planar regular image T, it is only necessary to determine a cutting condition such as from which position of the distorted wide-angle image S to which direction and how large the partial image is cut out. Geometrically, this cutting condition is constituted by three conditions: a cutting center point P, a parameter φ indicating the cutting direction, and a magnification m.

  In the example shown in FIG. 3, the point P (xp, yp) is the cut center point P. This cut-out center point P (xp, yp) can be defined by a coordinate value (xp, yp) on the XY coordinate system. As shown in FIG. 4, the cut-out center point P is an image center point on the planar regular image T, and occupies an origin position having a coordinate value (0, 0) on the UV coordinate system.

  On the other hand, the parameter φ is a parameter for determining an angle generally called a “planar inclination angle”, and is defined as an angle formed by the cutting direction line D and the Y axis (or X axis) as shown in FIG. The This angle φ is a parameter for determining the orientation of the planar regular image T, and the differential direction of the cut-out direction line D at the position of the point P in FIG. 3 coincides with the direction of the U axis at the position of the point P in FIG. To do. When the value of the angle φ is changed and, for example, the cutting direction line D is set so as to face the direction from the point P toward the origin O, the obtained planar regular image T is obtained as shown in FIG. Become an image. 4 and 5 are images obtained by designating the same cut-out center point P (xp, yp), so both are images having the point P as the center point, but the cut-out direction is determined. Since the parameter φ shown is different, the female orientation is different.

  The magnification m is a parameter that determines a scaling factor at the time of conversion. In the above example, if m is increased, the image of a woman on the planar regular image T is enlarged, and if m is decreased, the image on the planar regular image T is increased. Women's images are reduced. In other words, the magnification m is a parameter for determining the size of the cutout area E shown in FIG. 3. When m is increased, the area of the cutout area E is reduced, and when m is reduced, the cutout area E is reduced. The area of becomes larger. Comparing the image shown in FIG. 4 with the image shown in FIG. 5, it can be seen that the image of the woman is enlarged in the latter case. Therefore, both have not only different parameters φ indicating the cutting direction but also different magnifications m (the latter has a larger magnification m and the area of the cutting region E is smaller).

  Since the screen size of the display device that displays the planar regular image T is usually constant, both the image T shown in FIG. 4 and the image T shown in FIG. 5 have the horizontal dimension a (the number of pixels in the horizontal direction) and the vertical direction. The dimension b (number of pixels in the vertical direction) is the same. Therefore, the resolution of the planar regular image T becomes higher or lower than the resolution of the pixels in the cutout region E depending on the setting of the magnification m. In the former case, pixel interpolation is performed during conversion, and in the latter case, pixel thinning is performed during conversion.

  As described above, in order to obtain a specific planar regular image T from the distorted wide-angle image S, it is only necessary to set three cutting conditions, that is, the cutting center point P, the parameter φ indicating the cutting direction, and the magnification m. Recognize. When these three cut-out conditions are determined, geometrically, the UV plane shown in FIG. 4 with respect to the point Si (xi, yi) on the distorted wide-angle image S defined on the XY plane shown in FIG. The points Ti (ui, vi) on the planar regular image T defined above correspond one-to-one. That is, a coordinate conversion formula between the point Si (xi, yi) on the XY coordinate system and the point Ti (ui, vi) on the UV coordinate system can be defined.

  The conversion process from the distorted wide-angle image S to the planar regular image T is an arithmetic process based on such a coordinate conversion formula. Actually, the corresponding coordinates (x, y) on the XY plane corresponding to the coordinates (u, v) indicating the center position of each of the a × b pixels constituting the planar regular image T shown in FIG. ) On the basis of the above coordinate conversion formula, the pixel at the position of the corresponding coordinate (x, y) on the distorted wide-angle image S is taken as the corresponding pixel, and the pixel value of this corresponding pixel is determined as the pixel on the planar regular image T. It is sufficient to perform a process of setting the pixel value to (if necessary, an interpolation process is performed). The cutout area E shown in FIG. 3 is determined as the distribution area of the pixel group that has become the corresponding pixel in the above processing (in the figure, for convenience, the cutout area E is shown in a fan shape, but in practice, , Become more complex shapes).

  In the actual shooting example shown in FIG. 2, the camera is arranged so that the XY plane in the model shown in FIG. 1 coincides with the horizontal plane and the street scene is photographed. Of course, the XY plane is on a vertical plane. It is also possible to shoot by arranging the cameras so that they match. FIG. 6 is a side view showing a state in which an outdoor landscape composed of the road surface 10, the tree 20, and the guard rail 30 is photographed with a photographing device in which a fisheye lens 50 is attached to the video camera 40. FIG. It is a top view which shows the obtained distortion wide angle image S. FIG. Since the XY plane of the fisheye lens 50 is a vertical plane, focusing on the subject located at the origin O, the X axis is in the horizontal direction and the Y axis is in the vertical direction. However, the further away from the origin O, the more severe the distortion of the image.

  No matter what direction the XY plane shown in FIG. 1 is arranged and photographed, there is no change in the geometric optical phenomenon, so the distorted wide-angle image S shown in FIG. A part can be cut out and converted into a planar regular image. FIG. 8 is a plan view showing an example in which two cutout areas E1 and E2 are set on the distorted wide-angle image S shown in FIG. 7, and FIG. 9 shows the two cutout areas E1 and E2. It is a top view which shows the plane regular image T1, T2 obtained correspondingly. As illustrated, the planar regular image T1 is an image of the portion of the tree 20, and the planar regular image T2 is an image of the portion of the guardrail 30.

  As described above, in practice, the cutout areas E1 and E2 are determined by setting three cutout conditions. That is, the cutout area E1 shown in FIG. 8 specifies the coordinates of the cutout center point P1, the cutout direction φ1 (in the illustrated example, the direction in which the cutout direction line D faces right), and the magnification m1. The cut-out area E2 is determined by the coordinates of the cut-out center point P2, the cut-out direction φ2 (in the illustrated example, the direction in which the cut-off direction line D faces slightly upward to the right), and the magnification m2. It is uniquely determined by specifying. In the case of this example, since the magnification m2> m1, the area of the cutout region E2 is smaller than the area of E1. Note that the shape of the cutout areas E1 and E2 is actually a complicated shape surrounded by a curved outline, but in FIG. 8, for convenience, it is shown as a rectangular area. In the present application, hereinafter, for convenience of explanation, each cutout region is illustrated as a rectangular region.

  So far, the relationship between the distorted wide-angle image S as one still image and the planar regular image T obtained by cutting out a part of the image has been described. If these still images are presented sequentially in time series, Can be presented. For example, if a video camera equipped with a fisheye lens is used to shoot moving images at a rate of 30 frames per second, 30 distorted wide-angle images S as shown in FIG. 7 can be obtained per second. If a plane regular image T is cut out from each of the 30 distorted wide-angle images S and presented sequentially, a moving image can be presented for 1 second composed of the plane regular image.

  In addition, since it is possible to change the cutting conditions during the reproduction of the moving image, even if the moving image data consisting of the same distorted wide-angle image S is used, the cutting conditions freely set by the viewer at the time of reproduction are used. The content of a moving image consisting of a planar regular image presented on the display screen changes. For example, in the first half of the video, by specifying the cut-out center point P1, as shown in FIG. 9A, the tree 20 is displayed, and in the second half, the cut-out center point P2 is specified. As shown in FIG. 9 (b), a moving image presentation mode in which the guardrail 30 is displayed becomes possible.

  An object of the present invention is to perform processing for automatically tracking an arbitrary tracking target when a part of a moving image composed of a distorted wide-angle image is cut out and reproduced. For example, assume that a cat 60 appears in the distorted wide-angle image S (1) at a certain time t1 as shown in FIG. In this case, as shown in the drawing, if the cut center point P (1) is set on the cat 60 and the cut direction φ and the magnification m are appropriately set, the cut region E (1) including the cat 60 is determined. be able to. Then, it is assumed that the cat moves to the left at the subsequent time points t2, t3,..., T9, and at the time point t9, a distorted wide-angle image S (9) as shown in FIG. Also in this case, as shown in the figure, if the cutting center point P (9) is set on the cat 60 and the cutting direction φ and the magnification m are appropriately set, the cutting region E (9) including the cat 60 is set. Can be determined.

  FIGS. 12A and 12B are plan views showing the planar regular images T (1) and T (9) obtained based on such a cutting condition. The planar regular image T (1) is an image corresponding to the cutout region E (1) in the distorted wide-angle image S (1) at the time point t1 shown in FIG. 10, and the planar regular image T (9) is illustrated in FIG. Is an image corresponding to the cutout region E (9) in the distorted wide-angle image S (9) at the time point t9 shown in FIG. All the images are images including the cat 60 in the screen.

  If the viewer gives an instruction to sequentially change the position of the cut-out center point P while playing back a moving image, the viewer performs a manual tracking operation so that the cat 60 is always displayed in the planar regular image T. Can do. For example, assume that a cutting condition is set such that a planar regular image T (1) as shown in FIG. In this case, if the cutting condition remains constant, the cat 60 disappears to the left side of the screen as time elapses at time points t2 and t3. Therefore, if the viewer performs an instruction operation on the planar regular image T to move the cut center point P to the left, the viewer can follow the cat 60 with the screen frame.

  However, since it is difficult to accurately predict the destination of the cat 60, such a manual tracking operation is inaccurate, and the burden on the viewer must be increased. The present invention proposes an efficient method capable of automatically tracking the cat 60 in such a case. When the moving picture presentation device according to the present invention is used, for example, when a cat 60 shown in FIG. 10 is designated as a tracking target and a tracking start instruction is given at time t1, time t2, t3,. Accordingly, the position of the cut-out center point is automatically updated to P (2), P (3),..., P (9), and the cat 60 that is the tracking object is displayed on the planar regular image T. It will continue to be displayed. The specific mechanism will be described below.

<<< §2. Basic configuration and operation of moving picture presentation apparatus according to the present invention >>
FIG. 13 is a block diagram showing the configuration of the video presentation device according to the basic embodiment of the present invention. In the figure, an image input unit 110 is a component that inputs a moving image captured by a wide-angle imaging device such as a video camera equipped with a fisheye lens. As described above, such a moving image is composed of a plurality of distorted wide-angle images S given in time series. On the other hand, the image output unit 120 is a component that outputs a planar regular image T that is cut out from a part of the input distorted wide-angle image S and converted. The fundamental function of this moving picture presentation apparatus is to create a planar regular image T based on the distorted wide-angle image S input by the image input unit 110 and output it.

  The distorted wide-angle image memory 130 is a component that stores the distorted wide-angle image S input by the image input unit 110. As described in §1, the distorted wide-angle image S is an image composed of a collection of a large number of pixels arranged at the position indicated by the coordinates (x, y) on the two-dimensional XY coordinate system. When shooting using is performed, a circular image is obtained. On the other hand, the planar regular image memory 140 is a component that stores the planar regular image T obtained by the conversion. As described in §1, the planar regular image T is an image constituted by an aggregate of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system, In the case of the example shown in (2), it is a rectangular image suitable for displaying on a normal display screen.

  Here, the following description will be given by taking as an example a case where the image input unit 110 inputs a distorted wide-angle image S constituting a moving image shot at a rate of 30 frames per second. In this case, the image input unit 110 receives the i-th (i is the first frame, the second frame, the third frame,... In other words, the distorted wide-angle images of the integers that increase sequentially in time series are sequentially stored in the distorted wide-angle image memory 130.

  The distorted wide-angle image memory 130 has a function of storing at least one frame of the distorted wide-angle image S sequentially given from the image input unit 110 in units of frames. The distorted wide-angle image S stored in the distorted wide-angle image memory 130 can be deleted after the processing is completed. Similarly, the planar regular image memory 140 has a function of storing the planar regular image T in units of frames after conversion for at least one frame. The planar regular image T stored in the planar regular image memory 140 is read by the image output unit 120 and output to an external device (such as a display device). The planar regular image T stored in the planar regular image memory 140 can also be deleted after the processing is completed.

  The image cutout conversion unit 150 cuts out a part of the distorted wide-angle image S (i) of the i-th frame stored in the distorted wide-angle image memory 130, converts it into a planar regular image, and converts this to the i-th frame. A process of storing the frame regular image T (i) in the plane regular image memory 140 is performed. As described in §1, in order to perform the process of cutting out the planar regular image T from a part of the distorted wide-angle image S, the cut-out center point P that is one point on the distorted wide-angle image S and the cut-out direction of the image are set. What is necessary is just to determine the cutting condition containing the parameter (phi) shown and the predetermined magnification m.

  The cutting condition storage unit 170 is a component that stores these three cutting conditions P, φ, and m. The image cut-out conversion unit 150, based on the cut-out conditions stored in the cut-out condition storage unit 170, the distorted wide-angle image S (i) of the i-th frame stored in the distorted wide-angle image memory 130. ) From the cut-out position indicated by the cut-out central point P in the cut-out direction indicated by the parameter φ, cut out an image of the cut-out size indicated by the magnification m, and converts this to a planar regular image to convert the i-th frame. A process of storing the planar regular image T (i) in the planar regular image memory 140 is performed.

  As described in §1, the conversion process from the distorted wide-angle image S to the planar regular image T executed by the image cutout conversion unit 150 is actually performed for each individual pixel constituting the planar regular image T. It can be said that the pixel value of the corresponding pixel in the distorted wide-angle image S stored in the distorted wide-angle image memory 130 is given.

  For example, in order to obtain the pixel value of a specific pixel constituting the planar regular image T, coordinates (u, v) indicating the center position of the specific pixel are obtained, and the XY plane corresponding to the coordinates (u, v) is obtained. The corresponding coordinates (x, y) above are obtained based on a predetermined coordinate conversion formula. Then, the pixel value of the corresponding pixel may be obtained by setting the pixel at the position of the corresponding coordinate (x, y) on the distorted wide-angle image S stored in the wide-angle image memory 130 as the corresponding pixel. The image cut-out conversion unit 150 performs a process of writing the obtained pixel value for the specific pixel in the planar regular image memory 140.

  Of course, if necessary, a plurality of corresponding pixels in the vicinity of the corresponding coordinates (x, y) are determined, and an interpolation operation is performed on the pixel values of the plurality of corresponding pixels to determine a final pixel value. By doing so, more accurate image conversion can be performed. Note that coordinate conversion formulas for obtaining coordinates (x, y) on the XY plane corresponding to arbitrary coordinates (u, v) on the UV plane are known as described in the above-mentioned patent documents. The detailed description is omitted here.

  As described above, the image cut-out conversion unit 150 cuts out a part of the distorted wide-angle image S based on the three cut-out conditions P, φ, and m stored in the cut-out condition storage unit 170 to form a planar regular image T. Although the process to convert is performed, these three extraction conditions P, (phi), m can be changed by operation of the viewer who receives presentation of a moving image. That is, when the viewer inputs a cutting condition change instruction for changing the cutting condition to the instruction input unit 240, the cutting condition manual changing unit 230, based on the instruction, the cutting condition storage unit 170. The process of changing the cutting conditions P, φ, m stored in is performed.

  Although a specific method for inputting an instruction to change the cutting conditions P, φ, m will be described later, the viewer can change the cutting center point P to change the arbitrary image captured in the distorted wide-angle image S. The plane regular image T centered on the target object can be displayed, and by changing the parameter φ indicating the cutout direction, the plane regular image T can be displayed in an arbitrary direction, and the magnification m is changed. By doing so, the planar regular image T can be displayed at an arbitrary magnification.

  Each component described above is a component that executes the basic function of this moving image presentation device, and is known in a conventional general moving image presentation device having a function of presenting a moving image obtained by wide-angle shooting. Is a component of The feature of the present invention is that an automatic tracking function for an arbitrary tracking target designated by the viewer is realized by adding some new components described later to such a known video presentation device. It is in.

  That is, the moving picture presentation device according to the present invention further includes a neighboring block extraction unit 160, a block selection unit 180, a reference color storage unit 190, a reference color extraction unit 200, an extraction condition automatic change unit 210, and an automatic tracking flag setting unit. 220 is added, and an automatic tracking function is realized by these components.

  Before describing the functions of these new components, here, a controller device that is a hardware component of the instruction input unit 240 will be described. FIG. 14 is a front view showing an example of the controller device 300. In practice, the controller device 300 can be used as it is as a general-purpose game controller device that is used by being connected to a personal computer or a dedicated game machine. The illustrated connection cable 350 is a cable for connecting to a personal computer, a game dedicated machine, or the like. If a dedicated interface for accepting an input from the game controller device is prepared in a program that is a software component of the instruction input unit 240, the viewer can use the game controller device to use the moving image. A desired instruction can be input to the presentation device.

  As illustrated, a number of buttons are arranged on the operation panel unit 310 provided in front of the controller device 300. The screen movement button 320 is a button used to input a cutting condition change instruction for changing the above-described cutting center point P. In the illustrated example, the screen movement button 320 is configured by a single circular operation panel, but the operation panel is inclined vertically and horizontally on the operation panel unit 310 and is substantially vertically and horizontally disposed. Functions as four screen movement buttons. That is, when the upper part of the circular operation panel is pushed in and tilted, it is recognized that the upper screen movement button is pressed, and when the lower part is pushed and tilted, it is recognized that the lower screen movement button is pressed, When the part is pushed and tilted, it is recognized that the left screen movement button is pressed, and when the right part is pushed and tilted, it is recognized that the right screen movement button is pressed.

  The cutting condition manual changing unit 230 is stored in the cutting condition storage unit 170 so that the cutting position on the planar regular image T moves up and down and left and right based on the operation of the four screen movement buttons. A process of changing the cut center point P is performed (specifically, a process of changing the coordinate values xp and yp of the cut center point P is performed).

  Here, as shown in FIG. 15, let us consider a state in which a planar regular image T (i) of the i-th frame is cut out from the distorted wide-angle image S (i) of the i-th frame. Such a cutting process is performed based on the cutting conditions P (i), φ (i), and m (i) stored in the cutting condition storage unit 170. That is, the image cut-out conversion unit 150 performs magnification m (i) from the cut-out position indicated by the cut-out center point P (i) of the distorted wide-angle image S (i) in the cut-out direction indicated by the parameter φ (i). The image of the cut-out size indicated by is cut out and converted into a planar regular image T (i). The origin G of the planar regular image T (i) obtained in this way is a point corresponding to the cut-out center point P (i).

  FIG. 16 is a plan view of a planar regular image T (i) of the i-th frame obtained by such a cutting process. Consider a case where the viewer operates the circular operation panel 320 in a state where such a planar regular image T (i) is output from the image output unit 120 and displayed on the display screen. For example, when the viewer pushes in on the upper part of the circular operation panel 320, the cutout position moves upward on this screen so that the upper space in FIG. That is, the extraction condition storage unit so that the extraction center point P (i + 1) used for the extraction of the (i + 1) th frame becomes a point moved in the direction of the origin O on the XY plane shown in FIG. 170 is rewritten.

  Similarly, in the state where the planar regular image T (i) as shown in FIG. 16 is displayed, when the viewer pushes in the right part of the circular operation panel 320, the right space of FIG. In addition, the cutout position moves to the right side of this screen. That is, the cutting center point P (i + 1) used for cutting out the (i + 1) th frame is counterclockwise along the circumferential direction from the position of the cutting center point P (i) shown in FIG. The clipping condition storage unit 170 is rewritten so as to be the moved point. As a result, in the planar regular image T (i + 1) of the (i + 1) th frame, the cat 60 is positioned closer to the center of the screen.

  In this way, the viewer can move the line-of-sight direction in a desired direction by tilting the screen movement button 320 up, down, left, and right while watching the moving image composed of the planar regular image T displayed on the display screen. (Operation for manually changing the cutting center point P) can be performed. Therefore, as described in §1, a manual tracking operation can be performed so that the cat 60 is always displayed on the display screen.

  Further, the viewer can change the magnification and direction of the planar regular image T displayed on the display screen by operating the magnification direction change button 330. Specifically, the magnification direction change button 330 is composed of four individual buttons as illustrated. Here, the magnification change button 331 is a button for increasing the magnification m. When the button is continuously pressed, the extraction condition manual change unit 230 sets the magnification m stored in the extraction condition storage unit 170. A process of gradually increasing is performed. The magnification change button 332 is a button for decreasing the magnification m. When the button is continuously pressed, the cutting condition manual changing unit 230 gradually increases the magnification m stored in the cutting condition storage unit 170. The process to decrease is performed. With these buttons, the viewer can perform zoom-in and zoom-out operations on the planar regular image T displayed on the display screen.

  On the other hand, the direction change buttons 333 and 334 are buttons for increasing or decreasing the parameter φ indicating the image cut-out direction. When these buttons are kept pressed, the value of the parameter φ gradually increases or decreases. With these buttons, the viewer can perform an operation of rotating the planar regular image T displayed on the display screen clockwise or counterclockwise.

  In short, the cutting condition manual changing unit 230 assumes that the cutting condition changing instruction has been input when the magnification changing buttons 331 and 332 or the direction changing buttons 333 and 334 are pressed, and the cutting condition storing unit 170 Processing to change the stored magnification m or parameter φ is performed.

  On the other hand, a cursor movement button 340 arranged on the right side of the operation panel unit 310 in FIG. 14 is a button for using the automatic tracking function that is a feature of the present invention. As shown in the figure, this cursor movement button 340 is composed of five individual buttons: an upward movement button 341, a downward movement button 342, a leftward movement button 343, a rightward movement button 344, and a tracking start instruction button 345. Yes. In the embodiment shown here, when one of these five individual buttons is pressed, a cross cursor C is displayed at the center on the display screen of the display as shown in FIG. The crosshair cursor C is used to specify a tracking target.

  The viewer can move the cross cursor C up, down, left, and right by pressing individual cursor movement buttons 341 to 344 arranged in the up, down, left, and right directions. The cursor C can be moved on the top. Specifically, as a software component part of the instruction input unit 240, a cursor that instructs the image output unit 120 on the position of the cursor that moves up, down, left, and right based on the operation of the four cursor movement buttons 341-344. A program that functions as a position instruction unit is prepared, and the image output unit 120 is provided with a function of outputting an image in which the cross cursor C is superimposed on the position specified by the cursor position instruction unit on the planar regular image T. Just keep it.

  Here, it is assumed that the viewer is interested in the cat 60 appearing on the screen, and the screen frame automatically tracks the cat 60 so that the cat is always displayed on the screen. . In this case, the viewer may specify the cat 60 as the tracking target and perform an operation for giving a tracking start instruction. In the case of the embodiment shown here, an operation of pressing the tracking start instruction button 345 by moving the cross cursor C onto the cat 60 as the tracking target while pressing the cursor movement buttons 341 to 344 may be performed.

  FIG. 18 is a plan view showing a state in which the cross cursor C shown in FIG. 17 is moved onto the cat 60 to be tracked. When the viewer presses the tracking start instruction button 345 in this state, the automatic tracking process starts. Here, the position of the cross cursor C when the tracking start instruction button 345 is pressed is referred to as a tracking start point Q. Also, assume that the image (image in FIG. 18) displayed when the tracking start instruction button 345 is pressed is the planar regular image T (i) of the i-th frame. In this case, the instruction input unit 240 has input a tracking start instruction including the position of the tracking start point Q on the specific planar regular image T (i).

  When the moving image presentation apparatus shown in FIG. 13 receives such a tracking start instruction, when the distorted wide-angle image S (i + 1) of the (i + 1) th frame arrives, it is stored in the extraction condition storage unit 170. It has a function of performing a process of updating the cut-out center point P (i) that has been performed to a new cut-out center point P (i + 1). As a result, the extraction process for the distorted wide-angle image S (i + 1) of the (i + 1) th frame is performed based on the new extraction center point P (i + 1), as shown in FIG. A planar regular image T (i + 1) of the (i + 1) th frame in which the cat 60 is located at the center of the screen is obtained.

  Thereafter, until the viewer inputs a tracking end instruction (or until the automatic tracking process fails as described later), the automatic tracking mode is maintained, and the cat 60 as the tracking target is always at the center of the screen. Thus, the cutting center point P is automatically updated. That is, the cutting process for the distorted wide-angle image S (i + 2) of the (i + 2) th frame is performed based on the new cutting center point P (i + 2), and the planar regular image T (i + 2) is obtained. Subsequently, the cutting process for the distorted wide-angle image S (i + 3) of the (i + 3) th frame is performed based on the new cutting center point P (i + 3), and the planar regular image T (i + 3) is obtained. , ... and so on.

  The automatic tracking flag setting unit 220 in the video presentation device shown in FIG. 13 has a function of switching the ON / OFF state of this automatic tracking mode. That is, the automatic tracking flag setting unit 220 has a built-in flag indicating the ON / OFF state, and the instruction input unit 240 has input a tracking start instruction including the position of the tracking start point Q on a specific planar regular image. Sometimes, an automatic tracking start process for switching the automatic tracking flag to ON is executed, and when the instruction input unit 240 inputs a tracking end instruction, an automatic tracking end process for switching the automatic tracking flag to OFF is executed.

  When the controller device 300 shown in FIG. 14 is used as a hardware component of the instruction input unit 240, when the tracking start instruction button 345 is pressed, the automatic tracking flag setting unit 220 displays the crosshair cursor C at that time. Assuming that a tracking start instruction having the position as the tracking start point Q is input, the automatic tracking start process is executed.

  Note that the automatic tracking start process executed when the automatic tracking flag setting unit 220 inputs a tracking start instruction includes another important process in addition to the process of switching the automatic tracking flag to ON. In other words, when the tracking start instruction including the position of the tracking start point Q on the specific planar regular image is input, the automatic tracking flag setting unit 220 switches the automatic tracking flag to ON and corresponds to the tracking start point Q. The process of updating the cutting center point P stored in the cutting condition storage unit 170 is executed with the point on the two-dimensional XY coordinate system to be set as a new cutting center point P.

  For example, as in the example shown in FIG. 18, when a tracking start instruction specifying the tracking start point Q on the cat 30 is input, the automatic tracking flag setting unit 220 switches the automatic tracking flag to ON, A point on the two-dimensional XY coordinate system corresponding to the tracking start point Q is set as a new cut center point P (i) new, and the cut center point (here, P ( i) (referred to as old) is updated (that is, P (i) old is rewritten to P (i) new). In the block diagram shown in FIG. 13, an arrow with a letter P directed from the automatic tracking flag setting unit 220 to the extraction condition storage unit 170 indicates such update processing of the extraction center point P. Note that P (i) new is different from P (i + 1) shown in FIG.

  Here, in order to facilitate understanding, differences between P (i) old, P (i) new, and P (i + 1) will be briefly described with reference to FIGS. Now, in a state where the distorted wide-angle image S (i) of the i-th frame input by the image input unit 110 is stored in the distorted wide-angle image memory 130, the clipping process by the image clipping conversion unit 150 is performed. Suppose that the i-th frame regular image T (i) is obtained. The two images shown on the left and right in FIG. 20 show the distorted wide-angle image S (i) and the planar regular image T (i) for the i-th frame. At this time, the cutting conditions stored in the cutting condition storage unit 170 are P (i) old, φ (i), and m (i) as illustrated. However, the part of the symbol “old” of P (i) old is given to distinguish it from P (i) new that is generated later, and is simply represented as P (i) when the cutting process is executed. You can leave it.

  The cut-out area E (i) shown in FIG. 20 extends from the cut-out position indicated by the cut-out center point P (i) old in the distorted wide-angle image S (i) to the cut-out direction indicated by the parameter φ (i). This is an area for cutting out an image having a size indicated by the magnification m (i), and a plane regular image T (i) is obtained by converting the image cut out from the cut-out area E (i). . The center point (the origin G of the UV coordinate system) of the planar regular image T (i) corresponds to the cut center point P (i) old on the distorted wide-angle image S (i).

  Here, it is assumed that the viewer designates the tracking start point Q on the planar regular image T (i) and inputs a tracking start instruction (specifically, as shown in FIG. 18, a cross cursor It is only necessary to move C to the position of the tracking start point Q and press the tracking start instruction button 345). Then, the automatic tracking flag setting unit 220 switches the automatic tracking flag to ON and updates the point on the two-dimensional XY coordinate system corresponding to the tracking start point Q as a new cut-out center point P (i) new. Execute. That is, on the planar regular image T (i) shown in FIG. 20, P (i) new (tracking start point Q) is a point located at the lower right of P (i) old (origin G), and has a distorted wide angle. Also on the image S (i), P (i) new is a point slightly shifted from P (i) old (position where the cat 60 is present).

  Comparing T (i) in FIG. 20 with T (i + 1) in FIG. 21, the position of the cutting center point P (i + 1), which is the cutting condition of the (i + 1) th frame, is represented by P (i ) It may seem that it may be set at the same position as new, but actually, unless the cat is stationary, the position of the cat on the distorted wide-angle image S (i) and the distorted wide-angle image S ( Since it is different from the position of the cat on i + 1), it is not preferable to set the position of P (i) new as the position of the cut center point P (i + 1) as it is. For the (i + 1) th frame, P (i) new is only a coordinate indicating the past position of the cat (position one frame before).

  Therefore, in the present invention, by using P (i) new and the planar regular image T (i), a new position P (of the cat on the distorted wide-angle image S (i + 1) of the (i + 1) th frame is used. i + 1) is predicted, and this is updated as a new cutting center point P in the cutting condition storage unit 170. In the present invention, since the parameter φ indicating the cutting direction and the magnification m are not automatically changed, these are not changed unless the manual change operation described above is performed. That is, unless a manual change operation is performed, the parameter φ (i + 1) is the same as φ (i), and the magnification m (i + 1) is the same as m (i). The cutting conditions P (i + 1), φ (i + 1), m (i + 1) in the cutting condition storage unit 170 shown in FIG. 21 are determined in this way. Of course, if necessary, an operation of automatically updating φ and m may be adopted. For example, the parameter φ may be automatically adjusted so that the landscape horizontal plane is always parallel to the U axis.

  When the image cropping conversion unit 150 performs the cropping process on the (i + 1) th frame, these cropping conditions P (i + 1), φ (i + 1), and m (i + 1) are used. The two images shown on the left and right in FIG. 21 show the distorted wide-angle image S (i + 1) and the planar regular image T (i + 1) for the (i + 1) th frame. Here, the cut-out area E (i + 1) has a magnification m from the cut-out position indicated by the cut-out center point P (i + 1) in the distorted wide-angle image S (i + 1) to the cut-out direction indicated by the parameter φ (i + 1). This is an area for cutting out an image of the size indicated by (i + 1), and a plane regular image T (i + 1) is obtained by converting the image cut out from this cutout area E (i + 1). The center point (origin point G of the UV coordinate system) of the planar regular image T (i + 1) corresponds to the cut center point P (i + 1) on the distorted wide-angle image S (i + 1).

  A cutout area E (i) indicated by a broken line in FIG. 21 is a cutout area for the i-th frame shown in FIG. If the new cut-out center point P (i + 1) indicates the correct position, that is, the correct cat position on the distorted wide-angle image S (i + 1), as shown in the drawing, the planar regular image T displayed with the cat at the center. (I + 1) is obtained.

  In the neighboring block extraction unit 160, the block selection unit 180, the reference color storage unit 190, the reference color extraction unit 200, and the extraction condition automatic change unit 210 illustrated in FIG. 13, the automatic tracking flag in the automatic tracking flag setting unit 220 is ON. (I + 1) by using P (i) new and T (i) for the i-th frame. ) A component for executing a process for obtaining a cut-out center point P (i + 1) for the second frame. In FIG. 13, an arrow with a character ON from the automatic tracking flag setting unit 220 to another block indicates a signal indicating that the automatic tracking flag is ON. Hereinafter, basic functions of these components will be described.

  First, when the automatic tracking flag is ON, the reference color extraction unit 200 stores the extraction condition based on the planar regular image T (i) of the i-th frame stored in the planar regular image memory 140. I updates the reference color α (i) of the i-th frame representing the color of the tracking object located at “the cut-out center point P for the i-th frame” stored in the unit 170. A sequential extraction process is performed every time.

  For example, as shown in FIG. 20, on the planar regular image T (i) of the i-th frame, the viewer designates the tracking start point Q on the cat to be tracked and inputs the tracking start instruction. Consider the case. Upon receiving this tracking start instruction, the automatic tracking flag setting unit 220 sets the automatic tracking flag to ON and performs a rewriting process with the tracking start point Q as a new cutting center point P (i) new. This is performed for the storage unit 170. In addition, a signal indicating that the automatic tracking flag is ON is transmitted to the reference color extraction unit 200.

  Receiving this, the reference color extraction unit 200 reads the rewritten cut center point P (i) new stored in the cut condition storage unit 170, and stores the i th stored in the planar regular image memory 140. For the planar regular image T (i) of the th frame, a representative color representative of the color of the tracking object (in this example, cat) located at the cut-out center point P (i) new is set to the i th A process of extracting the frame as the reference color α (i) is performed. The reference color α is, for example, a combination of pixel values of three primary colors (R, G, B) when expressed in the RGB color system. When one primary color is expressed by an 8-bit pixel value, the reference color α is composed of 24-bit data.

  Note that the algorithm for determining the representative color of the tracking object is based on the pixel values of the pixels in the vicinity of the cut center point P (i) new, and the colors representing the vicinity of the cut center point P (i) new are selected. Any algorithm that can be determined may be used. A specific example of the processing algorithm will be described in detail in §3.

  The reference color storage unit 190 is a component having a function of storing the latest reference color extracted by the reference color extraction unit 200. As described above, when the automatic tracking flag is ON, the reference color extraction unit 200 performs a process of sequentially extracting the reference color every time the frame number i is updated. Therefore, the reference color storage unit 190 stores α (i), α (i + 1), α (i + 2),... And the frame every time (in the example shown here, every 1/30 seconds). Although a new reference color α is given, the reference color storage unit 190 may always store the latest reference color α.

  As described above, the reason why the reference color α representing the color of the tracked object is updated in units of frames is that the color may change on the captured image even for the same physical object. It is. For example, when “brown-haired cat” is set as a tracking target, the actual color of the cat does not change in a short time, but the background lighting environment may change with time. If the video camera used for shooting has a function for automatically adjusting the white balance, the color of the cat also changes on the captured image when the background hue changes. Under such circumstances, in the present invention, the value of the reference color α indicating the representative color of the tracking object is also updated sequentially for each frame.

  When the reference color α (i) relating to the i-th frame is thus obtained, subsequently, the distorted wide-angle image S (i + 1) of the (i + 1) -th frame stored in the distorted wide-angle image memory 130 is subsequently obtained. From among them, the process of searching for the tracking object is performed using the reference color α (i) as a clue. However, since it is difficult to search using only the reference color α (i) as a clue, in the present invention, the position of the cut-out center point P (i) new of the i-th frame (the past position of the cat) In addition to clues, search. Moreover, the distorted wide-angle image S (i + 1) is divided into a plurality of blocks, and processing for narrowing down candidate blocks is performed.

  The neighboring block extraction unit 160 performs a function of extracting several neighboring blocks that are estimated to have the tracking target, using the extracted center point P (i) new as a clue. That is, when the automatic tracking flag is ON, the neighboring block extraction unit 160 divides the distorted wide-angle image S (i + 1) of the (i + 1) th frame stored in the distorted wide-angle image memory 130 into a plurality of blocks. Then, a process of extracting a plurality of blocks near the extraction center point P (i) new for the i-th frame stored in the extraction condition storage unit 170 as a neighboring block is performed. In the case of the embodiment described here, as will be described in detail in §3, a total of nine neighboring blocks B1 to B9 are extracted as candidates including the block including the cut center point P (i) new.

  The plurality of neighboring blocks extracted by the neighboring block extraction unit 160 in this way indicate an area where the tracking target (cat 60) located at the position P (i) new in the i-th frame may move. Become. In this way, if the method of narrowing down candidate blocks to only neighboring blocks is adopted, the calculation processing burden is reduced, and an area having a color similar to the reference color α (i) in the remote block is erroneously detected as a tracking target. It can be prevented from being detected.

  On the other hand, when the automatic tracking flag is ON, the block selection unit 180 includes, for each of the plurality of neighboring blocks extracted by the neighboring block extraction unit 160, the reference color storage unit 190 among the individual pixels included in the block. The number of approximate color pixels indicating the number of pixels having a color that falls within a predetermined approximate range with respect to the reference color α (i) of the i-th frame stored in is calculated. This is a component with the function of selecting the largest block.

  In the case of the above example, the neighboring block extraction unit 160 extracts a total of nine neighboring blocks B1 to B9, and each of these nine blocks has a predetermined approximation to the reference color α (i). The number of pixels with colors that fall within the range is counted. For example, when the reference color α (i) indicates “specific brown” by setting “brown-haired cat” as the tracking target, the block selecting unit 180 performs the following operation on each of the neighboring blocks. The number of pixels having a color similar to “specific brown” is checked and counted. Then, the block Bmax having the maximum number of approximate color pixels is selected from the nine neighboring blocks B1 to B9. The block Bmax selected in this way is the block most likely to have a tracking target (ie, “brown cat”).

  The extraction condition automatic changing unit 210 searches the block Bmax and identifies the position of the tracking target on the assumption that the tracking target exists in the block Bmax selected by the block selection unit 180. It is a component that performs the processing. The position specified in this way is a position where the tracking target object (cat) is estimated to exist on the distorted wide-angle image S (i + 1) of the (i + 1) th frame, and the (i + 1) th cut center. It will be used as the point P (i + 1). Therefore, the cutting condition automatic changing unit 210 performs a process of rewriting the cutting center point P in the cutting condition storage unit 170 with the obtained cutting center point P (i + 1).

  Here, the specific process performed by the extraction condition automatic changing unit 210 is to define a plurality of candidate regions having different positions in the block Bmax selected by the block selection unit 180 when the automatic tracking flag is ON, A candidate area having a color that best matches the i-th reference color α (i) stored in the reference color storage unit 190 is selected as the optimal candidate area from the plurality of candidate areas, and the selected optimal candidate This is a process of updating the information indicating the extraction center point P stored in the extraction condition storage unit 150 with the center point of the region as the extraction center point P (i + 1) for the (i + 1) th frame. Can do. A specific algorithm for such processing will be described in detail in Section 3.

  Thus, when the extraction center point P (i + 1) for the (i + 1) -th frame is written into the extraction condition storage unit 170, the image extraction conversion unit 150 causes the extraction center point P (i + 1) to be written. , Using the parameter φ (i + 1) and the magnification m (i + 1), as shown in FIG. 21, the cut-out area E (i + 1) in the distorted wide-angle image S (i + 1) stored in the distorted wide-angle image memory 130 ) Is converted into a planar regular image and stored in the planar regular image memory 140 as a planar regular image T (i + 1) of the (i + 1) th frame. If the automatic tracking process is correctly performed, the tracking target object (cat) is arranged at the center (origin G) of the planar regular image T (i + 1).

  The automatic tracking process according to the present invention has been described above with respect to the transition from the i-th frame to the (i + 1) -th frame. Thereafter, the same operation is performed while updating i until an automatic tracking end instruction is given. This process will be repeated. That is, if the automatic tracking flag is in the ON state, the reference color extracting unit 200 detects the tracking object located at the cut-out center point P (i + 1) based on the planar regular image T (i + 1) shown in FIG. The reference color α (i + 1) of the (i + 1) th frame representing the color is extracted, and the reference color storage unit 190 stores it as the latest reference color.

  On the other hand, the neighboring block extraction unit 160 extracts blocks near the cut-out center point P (i + 1) from the distorted wide-angle image S (i + 2) of the (i + 2) th frame, and the block selection unit 180 Among the neighboring blocks, the block having the largest number of pixels having a color that falls within a predetermined approximate range with respect to the reference color α (i + 1) is selected, and the extraction condition automatic changing unit 210 selects the selected block. An optimum candidate region having a color most suitable for the reference color α (i + 1) is identified from the inside, and a process of writing the center point as a new cut center point P (i + 2) in the cut condition storage unit 170 is performed.

  In the above description, the description has been made using two points, P (i) old and P (i) new, as the i-th cutting center point. Since the tracking start instruction including the position of the tracking start point Q is input on the planar regular image T (i), the cut center point is changed from P (i) old to P (i) new ( = Q). As described above, with respect to the initial frame switched to the automatic tracking mode, the cutting center point is changed in the same frame, but thereafter, one cutting center point is defined for one frame. Will be. Therefore, there is no distinction between P (i) old and P (i) new for the cut-out center point of any i-th frame in the automatic tracking mode.

  The basic configuration and operation of the moving picture presentation apparatus according to the basic embodiment of the present invention have been described above with reference to the block diagram of FIG. 13. This moving picture presentation apparatus incorporates a dedicated program into a computer. Can be configured. In this case, the image input unit 110, the image output unit 120, and the instruction input unit 240 can be configured by the input / output device and the input / output interface for the computer, and the distorted wide-angle image memory 130 and the planar regular image memory 140 are configured. The cutting condition storage unit 170 and the reference color storage unit 190 can be configured by a storage device for the computer, and the remaining components are a program built in the computer and a computer that executes the program. It can comprise by each means within.

  Finally, the features of the video presentation device according to the above embodiment are briefly summarized. A major feature of the automatic tracking algorithm employed by this apparatus is that a reference color α representing the tracking object is extracted from the image of the i-th frame, and the (i + 1) -th frame is used as a clue to this reference color α. The tracking method is to estimate the position of the tracking object on the image. Since a search method using the reference color α as a clue is adopted, analysis regarding the shape of the tracking target is not required.

  As described above, on the distorted wide-angle image S photographed using a fisheye lens or the like, the shape of any object is distorted, and the degree of distortion varies depending on the position on the photographed image. For this reason, even if the tracking target is physically the same, the shape changes variously depending on the position on the captured image, and it is difficult to grasp the tracking target from the characteristics of the shape. In the present invention, since the tracking target is not specified by the shape but is specified based on the feature amount of the reference color α, even if the tracking target changes in shape on the distorted wide-angle image S, the tracking target is accurate. Tracking becomes possible.

  However, if tracking is performed only with the clue of the reference color α, misunderstanding confusion occurs when another object having a similar color happens to exist. Therefore, in the present invention, the distorted wide-angle image S is divided into a plurality of blocks, and a block in the vicinity of the position of the tracking target object (cutout center point P) in the previous frame is extracted. A method of selecting a block having the highest possibility of containing the tracking target is adopted. Unless the tracked object moves very fast, the position of the tracked object in the next frame is estimated to be in the same block as the current frame, or at most in the neighboring block, so there are many methods described above. Very effective in cases.

  By using this method, even if another object with a similar color happens to exist in a remote block, the problem of misidentifying the other object as a tracking object is solved. Can do. In addition, by using only neighboring blocks as candidates, an advantage of reducing the calculation burden can be obtained. Furthermore, when selecting one block from neighboring blocks, an algorithm is employed to select a block having the maximum number of pixels having a color that falls within a predetermined approximate range with respect to the reference color α, so that the computation burden is increased. The block selection can be performed by a process having a relatively low value.

  On the other hand, after one block is selected, it is necessary to search within the selected block and specify a position where the tracking target is estimated to exist. For this purpose, in the present invention, a plurality of candidate areas are defined in the block, and a candidate area having a color that best matches the reference color α is selected as the optimum candidate area. Since this process only needs to be performed on the selected block, the calculation load is not so great.

  As described above, in the video presentation device according to the present invention, when a video composed of a distorted wide-angle image obtained by wide-angle shooting is cut out and reproduced, a process for automatically tracking an arbitrary tracking target is performed. Can be performed accurately with a simpler process.

<<< §3. Specific processing algorithm >>
Here, a specific algorithm for each process executed in the video presentation device according to the basic embodiment of the present invention described in §2 is exemplified.

<3-1. Processing Algorithm of Standard Color Extraction Unit 200>
When the automatic tracking flag is ON, the reference color extraction unit 200 illustrated in FIG. 13 is based on the planar regular image T stored in the planar regular image memory 140, and the tracking target located at the cut-out center point P is detected. The representative color (reference color α) is extracted. Here, some examples of specific processing algorithms in which the reference color extraction unit 200 extracts the reference color α will be described below.

  The simplest processing algorithm is a method of extracting the color of one pixel including the cut-out center point P as it is as the reference color α. For example, as shown in FIG. 20, when the tracking start point Q is designated on the planar regular image T (i), the tracking start point Q becomes the cut center point P (i) new. The color of the pixel including the center point P (i) new may be extracted as the reference color α (i) as it is. Since the viewer has performed the operation of specifying the tracking start point Q as shown in FIG. 20 in order to specify “brown-haired cat” as the tracking target, the pixel including the tracking start point Q is displayed. Is the simplest method for determining the representative color of the tracking object.

  In the subsequent frames, as in the example shown in FIG. 21, the cut-out center point P (i + 1) coincides with the origin G of the planar regular image T (i + 1), so that the color of the pixel including the origin G remains as the reference color. It will be extracted as α (i + 1).

  However, actually, the tracking target is an object having a certain size, and constitutes an aggregate region of a plurality of pixels. Then, the viewer recognizes the entire aggregate area as a tracking target, and performs an operation of designating one point as the tracking start point Q. Accordingly, the color of the pixel located at one point that the viewer happens to specify as the tracking start point Q is not necessarily a color that represents the entire tracking target. For example, if the target to be tracked is a “brown cat”, since there are various variations in brown, one brown color designated as the tracking start point Q is not necessarily representative of the entire cat. Not necessarily.

  In consideration of such points, in practice, it is preferable to adopt an algorithm that extracts an average value of colors of a plurality of pixels located near the cut-out center point P (tracking start point Q) as the reference color α. That is, the reference color extraction unit 200 defines a neighboring area including a central pixel including the cut-out center point P and surrounding peripheral pixels, and an average value of the colors of the pixels in the neighboring area is extracted as the reference color α. That's fine.

  FIG. 22 is a plan view showing an example of the reference color extraction process when a tracking start instruction is given in the state shown in FIG. 20 based on such an algorithm. If the viewer designates one point near the abdomen of the cat 60 as the tracking start point Q, this tracking start point Q becomes a new cut-out center point P (i) new. Therefore, a pixel including the cut-out center point P (i) new is defined as a central pixel, and a neighborhood region N composed of the central pixel and surrounding peripheral pixels is defined. In the example shown in the figure, a neighborhood region N composed of 7 × 7 pixels including the central pixel is defined. Then, the average value of the colors of all 49 pixels in the vicinity region N may be extracted as the reference color α. By adopting such a method, it is possible to extract a more appropriate color as the reference color α as a color representative of the entire tracked object.

  In the example shown in FIG. 22, the neighborhood area N that happens to consist of 7 × 7 pixels is an area that covers almost the entire cat 60 that is the tracking target, but in reality, the neighborhood area N is the tracking target. There may be a case where only a part of the object can be covered, or a case where not only the tracked object but also the surrounding background part is covered. In the former case, there is no problem even if the average value of the colors of all the pixels in the vicinity region N is used as the reference color α. However, in the latter case, the average value including the surrounding background color becomes the reference color α, and therefore tracking There is a possibility that a color that is not appropriate as a color representing the object is extracted as the reference color α.

  In order to prevent such a problem, the size of the neighborhood region N may be set to be less than or equal to some extent. That is, as the neighboring region N, generally, an n × n square pixel array centered on the central pixel including the cut-out center point P (tracking start point Q) may be defined. If is too large, a wide neighboring region N that covers the surrounding background portion is set, so that the optimal value of n may be determined in consideration of the resolution of the planar regular image T. When presenting a moving image having a general resolution, it is practically large if n = 7 is set and a neighboring region N having a size of about 7 × 7 pixels is set as in the example shown in the figure. There will be no hindrance.

  In order to eliminate the surrounding background color and extract a more accurate reference color α, the following algorithm may be employed. That is, the reference color extraction unit 200 defines a neighboring area N composed of a central pixel including the cut-out center point P and surrounding peripheral pixels. A pixel having a color that falls within a predetermined approximate range may be used as a reference pixel, and an average value of the colors of the reference pixels may be extracted as a standard color α.

  An actual example of the reference color extraction process based on such an algorithm will be specifically described with reference to FIG. FIG. 23 is an enlarged view of the vicinity of the vicinity region N of FIG. As described above, in the case of this example, the neighborhood region N is configured by a total of 49 pixels arranged in 7 rows and 7 columns. Here, these 49 pixels are referred to as pixels W1, W2,..., W7 from the left to the right in the first row, and hereinafter the pixels W8 to W7 in the same order for the second to seventh rows. Let's call it W49. However, since the pixel W25 that is the central pixel is a pixel including the tracking start point Q, it is specifically referred to as a pixel Wq. In FIG. 23, for the sake of convenience, the contour line of the cat 60 is drawn with a smooth curve, and therefore there are pixels whose interior is divided by the contour line. Only one color is defined for each pixel.

  Here, considering the colors of the 49 pixels constituting the neighboring region N, for example, the pixel W27 is a pixel having the color of the original tracking target (cat 60), but the pixel W7 is a background. This means that the pixel has the color of the image. Therefore, when extracting the reference color α representing the color of the tracking object, the color of the pixel W27 should be considered, but the color of the pixel W7 should not be considered. Therefore, first, of all 49 pixels, screening is performed between a pixel to be considered (reference pixel) and a pixel that should not be considered (non-reference pixel). Specifically, a pixel having a color that falls within a predetermined approximate range with respect to the color of the center pixel Wq may be used as a reference pixel, and a pixel having a color outside the approximate range may be set as a non-reference pixel. For example, when 30 pixels among 49 pixels are set as reference pixels, an average value of colors of the 30 reference pixels may be extracted as the reference color α.

  If there is a brown cat 60 on the paved gray road surface, the hatched part in FIG. 23 is brown, and the other part is gray, so the average of all 49 pixels is averaged. If it is obtained, the mixed color of gray and brown will be extracted as the standard color α, but if the above algorithm is used for screening, the average color for the color of the brown reference pixel will be extracted as the standard color α. Therefore, a more appropriate reference color α can be obtained.

  The specific algorithm for extracting the reference color α when the tracking start instruction with the specification of the tracking start point Q is input in the i-th frame has been described above with reference to FIG. The same algorithm can be applied to the reference color extraction processing of the (i + 1) th and subsequent frames that are repeated until an instruction to end automatic tracking is given. That is, generally, the center pixel Wq shown in FIG. 23 is a pixel including the cut-out center point P (i), and is a pixel including the origin G of the UV coordinate system in the middle of the automatic tracking process. Become.

<3-2. Color approximation and averaging algorithm>
In the above-described algorithm, it is necessary to determine whether or not the color of the peripheral pixel falls within a predetermined approximate range with respect to the color of the center pixel Wq in order to perform the screening of the reference pixel or the non-reference pixel. In addition, it is necessary to obtain an average for the colors of a plurality of reference pixels. Here, a specific algorithm for performing approximate color judgment and a specific algorithm for obtaining an average of colors are exemplified.

  First, an example of an algorithm for obtaining a color average will be described. In general, each pixel constituting image data obtained by photographing with a video camera has pixel values (R, G, B) expressed in the RGB color system, and one primary color is represented by 8 bits. When expressed in terms of pixel values, one pixel is composed of 24-bit data. Thus, in order to calculate the average value of the colors of a plurality of pixels having pixel values (R, G, B) expressed in the RGB color system, the average value Rav of the pixel value R, the pixel value The average value Gav of G and the average value Bav of the pixel value B are obtained, and the color indicated by these pixel values (Rav, Gav, Bav) may be used as the average value of the colors of the plurality of pixels. Further, when the pixel value is expressed in a color system other than the RGB color system, the average may be obtained by the above method after performing conversion for expressing the pixel value in the RGB color system. .

  Next, an example of an algorithm for performing color approximation determination will be described. The algorithm described here is effective for pixels having pixel values (H, S, V) expressed in the HSV color system. Specifically, as a condition for the color of the second pixel to fall within a predetermined approximate range of the color of the first pixel, the pixel value of each pixel is expressed in the HSV color system and the value of hue H is set to 0. When expressed by an angle of ˜360 °, “the angle difference between the angle representing the hue H of the color of the first pixel and the angle representing the hue H of the color of the second pixel is equal to or less than a predetermined allowable angle θ. The color approximation determination may be performed using the condition “”.

  As described above, generally, image data obtained by photographing with a video camera is expressed in the RGB color system. As described above, in order to perform color approximation determination for pixels expressed in the RGB color system using the above algorithm, the pixel values (R, G, B) of individual pixels are expressed in the HSV color system. The pixel value (H, S, V) may be converted and the hue H value may be compared with each other.

  Here, for the example shown in FIG. 23, a procedure for extracting the reference value α by performing the color approximation determination based on the above algorithm and obtaining the average value of the colors will be described. Assume that the 49 pixels W1 to W49 shown in FIG. 23 each have pixel values (R, G, B) expressed in the RGB color system. For example, the first pixel W1 has a pixel value of (R1, G1, B1), and the jth pixel Wj has a pixel value of (Rj, Gj, Bj). The 49th pixel W49 has a pixel value of (R49, G49, B49). The 25th pixel W25, that is, the center pixel Wq has a pixel value of (Rq, Gq, Bq).

  Here, as shown in FIG. 24, the pixel values of the pixels W1 to W49 are converted from the RGB color system to the HSV color system. For example, the pixel value (Rj, Gj, Bj) of the jth pixel Wj is converted to (Hj, Sj, Vj), and the pixel value (Rq, Gq, Bq) of the center pixel Wq is (Hq , Sq, Vq).

The conversion process from the RGB color system to the HSV color system can be performed using a known conversion formula. Specifically, when the pixel value (R, G, B) is converted to the pixel value (H, S, V), the maximum of the three values R, G, B is MAX, and the minimum is As MIN, the pixel values (H, S, V) are
H = 60 × (GB) / (MAX−MIN) +0, if MAX = R
H = 60 × (BR) / (MAX−MIN) +120, if MAX = G
H = 60 × (RG) / (MAX−MIN) +240, if MAX = B
S = (MAX−MIN) / MAX
V = MAX
Given in. In the HSV color system, the color is indicated by the value of the hue H, so that the approximate determination of the color can be made by comparing the values of the hue H with each other. Since only the hue H value is used for color comparison, it is not actually necessary to obtain the saturation S and lightness V values when converting to the HSV color system.

  After all, in order to determine whether the colors of the 48 peripheral pixels W1 to W24 and W26 to W49 shown in FIG. 23 fall within the approximate range of the color of the center pixel Wq (W25), the color is converted to the HSV color system. It is only necessary to determine whether or not the hues H1 to H24 and H26 to H49 of each peripheral pixel are within the approximate range of the hue Hq of the center pixel. Since the hue H indicates an angle of 0 to 360 ° on the color wheel, “an angle representing the hue H1 to H24, H26 to H49 of the color of each peripheral pixel and an hue representing the hue Hq of the color of the center pixel”. If the condition that the angle difference between the two is less than or equal to the predetermined allowable angle θ is set, the color approximation can be determined.

  FIG. 25 is a diagram illustrating an example of setting the approximate condition of the hue H in the HSV color system. As shown in the drawing, a pixel having a hue whose angle difference is equal to or smaller than the allowable angle θ with respect to the angle of the hue Hq of the central pixel, that is, within a range of hue Hq−θ to Hq + θ Therefore, if it is negative, it is a value obtained by adding 360 °, and if it exceeds 360 °, the value obtained by subtracting 360 ° is determined as an approximate range. The allowable angle θ that influences the approximate range is considered to be appropriate in consideration of the content of the video that is actually reproduced using this video presentation device, for example, θ = 10 ° or θ = 15 °. Can be set to a value that can be

  Thus, when the reference pixels determined to be within the approximate range among the 49 pixels including the center pixel Wq are obtained, the average Wav of the colors of these reference pixels is obtained as described above, and this is used as the reference color α. And it is sufficient. Here, when obtaining the average color Wav, as described above, the average value Rav of the pixel values R and the average of the pixel values G are used using the pixel values (R, G, B) expressed in the RGB color system. The average value Bav of the value Gav and the pixel value B is obtained, and the color indicated by these pixel values (Rav, Gav, Bav) may be used as the average color Wav.

<3-3. Processing Algorithm of Neighboring Block Extraction Unit 160>
When the automatic tracking flag is ON, the neighboring block extraction unit 160 divides the distorted wide-angle image S (i + 1) of the (i + 1) th frame stored in the distorted wide-angle image memory 130 into a plurality of blocks, A process of extracting a plurality of blocks adjacent to the cut center point P (i) new for the i-th frame stored in the cut condition storage unit 170 as a neighboring block is performed.

  FIG. 26 is a plan view showing an example of a neighboring block extraction process performed by the neighboring block extraction unit 160. In the case of the example shown here, the neighboring block extraction unit 160 divides the distorted wide-angle image S (i + 1) into a plurality of rectangular blocks arranged vertically and horizontally, and the cut-out center point P (i) new As a central block B5, a total of nine blocks including the central block B5 and its adjacent blocks located in the upper, lower, left, and right and diagonal four directions are extracted as neighboring blocks B1 to B9.

  In this way, if a total of nine blocks are extracted as neighboring blocks, no matter which direction the tracking object moves from the central block, it must pass through one of the neighboring blocks. It is effective to prevent losing sight. Of course, if the movement speed of the tracked object is extremely high, it may move out of the 9 neighboring blocks. Therefore, automatic tracking of the tracked object with such a high moving speed is appropriate. Therefore, a total of 25 blocks of 5 × 5 may be extracted as neighboring blocks centering on the central block including the cut center point P (i) new. In addition, when the cut-out center point P (i) new is located near the circumference of the distorted wide-angle image S (i + 1), some of the adjacent blocks located in the upper, lower, left, and right and diagonal four directions are present at the corresponding position. However, in such a case, as a matter of course, a block that does not exist at the corresponding position is not extracted.

  The neighboring blocks to be extracted are not necessarily arranged in an n-row and n-column arrangement. For example, the movement speed in the X-axis direction is higher than the movement speed in the Y-axis direction due to the characteristics of the tracking target. If there is a circumstance such as large, a total of 15 neighboring blocks constituting a 3 × 5 array may be extracted. In addition, FIG. 26 shows an example in which each block is a square, but the shape of each block is arbitrary, and any shape can be used as long as it can divide a plane without gaps, such as a rectangle, a regular triangle, or a regular hexagon. Shaped blocks can be employed. Of course, the size of each block is also arbitrary, and can be set to an arbitrary size that seems to be optimum in practice. Also, since each block does not necessarily have the same shape and the same size, theoretically, the distorted wide-angle image S is divided into blocks by arbitrarily shaped blocks such as each piece of a jigsaw puzzle. It doesn't matter.

  The embodiment described here is an example in which the image input unit 110 inputs a distorted wide-angle image S obtained by photographing using a fisheye lens. Therefore, as shown in FIG. 26, the distorted wide-angle image S (i + 1) is circular. Become an image. In such a circular distorted wide-angle image, the compression rate of the image information near the outer periphery is extremely high. In other words, in the case of a circular distorted wide-angle image, the degree of distortion is greater in the vicinity of the circumference, and the recording accuracy of the image is reduced. Therefore, in the case of the embodiment shown here, the neighboring block extraction unit 160 divides the circular distorted wide-angle image into a plurality of rectangular blocks, and then eats even part of the outline of the circular distorted wide-angle image. The blocks to be output are not extracted as neighboring blocks.

  In the example shown in FIG. 26, each individual block drawn in a square is a block located inside the contour line of a circular distorted wide-angle image, and a block that partially protrudes from the contour line is drawn. Absent. The neighboring block extraction unit 160 extracts neighboring blocks from the blocks drawn in the square. If such an operation is performed, the region near the outer periphery of the circular distorted wide-angle image (the region where the image recording accuracy is low) is excluded from the target of automatic tracking, so erroneous recognition based on the image with low accuracy It is effective in preventing the occurrence of

<3-4. Processing Algorithm of Block Selection Unit 180>
When the automatic tracking flag is ON, the block selection unit 180 selects each of a plurality of neighboring blocks extracted by the neighboring block extracting unit 160 (in the example shown in FIG. 26, nine neighboring blocks B1 to B9). The number of pixels having a color that falls within a predetermined approximate range with respect to the reference color α stored in the reference color storage unit 190 among the individual pixels included in the block is counted, and the number of approximate color pixels is the maximum. The process which selects the block which becomes is performed.

  Here, in order to determine whether the color of each pixel falls within a predetermined approximate range with respect to the reference color α, the algorithm for performing the approximate color determination described in §3-2 is used. That's fine. For example, in the case of counting the number of approximate color pixels for the block B1 shown in FIG. 26, processing according to the following procedure may be performed.

  First, for convenience of explanation, it is assumed that each block is composed of an area of 30 × 30 pixels. In this case, the block B1 includes a total of 900 pixels. Therefore, as shown in FIG. 27, for each of the 900 pixels W1 to W900, the RGB color system pixel values (R, G, B) are changed to HSV color system pixel values (H, S, V). Convert. For example, the pixel values (R1, G1, B1) of the first pixel W1 are converted into pixel values (H1, S1, V1), and the pixel values (R900, G900, B900) of the 900th pixel W900 are Are converted into pixel values (H900, S900, V900) (as described above, it is not actually necessary to determine the values of saturation S and brightness V).

  Subsequently, the reference color α stored in the reference color storage unit 190 is also converted into the HSV color system. In the case of the example described in §3-2, the reference color α is given as an average color Wav (Rav, Gav, Bav) obtained by averaging pixel values expressed in the RGB color system. What is necessary is just to convert into the color Wav (Hav, Sav, Vav) expressed in the color system (in this case also, it is not actually necessary to determine the values of the saturation Sav and the lightness Vav).

  Then, as shown in FIG. 27, a process of comparing the hues H1 to H900 of 900 pixels with the hue Hav of the reference color α is performed to determine whether or not they are within the approximate range. Specifically, a pixel having a hue whose angle difference is equal to or smaller than the allowable angle θ with respect to the angle of the hue Hav, that is, the hue is within a range of Hav−θ to Hav + θ (an angle of 0 to 360 ° on the color wheel). Therefore, if it is negative, it is a value obtained by adding 360 °, and if it exceeds 360 °, it is a value obtained by subtracting 360 °). If the number of pixels determined to be in the approximate range among the 900 pixels is counted, the count value is the number of approximate color pixels for the block B1.

  The block selection unit 180 performs the same counting process for the blocks B2 to B9, and performs the process of selecting the block Bmax that maximizes the number of approximate color pixels. If the tracked object has a characteristic hue that does not exist in the background image, it is highly likely that the block including many pixels having colors close to the characteristic hue includes the tracked object. it is conceivable that. The block selection process by the block selection unit 180 can be said to be a process of selecting a block that is most likely to contain a tracking target based on such a concept.

  Of course, in an environment where the typical hue of the tracking target is also scattered in the background image (for example, an environment where a brown cat is walking on brown soil), block selection processing based on the above algorithm Cannot lead to correct results. Even in an environment where there is another object with a color close to the typical hue of the tracked object (for example, an environment where several brown cats hang out), block selection processing based on the above algorithm Cannot lead to correct results. The automatic tracking processing of the tracking object according to the present invention fails in the special environment as in the above example, and a part of the background or another object is erroneously detected as the tracking object. As long as the basic principle of the present invention is to track a moving object using a color as a clue, it is unavoidable that a tracking failure example as described above occurs.

  Note that erroneous detection also occurs when the tracking target has already moved out of the neighboring block. In order to avoid such erroneous detection, the block selection unit 180 sets a minimum reference value for the number of approximate color pixels in advance, and there is no block in which the number of approximate color pixels exceeds the minimum reference value. The automatic tracking flag setting unit 220 gives an automatic tracking failure signal without selecting a block, and the automatic tracking flag setting unit 220 turns off the automatic tracking flag when the automatic tracking failure signal is given. It is preferable to perform the process of switching to.

  In other words, if a block is selected only by the algorithm of “a block having the maximum number of approximate color pixels”, for example, when the number of approximate pixels in another block is 0, the number of approximate color pixels is only one. Even a block with only one will be selected. In this way, there is a high possibility that a block having an extremely small number of approximate color pixels does not actually contain a tracking object. Therefore, if the minimum reference value of the approximate color pixel number is set and there is no block whose approximate color pixel number is equal to or greater than the minimum reference value, the tracking target has already moved out of the neighboring block. It is preferable that the automatic tracking flag is switched to OFF and the subsequent automatic tracking processing is stopped after determining that it is closed.

<3-5. Processing algorithm of cutting condition automatic changing unit 210>
When the automatic tracking flag is ON, the extraction condition automatic change unit 210 defines a plurality of candidate areas whose positions are different from each other in the block selected by the block selection unit 180, and selects a reference area from the plurality of candidate areas. A candidate area having a color that best matches the color α is selected as the optimum candidate area, and the center point of the selected optimum candidate area is stored in the extraction condition storage unit 170 as the extraction center point P for the next frame. A process for updating information indicating the cut-out center point P is performed.

  FIG. 28 is a plan view showing an example of the optimum candidate region selection process performed by the automatic extraction condition changing unit 210. FIG. The block Bmax shown in the drawing is a block selected by the block selection unit 180. In the embodiment shown here, the extraction condition automatic changing unit 210 defines a pixel array of a predetermined size, and the area in the frame when the frame of the pixel array is arranged at a predetermined position in the block is set as a candidate area. A plurality of candidate regions whose positions are different from each other are defined by shifting the arrangement of the frame of the pixel array by one pixel in the vertical and horizontal directions.

  In the example shown in FIG. 28, a 7 × 7 pixel array (a pixel array having the same size as the neighboring region N used for extraction of the reference color α) is defined, and the frame of this pixel array is indicated by a thick line. Yes. A region surrounded by the bold frame is one candidate region. In the figure, for the sake of convenience, only the first candidate area A (1) and the kth candidate area A (k) are illustrated, but actually, this 7 × 7 pixel array is arranged in the horizontal direction. By shifting one pixel at a time and shifting one pixel at a time in the vertical direction, a large number of candidate regions having different positions are defined.

  For example, if the block Bmax is a block having a 30 × 30 pixel array, there are 24 variations in the horizontal direction and 24 variations in the vertical direction in which the candidate region having the 7 × 7 pixel array is arranged. Therefore, 24 × 24 = 576 candidate regions having different positions are defined on the block Bmax. The cutting condition automatic changing unit 210 performs a process of selecting a candidate area having a color most suitable for the reference color α from among the 576 candidate areas as an optimal candidate area.

  Here, as a specific algorithm for selecting a candidate region having a color most suitable for the reference color α, a candidate region whose average color value of pixels in the candidate region is closest to the reference color α is determined as an optimal candidate region. The method of selecting as can be taken. For example, in the case of the illustrated example, for the first candidate area A (1), the average value of the color of a total of 49 pixels located within the bold line frame is calculated (as described in §3-2). In addition, the pixel values of the three primary colors expressed in the RGB color system may be averaged), and this may be compared with the reference color α to check the degree of color approximation. In order to obtain the approximate degree of the two colors, as described in §3-2, both colors may be converted into the HSV color system and the angle difference between the hues H may be obtained. If the angle difference of hue H is obtained for all 576 candidate areas, the candidate area with the smallest angle difference can be selected as the optimum candidate area.

  However, when the average value of colors is obtained for all 49 pixels in this way, as described in §3-1, not only the color of the tracking object but also the color of the surrounding background is averaged. Will contribute to the value. For example, consider the case where the positional relationship between the kth candidate area A (k) and the tracking target 60 is as shown in the example shown in FIG. That is, the neighborhood area N shown in FIG. 23 corresponds to the candidate area A (k). In this case, if the average value of the colors is obtained for all 49 pixels, the background color will be mixed into the color of the tracking target (cat), and therefore the candidate area A (k). There is a possibility that the degree of approximation with the reference color α does not increase so much even though the tracking object (cat) is inside.

  In order to solve such a problem, the clipping condition automatic changing unit 210 uses, as a reference pixel, a pixel having a color that falls within a predetermined approximate range with respect to the reference color α among the pixels in the candidate area. A candidate area whose pixel color average value is closest to the reference color α may be selected as the optimum candidate area.

  A specific processing procedure based on the above algorithm will be described with reference to FIG. FIG. 29 is a diagram showing a processing procedure for examining the degree of approximation with the reference color α for the kth candidate area A (k) shown in FIG. As described above, the kth candidate area A (k) includes a total of 49 pixels. Here, these 49 pixels are referred to as pixels W1k to W49k. Each of these pixels has an RGB color system pixel value (R, G, B). For example, the first pixel W1k has a pixel value (R1k, G1k, B1k), and the 49th pixel W49k has a pixel value (R49k, G49k, B49k). Therefore, first, these RGB color system pixel values are converted into HSV color system pixel values. For example, the pixel value of the first pixel W1k is converted into a pixel value (H1k, S1k, V1k), and the pixel value of the 49th pixel W49k is converted into a pixel value (H49k, S49k, V49k). (As described above, in practice, it is not necessary to obtain the values of saturation S and brightness V).

  Subsequently, the reference color α stored in the reference color storage unit 190 is also converted into the HSV color system. In the case of the example described in §3-2, the reference color α is given as an average color Wav (Rav, Gav, Bav) obtained by averaging pixel values expressed in the RGB color system. What is necessary is just to convert into the color Wav (Hav, Sav, Vav) expressed in the color system (in this case also, it is not actually necessary to determine the values of the saturation Sav and the lightness Vav).

  Then, as shown in FIG. 29, a process of comparing the hues H1k to H49k of the 49 pixels with the hue Hav of the reference color α is performed to determine whether or not they are within the approximate range. Specifically, a pixel having a hue whose angle difference is equal to or smaller than the allowable angle θ with respect to the angle of the hue Hav, that is, the hue is within a range of Hav−θ to Hav + θ (an angle of 0 to 360 ° on the color wheel). Therefore, if it is negative, it is a value obtained by adding 360 °, and if it exceeds 360 °, it is a value obtained by subtracting 360 °). Then, an average Wavk of the colors of the reference pixels is obtained by using, as a reference pixel, a pixel that has been determined to be an approximate range among the 49 pixels. When obtaining the average value of the colors, the average value for each primary color of the RGB color system pixel values is taken, so that the average color Wavk is the pixel values (Ravk, Gavk, Bavk).

  The meaning of the color Wavk (Ravk, Gavk, Bavk) obtained in this way is that the reference color α (Wav (Wav ()) among the 49 pixels included in the kth candidate area A (k) shown in FIG. Rav, Gav, Bav)) is the average of the colors of pixels having colors that fall within a predetermined approximate range. Therefore, finally, the degree of approximation between the color Wavk (Ravk, Gavk, Bavk) and the reference color α (Wav (Rav, Gav, Bav)) is calculated. Specifically, the color Wavk (Ravk, Gavk, Bavk) is converted into the HSV color system to obtain Wavk (Havk, Savk, Vavk), and the reference color α (Wav (Rav, Gav, Bav)) is determined as the HSV table. It is only necessary to obtain Wav (Hav, Sav, Vav) after conversion to a color system and obtain the angle difference between the hue Hav and the hue Havk (in this case also, it is actually necessary to obtain the values of the saturation S and the lightness V) Not) The angle difference is a parameter indicating the degree of matching between the color of the kth candidate area A (k) and the reference color α (the smaller the angle difference, the higher the degree of matching).

  The above is the procedure for obtaining the degree of fitness for the kth candidate region A (k). The same procedure is used for the first candidate region A (1) to the 576th candidate region A (576). The candidate area having the highest degree of matching (the candidate area having the smallest angle difference) is selected as the optimum candidate area. The center point of the optimum candidate area selected in this way becomes the cut center point P for the next frame. For example, as shown in FIG. 28, when the kth candidate area A (k) is selected as the optimal candidate area for the selected block Bmax in the (i + 1) th distorted wide-angle image S (i + 1), The center point of the optimal candidate area A (k) becomes a new cut center point P (i + 1) and is written in the cut condition storage unit 170.

  Even when the optimal candidate area is selected by the above algorithm, the extraction condition automatic changing unit 210 sets a minimum matching criterion for the reference color α in advance, and the candidate area whose degree of matching is equal to or higher than the minimum matching criterion. Is not present (for example, when there is no candidate region having an angle difference of 5 ° or less), the automatic tracking flag is selected without selecting the optimum candidate region and updating the information indicating the cut-out center point P. It is preferable to give an automatic tracking failure signal to the setting unit 220. When an automatic tracking failure signal is given, the automatic tracking flag setting unit 220 may switch the automatic tracking flag to OFF and stop the subsequent automatic tracking processing.

<3-6. Processing algorithm after moving to adjacent block>
In the automatic tracking mode in the video presentation device according to the present invention, even when the tracking target moves to an adjacent block, the automatic tracking can be continued without any trouble by performing the same processing as the algorithm described so far. it can.

  For example, as shown in FIG. 26, when nine neighboring blocks B1 to B9 are extracted around the block including the cut center point P (i) new on the distorted wide-angle image S (i + 1), Consider the case where the tracking target has already moved from the block B5 to the left block B4. In this case, the block selection unit 180 selects the block B4 in which the tracking target exists, and the extraction condition automatic change unit 210 specifies the position of the tracking target on the block B4 and determines the position. A process of writing into the cutting condition storage unit 170 as a new cutting center point P (i + 1) is performed.

  In this case, on the distorted wide-angle image S (i + 2) as the next frame, as shown in FIG. 30, nine neighboring blocks B10, B1, centering on the block B4 including the cut-out center point P (i + 1). B2, B11, B4, B5, B12, B7, and B8 are extracted, and automatic tracking of the tracking object is continued by the same processing procedure as the algorithm described so far.

  When the cut-out center point P (i + 1) for the (i + 1) th frame is obtained, the planar regular image T (i + 1) of the (i + 1) th frame is stored in the planar regular image memory 140. The reference color extraction unit 200 executes a process of extracting the (i + 1) th reference color α (i + 1) based on the planar regular image T (i + 1). You can follow the algorithm just described. The reference color α extraction algorithm described with reference to FIG. 22 is an algorithm for the initial frame to which the tracking start instruction is given, and therefore the neighborhood region N is defined at the position of the tracking start point Q. As for the frame, since the tracking object is always displayed at the center of the screen, the vicinity region N is defined at the center position of the screen.

  FIG. 31 is a plan view showing an example of the reference color extraction process performed on the planar regular image T (i + 1) of the (i + 1) th frame. As described above, in the case of a frame in the middle of the automatic tracking mode, the cut-out center point P (i + 1) comes to the center position of the planar regular image T (i + 1), and the neighboring region N also includes the planar regular image T ( defined in the middle of i + 1). Even in this case, the algorithm of the reference color extraction process using the neighborhood region N is not changed. In the illustrated example, the average color Wav (Rav, Gav, Bav) is the same as the example described in §3-1. Are extracted as the (i + 1) th reference color α (i + 1).

<<< §4. Some variations >>>
Finally, some modifications of the moving picture presentation apparatus according to the present invention will be described.

(1) Modified Example of Color Approximation In §3-2, as an algorithm for color approximation, pixel values are converted into HSV color system pixel values, and approximation is determined based on the angle difference of hue H Although an example has been described, the algorithm for performing the approximate determination of the colors of two pixels is not limited to the method based on the angle difference between the hues H.

  For example, as a condition for the color of the second pixel to fall within a predetermined approximate range of the color of the first pixel, the pixel value of each pixel is expressed in the RGB color system, and the color of each pixel is expressed in 3D RGB When plotting as coordinate points on the color space indicated by the coordinate system, it is also possible to use the condition that “the Euclidean distance between the coordinate points of two pixels is equal to or less than a predetermined value”.

  FIG. 32 is a diagram illustrating a setting example of color approximation conditions in the RGB color system. When the pixel value of each pixel is expressed in the RGB color system, the color of each pixel can be plotted as a coordinate point on a color space represented by a three-dimensional RGB coordinate system. For example, the point Qα and the vector Vα shown in FIG. 32 are one coordinate point and vector indicated by numerical values (Rα, Gα, Bα), but when expressed in the RGB color system, the pixel values (Rα, Gα , Bα). Similarly, the point Qβ and the vector Vβ indicate colors represented by pixel values (Rβ, Gβ, Bβ).

  Here, the degree of approximation of the two colors Qα and Qβ can be expressed by the Euclidean distance d between the two points Qα and Qβ, and it can be said that the smaller the distance d, the higher the degree of approximation. Since the Euclidean distance d can be defined by the arithmetic expression shown in FIG. 33, if the pixel values of the RGB color system of two pixels are obtained after all, the Euclidean distance d can be obtained using the expression shown in FIG. The degree of approximation of the colors of both pixels can be obtained. Therefore, as a condition for the color Qβ to fall within the predetermined approximate range of the color Qα, a condition that “Euclidean distance d is equal to or smaller than a predetermined value” can be set, and the color approximation can be determined.

(2) Modification Example of Reference Color Extraction Object In the embodiments described so far, the color of the tracking object is determined based on the planar regular image T stored in the planar regular image memory 140 by the reference color extraction unit 200. An example of extracting the reference color α representing the above has been described. For example, the neighborhood area N shown in FIG. 22 and the neighborhood area N shown in FIG. 31 are both areas defined on the planar regular image T, and the extraction process of the reference color α is performed on pixels on the planar regular image T. It has been done using.

  However, the reference color extraction target is not necessarily limited to the planar regular image T, and the reference color α can be extracted based on the distorted wide-angle image S stored in the distorted wide-angle image memory 130. It is. The cut-out center point P is a point that can be defined on both the planar regular image T and the distorted wide-angle image S. Therefore, the neighboring region is defined by the pixel including the cut-out center point P on the distorted wide-angle image S and its peripheral pixels. N can also be defined, and based on the neighboring region N on the distorted wide-angle image S, the reference color α can be extracted by the same method as described above.

(3) Modified Example of Controller Device 300 FIG. 14 shows an example of the controller device 300 that is a hardware component of the instruction input unit 240. Of course, the configuration of the controller device is limited to the example shown in FIG. Is not to be done.

  In the controller device 300 shown in FIG. 14, the cursor movement buttons 341 to 344 are used to move the cross cursor C on the screen, and an operation for designating the position of the tracking start point Q is performed by the cross cursor C. However, it is not always necessary to specify the position of the tracking start point Q using such cursor movement buttons 341 to 344.

  The controller device 300 shown in FIG. 14 includes a screen movement button 320. By tilting the screen movement button 320 up, down, left, and right, the cutout position on the planar regular image can be moved up, down, left, and right. it can. Therefore, it is possible to perform an operation for designating the position of the tracking start point Q by using the screen movement button 320. For example, it is possible to input a tracking start instruction including the position of the tracking start point Q using the screen movement button 320 and the tracking start instruction button 345. For this purpose, it is assumed that when the tracking start instruction button 345 is pressed, the automatic tracking flag setting unit 220 receives a tracking start instruction with the position of the cut-out center point P at that time as the tracking start point Q. The automatic tracking start process may be executed.

  Let's show the specific operation in this case with an example. For example, when the viewer wants to enter the automatic tracking mode with the cat 60 as the tracking target, the screen moving button 320 is operated so that the cat 60 is displayed at the center of the screen, as shown in FIG. When the cat 60 is displayed at the center of the screen, the tracking start instruction button 345 may be pressed. Then, the automatic tracking flag setting unit 220 starts automatic tracking on the assumption that a tracking start instruction having the tracking start point Q as the position of the cat 60 at the center of the screen (that is, the position of the cut center point P) is input. Processing can be executed.

  In addition, although the tracking end instruction button is not shown in the controller device 300 shown in FIG. 14, in practice, a tracking end instruction button is prepared in the controller device 300, and the automatic tracking flag setting unit 220 performs this operation. When the tracking end instruction button is pressed, it is preferable to execute the automatic tracking end process by switching the automatic tracking flag to OFF, assuming that the tracking end instruction is input. Of course, other buttons can also be used as tracking end instruction buttons. For example, if the tracking start instruction button 345 is also used as a tracking end instruction button and the tracking start instruction button 345 is pressed when the automatic tracking flag is ON, the tracking end instruction is input and the automatic tracking end Processing may be executed.

  Furthermore, if the image data input by the image input unit 110 is some digital content data and can be fast forwarded, rewound, paused, slow playback, etc., as necessary, the controller device 300 A fast-forward button, a rewind button, a pause button, a slow playback button, and the like may be provided so that these operations can be performed.

(4) Wide Angle Shooting Device and Modification Examples of Shooting Environment The example described so far is an example using a video camera equipped with a fisheye lens as a wide angle shooting device, but a moving image that can be presented by the moving image presentation device according to the present invention. Is not necessarily limited to an image photographed by a photographing apparatus equipped with a fisheye lens. For example, the present invention can also be applied to a case where a moving image shot by a shooting device equipped with an omnidirectional mirror is presented.

  In addition, there are cases in which movie shooting is performed in a fixed-point shooting with a wide-angle shooting device fixed and cases in which moving shooting is performed for shooting surrounding scenery while moving the wide-angle shooting device. The present invention can be applied to the presentation of moving images obtained in either case. In the case of a fixed-point video, the tracking object (moving object) is a moving object such as a cat, a person, or a car. However, in the case of a moving image, the tracking object (moving object) is not limited to a moving object. Also, fixed objects such as buildings and signs can be designated as tracking objects.

10: Road surface 20: Tree 30: Guard rail 40: Video camera 50: Fisheye lens 60: Tracking object (cat)
110: Image input unit 120: Image output unit 130: Distorted wide-angle image memory 140: Planar regular image memory 150: Image extraction conversion unit 160: Neighboring block extraction unit 170: Extraction condition storage unit 180: Block selection unit 190 : Reference color storage unit 200: reference color extraction unit 210: extraction condition automatic change unit 220: automatic tracking flag setting unit 230: extraction condition manual change unit 240: instruction input unit 300: controller device 310: operation panel unit 320: Screen movement button (circular operation panel)
330: Magnification direction change button 331: Magnification change button (increase)
332: Magnification change button (decrease)
333: Direction change button (counterclockwise)
334: Direction change button (clockwise)
340: Cursor movement button 341: Upward movement button 342: Downward movement button 343: Leftward movement button 344: Rightward movement button 345: Tracking start instruction button 350: Connection cable A (1), A (k): Candidate Area a: horizontal dimension of display screen (number of pixels in horizontal direction)
B, Bα, Bβ: Blue pixel values B1 to B12, Bmax: Block b: Vertical dimension of display screen (number of pixels in the vertical direction)
C: Crosshair cursor D: Cutout direction line d: Euclidean distance E, E1, E2, E (1), E (9): Cutout area E (i), E (i + 1): Cutout area G: Two-dimensional Origin G, Gα, Gβ of UV coordinate system: Green pixel value H: Virtual spherical surface Hq: Hue of reference color H (x, y, z): Incident points L1, L2 on virtual spherical surface H: incident rays m, m ( i), m (i + 1): magnification N: neighboring region O: origin P, P1, P2, P (1), P (9) of the three-dimensional XYZ Cartesian coordinate system: cutting center points P (i), P ( i) old, P (i) new, P (i + 1): cutting center point P (xp, yp): cutting center point Q: tracking start point Qα, Qβ: coordinate points R, Rα, RGB color space Rβ: red pixel value r: radius of the distorted wide-angle image S (radius of the phantom spherical surface H)
S, S (1), S (9): Distorted wide-angle image (circular image photographed with a fisheye lens)
S (i), S (i + 1): Distorted wide-angle image (circular image taken with a fisheye lens)
S (x, y): point Si (xi, yi) in the distorted wide-angle image S on the two-dimensional XY orthogonal coordinate system: points T, T1, T2 in the distorted wide-angle image S on the two-dimensional XY orthogonal coordinate system: Planar regular images T (i), T (i + 1): Planar regular images Ti (ui, vi): Points in the planar regular image T on the two-dimensional UV orthogonal coordinate system: Coordinate axes u of the two-dimensional UV coordinate system: Two Coordinate value V regarding coordinate axis U of two-dimensional UV coordinate system V: Coordinate axis V of two-dimensional UV coordinate system V: Coordinate value Vα, Vβ regarding coordinate axis V of two-dimensional UV coordinate system: Vector W1-W49: Pixel Wq: Center pixel Wj (Rj, Gj, Bj): Pixel color (RGB color system)
Wj (Hj, Sj, Vj): Pixel color (HSV color system)
X: coordinate axis x of three-dimensional XYZ orthogonal coordinate system, xp: coordinate value related to coordinate axis X of two-dimensional XY orthogonal coordinate system Y: coordinate axis y of three-dimensional XYZ orthogonal coordinate system, yp: related to coordinate axis Y of two-dimensional XY orthogonal coordinate system Coordinate value Z: coordinate axis α of three-dimensional XYZ orthogonal coordinate system α: reference color φ, φ (i), φ (i + 1): parameter indicating the cutting direction (plane inclination angle)
θ: Allowable angle

Claims (20)

A video presentation device that cuts out a part of a distorted wide-angle image constituting each frame of a video obtained by wide-angle shooting, converts it into a planar regular image, and sequentially outputs this to present a video,
A distorted wide-angle image memory for storing a distorted wide-angle image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY coordinate system;
An image input unit that sequentially stores a distorted wide-angle image of an i-th frame (i is an integer that sequentially increases in time series) sequentially given as time-series data in units of frames;
A planar regular image memory for storing a planar regular image composed of a collection of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system;
An image output unit that reads and outputs a planar regular image stored in the planar regular image memory;
As a condition for cutting out a planar regular image from a part of a distorted wide-angle image, a cutting center point P that is one point on the distorted wide-angle image, a parameter φ indicating the cutting direction of the image, and a predetermined magnification m are: A cutting condition storage unit for storing cutting conditions including,
Based on the cutting condition stored in the cutting condition storage unit, from the cutting position indicated by the cutting center point P of the distorted wide-angle image of the i-th frame stored in the distorted wide-angle image memory. Then, an image having a cut-out size indicated by the magnification m is cut out in the cut-out direction indicated by the parameter φ, converted into a plane regular image, and stored in the plane regular image memory as a plane regular image of the i-th frame. An image cropping conversion unit;
An instruction input unit for inputting instructions from the viewer;
When the instruction input unit inputs a cutting condition change instruction for changing the cutting condition, a cutting condition manual for changing the cutting condition stored in the cutting condition storage unit based on the instruction Change part,
When the instruction input unit inputs a tracking start instruction including the position of the tracking start point Q on a specific planar regular image, the automatic tracking flag is turned ON and the two-dimensional XY corresponding to the tracking start point Q A point on the coordinate system is set as a new cut center point P, an automatic tracking start process for updating the cut center point P stored in the cut condition storage unit is executed, and the instruction input unit performs tracking end An automatic tracking flag setting unit for executing an automatic tracking end process for switching the automatic tracking flag to OFF when an instruction is input;
When the automatic tracking flag is ON, the i-th frame regular image stored in the plane regular image memory or the distorted wide-angle image of the i-th frame stored in the distorted wide-angle image memory. Based on the image, the reference color α of the i-th frame representing the color of the tracking object located at the cut-out center point P for the i-th frame stored in the cut-out condition storage unit, a reference color extraction unit that sequentially extracts each time i is updated;
A reference color storage unit for storing the latest reference color extracted by the reference color extraction unit;
When the automatic tracking flag is ON, the distorted wide-angle image of the (i + 1) th frame stored in the distorted wide-angle image memory is divided into a plurality of blocks and stored in the extraction condition storage unit. A neighboring block extraction unit that extracts a plurality of blocks near the cut-out center point P for the i-th frame as neighboring blocks;
When the automatic tracking flag is ON, for each of the plurality of neighboring blocks, among the individual pixels included in the block, the reference color α of the i-th frame stored in the reference color storage unit is set. A block selection unit that obtains an approximate color pixel number indicating the number of pixels having a color that falls within a predetermined approximate range, and selects a block having the maximum approximate color pixel number;
When the automatic tracking flag is ON, a plurality of candidate areas whose positions are different from each other are defined in the block selected by the block selection unit, and stored in the reference color storage unit from among the plurality of candidate areas A candidate area having a color that best matches the i-th reference color α is selected as the optimum candidate area, and the center point of the selected optimum candidate area is set as a cut-out center point P for the (i + 1) -th frame. A cutting condition automatic changing unit for updating information indicating the cutting center point P stored in the cutting condition storage unit;
A moving picture presentation apparatus comprising: The video presentation device according to claim 1,
The moving picture presentation apparatus, wherein the reference color extraction unit extracts the color of one pixel including the cut-out center point P as the reference color α. The video presentation device according to claim 1,
The reference color extraction unit defines a neighboring area including a central pixel including the cut-out center point P and surrounding peripheral pixels, and extracts an average value of the colors of the pixels in the neighboring area as a reference color α. A video presentation device. The video presentation device according to claim 1,
The reference color extraction unit defines a neighborhood area including a center pixel including the cut-out center point P and surrounding pixels around the center pixel, and a predetermined approximate range with respect to the color of the center pixel among the pixels in the neighborhood area A moving picture presenting apparatus characterized in that a pixel having a color falling within is used as a reference pixel, and an average value of colors of the reference pixel is extracted as a standard color α. In the moving image presentation device according to claim 3 or 4,
The moving picture presentation device, wherein the reference color extraction unit defines a square pixel array of n rows and n columns centering on a central pixel including the cut-out center point P as a neighborhood region. In the moving image presentation device according to any one of claims 1 to 5,
The neighboring block extraction unit divides the distorted wide-angle image into a plurality of rectangular blocks arranged in vertical and horizontal directions, and uses the block including the cut-out center point P as a central block, and the central block and its upper, lower, left, and right 9. A moving picture presentation apparatus that extracts a total of nine blocks (excluding blocks that do not exist at a corresponding position) including neighboring blocks located in diagonal directions as neighboring blocks. In the moving image presentation device according to any one of claims 1 to 6,
When the cutting condition automatic changing unit defines a pixel array of a predetermined size and the frame of the pixel array is arranged at a predetermined position in the block, the area in the frame is set as a candidate area, and the arrangement of the frame of the pixel array A moving picture presentation apparatus that defines a plurality of candidate areas having different positions from each other by shifting the vertical and horizontal one pixel at a time. The video presentation device according to claim 7,
The moving picture presentation device, wherein the extraction condition automatic change unit selects a candidate area whose average color value of pixels in the candidate area is closest to the reference color α as the optimum candidate area. The video presentation device according to claim 7,
The cutting condition automatic changing unit uses, as a reference pixel, a pixel having a color that falls within a predetermined approximate range with respect to the reference color α among the pixels in the candidate region, and the average value of the color of the reference pixel is set as the reference color α. An apparatus for presenting a moving image, wherein the nearest candidate area is selected as an optimal candidate area. In the moving image presentation apparatus in any one of Claims 1-9,
The block selection unit sets the minimum reference value of the approximate color pixel number in advance, and if there is no block whose approximate color pixel number is equal to or greater than the minimum reference value, the automatic tracking flag setting unit does not perform block selection Gives an automatic tracking failure signal to
The moving picture presenting apparatus, wherein the automatic tracking flag setting unit switches the automatic tracking flag to OFF when the automatic tracking failure signal is given. In the moving image presentation apparatus in any one of Claims 1-10,
When the cutting condition automatic changing unit sets the minimum matching criterion for the reference color α in advance, and there is no candidate area whose degree of matching is equal to or higher than the minimum matching criterion, the selection of the optimal candidate area and the cutting center point An automatic tracking failure signal is given to the automatic tracking flag setting unit without updating the information indicating P,
The moving picture presenting apparatus, wherein the automatic tracking flag setting unit switches the automatic tracking flag to OFF when the automatic tracking failure signal is given. The moving picture presentation device according to claim 1, 4 or 9,
As a condition for the color of the second pixel to fall within a predetermined approximate range of the color of the first pixel, the pixel value of each pixel is expressed in the HSV color system, and the value of hue H is an angle of 0 to 360 °. The condition that “the angle difference between the angle representing the hue H of the color of the first pixel and the angle representing the hue H of the color of the second pixel is equal to or less than a predetermined allowable angle θ” A moving picture presentation apparatus characterized by being used. The moving picture presentation device according to claim 1, 4 or 9,
As a condition for the color of the second pixel to fall within a predetermined approximate range of the color of the first pixel, the pixel value of each pixel is expressed in the RGB color system, and the color of each pixel is represented in the three-dimensional RGB coordinate system. A moving picture presentation apparatus using a condition that “Euclidean distance between coordinate points of two pixels is equal to or less than a predetermined value” when plotted as coordinate points on a color space indicated by. In the moving image presentation device according to claim 3, 4, 8, or 9,
When calculating the average value of the colors of a plurality of pixels, the pixel value of each pixel is expressed in the RGB color system, the average value Rav of the pixel value R, the average value Gav of the pixel value G, and the average of the pixel value B A moving picture presentation apparatus characterized in that a value Bav is obtained, and a color indicated by these pixel values (Rav, Gav, Bav) is used as an average value of colors of the plurality of pixels. In the moving image presentation apparatus in any one of Claims 1-14,
The instruction input unit
A controller device having four cursor movement buttons arranged vertically and horizontally and a tracking start instruction button;
Cursor position indicating means for instructing the image output unit of the position of the cursor that moves up, down, left and right based on the operation of the four cursor movement buttons;
Have
The image output unit outputs an image in which the cursor is superimposed at a position indicated by the cursor position indicating means on the planar regular image,
When the tracking start instruction button is pressed, the automatic tracking flag setting unit executes an automatic tracking start process on the assumption that a tracking start instruction with the tracking position as the tracking start point Q is input. A video presentation device characterized by the above. In the moving image presentation apparatus in any one of Claims 1-14,
The instruction input unit includes a controller device having four screen movement buttons arranged vertically and horizontally, and a tracking start instruction button,
The cutting condition manual change unit stores the cutting condition stored in the cutting condition storage unit so that the cutting position on the planar regular image moves up, down, left, and right based on the operation of the four screen movement buttons. It has a function to change the center point P,
When the tracking start instruction button is pressed, the automatic tracking flag setting unit assumes that a tracking start instruction with the position of the cutting center point P at that time as the tracking start point Q is input, and automatic tracking start processing The moving picture presentation apparatus characterized by performing. The moving picture presentation device according to claim 15 or 16,
When the tracking end instruction button provided in the controller device is pressed by the automatic tracking flag setting unit, or when the tracking start instruction button is pressed when the automatic tracking flag is ON, the tracking end instruction is input. As an example, a moving picture presentation apparatus that performs an automatic tracking end process. In the moving image presentation apparatus in any one of Claims 15-17,
The controller device further includes a magnification change button for increasing / decreasing the magnification m, and a direction changing button for increasing / decreasing a parameter φ indicating the image cutting direction,
When the cutting condition manual changing unit has pressed the magnification changing button or the direction changing button, the magnification m or parameter stored in the cutting condition storage unit is assumed to be an instruction for changing the cutting condition. A video presentation device characterized by changing φ. In the moving image presentation device according to any one of claims 1 to 18,
The image input unit inputs a circular distorted wide-angle image obtained by photographing using a fisheye lens, stores this in a distorted wide-angle image memory,
A neighboring block extraction unit divides the circular distorted wide-angle image into a plurality of rectangular blocks, and a block that partially protrudes from the outline of the circular distorted wide-angle image is extracted as a neighboring block. A video presenting apparatus characterized in that it is not performed.   The program for functioning a computer as a moving image presentation apparatus in any one of Claims 1-19.

Images