JP2011061511A - Fish-eye monitoring system - Google Patents

Abstract

An object of the present invention is to make it possible to grasp the spatial positional relationship between images before and after switching even when the cutout location from a distorted circular image is switched.
A part of a distorted circular image S obtained by photographing using a fisheye lens is cut out, converted into a planar regular image, and displayed on a monitor. In a normal monitoring state, the moving object is detected by analyzing the image for each frame while cutting out and displaying the image in the vicinity region E0 of the point P0 on the monitor. When the moving body 60 is detected, an image in the vicinity area En of the center point Pn of the moving body is cut out and displayed on the monitor. When switching the display, a predetermined transition period is provided, and panning for gradually moving the cutout position along a predetermined movement path from the point P0 to the point Pn and zooming for gradually changing the magnification of the image are performed. The panning speed is set so that the moving speed on the virtual spherical surface indicating the optical characteristics of the fisheye lens becomes a desired speed. Thereafter, the screen is moved while tracking the moving body 60.
[Selection] Figure 22

Description

  The present invention relates to a fish-eye monitoring system that displays an image taken by a video camera with a fish-eye lens on a monitor screen.

  Recently, in order to ensure security, an image taken by a security camera installed on the ceiling or wall of a building is displayed on a monitor screen to help monitor the surroundings. Such a camera for a surveillance system often uses a fisheye lens. When a fisheye lens is used, a circular image showing all directions of a hemisphere can be obtained without a mechanical operation mechanism. For this reason, a fish-eye lens is most suitable for the purpose of monitoring the surroundings. In particular, if a video camera with a fisheye lens is used, a system capable of monitoring an image in a very wide field of view in real time can be realized.

  However, since an image obtained by photographing using a fisheye lens is a distorted circular image, it is necessary to perform processing for converting the distorted circular image into a planar regular image with less distortion. For example, Patent Documents 1 and 2 below disclose a technique for converting a part of a distorted circular image into a planar regular image in real time using a computer. By using such a conversion technique, it becomes possible to observe a moving image composed of a distorted circular image captured by a video camera with a fisheye lens in real time as a moving image composed of a planar regular image, and has an angle of view of 180 °. A monitoring system can be constructed.

  On the other hand, in the industrial field of surveillance systems, moving body tracking techniques have been studied for a long time. This is a technique for analyzing captured images obtained in time series in units of frames by a computer and automatically recognizing and tracking moving objects existing in the field of view of the camera. By using this technique, it is possible to automatically zoom up and display a moving object on the monitor screen, and even when the moving object moves, this can be automatically tracked to keep the zoomed up state.

  Attempts have also been made to introduce such moving body tracking technology into fisheye monitoring systems. For example, the following Patent Document 3 discloses a technique for recognizing a moving object by analyzing a distorted circular image photographed by a fisheye lens, cutting out a part including the moving object, converting it into a planar regular image, and displaying it on a monitor screen. It is disclosed.

Japanese Patent No. 3012142 Japanese Patent No. 3051173 JP 2002-329207 A

  As described above, a distorted circular image obtained by shooting using a fisheye lens is an image including hemispherical omnidirectional information, and has an extremely wide angle of view compared to the angle of view obtained by normal macroscopic observation. Become an image. For this reason, the planar regular image displayed on the monitor screen must be a partial image obtained by cutting out only a part of the distorted circular image and correcting the distortion. Of course, if necessary, it is possible to display a distorted circular image on the monitor screen and present the entire image. Since it cannot correctly recognize the positional relationship between objects and each other, it is unsuitable for use in a monitoring system.

  After all, the image presented on the monitor screen in the fish-eye monitoring system is always an image obtained by cutting out a specific part of the distorted circular image obtained by photographing. Here, when the cutout part on the distorted circular image is switched from the first part to the second part, the planar regular image presented on the monitor screen is also switched, but the user who is watching the monitor screen This causes a problem that it is difficult to grasp the spatial positional relationship between images before and after switching.

  In particular, in the case of a fish-eye monitoring system incorporating a moving body tracking technology, when a moving body is detected, the cutting conditions such as a cutout portion on the distorted circular image and a display magnification are cut out to be suitable for displaying the moving body. Processing to automatically switch to conditions is performed, but if such switching is performed, a planar regular image suitable for moving object gaze can be presented on the monitor screen, but the video is discontinuous in time Therefore, it is difficult for the user to recognize which place the video is switched to.

  Therefore, the present invention provides a fish-eye monitoring system and an image presentation system that allow a user to easily grasp the spatial positional relationship between images before and after switching even when the cutting condition of a distorted circular image is switched. For the purpose.

(1) A first aspect of the present invention is a fisheye monitoring system in which a part of a distorted circular image obtained by photographing using a fisheye lens is cut out, converted into a planar regular image, and displayed on a screen.
A video camera with a fisheye lens that captures images of the outside world as a distorted circular image with the attached fisheye lens,
A distorted circular image having a radius R centered at the origin O of the two-dimensional XY coordinate system, which is composed of a collection of a large number of pixels arranged at positions indicated by coordinates (x, y) on the two-dimensional XY coordinate system. A distorted circular image memory for storing at least one frame;
An image input unit for storing a distorted circular image sequentially given as time-series data for each frame from a video camera with a fisheye lens in a distorted circular image memory;
A planar regular image memory for storing a planar regular image composed of a collection of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system;
An image output unit that reads and outputs a planar regular image stored in the planar regular image memory;
A monitor device for displaying a planar regular image output from the image output unit on the screen;
An actual cutting condition constituted by a cutting center point P that is one point on the distorted circular image, a predetermined plane inclination angle φ, and a predetermined magnification m is determined from a part of the distorted circular image as a planar regular image. An actual cutting condition determination unit that determines as a condition for cutting
An image having a cut-out size indicated by the magnification m is cut out from the cut-out position indicated by the cut-out center point P of the distorted circular image stored in the distorted circular image memory in the cut-out direction indicated by the plane inclination angle φ. Is converted to a planar regular image and stored in the planar regular image memory,
A standard cutting condition storage unit for storing a standard cutting condition composed of a standard cutting center point P0, a standard plane inclination angle φ0, and a standard magnification m0;
When moving object detection processing is performed by comparing the distorted circular images sequentially stored in the distorted circular image memory in time series, and when moving object detection is performed, the target cut-out center point Pn indicating the cut-out position of the detected moving object is Of the target cutting conditions constituted by the target plane inclination angle φn indicating the cutout direction of the detected moving body and the target magnification mn indicating the cutout size of the detected moving body, at least the target cutout center point Pn is actually cut out. A moving object detection unit to be provided to the condition determination unit;
After the moving object detection, a moving object tracking process for tracking the detected moving object is performed, and a tracking cut-out center point P (i) indicating the cut-out position of the tracking moving object, and a tracking plane inclination angle φ (i) indicating the cut-out direction of the tracking moving object Of the tracking cutout conditions configured by the tracking magnification m (i) indicating the cutout size of the tracking moving object, at least the tracking cutout central point P (i) is determined to disappear. Until the actual tracking condition determining unit,
Provided,
The actual cutting condition determination unit is in charge of a static monitoring period in charge of a static monitoring period in which no moving object is detected, a transition period in charge of a predetermined transition period after moving object detection, and a transition A dynamic monitoring period responsible section that takes charge of the dynamic monitoring period from the completion of the period until the decision to disappear is made, and the static monitoring period responsible section defines the standard cutting condition as the actual cutting condition The transition period department sets the actual cutting conditions that change step by step so that the standard cutting conditions shift to the target cutting conditions, and the dynamic monitoring period section sets the tracking cutting conditions as the actual cutting conditions. It is what you specify.

(2) According to a second aspect of the present invention, in the fish-eye monitoring system according to the first aspect described above,
In a three-dimensional XYZ Cartesian coordinate system including a two-dimensional XY Cartesian coordinate system in which a distorted circular image is arranged, a virtual sphere having a radius R is defined with the origin O as the center, and one point on the virtual sphere is directed toward the origin O. When incident external light reaches one point on the XY plane by the optical action of the fisheye lens used for photographing, a point on the virtual sphere and a point on the XY plane correspond to each other A corresponding point Q that is a point on the virtual sphere corresponding to the cut-out center point P, a vector passing through the corresponding point Q starting from the origin O is defined as a line-of-sight vector n, and the line-of-sight vector n A plane inclination angle on a plane passing through the point G and perpendicular to the line-of-sight vector n or a curved surface obtained by curving the plane, with the point G separated from the origin O by “product m · R of the magnification m and the radius R” as the origin. Two arranged with orientation according to φ When it defined the original UV coordinate system,
The image cut-out conversion unit uses a predetermined correspondence relation expression indicating the correspondence between the coordinates (u, v) on the two-dimensional UV coordinate system and the coordinates (x, y) on the two-dimensional XY orthogonal coordinate system, By calculating the corresponding coordinates (x, y) corresponding to the coordinates (u, v), processing for converting an image cut out from the distorted circular image into a planar regular image is performed.

(3) According to a third aspect of the present invention, in the fisheye monitoring system according to the first or second aspect described above,
In the first half of the transition period, the transition period responsible section determines the actual cutting conditions in which the cutting center point P gradually changes from the standard cutting center point P0 to the target cutting center point Pn. In the zooming period, an actual cutting condition in which the magnification m changes stepwise from the standard magnification m0 to the target magnification mn is determined.

(4) According to a fourth aspect of the present invention, in the fisheye monitoring system according to the first or second aspect described above,
The transition period responsible section sets the entire transition period as a panning period and a zooming period, and the cutting center point P changes from the standard cutting center point P0 to the target cutting center point Pn step by step. An actual cutting condition in which m gradually changes from the standard magnification m0 to the target magnification mn is determined.

(5) According to a fifth aspect of the present invention, in the fisheye monitoring system according to the second aspect described above,
The transition period responsible section sets all or part of the transition period as the panning period, and in the panning period, the cutting center point P is gradually changed from the standard cutting center point P0 to the target cutting center point Pn. When changing, when the corresponding point Q0 on the virtual sphere for the standard cut center point P0 and the corresponding point Qn on the virtual sphere for the target cut center point Pn are defined, the cut center point P The moving path of the corresponding point Q on the virtual spherical surface on the virtual spherical surface is the shortest path connecting the corresponding point Q0 and the corresponding point Qn on the virtual spherical surface.

(6) According to a sixth aspect of the present invention, in the fisheye monitoring system according to the fifth aspect described above,
The standard cut-out center point P0 is set on the origin O of the two-dimensional XY orthogonal coordinate system,
The transition period responsible section moves the cut center point P from the origin O toward the target cut center point Pn along the radius of the distorted circular image.

(7) According to a seventh aspect of the present invention, in the fisheye monitoring system according to the second aspect described above,
The standard cutting center point P0 is set at a position other than the origin O of the two-dimensional XY orthogonal coordinate system,
The transition period responsible section sets all or part of the transition period as the panning period, and in the panning period, the cutting center point P is gradually changed from the standard cutting center point P0 to the target cutting center point Pn. When changing, after moving the cutting center point P along the radius of the distorted circular image from the standard cutting center point P0 to the origin O, the origin O to the target cutting center point Pn, The cut center point P is moved along the radius of the distorted circular image.

(8) The eighth aspect of the present invention is the fish-eye monitoring system according to the fifth to seventh aspects described above,
The transition period charge section changes the position of the cut-out center point P so that the movement of the corresponding point Q on the virtual spherical surface is a constant speed movement.

(9) According to a ninth aspect of the present invention, in the fisheye monitoring system according to the fifth to seventh aspects described above,
The transition period charge unit is such that the movement of the corresponding point Q on the phantom spherical surface is an inconstant speed movement such that the speed of the start part and the end part of the panning period is slower than the speed of the intermediate part. The position of the point P is changed.

(10) According to a tenth aspect of the present invention, in the fisheye monitoring system according to the fifth to seventh aspects described above,
The transition period department
A function storage for storing a function indicating the relationship between the movement distance b of the corresponding point Q on the virtual sphere and the time t;
A corresponding point conversion unit that outputs a point on the XY plane corresponding to the “position of the corresponding point Q at time t” obtained using this function as a cut-out center point P at the time t;
It is made to have.

(11) An eleventh aspect of the present invention is the fisheye monitoring system according to the second aspect described above,
The transition period responsible section sets all or part of the transition period as the panning period, and in the panning period, the cutting center point P is gradually changed from the standard cutting center point P0 to the target cutting center point Pn. When changing, the corresponding point Q on the virtual spherical surface with respect to the cut-out center point P moves on a predetermined spherical moving path set in advance at a predetermined spherical moving speed set as a function related to time. The position of the cutting center point P is determined.

(12) According to a twelfth aspect of the present invention, in the fisheye monitoring system according to the second aspect described above,
The transition period department sets the whole or part of the transition period as the zooming period, and sets the magnification m from the standard magnification m0 so that it monotonously increases or decreases along the time axis during the zooming period. The magnification is changed to mn.

(13) According to a thirteenth aspect of the present invention, in the fisheye monitoring system according to the second aspect described above,
The transition period responsible section linearly changes the magnification m along the time axis during the zooming period.

(14) According to a fourteenth aspect of the present invention, in the fisheye monitoring system according to the twelfth aspect described above,
The transition period charge section changes the magnification m along the time axis so that the magnification change rate at the start and end of the zooming period changes at an inconstant speed that is slower than the change rate at the intermediate part. It is a thing.

(15) According to a fifteenth aspect of the present invention, in the fisheye monitoring system according to the twelfth aspect described above,
The transition period department
A function storage unit for storing a function indicating the relationship between the magnification m and the time t;
A magnification determining unit that outputs a magnification at time t obtained using this function as a magnification m at time t;
It is made to have.

(16) According to a sixteenth aspect of the present invention, in the fisheye monitoring system according to the second aspect described above,
In the standard cut-out condition storage unit, a standard plane inclination angle φ0 is set so that a plane regular image is cut out in a direction in which the vertical axis direction of the real world in the photographed image is downward.
The moving body detection unit sets a target plane inclination angle φn such that a plane regular image is cut out in a direction in which the vertical axis direction is a downward direction,
The transition period responsible section changes the plane inclination angle φ from the standard plane inclination angle φ0 to the target plane inclination angle φn so that the plane regular image is cut out in the direction in which the vertical axis direction is the downward direction.
The moving body tracking unit sets the tracking plane inclination angle φ (i) so that a planar regular image is cut out in a direction with the vertical axis direction as a downward direction.

(17) According to a seventeenth aspect of the present invention, in the fisheye monitoring system according to the sixteenth aspect described above,
The video camera with a fisheye lens is installed so that the XY plane as the imaging surface coincides with the vertical plane and the Y axis faces the direction that forms an angle ξ with respect to the vertical axis W.
The angle formed by the rotation reference axis J that passes through the origin G of the two-dimensional UV coordinate system, is parallel to the XY plane, and is orthogonal to the line-of-sight vector n, and the U axis of the two-dimensional UV coordinate system is the plane inclination angle φ. When the angle between the orthographic projection of the line-of-sight vector n on the XY plane and the Y axis is defined as the azimuth angle α, and the U axis direction is defined as the lateral direction of the planar regular image, φ = −α The setting to be −ξ is performed.

(18) According to an eighteenth aspect of the present invention, in the fisheye monitoring system according to the seventeenth aspect described above,
The standard cut-out center point P0 is set on the origin O of the two-dimensional XY orthogonal coordinate system,
In the standard cutout condition storage unit, a standard cutout condition indicating the cutout direction is set so that the planar regular image is cut out so that the V axis corresponds to the vertical axis in the real world at the position of the origin O of the photographed image. Every
The moving object detection unit sets the target plane inclination angle φn = −αn−ξ when the azimuth angle with respect to the target cutting center point Pn is αn,
The transition period person in charge always sets the plane inclination angle φ = φn.

(19) According to a nineteenth aspect of the present invention, in the fisheye monitoring system according to the sixteenth aspect described above,
A video camera with a fisheye lens is installed so that the XY plane as the imaging surface coincides with the horizontal plane.
The angle formed by the rotation reference axis J that passes through the origin G of the two-dimensional UV coordinate system, is parallel to the XY plane, and is orthogonal to the line-of-sight vector n, and the U axis of the two-dimensional UV coordinate system is the plane inclination angle φ. In this case, when the U-axis direction is defined as the horizontal direction of the planar regular image, the setting is always set to φ = 0 °.

(20) According to a twentieth aspect of the present invention, in the fisheye monitoring system according to the second aspect described above,
The image cut-out conversion unit calculates the corresponding coordinates (x, y) for the coordinates (u, v) of one pixel of interest constituting the planar regular image, and the corresponding coordinates (x, y in the distorted circular image memory). ) Is performed on each pixel constituting the planar regular image, and the pixel value of the pixel of interest is determined based on the readout pixel value, and is determined in the planar regular image memory. In this case, the pixel value of each pixel is written in to perform a process of converting into a planar regular image.

(21) According to a twenty-first aspect of the present invention, in the fisheye monitoring system according to the twentieth aspect described above,
When the image cut-out conversion unit determines the pixel value of the pixel of interest arranged at the position indicated by the coordinates (u, v), the distortion arranged near the position indicated by the corresponding coordinates (x, y) An interpolation operation is performed on pixel values of a plurality of reference pixels on the circular image.

(22) According to a twenty-second aspect of the present invention, in the fish-eye monitoring system according to the second aspect described above,
On the virtual sphere, the image cut-out conversion unit corresponds to the point on the sphere corresponding to the point Si indicated by the coordinates (xi, yi) on the two-dimensional XY orthogonal coordinate system, according to the projection method of the fisheye lens used for shooting. When Qi is taken and the coordinates on the two-dimensional UV coordinate system of the intersection Ti between the straight line connecting the origin O and the corresponding point Qi on the spherical surface and the arrangement surface of the two-dimensional UV coordinate system are (ui, vi), Corresponding relational expressions in which coordinates (xi, yi) are obtained as corresponding coordinates corresponding to coordinates (ui, vi) are used.

(23) According to a twenty-third aspect of the present invention, in the fish-eye monitoring system according to the twenty-second aspect described above,
Image cropping conversion unit
When the distorted circular image stored in the distorted circular image memory is an orthographic image captured by the orthographic fisheye lens, with respect to the cut-out center point P indicated by the coordinates (xp, yp). A point indicated by coordinates (xp, yp, zp) given as an intersection of a straight line passing through the point P and parallel to the Z axis and the virtual sphere is designated as a corresponding point Q on the sphere, and is indicated by coordinates (xi, yi). For an orthogonal projection image with respect to the point Si, a point indicated by coordinates (xi, yi, zi) given as an intersection of a straight line passing through the point Si and parallel to the Z axis and the virtual sphere is a corresponding point Qi on the sphere. Using the correspondence equation,
When the distorted circular image stored in the distorted circular image memory is a non-orthographic image captured by a non-projection type fisheye lens, the coordinates on the ortho-projection image and the coordinates on the non-ortho-projection image The correspondence relation formula for non-orthographic projection images obtained by correcting the correspondence relation formula for orthogonal projection images using the coordinate transformation formula between is used.

(24) According to a twenty-fourth aspect of the present invention, in the fisheye monitoring system according to the twenty-third aspect described above,
The image cut-out conversion unit defines an angle formed by the orthogonal projection of the line-of-sight vector n on the XY plane and the Y axis as an azimuth angle α, and an angle formed between the line-of-sight vector n and the Z axis positive direction as a zenith angle β.
A = cosφ cosα-sinφ sinα cosβ
B = −sinφ cosα − cosφ sinα cosβ
C = sinβ sinα
D = cosφ sinα + sinφ cosα cosβ
E = -sinφ sinα + cosφ cosα cosβ
F = -sinβ cosα
w = mR
As a correspondence relation expression for an orthographic image indicating the correspondence between coordinates (u, v) and coordinates (x, y),
x = R (uA + vB + wC) / √ (u ² + v ² + w ² )
y = R (uD + vE + wF) / √ (u ² + v ² + w ² )
This formula is used.

(25) According to a twenty-fifth aspect of the present invention, in the fisheye monitoring system according to the twenty-fourth aspect described above,
Image cropping conversion unit
When the distorted circular image stored in the distorted circular image memory is an equidistant projection image photographed by a fisheye lens of the equidistant projection method, the coordinates (xa, ya) on the orthographic projection image are equidistantly projected. Coordinate conversion formula to convert to coordinates (xb, yb) on the image xb = xa (2R / πr) sin ⁻¹ (r / R)
yb = ya (2R / πr) sin ⁻¹ (r / R)
However, r = √ (xa ² + ya ² )
Is used to correct the orthographic image correspondence relation.

(26) According to a twenty-sixth aspect of the present invention, in the fish-eye monitoring system according to the first to the twenty-fifth aspects described above,
The moving object detector
A background image holding unit that holds an image stored in a distorted circular image memory at a predetermined initial time point as a background distorted circular image;
Compare the pixel values of the individual pixels on the distorted circular image that are sequentially stored in the distorted circular image memory for each frame in time series with the pixel values of the corresponding pixels on the background distorted circular image. A corresponding pixel comparison unit that extracts a pixel having a threshold value or more as a target pixel;
A focused area search unit that searches for a focused area that is a continuous area composed of focused pixels, and that satisfies a condition that satisfies a reference area that is equal to or larger than a reference area and that has a vertical width and a horizontal width that are equal to or greater than a predetermined reference dimension;
When a distorted circular image including the region of interest is continuously obtained for the number of frames equal to or greater than the reference number of frames, it is determined that the moving object has been detected, and the moving object detection that generates the target cut-out condition based on the final region of interest A determination unit;
It is made to have.

(27) According to a twenty-seventh aspect of the present invention, in the fisheye monitoring system according to the twenty-sixth aspect described above,
The moving object is detected when the moving object detection determination unit continuously obtains a distorted circular image including a specific region of interest that is expected to be formed due to the same moving object for the number of frames that is the reference number or more. When there are a plurality of regions of interest separated from each other, it is determined that the region of interest having a larger area is the specific region of interest.

(28) According to a twenty-eighth aspect of the present invention, in the fisheye monitoring system according to the twenty-sixth or twenty-seventh aspect described above,
The moving object detection determination unit sets the center of gravity position of the final target region as the target cut-out center point Pn, and sets the target magnification mn based on the vertical width and horizontal width of the final target region.

(29) According to a twenty-ninth aspect of the present invention, in the fish-eye monitoring system according to the first to twenty-eighth aspects described above,
The moving body tracking unit
A known moving object region holding unit for holding information of a known moving object region that has already been recognized on the (i-1) th frame image stored in the distorted circular image memory;
A candidate area extraction unit that extracts a plurality of areas located in the vicinity of the known moving body area as candidate areas from the i-th frame image stored in the distorted circular image memory;
A feature amount calculation unit that performs an operation for obtaining a feature amount indicating a feature of an image for each of the known moving body region and the plurality of candidate regions;
The feature amount of the known moving object region calculated by the feature amount calculating unit is compared with the feature amounts of the plurality of candidate regions, respectively, and the similarity to the feature amount of the known moving object region is equal to or higher than a predetermined reference and is the most A new moving body region recognition unit for recognizing a candidate region having a high feature amount as a new moving body region on the i-th frame image;
When the recognition by the new moving body region recognition unit is successful, a tracking cut-out condition for the i-th frame image is generated based on the new moving body region, and when the recognition fails, the tracking moving body disappears. A moving body tracking determination unit for determining the disappearance of
Have
The existing moving object region holding unit holds the detected moving object region information given from the moving object detecting unit as the first known moving object region information, and thereafter holds the new moving object region information as the new known moving object region information. It is what I did.

(30) According to a thirtieth aspect of the present invention, in the fisheye monitoring system according to the twenty-ninth aspect described above,
The moving object tracking determination unit sets the gravity center position of the new moving object region recognized for the i-th frame image as the tracking cut-out center point P (i), and the tracking magnification based on the vertical width and the horizontal width of the new moving object region m (i) is set.

(31) According to a thirty-first aspect of the present invention, in the fisheye monitoring system according to the twenty-ninth or thirtieth aspect described above,
The feature amount calculation unit obtains the color histogram or edge direction histogram of the pixels constituting the region given as the calculation target as the feature amount.

(32) According to a thirty-second aspect of the present invention, in the fisheye monitoring system according to the first to thirty-first aspects described above,
The moving object detection unit pauses the new moving object detection process while the moving object tracking process is performed by the moving object tracking unit.

(33) According to a thirty-third aspect of the present invention, in the fisheye monitoring system according to the first to thirty-first aspects described above,
If the moving object detection unit continues the moving object detection process while the moving object tracking process is being performed by the moving object tracking unit, and a new moving object that is different from the old moving object that is the target of the moving object tracking process is detected, Perform processing to give the target cutting condition for the new moving object to the actual cutting condition determination unit,
When the target cutting condition for the new moving object is given to the actual cutting condition determination unit, the transition period charge section shifts from the tracking cutting condition for the old moving object to the target cutting condition for the new moving object. The actual cutting conditions that change
The moving body tracking unit is configured to perform tracking processing for a new moving body.

(34) According to a thirty-fourth aspect of the present invention, in the fisheye monitoring system according to the first to thirty-first aspects described above,
The moving object detection unit continues the moving object detection process while the moving object tracking process is performed by the moving object tracking unit, and is different from the old moving object that is the target of the moving object tracking process and has a larger area than the old moving object. When another moving object that satisfies the condition is detected, the target cutting condition for the new moving object that satisfies the condition is given to the actual cutting condition determining unit,
When the target cutting condition for the new moving object is given to the actual cutting condition determination unit, the transition period charge section shifts from the tracking cutting condition for the old moving object to the target cutting condition for the new moving object. The actual cutting conditions that change
The moving body tracking unit is configured to perform tracking processing for a new moving body.

(35) According to a thirty-fifth aspect of the present invention, in the fisheye monitoring system according to the thirty-third or thirty-fourth aspect described above,
When the moving object tracking unit performs the tracking process for the new moving object, also performs the tracking process for the old moving object, sets the tracking cut-out condition based on the new moving object, and the new moving object disappears earlier than the old moving object, In place of the moving object detection unit, the target cutting condition for the old moving object is given to the actual cutting condition determination unit,
When the target cutting condition for the old moving object is given to the actual cutting condition determination unit, the transition period responsible section shifts from the tracking cutting condition for the new moving object to the target cutting condition for the old moving object. The actual cutting conditions that change
The moving object tracking unit is configured to continue the tracking process for the old moving object.

(36) According to a thirty-sixth aspect of the present invention, in the fisheye monitoring system according to the thirty-third to thirty-fifth aspects described above,
When the transition period responsible department performs transition processing to switch from the old moving object to the new moving object, if the dynamic monitoring period for the old moving object is less than the predetermined minimum reference time, the dynamic monitoring period is the minimum reference time. The process is started after a waiting time until it reaches.

(37) The thirty-seventh aspect of the present invention is the fisheye monitoring system according to the first to thirty-sixth aspects described above,
Manual condition setting unit with a function to arbitrarily set the actual cutting condition to be determined by the actual cutting condition determination unit and the standard cutting condition stored in the standard cutting condition storage unit based on the user's setting operation Is further provided.

  (38) According to a thirty-eighth aspect of the present invention, the components other than the video camera with a fisheye lens and the monitor device in the fisheye monitoring system according to the first to thirty-seventh aspects described above are configured by incorporating a dedicated program into a computer. It is what you do.

  (39) According to a thirty-ninth aspect of the present invention, the components other than the video camera with a fisheye lens and the monitor device in the fisheye monitoring system according to the first to thirty-seventh aspects described above have functions corresponding to these components. It is constituted by a semiconductor integrated circuit in which an electronic circuit is incorporated.

(40) According to a 40th aspect of the present invention, in the image presentation system that cuts out a part of a distorted circular image obtained by photographing using a fisheye lens, converts the image into a planar regular image, and presents the image on a screen.
It is composed of a collection of many pixels arranged at the position indicated by coordinates (x, y) on the two-dimensional XY coordinate system, has a radius R around the origin O of the two-dimensional XY coordinate system, and uses a fisheye lens A distorted circular image memory for storing a distorted circular image obtained by shooting,
A planar regular image memory for storing a planar regular image composed of a collection of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system;
An image display device for displaying a planar regular image stored in a planar regular image memory on a screen;
An actual cutting condition constituted by a cutting center point P that is one point on the distorted circular image, a predetermined plane inclination angle φ, and a predetermined magnification m is determined from a part of the distorted circular image as a planar regular image. An actual cutting condition determination unit that determines as a condition for cutting
Based on the actual cutting condition given from the actual cutting condition determination unit, the plane inclination angle φ is indicated from the cutting position indicated by the cutting center point P of the distorted circular image stored in the distorted circular image memory. An image cut-out conversion unit that cuts out an image having a cut-out size indicated by the magnification m in the cut-out direction, converts the image into a plane regular image, and stores the image in a plane regular image memory;
Provided,
When the actual cutting condition determination unit changes the current cutting condition including the current cutting center point P0 to the target cutting condition including the target cutting center point Pn, the image cutting conversion unit Is performed in such a manner that the position of the cut-out center point P given to is gradually shifted from the point P0 to the point Pn.

(41) The forty-first aspect of the present invention is the image presentation system according to the forty-first aspect described above,
In a three-dimensional XYZ Cartesian coordinate system including a two-dimensional XY Cartesian coordinate system in which a distorted circular image is arranged, a virtual sphere having a radius R is defined with the origin O as the center, and one point on the virtual sphere is directed toward the origin O. When incident external light reaches one point on the XY plane by the optical action of the fisheye lens used for photographing, a point on the virtual sphere and a point on the XY plane correspond to each other A corresponding point Q that is a point on the virtual sphere corresponding to the cut-out center point P, a vector passing through the corresponding point Q starting from the origin O is defined as a line-of-sight vector n, and the line-of-sight vector n A plane inclination angle on a plane passing through the point G and perpendicular to the line-of-sight vector n or a curved surface obtained by curving the plane, with the point G separated from the origin O by “product m · R of the magnification m and the radius R” as the origin. Two arranged with orientation according to φ When it defined the original UV coordinate system,
The image cut-out conversion unit uses a predetermined correspondence relation expression indicating the correspondence between the coordinates (u, v) on the two-dimensional UV coordinate system and the coordinates (x, y) on the two-dimensional XY orthogonal coordinate system, By calculating the corresponding coordinates (x, y) corresponding to the coordinates (u, v), processing for converting an image cut out from the distorted circular image into a planar regular image is performed.

(42) In a forty-second aspect of the present invention, in the image presentation system according to the forty-first aspect described above,
The actual cutting condition determination unit moves the corresponding point Q on the virtual spherical surface with respect to the cutting center point P on the predetermined spherical moving path set at a predetermined spherical moving speed set as a function related to time. As described above, the position of the cutting center point P is determined.

(43) According to a 43rd aspect of the present invention, in the image presentation system according to the 40th to 42nd aspects described above,
The actual cutting condition determination unit performs a process of shifting the position of the cutting center point P from the point P0 to the point Pn in a stepwise manner, and changes the value of the magnification m from the current magnification m0 to the target magnification mn. In this way, the process of shifting in stages is performed.

(44) According to a 44th aspect of the present invention, in the image presentation system according to the 43rd aspect described above,
The actual cutting condition determination unit shifts the value of the magnification m at a predetermined magnification changing speed set as a function related to time.

  According to the fisheye monitoring system and the image presentation system of the present invention, when switching the cutting condition of the distorted circular image, a predetermined transition period is provided, and the cutting condition is changed stepwise during this transition period. The planar regular image on the monitor screen is subjected to panning and zooming effects at the time of switching. Therefore, the user can easily grasp the spatial positional relationship between the images before and after switching.

It is a perspective view which shows the basic model which forms the distortion circular image S by imaging | photography using the fisheye lens of an orthogonal projection system. It is a top view which shows an example of the distortion circular image S obtained by imaging | photography using a fisheye lens (It shows the general image of the distortion circular image S, and does not show an exact image.). 6 is a plan view showing an example in which a cutout region E is defined in a part of a distorted circular image S. FIG. 3 is a perspective view showing a relationship between a two-dimensional XY orthogonal coordinate system including a distorted circular image S and a two-dimensional UV orthogonal coordinate system including a planar regular image T. FIG. It is a top view which shows the relationship between the plane regular image T defined on the two-dimensional UV orthogonal coordinate system, and plane inclination-angle (phi). It is a perspective view which shows the principle of the coordinate transformation from a two-dimensional XY orthogonal coordinate system to a two-dimensional UV orthogonal coordinate system. It is the figure which looked at each component shown by the perspective view of FIG. 6 from the horizontal direction. It is the figure which looked at each component shown by the perspective view of FIG. 6 from upper direction. It is a perspective view which shows the correspondence of point Si (xi, yi) on a two-dimensional XY orthogonal coordinate system, and point Ti (ui, vi) on a two-dimensional UV orthogonal coordinate system in the model which considered the magnification m. It is a side view which shows arrangement | positioning of the two-dimensional UV orthogonal coordinate system at the time of setting magnification m = 1, and the range of cut-out area | region E1. It is a side view which shows arrangement | positioning of the two-dimensional UV orthogonal coordinate system at the time of setting magnification | magnification m> 1, and the range of the cutting-out area | region E2. It is a top view which shows an example of the plane regular image T obtained on the two-dimensional UV orthogonal coordinate system by cutting out a part of the distortion circular image S shown in FIG. It is a top view which shows another example of the plane regular image T obtained on the two-dimensional UV orthogonal coordinate system by cutting out a part of the distortion circular image S shown in FIG. It is a top view which shows the plane regular image T defined on the two-dimensional UV orthogonal coordinate system. It is a top view which shows the distortion circular image S defined on the two-dimensional XY orthogonal coordinate system. Correspondence between points S1 (x1, y1) and S2 (x2, y2) on the two-dimensional XY orthogonal coordinate system and points T1 (u1, v1) and T2 (u2, v2) on the two-dimensional UV orthogonal coordinate system It is a side view of the virtual spherical surface for showing. 4 is an orthographic image correspondence relational expression showing a correspondence relation between coordinates (x, y) on a two-dimensional XY orthogonal coordinate system and coordinates (u, v) on a two-dimensional UV orthogonal coordinate system. It is a side view which shows the example which installed the video camera with a fisheye lens for the outdoor monitoring. It is a top view which shows an example of the distorted circular image obtained by imaging | photography using the video camera with a fisheye lens shown in FIG. FIG. 20 is a plan view showing an example in which two cutout areas E0 and En are set on the distorted circular image shown in FIG. FIG. 21 is a plan view showing a planar regular image obtained corresponding to the two cutout regions E0 and En shown in FIG. 20. FIG. 20 is a plan view showing a state in which moving object detection is performed on the distorted circular image shown in FIG. 19. It is a top view which shows the state by which the moving body tracking was further performed with respect to the detection moving body shown in FIG. It is a top view which shows the planar regular image obtained corresponding to the two types of cutting area | regions E0 and En shown in FIG. It is a top view which shows the principle which produces a panning effect by moving a cutting area gradually, when the moving body detection shown in FIG. 22 is performed. It is a top view which shows the principle which produces the zooming effect after the panning effect shown in FIG. It is a top view which shows the principle which performs a moving body tracking after the zooming effect shown in FIG. 1 is a block diagram showing a basic configuration of a fisheye monitoring system 100 according to an embodiment of the present invention. It is a flowchart which shows the procedure of the moving body detection process performed by the moving body detection part 160 of the system shown in FIG. It is a top view which shows the specific process of step S13, S14 in the flowchart of FIG. FIG. 30 is a plan view showing an example in which a cat is detected as a moving object by a specific process of steps S13 and S14 in the flowchart of FIG. 29. It is a block diagram which shows the detailed structure of the moving body detection part 160 of the system shown in FIG. It is a flowchart which shows the procedure of the moving body tracking process performed by the moving body tracking part 190 of the system shown in FIG. It is a block diagram which shows the detailed structure of the moving body tracking part 190 of the system shown in FIG. It is a flowchart which shows the procedure of the determination process of the actual cutting condition performed by the actual cutting condition determination part 170 of the system shown in FIG. It is a block diagram which shows the detailed structure of the actual cutting condition determination part 170 of the system shown in FIG. FIG. 37 is a plan view showing a specific transition process performed by a transition period charge unit 172 shown in FIG. 36. It is a table | surface which shows the specific transition example of the actual extraction conditions of the transition period when the moving body detection shown in FIG. 22 is performed. It is a side view which shows the specific method of the constant velocity panning which concerns on this invention. It is the top view and graph which show the specific method of the constant velocity panning which concerns on this invention. It is the top view and graph which show the specific method of the inconstant speed panning which concerns on this invention. It is a graph which shows the specific method of the constant speed and inconstant speed zooming which concerns on this invention. It is a table | surface which shows the specific step of the uniform velocity and non-uniform velocity panning which concern on this invention. It is a table | surface which shows the specific step of constant speed and inconstant speed zooming which concerns on this invention. It is a block diagram which shows the detailed structure of the transition period charge part 172 shown in FIG. It is a block diagram which shows the detailed structure of the actual cutting center point determination part 70 shown in FIG. It is a block diagram which shows the detailed structure of the real magnification determination part 90 shown in FIG. It is a top view which shows the direction of the cut-out area | region at the time of installing a video camera with a fisheye lens so that an imaging surface may correspond to a vertical surface and an X-axis may correspond to the vertical axis W of the real world. It is a top view which shows the direction of a cut-out area | region when a video camera with a fisheye lens is installed in the direction where an imaging surface corresponds to a vertical surface and the Y axis makes an angle ξ with respect to the vertical axis W in the real world. It is a top view which shows direction of the cut-out area | region at the time of installing a video camera with a fisheye lens so that an imaging surface may correspond to a horizontal surface. It is a top view which shows the horizontal surface direction of the picked-up image at the time of installing a video camera with a fisheye lens so that an imaging surface may correspond to a horizontal surface. It is a top view which shows an example of a panning path | route in case the standard cutting center point P0 is set in positions other than the origin O of a two-dimensional XY orthogonal coordinate system. It is a top view which shows another example of the panning path | route in case the standard cutting center point P0 is set in positions other than the origin O of a two-dimensional XY orthogonal coordinate system. It is a perspective view which shows the principle which calculates | requires the movement path | route of the cutting center point P in the case of taking the panning path | route shown in FIG. It is a figure which shows the computing equation for determining the position of the cutting center point P based on the principle shown in FIG. It is a time chart which shows the 1st handling policy in case a some moving body exists. It is a time chart which shows the 2nd handling policy in case a some moving body exists. It is a time chart which shows the 3rd handling policy in case a some moving body exists. It is a time chart which shows the operation | movement which ensures standby | waiting time before performing a migration process. It is a side view which shows the projection state of the incident light in the orthographic fisheye lens. It is a perspective view which shows the relationship between the orthographic projection image formed using the orthographic projection fisheye lens, and the equidistant projection image formed using the equidistant projection fisheye lens. It is a figure which shows the conversion type for performing coordinate conversion between the coordinate on an equidistance projection image, and the coordinate on an orthographic projection image. It is a perspective view which shows the correspondence of point Si (xi, yi) on a two-dimensional XY orthogonal coordinate system, and point Ci (ui, vi) on the two-dimensional UV coordinate system defined on the curved surface.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic model of image conversion processing >>>
Since an image obtained by photographing with a fisheye lens is a distorted circular image, it is necessary to perform processing for converting the distorted circular image into a planar regular image with less distortion in order to display it on the monitor screen. is there. Therefore, here, a general characteristic of a distorted circular image obtained by photographing using a fisheye lens and a basic principle of processing for cutting out a part of the image and converting it into a planar regular image will be described.

  FIG. 1 is a perspective view showing a basic model for forming a distorted circular image S by photographing using an orthographic fisheye lens. In general, fish-eye lenses are classified into a plurality of types according to the projection method, but the model shown in FIG. 1 is for a fish-eye lens of an orthographic projection method (a method for applying the present invention to a fish-eye lens other than an orthographic projection method) , Described in §9-3).

  FIG. 1 shows an example in which a distorted circular image S is formed on an XY plane in a three-dimensional XYZ orthogonal coordinate system. Here, as shown in the figure, an example is shown in which the Z axis is taken upward in the figure and a dome-like virtual spherical surface H (hemisphere) is defined on the positive region side of the Z axis.

  The distorted circular image S formed on the XY plane is an image forming a circle with a radius R centered on the origin O of the coordinate system, and is an area having an angle of view of 180 ° on the positive area side of the Z axis. Is equivalent to the image recorded with distortion. FIG. 2 is a plan view showing an example of a distorted circular image S obtained by photographing using a fisheye lens. As described above, in the distorted circular image S, all the images existing on the positive region side of the Z-axis are recorded. However, the scale magnification of the image is different between the central portion and the peripheral portion. The shape of the recorded image is distorted. Note that the distorted circular image S shown in FIG. 2 shows a general image of a distorted circular image obtained by photographing using a fisheye lens, and does not show an accurate image obtained using an actual fisheye lens. Absent.

  An actual fisheye lens is constituted by an optical system that is a combination of a plurality of convex lenses and concave lenses, and it is known that its optical characteristics can be modeled by a virtual spherical surface H as shown in FIG. In other words, considering a model in which a dome-shaped virtual spherical surface H (hemisphere) having a radius R is arranged on the upper surface of the distorted circular image S, the optical characteristics of the orthographic fisheye lens are arbitrary on the virtual spherical surface H. The incident light beam L1 incident from the normal direction to the point H (x, y, z) behaves toward the point S (x, y) on the XY plane as an incident light beam L2 parallel to the Z axis. You may think. In other words, the pixel located at the point S (x, y) on the distorted circular image S in FIG. 2 indicates one point on the object existing on the extension line of the incident light beam L1 shown in FIG. become.

  Of course, an optical phenomenon occurring in an actual fisheye lens is that a specific point of an object to be imaged is connected to a specific point S (x, y) on the XY plane due to refraction by a plurality of convex lenses and concave lenses. Although this is an image phenomenon, there is no problem in performing an image conversion process or the like even if the argument is replaced with a model using a virtual spherical surface H as shown in FIG. Therefore, even in the image conversion processing disclosed in the above-mentioned patent document, a technique based on such a model is shown. In the following description of the present invention, description based on such a model is also given. To do.

  In the apparatus according to the present invention, it is necessary to cut out a part of the distorted circular image S and convert it into a planar regular image. For example, assume that a user who looks at the distorted circular image S shown in FIG. 2 wants to observe the image of a woman drawn on the lower left of the image with a correct image without distortion. In such a case, the user needs to specify which part of the distorted circular image S should be cut out and converted. For example, if the cutout area E as shown by hatching in FIG. 3 is designated as the area to be converted, the most intuitive designation method is to specify the position of the center point P (xp, yp). It would be the way to specify. In the present application, the point P designated by the user or the system in this way is referred to as a cut center point P. However, since the distorted circular image S is a distorted image, the cut-out center point P does not become an accurate geometric center with respect to the cut-out region E.

  Here, the following basic model is considered in order to convert an image in the cut-out area E centered on the cut-out center point P (xp, yp) into a planar regular image. FIG. 4 is a perspective view showing the relationship between the two-dimensional XY orthogonal coordinate system including the distorted circular image S and the two-dimensional UV orthogonal coordinate system including the planar regular image T in this basic model. As illustrated, since the distorted circular image S is defined on the XY plane of the three-dimensional XYZ orthogonal coordinate system, the distorted circular image S itself is an image defined on the two-dimensional XY orthogonal coordinate system. Therefore, consider an intersection G between a virtual spherical surface H and a straight line passing through the cut-out center point P (xp, yp) defined on the distorted circular image S and parallel to the Z axis. This intersection point G is, so to speak, a point immediately above the cut-out center point P (xp, yp), and its position coordinates are (xp, yp, zp).

  Next, a tangent plane in contact with the phantom spherical surface H is defined at the intersection point G (xp, yp, zp), and a two-dimensional UV orthogonal coordinate system is defined on the tangent plane. Then, the planar regular image T is obtained as an image on the two-dimensional UV orthogonal coordinate system. In the example shown in FIG. 4, the two-dimensional UV orthogonal coordinate system is defined so that the intersection point G (xp, yp, zp) is the origin. Eventually, the origin G of the UV coordinate system in this model is set anywhere on the virtual spherical surface H, and the UV plane constituting the UV coordinate system coincides with the tangential plane with respect to the virtual spherical surface H at the position of the origin G.

  The position of the point G (xp, yp, zp) serving as the origin of the UV coordinate system can be specified by the azimuth angle α and the zenith angle β, as shown. Here, the azimuth angle α (0 ≦ α <360 °) is an angle formed by a straight line connecting the cut-out center point P (xp, yp) and the origin O of the XY coordinate system and the Y axis, and the zenith angle β (0 ≦ β ≦ 90 °) is an angle (acute angle) formed by a straight line connecting the point G (xp, yp, zp) serving as the origin of the UV coordinate system and the origin O of the XY coordinate system and the Z axis.

  As described above, the UV plane can be specified by designating the azimuth angle α and the zenith angle β. However, in order to determine the UV coordinate system, it is necessary to designate another angle φ. This angle φ is a parameter indicating the direction of the UV coordinate system with the straight line OG as the rotation axis, and is defined as the angle formed by the U axis and the J axis in the example of FIG. Here, the J-axis is an axis that passes through the point G (xp, yp, zp), is parallel to the XY plane and is orthogonal to the straight line OG, and is hereinafter referred to as a rotation reference axis. In short, the angle φ is defined as an angle formed by the vector U and the vector J when the vector U facing the U-axis direction and the vector J facing the rotation reference axis J are defined in the UV coordinate system. This angle is usually called “planar inclination angle”.

  FIG. 5 is a plan view showing the relationship between the planar regular image T defined on the UV coordinate system and the planar tilt angle φ. In the case of the example shown here, the planar regular image T is defined as a rectangle on the UV plane centered on the origin G (xp, yp, zp) of the UV coordinate system, and its long side is parallel to the U axis. The short side is parallel to the V axis. Since the plane inclination angle φ is an angle formed by the U axis and the J axis as described above, in the example shown in FIG. 5, this is a parameter indicating the rotation factor of the planar regular image T on the UV plane. .

  As a result, the position and orientation of the UV coordinate system for forming the planar regular image T shown in FIG. 4 are uniquely set by setting parameters composed of three angles, the azimuth angle α, the zenith angle β, and the plane tilt angle φ. To be determined. These three angles are generally called Euler angles.

  Now, the image conversion processing to be performed here is coordinate conversion from the XY coordinate system to the UV coordinate system. Let's take a closer look at the geometric positional relationship between the XY coordinate system and the UV coordinate system. As shown in the perspective view of FIG. 6, when the distorted circular image S on the XY plane is inclined by the zenith angle β with respect to the direction indicated by the azimuth angle α, an inclined surface S1 is obtained. Here, as shown in the drawing, when the line-of-sight vector n is defined in the direction from the origin O of the XY coordinate system to the origin G of the UV coordinate system, and the inclined plane S1 is translated in the direction of the line-of-sight vector n by the distance R, A tangential plane S2 is obtained. The moving distance R is the radius of the distorted circular image S and the radius of the phantom spherical surface H.

  The tangent plane S2 is a plane in contact with the virtual spherical surface H at the point G, and the line-of-sight vector n is a vector indicating the normal direction of the virtual spherical surface H at the point G. The UV coordinate system is a coordinate system defined on the tangent plane S2, and has the point G as the origin, the U axis and the J axis (through the point G, parallel to the XY plane and orthogonal to the line-of-sight vector n. And is an axis parallel to the line of intersection between the inclined plane S1 and the XY plane in the drawing.) Is a two-dimensional orthogonal coordinate system defined so that the plane inclination angle φ is the plane inclination angle φ. FIG. 7 is a diagram of each component shown in the perspective view of FIG. 6 viewed from the horizontal direction. As described above, the point G (xp, yp, zp) is an intersection of the phantom spherical surface H and a straight line passing through the cut center point P (xp, yp) defined on the distorted circular image S and parallel to the Z axis. The position is determined by the azimuth angle α and the zenith angle β. On the other hand, FIG. 8 is a view of each component shown in the perspective view of FIG. 6 as viewed from above. An intersection point G (xp, yp, zp) shown in the drawing is a point on the phantom spherical surface H, and is located above the XY plane. Then, a UV coordinate system is defined on the tangent plane with respect to the virtual spherical surface H at the intersection point G (xp, yp, zp). At this time, the direction of the U axis is determined so that the angle formed by the U axis and the J axis is φ.

<<< §2. Basic Principles of Image Conversion Processing Considering Magnification >>>
In §1, the basic model that defines the UV coordinate system is described so that the origin G (xp, yp, zp) is one point on the phantom spherical surface H. In this case, the distance between the origin O of the XY coordinate system and the origin G of the UV coordinate system coincides with the radius R. On the other hand, a practical model in which a scaling factor is introduced into a planar regular image obtained by conversion is usually used. That is, a predetermined magnification m is set, a UV coordinate system is arranged at a position where the distance between the two points OG is m times the radius R, and a size corresponding to the magnification m is set on the UV coordinate system. A practical model that defines a planar regular image T having the same is used. Here, the basic principle of image conversion processing in this practical model will be described.

  FIG. 9 is a perspective view showing a correspondence relationship between the point Si (xi, yi) on the two-dimensional XY orthogonal coordinate system and the point Ti (ui, vi) on the two-dimensional UV orthogonal coordinate system for this practical model. . The difference from the basic model shown in FIG. 4 is the position of the origin G (xg, yg, zg) of the two-dimensional UV orthogonal coordinate system. That is, in the case of the practical model shown in FIG. 9, the distance between the two points OG is set to m · R. FIG. 9 shows an example in which m = 2 is set. The basic model shown in FIG. 4 is a special model in which m = 1 is set in the practical model, and the origin G (xg, yg, zg) is made to coincide with the corresponding point Q (xp, yp, zp) on the spherical surface. It corresponds to an example.

  Here, the corresponding point Q (xp, yp, zp) on the spherical surface is a point on the virtual spherical surface H corresponding to the cut-out center point P (xp, yp), and in the case of the orthographic fisheye lens, the cut-out center This is a point defined as the intersection of a straight line passing through the point P (xp, yp) and parallel to the Z axis and the phantom spherical surface H. Since the line-of-sight vector n is defined as a vector extending from the origin O to the corresponding point Q (xp, yp, zp) on the spherical surface, determining the cut-out center point P (xp, yp) determines the line-of-sight vector n Is equivalent to

  Of course, also in the practical model shown in FIG. 9, the origin G (xg, yg, zg) of the two-dimensional UV orthogonal coordinate system is a point on the line-of-sight vector n. It is defined on an orthogonal plane. The direction of the U axis is determined based on the plane inclination angle φ. Specifically, as shown in the figure, the orientation of the UV coordinate system is determined so that the angle formed by the rotation reference axis J passing through the origin G (xg, yg, zg) and the U axis coincides with the plane inclination angle φ. Will be.

  The purpose of the image conversion processing performed here is to cut out and deform the distorted image in the cut-out area centered on the cut-out center point P (xp, yp) on the distorted circular image S defined on the XY coordinate system. It is to obtain a planar regular image T on the UV coordinate system. Specifically, the pixel value of the pixel located at one point Ti (ui, vi) on the planar regular image T obtained on the UV coordinate system is converted into the corresponding one point Si (xi, xi, on the XY coordinate system. It is determined based on the pixel value of a pixel located in the vicinity of yi). For that purpose, as described in §3, a correspondence relation expression indicating a correspondence relation between coordinates (ui, vi) and coordinates (xi, yi) is required.

  In performing such image conversion processing, the line-of-sight vector n functions as a parameter indicating the cutout position of the planar regular image. When the line-of-sight vector n is set in the direction shown in the drawing, a planar regular image is cut out from the cut-out region centered on the cut-out center point P (xp, yp). If the direction of the line-of-sight vector n is changed, the position of the cut-out center point P is also changed, and the cut-out position of the planar regular image is also changed. On the other hand, the plane inclination angle φ functions as a parameter indicating the cutting direction of the planar regular image, and the magnification m (a factor that determines the distance between the two points OG) functions as a parameter indicating the cutting size of the planar regular image.

  10 and 11 are side views showing the relationship between the magnification m and the cutout region E. FIG. FIG. 10 corresponds to a model (model shown in FIG. 4) in which the magnification m is set to 1 (the model shown in FIG. 4), and the corresponding point Q on the spherical surface located directly above the cut-out center point P directly becomes the origin G1 of the UV coordinate system. The distance between the two points O and G1 is equal to the radius R. On the other hand, FIG. 11 corresponds to a model (model shown in FIG. 9) in which the magnification m> 1 is set, and UV is positioned at a position away from the corresponding point Q on the spherical surface located immediately above the cut-out center point P. The origin G2 of the coordinate system is set, and the distance between the two points O and G2 is equal to m times the radius R. In each case, the position of the cut-out center point P, the position of the corresponding point Q on the spherical surface, and the position of the line-of-sight vector n are the same, but the origin positions G1 and G2 are different, so the position of the UV coordinate system (formed above it) The positions of the planar regular images T1 and T2) are also different.

  If FIG. 10 is compared with FIG. 11, it will be understood that the cutout region E varies depending on the magnification m. That is, when the magnification m = 1, E1 becomes the cutout area as shown in FIG. 10, whereas when the magnification m> 1, the E2 becomes the cutout area as shown in FIG. As m increases, the area of the cutout region E decreases. Here, when the obtained planar regular images T1, T2 are displayed on the screen of the monitor device, the sizes of the planar regular images T1, T2 (determined according to the size of the monitor screen) are constant. Therefore, the planar regular images T1 and T2 are both images that are cut out with the cutout center point P as the center, but the flat regular image T2 set with the magnification m> 1 enlarges the cutout region E2 that is narrower. The displayed image is displayed. In other words, the planar regular image T1 is an image obtained by photographing a subject within the angle of view F1, whereas the planar regular image T2 is an image obtained by photographing a subject within a narrower angle of view F2.

  On the other hand, the plane inclination angle φ is a parameter indicating the cut-out direction of the planar regular image. When the plane inclination angle φ is changed, the positional relationship between the subject appearing in the planar regular image T and the image frame (related to the rotation direction). (Positional relationship) will change. In FIG. 5, when the angle φ is increased, the U axis (and the V axis) rotates counterclockwise, and the image frame of the planar regular image T also rotates counterclockwise. Like.

  Eventually, the user can obtain a desired planar regular image T on the UV coordinate system by setting three parameters: a line-of-sight vector n (cutting center point P), a magnification m, and a plane inclination angle φ. When the obtained planar regular image T is not satisfactory, the planar regular image T can be modified by appropriately modifying these three parameters. That is, if the orientation of the obtained image is not satisfied, the plane inclination angle φ may be corrected, and if the angle of view of the obtained image is not satisfied, the magnification m may be corrected. If the image cutout position is not satisfied, the line-of-sight vector n (cutout center point P) may be corrected.

  FIG. 12 is a plan view showing an example of a planar regular image T obtained on the UV coordinate system by appropriately setting the above three parameters for the distorted circular image S illustrated in FIG. FIG. In this example, the position of the female nose in the distorted circular image S shown in FIG. 2 is set as the cut-out center point P, and the image around the female face is cut out. However, since the cutout direction is set so that the female height direction is the U axis, as shown in the figure, when the U axis is displayed on the monitor screen in the horizontal direction, the woman is displayed sideways. .

  In such a case, the user may correct the plane inclination angle φ. For example, if the angle φ is decreased by about 90 °, the U-axis (and V-axis) is rotated clockwise, and the image frame of the planar regular image T is also rotated clockwise, whereby a female erect image is obtained. Will be. However, since the illustrated monitor screen has a rectangular frame in which the vertical dimension b (the number of pixels in the vertical direction) is smaller than the horizontal dimension a (the number of pixels in the horizontal direction), the monitor screen displays even the female breast. In order to do so, it is necessary to modify the magnification m slightly.

  FIG. 13 is a planar regular image T obtained by such correction. An erect image up to the breast of the woman is obtained as requested by the user. 12 and 13, the line-of-sight vector n (the cut-out center point P) has the same setting, and both are images centered on the cut-out center point P designated as the position of the female nose. . However, since the positions and orientations of the defined UV coordinate systems are different, the obtained planar regular images T are different.

  The “planar regular image” referred to in the present application does not necessarily mean “a complete image without distortion”, but uses “a normal lens compared to a distorted circular image S obtained by photographing using a fisheye lens”. This means a “planar image close to an image obtained by photographing”. Therefore, compared with the female image on the distorted circular image S shown in FIG. 2, the planar regular image T shown in FIGS. 12 and 13 looks like an image without distortion, but the distortion is not completely removed.

<<< §3. Correspondence formula of orthographic projection method >>>
FIG. 14 is a plan view showing a planar regular image T defined on the two-dimensional UV orthogonal coordinate system. Here, an arbitrary point Ti on the planar regular image T is represented as Ti (ui, vi) using the coordinate values ui, vi of the UV coordinate system. As shown in FIG. 9, the origin G of this UV coordinate system is indicated by a point G (xg, yg, zg) using coordinates on the three-dimensional XYZ orthogonal coordinate system. If it shows using a system, it will be point T (0, 0) as shown in FIG.

  On the other hand, FIG. 15 is a plan view showing a distorted circular image S defined on a two-dimensional XY orthogonal coordinate system. Here, an arbitrary point Si on the distorted circular image S is represented as Si (xi, yi) using the coordinate values xi, yi of the XY coordinate system. The image on the origin G (point T (0, 0)) shown in FIG. 14 corresponds to the image on the cut-out center point P (xp, yp) shown in FIG. 15, and an arbitrary point Ti (shown in FIG. 14). The image on ui, vi) corresponds to the image on the point Si (xi, yi) shown in FIG.

  As described above, in order to obtain the planar regular image T on the two-dimensional UV orthogonal coordinate system, the pixel value of the pixel located at the point Ti (ui, vi) shown in FIG. 14 is changed to the point Si (xi) shown in FIG. , Yi) must be determined based on the pixel value of a pixel (a pixel in the distorted circular image S on the two-dimensional XY orthogonal coordinate system). For this purpose, a correspondence expression that indicates a one-to-one correspondence between coordinates (ui, vi) on the two-dimensional UV orthogonal coordinate system and coordinates (xi, yi) on the two-dimensional XY orthogonal coordinate system is necessary. become. If such a correspondence relational expression is used, an arbitrary point on the planar regular image T shown in FIG. 14 is associated with any point in the cutout region E of the distorted circular image S shown in FIG. Thus, the distorted image in the cutout region E can be converted into a planar regular image T.

  Such a correspondence relation expression can be uniquely defined by a geometric method if the position and orientation of the UV coordinate system arranged in the space of the three-dimensional XYZ coordinate system are determined. For example, in the example shown in FIG. 9, if attention is paid to the positional relationship between the point Ti (ui, vi) on the planar regular image T and the point Si (xi, yi) on the distorted circular image S, the point Si (xi , Yi) where the point on the virtual sphere H just above the sphere is the corresponding point Qi (xi, yi, zi) on the sphere, the straight line ni connecting the origin O and the corresponding point Qi on the sphere and the UV coordinate system The point Ti (ui, vi) is located at the intersection with the coordinate plane.

  FIG. 16 is a side view for more clearly explaining such a positional relationship between two points. In FIG. 16, the line-of-sight vector n extends from the origin O in the direction having the zenith angle β, and the intersection point Q between the line-of-sight vector n and the phantom spherical surface H is located immediately above the cut-out center point P (xp, yp). is doing. A point G is defined at a position where the distance from the origin O on the line-of-sight vector n is mR. This point G is the origin of the two-dimensional UV orthogonal coordinate system, the coordinate plane of this UV coordinate system is orthogonal to the line-of-sight vector n, and the planar regular image T is defined on the coordinate plane of this UV coordinate system. Is done.

  The corresponding point S1 (x1, y1) on the distorted circular image S for an arbitrary point T1 (u1, v1) on the planar regular image T is defined as follows. In other words, the corresponding point Q1 on the spherical surface is obtained at the position of the intersection between the straight line connecting the point T1 (u1, v1) and the origin O and the virtual spherical surface H, and the distorted circular image S at the position directly below the corresponding point Q1 on the spherical surface. The upper point may be the corresponding point S1 (x1, y1). Similarly, for an arbitrary point T2 (u2, v2) on the planar regular image T, the corresponding point Q2 on the spherical surface is located at the intersection of the virtual sphere H with the straight line connecting the point T2 (u2, v2) and the origin O. And the point on the distorted circular image S at the position directly below the corresponding point Q2 on the spherical surface may be set as the corresponding point S2 (x2, y2).

  Such a correspondence between two points can be described by a geometrical correspondence expression. Specifically, it is known that the expressions (1) to (9) shown in FIG. 17 can be used. For example, in the above-mentioned Patent Documents 1 and 2, image conversion using such a correspondence relation expression is performed. A method is disclosed. In addition, because there are some differences in the sign of each term depending on how to define the coordinate system and how to define the angle, etc., there are some discrepancies in the published formulas depending on the literature, but essentially the same It is a formula which shows geometric operation.

  The correspondence relational expression shown in FIG. 17 is an expression when the coordinate system and angle are defined as shown in FIG. 9, and parameters α and β indicating the position of the UV coordinate system and parameters indicating the direction of the UV coordinate system. The formula includes φ. These three parameters are the Euler angles described above, that is, the azimuth angle α, the zenith angle β, and the plane tilt angle φ.

In particular,
x = R (uA + vB + wC) /
√ (u ² + v ² + w ² ) Formula (1)
Is an equation for obtaining the x coordinate value of the corresponding point S (x, y) on the XY coordinate system using the coordinate value u, v of one point T (u, v) on the UV coordinate system. , A, B, C are respectively
A = cosφ cosα−sinφ sinα cosβ Formula (3)
B = −sinφ cosα−cosφ sinα cosβ Formula (4)
C = sinβ sinα Formula (5)
And is determined by calculation using a trigonometric function of Euler angles α, β, and φ.

Similarly,
y = R (uD + vE + wF) /
√ (u ² + v ² + w ² ) Formula (2)
Is an expression for obtaining the y coordinate value of the corresponding point S (x, y) on the XY coordinate system using the coordinate values u, v of one point T (u, v) on the UV coordinate system. , D, E, F are respectively
D = cosφ sinα + sinφ cosα cosβ Equation (6)
E = −sinφ sinα + cosφ cosα cosβ Equation (7)
F = −sin β cos α Formula (8)
And is determined by calculation using a trigonometric function of Euler angles α, β, and φ.

In the formulas (1) and (2), w is
w = mR Formula (9)
The value given by. Here, as described above, R is the radius of the distorted circular image S, and m is the magnification. The magnification m indicates the relationship between the scaling of the coordinate values u and v and the scaling of the coordinate values x and y. The larger the magnification m, the larger the enlarged image in the planar regular image T. Although required, the cutout region E of the distorted circular image S becomes smaller.

  After all, in the equation shown in FIG. 17, since the value of R is known as the radius of the distorted circular image S, the azimuth angle α and the zenith angle β (in other words, the position of the cutting center point P), and the plane inclination If the angle φ and the magnification m are determined and the position and orientation of the UV coordinate system are determined, the unknowns for calculating the coordinate values x and y in the correspondence relationship shown in FIG. 17 are only u and v. . Therefore, by using this corresponding relational expression, it is possible to determine the corresponding point S (x, y) on the distorted circular image S corresponding to an arbitrary point T (u, v) in the planar regular image T.

<<< §4. Basic concept of the present invention >>
Next, the basic concept of the present invention will be described. FIG. 18 is a side view showing an example in which a general video camera with a fisheye lens is installed for outdoor monitoring. In this example, a video camera 30 with a fisheye lens is installed on the right wall surface of the building 20 located on the left side of the road surface 10. The imaging surface (fisheye lens imaging surface) of the video camera 30, that is, the XY plane shown in FIG. 1, is a surface along the side wall surface of the building 20, and the imaging center point 31 is at the origin O of the XY coordinate system. Correspond. In short, the video camera 30 is attached so that the XY plane (surface on which the distorted circular image S is formed) shown in FIG. 1 is a surface along the vertical plane in the real world.

  On the other hand, a tree 40 and a guard rail 50 are arranged on the right side of the road surface 10, and the video camera 30 captures a scene including the tree 40 and the guard rail 50 from the front. FIG. 19 is a plan view showing an example of a distorted circular image S obtained by photographing using such a video camera 30 with a fisheye lens. A state in which the trees 40 and the guard rails 50 are arranged in the left-right direction along the road surface 10 is shown, but the image is more distorted as it is closer to the circumference.

  Here, it is assumed that the orientation of the video camera 30 is set so that the Y axis of the XY coordinate system is parallel to the horizontal plane and the X axis is the vertical axis. Accordingly, as shown in FIG. 19, with respect to the immediate vicinity of the origin O of the distorted circular image S obtained on the XY plane, the Y-axis direction is the horizontal line direction of the real world scene, and the X-axis direction is the vertical line of the real world scene. It will be a direction. Such a directional characteristic is lost as it approaches the circumference of the image.

  In general fish-eye monitoring systems as disclosed in the above-mentioned Patent Documents 1 and 2, according to the user's request, an arbitrary part on the distorted circular image S as shown in FIG. A function of converting this into a planar regular image and displaying it on a monitor screen is provided. For example, as shown in FIG. 20, if the user gives an instruction to cut out a portion of the cutout area E0 with the point P0 at the center of the distorted circular image S as the cutout center point, the monitor screen displays FIG. A plane regular image as shown as a frame F0 in a) is obtained, and the details of the tree 40 can be confirmed. On the other hand, in FIG. 20, if the user gives an instruction to cut out the portion of the cut-out area En with the point Pn at the lower right of the distorted circular image S as the cut-out center point, the monitor screen displays FIG. A planar regular image as shown as a frame Fn in (b) is obtained, and the details of the guardrail 50 can be confirmed.

  In FIG. 20, each of the cutout areas E0 and En is shown as a rectangular area. However, in actuality, as shown in FIGS. 21 (a) and 21 (b), the planar regularity displayed on the monitor screen is displayed. When the image is a rectangular image, the corresponding cutout areas E0 and En have a distorted shape instead of a rectangle. However, in order to show the cutout direction (planar inclination angle φ) on the drawing, it is more convenient to draw a rectangular cutout region. For example, in the case of the example shown in FIG. 20, the cut-out areas E0 and En are both in the horizontal direction of the figure with the U axis (the axis in the long side direction of the rectangular planar regular image as in the example shown in FIG. 5). This is an area set in the orientation. Therefore, also in the following description, the cut-out area on the distorted circular image S is indicated as a rectangular area for convenience, and the direction of the U axis is indicated by the long side direction.

  As described in the model shown in FIG. 9, in order to cut out a part of the distorted circular image S and obtain the planar regular image T, the position P (xp, yp) of the cut-out center point for indicating the cut-out location It is necessary to provide a cutting condition consisting of three parameters: a plane inclination angle φ for indicating the cutting direction and a magnification m for indicating the cutting magnification. Specifically, in order to obtain the planar regular image shown in FIG. 21A, the position of the cut center point P0, the direction of the cut region E0 (plane tilt angle φ), and the cut region E0 shown in FIG. It is necessary to specify the extraction condition of the size (magnification m).

  Various types of man-machine interfaces for allowing the user to specify such specific cutting conditions are known. For example, the position of the cut-out center point P0 can be captured by a user operation by displaying a distorted circular image S as shown in FIG. 20 on the monitor screen and clicking an arbitrary position with a pointing device such as a mouse. is there. Further, the orientation and size of the cut-out area E0 can also be captured based on a user operation on the distorted circular image S displayed on the monitor screen. Of course, the position coordinate value of the point P0, the value of the angle φ, and the value of the magnification m can be input by the user.

  Therefore, for example, when a user engaged in security monitors a captured image in real time, a desired circular eye image S as shown in FIG. If the cutting condition is input, a planar regular image obtained by cutting an image at an arbitrary position in an arbitrary direction at an arbitrary magnification can be obtained. As shown in FIGS. 21 (a) and (b), the planar regular image obtained in this way is an image obtained by photographing a predetermined direction with a camera using a normal lens from the position of the imaging center point 31 shown in FIG. Since the images are equivalent to each other, the user can perform a monitoring operation equivalent to the case where the visual field of the monitoring camera equipped with a normal lens is moved by a mechanical mechanism.

  However, in the case of a surveillance camera equipped with a normal lens, since the visual field is moved by a mechanical mechanism, the process of switching the visual field always involves a panning operation or a zooming operation, whereas a surveillance camera equipped with a fisheye lens. In some cases, there is no such mechanical switching process. For example, when switching from the image of the frame F0 shown in FIG. 21 (a) to the image of the frame Fn shown in FIG. 21 (b), if the surveillance camera is equipped with a normal lens, the frames F1, F2, and F3 are in the middle. ,... Are inserted, and a stepwise switching process of frames F0, F1, F2, F3,..., Fn is displayed on the monitor screen. However, in the case of a monitoring system using a camera equipped with a fisheye lens, an electronic switching process for changing a cutting condition is performed instead of a mechanical switching process, so that an instantaneous switching from the frame F0 to the frame Fn is performed. .

  As described above, when the switching from the frame F0 to the frame Fn is instantaneously performed, the user switches between the scenery including the tree 40 displayed in the frame F0 and the scenery including the guardrail 50 displayed in the frame Fn. It becomes difficult to intuitively grasp the relative positional relationship. This problem is particularly noticeable in the case of a fish-eye monitoring system incorporating a moving body tracking technique.

  Generally, when the user himself / herself specifies a desired clipping condition and switches the planar regular image obtained on the monitor screen, the user consciously recognizes the spatial positional relationship between the images before and after the switching. Will be. For example, if the user gives a switching instruction from the cutout area E0 to En shown in FIG. 20, the planar regular image displayed on the monitor screen is changed from FIG. 21 (a) to FIG. 21 (b). The user can consciously understand that FIG. 21 (a) is a frontal monitoring image and FIG. 21 (b) is a lower right road surface monitoring image. (Of course, it is not enough to intuitively understand the movement of the field of view).

  However, when switching from the cut-out area E0 to En is performed by the monitoring system without regard to the user's intention, it is extremely difficult to grasp the spatial positional relationship between the images before and after the switching. For example, in the case of a fish-eye monitoring system having a moving body tracking function disclosed in the above-mentioned Patent Document 3, when a moving body is detected, cutting conditions such as a cutout portion on the distorted circular image S and a display magnification are set. Then, a process of automatically switching to a cutting condition suitable for display of the moving object is performed. For example, in the example shown in FIG. 20, normally, an image cut out from the region E0 is continuously displayed on the monitor, and when a moving object is detected in the region En, the image display cut out from the region En is automatically displayed. When the switching process is performed, the monitor screen is instantaneously switched from the frame F0 shown in FIG. 21 (a) to the frame Fn shown in FIG. 21 (b) regardless of the user's intention. Is discontinuous in time, and it is very difficult for a user who is watching the monitor screen to recognize which video has been switched to.

  The present invention proposes a new technique for solving such a problem, and the user can easily change the spatial positional relationship between the images before and after the switching even when the cutting condition of the distorted circular image is switched. The purpose is to be able to grasp.

  The fish-eye monitoring system according to the embodiment described below is a system having a moving object detection function and a moving object tracking function, a static monitoring period in which no moving object is detected, a transition period after the moving object is detected, and this transition A plane regular image can be displayed on the monitor screen in an optimum manner, divided into three periods, ie, a dynamic monitoring period in which a moving object is tracked after the period is completed.

  For example, during the static monitoring period in which no moving object is detected, as shown in FIG. 22, an image in the cutout area E0 centered on the cutout center point P0 is cut out and converted into a planar regular image. Let's consider the case where the setting to display on the monitor screen has been made. Here, as shown in the figure, when the moving body 60 is detected in the vicinity of the point Pn (here, it is assumed that the cat is detected as a moving body), the cutout region is gradually shifted from the region E0 to En. Next, as shown in FIG. 23, the transition period process is performed, and the dynamic monitoring period process for sequentially moving the cutout region from the region En to E (i) while tracking the moving body 60 is performed. .

  Here, if attention is paid to the position of the cut-out center point, as shown in FIG. 22, the position of the center point P0 of the distorted circular image S is continuously maintained during the static monitoring period in which no moving object is detected. When detection is performed, the point moves from point P0 to point Pn stepwise (transition period), and further, from point Pn to point P (i) in synchronization with the movement of moving body 60, as shown in FIG. And move in stages (dynamic monitoring period). Such moving body tracking processing is continued until it is determined that the moving body 60 has disappeared.

  On the other hand, the size of the cutout region (cutout magnification m) also changes as necessary. That is, the cutout area E0 in the static monitoring period maintains a constant size, but in the transition period, the cutout area size changes from the size of the area E0 to the size of the area En as shown in FIG. In the subsequent dynamic monitoring period, as shown in FIG. 23, the size of the cutout region changes stepwise from the size of the region En to the size of the region E (i). Here, the sizes of the region En and the region E (i) are automatically set so as to be a size suitable for display of the moving body 60.

  After all, if the fish-eye monitoring system according to this embodiment is used, a user engaged in a monitoring task can enter the monitoring range without performing any instruction operation as long as he / she is watching the monitor screen. Can be recognized. That is, during the static monitoring period in which no moving object within the monitoring range is detected, a static image including the tree 40 as shown in FIG. 24 (a) continues to be displayed on the monitor screen. However, when the moving object 60 is detected, the planar regular image displayed on the monitor screen is automatically switched to an image including the moving object 60 as shown in FIG. 24B, and in the subsequent dynamic monitoring period. The moving body tracking is performed so that an image including the moving body 60 is always displayed.

  Moreover, when switching from the image of frame F0 shown in FIG. 24 (a) to the image of frame Fn shown in FIG. 24 (b), a predetermined transition period is provided, as if the field of view is moved by a mechanical mechanism. Panning and zooming operations are performed. That is, during this transition period, intermediate frames of frames F1, F2, F3,... Are inserted, and on the monitor screen, stages of frames F0, F1, F2, F3,. A typical switching process will be displayed.

  An intermediate frame regular image inserted in the transition period can be obtained, for example, by gradually changing the cutting conditions as shown in FIGS. The example shown here is an example in which the transition period is configured by the first half of the panning period and the second half of the zooming period. FIG. 25 shows the transition of the position of the cutout region in the panning period, and FIG. The transition of the size of the cut-out area is shown.

  As shown in FIG. 25, in the first half of the panning period, a process of gradually moving the position of the cutting center point from the point P0 to the point Pj is performed. Here, the point Pj is a point at the same position as the point Pn shown in FIG. 22, and the point Pi is a point on the way from the point P0 to the point Pj. As shown in the drawing, the cut-out center point gradually moves along the movement path A on the XY plane. In the example shown here, since the first cut-out center point P0 is the position of the origin O of the XY coordinate system (the center point of the distorted circular image S), the movement path A follows the radius of the distorted circular image S. It becomes a route. The cut-out center point changes to P0, P1,..., Pi,..., Pj in synchronization with the frame period of the planar regular image displayed on the monitor screen.

  In response to this, each cutout region also changes to E0,..., Ei,. Since each cut-out area is an area centered on each cut-out center point, the cut-out area moves along the movement path A along with the movement of the cut-out center point. In the example shown here, since the magnification m is kept constant during the panning period, the sizes of the regions E0 to Ej indicated by rectangles are equal to each other for convenience. Note that, as described above, the cutout region illustrated here indicates the cutout position P, the orientation φ, and the magnification m for convenience in terms of a rectangular position, orientation, and size. It is not an exact indication.

  On the other hand, in the latter zooming period, as shown in FIG. 26, the size of the cut-out area is gradually reduced while the position of the cut-out center point is fixed (points Pj to Pn are points at the same position). The process will continue. As a result, the cut-out area changes to areas Ej,..., Ek,. Here, the region En is the region En shown in FIG. 22, and is set to a size suitable for displaying the detected moving object 60.

  Thus, as shown in FIG. 25, the cutout region changes to E0,..., Ei,..., Ej (first panning period), and further, as shown in FIG. , Ek,..., En (second zooming period). If the cutting condition is changed stepwise in this way during the transition period, panning in the lower right direction is performed on the monitor screen with the zoom magnification kept constant following the frame F0 shown in FIG. , Fi,..., Fj corresponding to the captured image, and then the frame F (j + 1),... Corresponding to the image zoomed to the telephoto side with the viewing direction fixed. , Fk,... Are displayed, and finally a frame Fn shown in FIG.

  By providing such a transition period, the user can visually recognize the transition process from the frame F0 to the Fn, so that the moving object detection function of the monitoring system is independent of the user's will. Even if the image is switched, the user can easily grasp the spatial positional relationship between the images before and after the switching. If the moving speed of the moving object is high, there may be a case where the moving object 60 runs away from the cut-out area En at the end of the transition period, but there is no problem because tracking is performed in the subsequent dynamic monitoring period.

  In addition, instead of setting the first half of the panning period and the second half of the zooming period separately, the panning operation (moving operation of the cutting center point) and the zooming operation (size changing operation of the cutting area) are performed in parallel. It doesn't matter. If only the panning operation is performed in the first half, there is a possibility that a part of the panned region Ej may protrude from the region of the distorted circular image S (in this case, a planar regular image with a part missing). If the panning operation and the zooming operation are performed in parallel, such an adverse effect can be prevented.

  In short, the entire transition period may be set as the panning period and the zooming period, and the panning operation and the zooming operation may be performed in parallel, or a part of the transition period may be set as the panning period. The section period may be set as a zooming period, and the panning operation and the zooming operation may be performed separately. Alternatively, it is possible to set so that the panning period and the zooming period partially overlap.

  Thus, when the transition period is completed and the image in the cutout area En shown in FIG. 26 is displayed on the monitor screen, the dynamic monitoring period starts, and the cutout is performed so as to follow the moving object. The location and size of the area is changed. FIG. 27 is a diagram showing the state of such moving object tracking. The cut center point P (0) and the cut region E (0) shown in FIG. 27 are the same as the cut center point Pn and the cut region En shown in FIG. The cut-out center point moves to P (i), P (j), and P (k) because the moving body 60 (cat) has moved to this position. By tracking the moving body 60, the cutout area also moves as E (i), E (j), and E (k). Here, for the sake of convenience, the transition in the transition period is indicated by subscripts “0, 1,..., I,..., N”, and the transition in the dynamic monitoring period is indicated by the subscript “(0) in parentheses. , (1),..., (I),.

  Eventually, the user is presented with a frame image that changes in stages from the frame F0 shown in FIG. 24A to the frame Fn shown in FIG. 24B, and then the frame obtained by tracking the moving object 60 in the frame Fn. The presentation of the image is continuously performed until the moving object disappears. When the moving body 60 disappears (when the moving object 60 moves outside the distorted circular image S), the monitoring system determines whether the moving object disappears and presents the frame F0 shown in FIG. 24 (a) again.

  The basic concept of the present invention has been described above. In the following §5 and later, specific embodiments of the fish-eye monitoring system designed based on such a basic concept will be described in detail.

<<< §5. Basic configuration of fish-eye monitoring system according to the present invention >>>
Here, the basic configuration of a fish-eye monitoring system according to an embodiment of the present invention will be described with reference to the block diagram of FIG. In this system, components shown in the upper half of the figure, that is, a video camera 110 with a fisheye lens, a monitor device 120, an image input unit 115, an image output unit 125, a distorted circular image memory 130, and a planar regular image memory 140, the image cut-out conversion unit 150 is a general component included in a conventional fish-eye monitoring system, and cuts out a part of a distorted circular image obtained by photographing using a fish-eye lens to form a planar regular image. It has the basic function of converting and displaying this on the monitor screen.

  The video camera 110 with a fisheye lens is a video camera that captures an image of the outside world as a distorted circular image with the attached fisheye lens. For example, in the case of the example shown in FIG. 18, the video camera attached to the wall surface of the building 20 This corresponds to the video camera 110 shown in FIG. Frame-based images taken by the video camera 110 are stored in the distorted circular image memory 130 via the image input unit 115. The captured image of the camera with a fisheye lens is a distorted circular image as in the example shown in FIG. The memory 130 stores the distorted circular image S as image data. Specifically, the distorted circular image S is constituted by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on the two-dimensional XY orthogonal coordinate system, and this two-dimensional XY orthogonal coordinate system. Is stored in the memory 130 as an image having a radius R around the origin O.

  From the video camera 110, such a distorted circular image S is output as image data in units of frames at a predetermined cycle (for example, 30 frames / second). The image input unit 115 performs a process of sequentially storing the distorted circular image S sequentially given as time-series data for each frame in the memory 130 in this way. The memory 130 is a buffer memory having a storage capacity for storing at least one frame of such a distorted circular image S. Of course, if a memory having a larger capacity is prepared, it is possible to simultaneously hold the data of a distorted circular image for a plurality of consecutive frames. Although not shown in the figure, when leaving the monitoring video as a record, the image data stored in the memory 130 is sequentially transferred to a separately prepared hard disk device or the like.

  On the other hand, the plane regular image memory 140 is a buffer that stores a plane regular image T formed by an aggregate of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system. The planar regular image T stored in the memory is read by the image output unit 125 and output to the monitor device 120. Normally, the planar regular image T is a rectangular image suitable for display on the display screen of the monitor device 120. Of course, the contour of the planar regular image T may have any shape. It doesn't matter.

  In the case of the embodiment shown here, the image output unit 125 reads the planar regular image T stored in the memory 140 and outputs it to the monitor device 120, and the distorted circular image S stored in the memory 130. Is also read and output to the monitor device 120. Which image is output to the monitor device 120 can be switched by a user's instruction operation, and the planar regular image T can be displayed on the display screen of the monitor device 120, or the distorted circular image S can be displayed. Can be displayed, and if necessary, the screen can be divided and both images can be displayed simultaneously.

  As described above, the distorted circular image S stored in the memory 130 is, for example, a fish-eye photographed image having a field angle of 180 ° as shown in FIG. The planar regular image T is an image that has been subjected to conversion processing for cutting out a part of the distorted circular image S and correcting the distortion, as shown in FIGS. 21 (a) and 21 (b), for example. The image cutout conversion unit 150 is a component that performs such partial image cutout and distortion correction conversion processing.

  Actually, the processing of the image cut-out conversion unit 150 is performed by an operation of determining the pixel value of each pixel constituting the planar regular image T based on the pixel value of the corresponding pixel on the distorted circular image S. . That is, for the specific pixel of interest on the planar regular image T arranged at the coordinate (u, v) on the UV coordinate system, the corresponding coordinate (x, y) on the XY coordinate system is obtained, and this corresponding coordinate (x, An operation may be performed in which the pixel value of the pixel on the distorted circular image S located at y) is set to the pixel value of the pixel of interest.

  However, the distorted circular image S is an image formed by an aggregate of a large number of pixels arranged at positions indicated by coordinates (x, y) on the two-dimensional XY orthogonal coordinate system. In this example, the position of a large number of grid points arranged vertically and horizontally is constituted by digital data defining unique pixel values. For this reason, the position of the corresponding coordinate (x, y) is usually a position between a plurality of lattice points. For example, when the distorted circular image S is composed of digital data defining pixel values at a large number of grid point positions arranged vertically and horizontally at a pitch 1, the coordinate value of any grid point is an integer value. . Therefore, when the calculated values of the corresponding coordinates x and y are values including decimal numbers (which will be the case in many cases), the position of the corresponding coordinates (x, y) is between the lattice points. The corresponding pixel value cannot be determined as one.

  Therefore, in actuality, when determining the pixel value of the pixel of interest on the planar regular image T arranged at the position indicated by the coordinates (u, v) in the image cut-out conversion unit, the corresponding coordinates (x, It is necessary to read out the pixel values of a plurality of reference pixels on the distorted circular image S arranged in the vicinity of the position indicated by y) and perform an interpolation operation on the pixel values of the plurality of reference pixels. Various methods such as bilinear interpolation and bicubic / spline interpolation are known as methods for performing such interpolation calculations, and thus detailed description thereof is omitted here. Of course, it is also possible to adopt a method in which the pixel value of the pixel closest to the position indicated by the corresponding coordinates (x, y) is directly determined as the pixel value of the target pixel without performing such interpolation.

  Eventually, the image cutout conversion unit 150 calculates the corresponding coordinates (x, y) for the coordinates (u, v) of one target pixel constituting the planar regular image T, and the corresponding image in the distorted circular image memory 130 is calculated. A process of reading the pixel value of a pixel arranged in the vicinity of the corresponding coordinate (x, y) and determining the pixel value of the pixel of interest based on the read pixel value is executed for each pixel constituting the planar regular image T Then, by writing the pixel value of each pixel into the planar regular image memory 140, a process of converting a partial image cut out from the distorted circular image S into a planar regular image T is performed.

  In order to calculate the corresponding coordinates (x, y) for the coordinates (u, v), as described in §3, equations (1) to (9) shown in FIG. 17 may be used. In order to perform the calculation based on these equations, it is necessary to determine three parameters, that is, the azimuth angle α and the zenith angle β (the position of the cutting center point P), the plane inclination angle φ, and the magnification m, as already described. . These parameters are parameters that determine the cutting conditions such as the cutting position, orientation, and size. After all, the image cut-out conversion unit 150 performs magnification from the cut-out position indicated by the cut-out center point P of the distorted circular image S stored in the distorted circular image memory 130 in the cut-out direction indicated by the plane inclination angle φ. An image having a cut-out size indicated by m is cut out, converted into a planar regular image T, and stored in the planar regular image memory 140.

  In the block diagram of FIG. 28, the components shown in the lower half of the figure, that is, the moving object detection unit 160, the actual cutting condition determination unit 170, the standard cutting condition storage unit 180, the manual condition setting unit 185, the moving object tracking The unit 190 is a component for designating the three parameters (cutout conditions) of the cutout center point P, the plane inclination angle φ, and the magnification m to the image cutout conversion unit 150.

  As described above, the distorted circular image memory 130 sequentially stores captured images in real time (for example, 30 frames / second). The image cut-out conversion unit 150 cuts out an image in accordance with a predetermined cut-out condition for each individual frame, and performs conversion processing into a planar regular image. Accordingly, the planar regular image T is also displayed on the screen of the monitor device 120 in real time. Therefore, it is necessary to give the image cut-out conversion unit 150 the cut-out conditions such as the cut-out center point P, the plane inclination angle φ, and the magnification m in real time. Here, the extraction condition given to the image extraction conversion unit 150 in real time in this way is referred to as an actual extraction condition.

  The actual cut condition determination unit 170 plays a role of determining the real cut condition in real time in synchronization with the frame and giving it to the image cut conversion unit 150. That is, the actual cutting condition determination unit 170 is configured by the cutting center point P that is one point on the distorted circular image S, the predetermined plane inclination angle φ, and the predetermined magnification m. Is determined as a condition for cutting out the planar regular image T from a part of the distorted circular image S.

  On the other hand, the standard extraction condition storage unit 180, the moving object detection unit 160, and the moving object tracking unit 190 provide the reference condition for determining the actual extraction condition to the actual extraction condition determination unit 170, thereby It plays a role of assisting the output condition determination unit 170. The actual cut condition determination unit 170 determines the actual cut condition based on the given reference condition, and executes a process of giving this to the image cut conversion unit 150.

  As described in §4, the fish-eye monitoring system according to this embodiment is a system having a moving object detection function and a moving object tracking function, and has a static monitoring period during which no moving object is detected (for example, the image shown in FIG. Obtained period), a transition period after the moving object is detected (for example, a period in which the cut region is shifted from E0 to En in FIG. 22), and dynamic monitoring for tracking the moving object after the transition period is completed. The planar regular image can be displayed on the monitor screen in an optimum manner by dividing the period into three periods (for example, a period in which the moving body 60 is tracked and displayed as shown in FIG. 23). The standard cut-out condition storage unit 180 is a component for giving the cut-out conditions in the static monitoring period to the real cut-out condition determining unit 170, and the moving object detection unit 160 sets the target cut-out conditions in the transition period. The moving object tracking unit 190 is a component for giving the actual cutting condition determination unit 170 the cutting condition for the dynamic monitoring period.

  More specifically, the standard cutting condition storage unit 180 includes a standard cutting center point P0, a standard plane inclination angle φ0, and a standard magnification m0 as cutting conditions (standard cutting conditions) for the static monitoring period. Is stored. For example, when the cutout area E0 shown in FIG. 20 is set as the cutout area in the static monitoring period, the standard cutout condition storage unit 180 stores the standard cutout center point P0 indicating the center point of the illustrated image. The coordinate value (in this example, the coordinate value (0, 0) of the origin O of the XY coordinate system) and the standard plane inclination angle φ0 indicating the direction of the cutout area E0 (the direction in which the horizontal direction in the figure is the U-axis direction) And the standard magnification m0 corresponding to the size of the cutout area E0 are stored in the standard cutout condition storage unit 180.

  If such a standard cutting condition is set in the standard cutting condition storage unit 180, the planar regular image displayed on the monitor device 120 during the static monitoring period in which no moving object is detected is shown in FIG. It becomes an image as shown in. The user shown in FIG. 21 (a) is presented to the user engaged in the monitoring work unless the moving object enters the monitoring field of view or unless the user intentionally designates a specific clipping condition. Will be.

  On the other hand, the moving object detection unit 160 performs the moving object detection process by comparing the distorted circular images S sequentially stored in the distorted circular image memory 130 in time series, and extracts the detected moving object when the moving object is detected. A target cut-out condition composed of a target cut-out center point Pn indicating the position, a target plane inclination angle φn indicating the cut-out direction of the detected moving object, and a target magnification mn indicating the cut-out size of the detected moving object is generated. Processing for giving the target cutting condition to the actual cutting condition determination unit 170 is performed.

  For example, as in the example shown in FIG. 22, the moving object detection unit 160 detects the moving object 60 in a state where static monitoring is performed based on the standard cutting condition that defines the illustrated cutting area E0. In this case, the moving object detection unit 160 generates a target cutout condition that defines the cutout area En shown in the figure, and performs processing of giving this to the actual cutout condition determination unit 170. Specifically, the target cutting center point Pn indicating the position of the cutout area En shown in the drawing, the target plane inclination angle φn indicating the direction of the cutout area En, and the magnification according to the size of the cutout area En are shown. The target magnification mn is generated and given to the actual cutting condition determination unit 170. In the case of the embodiment shown here, the target magnification mn is calculated as “appropriate value such that the cut-out area En becomes an area including the detected moving body 60” as described in detail in §6. The

  When the target cutting condition “Pn, φn, mn” is given from the moving object detection unit 160, the actual cutting condition determination unit 170 receives the standard cutting condition “P0, φ0” given from the standard cutting condition storage unit 180. , M0 ”, an actual cutting condition that changes stepwise is determined so as to shift to the target cutting condition“ Pn, φn, mn ”. Specifically, as shown in FIG. 25, the point moving from the point P0 to the point Pj (= Pn) toward the step point is determined as the actual cutting center point P, and as shown in FIG. A magnification value that changes stepwise from the standard magnification m0 corresponding to the size of the extraction area Ej (= size of E0) toward the target magnification mn corresponding to the size of the extraction area En is determined as the actual magnification m.

  As for the cutting direction, in the example shown in FIG. 22, in any of the cutting regions E0 to En, the U axis (horizontal axis of the planar regular image) is in a direction parallel to the Y axis (horizontal axis in the figure). Since the corresponding cutout direction is sufficient, the plane inclination angle φ0 = φn, and the plane inclination angle φ need not be substantially changed. Further, as described above, the target magnification mn is an amount determined according to the size of the detected moving object (apparent size on the shooting screen), and when mn = m0 happens to be, the magnification m is substantially There is no need to change. Alternatively, if the zooming is not performed intentionally, operation with the target magnification mn = standard magnification m0 is always possible regardless of the size of the detected moving object.

  After all, it is sufficient that the target cutting condition transmitted from the moving object detection unit 160 to the actual cutting condition determination unit 170 includes at least information on the target cutting center point Pn. For example, in the case of φ0 = φn, it is possible to omit transmitting the information of the target plane inclination angle φn from the moving object detection unit 160 to the actual cutting condition determination unit 170, and in the case of m0 = mn, the moving object detection is performed. It can be omitted to transmit the information of the target magnification mn from the unit 160 to the actual cutting condition determination unit 170.

  In addition, the moving object tracking unit 190 performs a moving object tracking process for tracking the detected moving object after the moving object is detected by the moving object detecting unit 160, and includes a tracking cut-out center point P (i) indicating a cut-out position of the tracking moving object, and the tracking moving object. It is determined that the tracking moving body has disappeared based on the tracking cut-out condition constituted by the tracking plane inclination angle φ (i) indicating the cutting direction and the tracking magnification m (i) indicating the tracking size of the tracking moving body. Until the actual cutting condition determination unit 170 continues.

  For example, as in the example illustrated in FIG. 22, when the moving object 60 is detected by the moving object detection unit 160, the cutout area En is determined as described above, and the actual cutout area is directed from the area E0 to En. The process of the transition period which changes in steps is executed. At this time, the moving object tracking unit 190 takes over the information of the moving object 60 from the moving object detection unit 160 and performs the operation of tracking the moving object 60. Then, as in the example shown in FIG. 27, the tracking cutout condition is set to “P (0), φ (0), m (0)” (= “Pn, φn, mn”) according to the position of the moving body 60. From “P (i), φ (i), m (i)”, “P (j), φ (j), m (j)”, “P (k), φ (k), m (k) ) ", ... and will change. As a result, the tracking cutout region moves as E (0), E (i), E (j), E (k),. The actual cutting condition determination unit 170 performs processing of adopting the tracking cutting condition given from the moving object tracking unit 190 as the actual cutting condition after completing the above-described transition period processing.

  In the case of the example shown here, the tracking cutout areas E (0), E (i), E (j), E (k),... The plane inclination angle φ (i) may be always set to a value corresponding to such a cut-out direction, as long as the cut-out direction corresponds to a direction parallel to the axis. As described in detail in §8, when a video camera with a fisheye lens is installed such that its imaging plane (XY plane) is a vertical plane, the plane inclination angle φ (i) is usually set according to the tracking position. Although it is necessary to change, the plane inclination angle φ (i) may be fixed when the imaging surface is set to be a horizontal plane.

  The tracking magnification m (i) is preferably determined according to the size of the moving object (apparent size on the shooting screen), similar to the target magnification mn. However, since the zooming is intentionally not performed. If so, regardless of the size of the moving object that is being tracked, it is possible to always perform a fixed operation such that the tracking magnification m (i) = target magnification mn.

  After all, it is sufficient that the tracking cutout condition transmitted from the moving body tracking unit 190 to the actual cutout condition determining unit 170 includes at least information of the tracking cutout center point P (i). For example, when a video camera with a fisheye lens is installed so that its imaging surface is a horizontal plane, φ (i) can be maintained at a predetermined fixed value (0 °) as will be described later. In addition, when adopting an operation in which the orientation of the obtained flat regular image is not required (operation in which the landscape displayed on the monitor may be upside down or oblique), regardless of the orientation of the video camera, φ (i) can be maintained at a fixed value. As described above, when φ (i) is maintained at a predetermined fixed value, it is possible to omit the information of the tracking plane inclination angle φ (i) from the moving body tracking unit 190 to the actual cutting condition determination unit 170. . Similarly, when the operation of fixing m (i) is performed, it is possible to omit the information of the tracking magnification m (i) from the moving body tracking unit 190 to the actual extraction condition determining unit 170.

  When the moving object tracking unit 190 loses sight of the moving object, the moving object tracking unit 190 reports to the actual cutting condition determination unit 170 that the moving object has disappeared, instead of transmitting the tracking cutting condition. When receiving the report of the disappearance of the moving object, the actual cutting condition determination unit 170 performs a process of switching the actual cutting condition to the standard cutting condition. As a result, the planar regular image shown in FIG. 24A is displayed on the screen of the monitor device 120 as before.

  On the other hand, the first function of the manual condition setting unit 185 is a function for forcibly setting the actual extraction condition to be determined by the actual extraction condition determination unit 170 to an arbitrary condition based on the user's setting operation. . As described above, the actual extraction condition determination unit 170 performs automatic determination processing that automatically determines the actual extraction condition by dividing into three periods of a static monitoring period, a transition period, and a dynamic monitoring period. When an arbitrary cutting condition is given from the manual condition setting unit 185, the automatic determination process is temporarily suspended during any period, and the cutting condition given from the manual condition setting unit 185 is preferentially executed. The processing is adopted as a cutting condition and given to the image cutting conversion unit 150.

  In other words, the manual condition setting unit 185 plays a role of transmitting the cutting condition input by the user's interruption operation to the actual cutting condition determining unit 170, and the actual cutting condition determining unit 170 The cutting condition instructed by the interrupt operation is unconditionally adopted as the actual cutting condition, and the process given to the image cutting conversion unit 150 is performed. Further, when the user gives an instruction to cancel the interrupt operation to the manual condition setting unit 185, the actual cutting condition determination unit 170 that has received the cancellation instruction returns to the original automatic determination process.

  The second function of the manual condition setting unit 185 is a function for arbitrarily setting a standard cutting condition stored in the standard cutting condition storage unit 180 based on a user's setting operation. For example, in the above-described example, the standard cutting condition storage unit 180 stores the standard cutting center point P0 indicating the cutting area E0 in FIG. 20, the standard plane inclination angle φ0, and the standard magnification m0 as the standard cutting conditions. Therefore, during the static monitoring period, the planar regular image shown in FIG. 21 (a) is displayed on the monitor screen. The user can give an instruction to the manual condition setting unit 185 to set an arbitrary cutting condition as a standard cutting condition. When such an instruction is given, the manual condition setting unit 185 performs a process of rewriting the standard cutting condition stored in the standard cutting condition storage unit 180 to a new cutting condition. Therefore, the user can change the planar regular image displayed during the static monitoring period to a desired image.

  As described above, the man-machine interface for allowing the user to specify an arbitrary cutting condition using the manual condition setting unit 185 is a pointing device such as a mouse on the distorted circular image displayed on the monitor screen. It is possible to appropriately design a format that can be specified by, a value that allows a user to input a numerical value, and the like. The manual condition setting unit 185 is not an essential component for the configuration of the present invention, and may be omitted in a system that does not require manual setting by the user.

  The basic configuration of the fish-eye monitoring system 100 according to the embodiment of the present invention has been described above with reference to the block diagram shown in FIG. 28. Actually, among the components shown in this block diagram, The components other than the video camera with fisheye lens 110 and the monitor device 120 can be realized by incorporating a dedicated program into the computer. Therefore, this fisheye monitoring system 100 can be constructed by connecting the video camera with fisheye lens 110 and the monitor device 120 to a computer in which the dedicated program is incorporated.

  Further, the components other than the video camera with fisheye lens 110 and the monitor device 120 in this system can be configured by a semiconductor integrated circuit in which an electronic circuit is embedded, instead of being configured by incorporating a program in a computer. In this case, the fish-eye monitoring system 100 can be realized by connecting a video camera with fish-eye lens 110 and a monitor device 120 to the semiconductor integrated circuit.

<<< §6. Motion detection processing >>>
Next, details of the moving object detection process of the moving object detection unit 160 in the fish-eye monitoring system 100 shown in FIG. 28 will be described.

  The moving object detection processing by the moving object detection unit 160 compares the distorted circular images S sequentially stored in the distorted circular image memory 130 in time series, detects the presence or absence of moving objects in the monitoring region, This is a process for recognizing a moving object area. In general, methods based on various algorithms are known as methods for recognizing moving objects by analyzing moving image data. Such a known arbitrary technique can be used for the moving object detection by the moving object detection unit 160.

  Here, an example of such a method will be described based on the flowchart of FIG. As already described, the distorted circular image memory 130 sequentially stores the distorted circular image S photographed by the video camera 110 as time-series image data for each frame. The basic principle of the technique shown here is that a background distorted circular image as a reference is stored in advance, and the distorted circular image sequentially stored in the distorted circular image memory 130 for each frame in time series is referred to as a background distorted circular image. In comparison, a region where the difference between the two is equal to or greater than a predetermined reference is recognized as a moving object region.

  First, in step S11, the frame counter C is initialized. Here, the frame counter is a parameter indicating what number of frame the current process is in performing the process in units of frames. In the example shown here, the initial value is set to C = 1. The In step S 12, the distorted circular image of the Cth frame stored in the distorted circular image memory 130 is read into the moving object detection unit 160. In step S13, the distorted circular image S (C) of the C-th frame is compared with the above-described background distorted circular image S (0), and pixels having a pixel value difference equal to or greater than the threshold value d are determined. A process of extracting the pixel of interest is performed.

  FIG. 30 is a plan view showing a specific process performed in step S13. Since both the background distorted circular image S (0) and the distorted circular image S (C) of the C-th frame are a set of a large number of pixels arranged on the XY plane, a pair of pixels arranged at the same coordinate position are used. A pixel can be recognized as a corresponding pixel. Therefore, the difference between the pixel values of the corresponding pixels is obtained. Then, a pixel whose difference is greater than or equal to a predetermined threshold value d is extracted as a target pixel. In the lower right part of FIG. 30, several pixels of interest extracted in this way are illustrated.

  In the case of a color image, pixel values of three primary colors such as R / G / B are defined for one pixel. In this case, for example, an average value for a difference between pixel values of the three primary colors Can be compared with the threshold value d, or the largest difference among the three primary colors can be compared with the threshold value d, and the respective pixel values can be stored in the RGB three-dimensional color space. Plot and compare the spatial distance of both plot points to the threshold d.

  Next, in step S14, the continuous region including the target pixel extracted in step S13 has an area equal to or larger than the reference area S, and has a vertical width and a horizontal width (width in the X-axis direction and width in the Y-axis direction). ) Are both searched for a region that satisfies the condition that is equal to or greater than the reference dimension D, and this is used as a region of interest. For example, in the case of the example shown in FIG. 30, there are two continuous regions A1 and A2 made up of the target pixel. The continuous area A1 has an area 8, vertical width 3, and horizontal width 4, and the continuous area A2 has area 2, vertical width 1, and horizontal width 2. Therefore, when the reference area S = 6 and the reference dimension D = 3 are set, the continuous area A1 becomes the attention area, but the continuous area A2 does not become the attention area.

  The reason why the continuous region that does not satisfy the reference area S is not set as the region of interest is to avoid detection of noise components and minute objects as moving objects. Moreover, the reason why both the vertical width and the horizontal width are set to the reference dimension D or more is to avoid detecting an elongated object that is generally unlikely to be a moving object as a moving object.

  FIG. 31 is a plan view showing a state in which the region of the moving body 60 (cat) shown in FIG. 22 is recognized as a region of interest by the processes of steps S13 and S14. A pixel shown by hatching in the figure is a pixel of interest, and a continuous region including the pixel of interest is a region having a reference area S or more, and satisfies both a criterion that both a vertical width and a horizontal width are a reference dimension D or more. It is an area.

  However, in the method shown in FIG. 29, even if a region of interest satisfying the condition of step S14 is searched, it is not immediately recognized as a moving object, and such region of interest is the reference frame number (Cmax). Only when the search is continuously performed over a period of time, processing for recognizing this as a moving object is performed. That is, if it is determined in step S15 that the region of interest has been searched, the process proceeds to step S16, where an update process for incrementing the frame counter C by 1 is performed, and the process from step S12 is repeatedly executed via step S17. Is done. That is, the next frame is read from the distorted circular image memory 130, and the above-described process is repeated. Thus, when the value of the frame counter C reaches the reference number of frames Cmax in step S17, the process proceeds to step S18, where it is determined that the moving object is detected. This means that when the attention area satisfying the condition of step S14 exists continuously for Cmax frames, the attention area is recognized as the moving object area.

  On the other hand, if the region of interest is not searched in step S15, the process returns to step S11, and the frame counter C is initialized to C = 1. Therefore, even if the region of interest is searched for several frames, if it is not searched continuously for Cmax frames, the process returns from step S15 to step S11, and moving object detection is not performed.

  It should be noted that, when it is determined that the region of interest exists continuously for Cmax frames, it is preferable to take into account that the identity of the region of interest existing over Cmax frames is ensured. . Specifically, in step S15, not only “whether or not the region of interest has been searched” is determined, but also “whether or not the region of interest having the same identity as the region of interest searched for last time has been searched” is determined. Make a decision. Here, the determination of identity is performed by, for example, examining the positional relationship between each pixel of interest that forms the region of interest searched last time and each pixel of interest that forms the region of interest searched this time. If the ratio is equal to or greater than a predetermined ratio, it may be determined that the two regions of interest are identical (regions formed due to the same moving object). Of course, in the identity determination, when the size or shape similarity of the region of interest is equal to or greater than a predetermined reference, the similarity of the feature amount (color histogram, edge direction histogram, etc.) of the region of interest Various criteria such as a case where the value is equal to or higher than a predetermined criterion can be used, and determination can be performed on the condition of a logical sum or logical product of these plural criteria.

  As described above, in the determination in step S15, a plurality of regions of interest appear alternately at different positions by adding a condition whether or not the identity of the regions of interest existing between frames is ensured. It is possible to avoid erroneously detecting an event as a moving object. In this case, when there are a plurality of regions of interest that are separated from each other, it is only necessary to make a determination for a region of interest having a larger area.

  In this way, when it is determined in step S18 that the moving object is detected, the moving object detection unit 160 generates the target cutting condition (Pn, φn, mn) and transmits the target cutting condition to the actual cutting condition determination unit 170. I do. In addition, the moving object tracking unit 190 performs processing for transmitting information indicating the final target region (the target region searched at the time of the frame counter C = Cmax in the flowchart of FIG. 29).

  As described in §5, the target cut-out conditions are the target cut-out center point Pn indicating the cut-out position of the detected moving object, the target plane inclination angle φn indicating the cut-out direction of the detected moving object, and the target indicating the cut-out size of the detected moving object. And a magnification mn. These pieces of information can be set based on the final target area. For example, the center of gravity position of the final target area is set as the target cut-out center point Pn, and the target target area is included so that the entire final target area is included in the target cut-out area based on the vertical and horizontal widths of the final target area. What is necessary is just to set the magnification mn.

  Specifically, for example, if the area formed by the set of hatched pixels shown in FIG. 31 is the final target area, the position of the center of gravity of the figure formed by the set of hatched pixels becomes the target cut-out center point Pn. The target magnification mn is determined so that the entire figure made up of a set of hatched pixels is included in the target cutout region. In the case of the embodiment shown here, as described above, the target plane inclination angle φn may be a fixed value (a value indicating the cutout direction in which the U axis of the plane regular image corresponds to the direction parallel to the Y axis).

  FIG. 32 is a block diagram showing a detailed configuration of the moving object detection unit 160 having a function of executing the moving object detection process shown in the flowchart of FIG. As shown in the figure, the moving object detection unit 160 operates by reading out a distorted circular image for each frame from the distorted circular image memory 130, and operates as a background image holding unit 161, a corresponding pixel comparison unit 162, a region of interest search unit 163, and a moving object detection. The determination unit 164 is configured. However, in practice, each of these components is configured by a computer program or a logic circuit on a semiconductor integrated circuit.

  The background image holding unit 161 is a component that holds an image stored in the distorted circular image memory 130 at a predetermined initial time point as a background distorted circular image S (0). It can be determined, for example, based on a user's instruction which image is retained as a background image. The user who is monitoring the regular plane image on the monitor screen may give an instruction to hold the image in the memory 130 at that time as a background image when there is no moving object. Receiving the instruction, the background image holding unit 161 captures and holds the image at that time as a background image.

  However, in the case where the video camera 110 is installed outdoors, the original background image itself that does not include moving objects varies greatly depending on the time of day and the weather. For example, even if images are taken of the same background, there is a large change in the overall luminance between daytime and nighttime. Therefore, in practice, it is preferable that the background image holding unit 161 periodically automatically updates the held background image. Specifically, images of all frames are sequentially fetched from the memory 130 to the background image holding unit 161, and an average image (images of each pixel) of a plurality of frames fetched in the static monitoring period at a predetermined cycle. An update process for holding an image obtained by averaging pixel values as a new background image may be performed. For example, by performing an update process of “every 100 minutes of the average image of 100 frames captured during the latest static monitoring period as a new background image” every minute, the latest background image can always be retained. it can. In addition, there is a method of creating a statistical background image (each pixel value is expressed by an upper limit and a lower limit) from a distribution model (normal distribution approximation or histogram) of pixel values for each pixel obtained from a plurality of previous frames. It is widely used as a general moving object detection method, and these methods can also be applied. In this case, the moving body region is detected as a region having a pixel value that is greater than or equal to a certain threshold value or less than a certain threshold value.

  On the other hand, the corresponding pixel comparison unit 162 holds, in the background image holding unit 161, pixel values of individual pixels on the distorted circular image S (C) that are sequentially stored in the distorted circular image memory 130 for each frame in time series. Compared with the pixel value of the corresponding pixel on the background distorted circular image S (0), a process is performed to extract a pixel whose difference is equal to or greater than the threshold value as the target pixel (see FIG. 30). This process corresponds to the process of step S13 in the flowchart of FIG. Information for identifying the extracted pixel of interest (for example, its coordinate value) is transmitted to the region of interest search unit 163.

  The region-of-interest search unit 163 searches for and searches a region of interest that is a continuous region composed of pixels of interest and satisfies a condition that satisfies the condition that the reference area S is greater than or equal to the predetermined reference dimension D. Information for specifying the region of interest (for example, coordinate values of individual pixels constituting the region) is transmitted to the moving object detection determination unit 164. This process corresponds to the process of step S14 in the flowchart of FIG. As described above, when there are a plurality of regions of interest separated from each other, information on the region of interest having a larger area may be transmitted to the moving object detection determination unit 164.

  The moving object detection determination unit 164 determines that the moving object is detected when the distorted circular image including the region of interest is continuously obtained for the reference number of frames Cmax or more. This processing corresponds to the processing in steps S15 to S18 in the flowchart of FIG. As already described, at this time, it is preferable to check whether or not the region of interest obtained continuously over the reference number of frames Cmax satisfies the condition of identity. In other words, the determination that the moving object is detected is performed only when the region of interest obtained continuously is expected to be an area formed due to the same moving object. The determination of identity is performed by examining the positional relationship between each pixel of interest that forms the target area searched last time and each pixel of interest that configures the target area searched this time, and the ratio of pixels at the same position is a predetermined ratio. In the case described above, the two regions of interest may be performed based on the criterion that they are the same or the various criteria described above.

  When moving object detection is performed in this way, the moving object detection determination unit 164 tracks information indicating the final region of interest (for example, coordinate values of individual pixels constituting the region of interest in the Cmaxth frame). Tell part 190. In addition, a target cutting condition is generated based on this final focus area, and this is given to the actual cutting condition determination unit 170. Specifically, as described above, the center of gravity position of the final target region may be set as the target cut-out center point Pn, and the target magnification mn may be set based on the vertical width and horizontal width of the final target region.

  In the case of the embodiment shown here, the target plane inclination angle φn is a fixed value, and therefore does not need to be set by the moving object detection determination unit 164. Further, when the target magnification mn is set to a fixed value without zooming, the moving object detection determination unit 164 does not need to set the target magnification mn, so only the target cutting center point Pn is selected. It is sufficient to generate it as the extraction condition and give it to the actual extraction condition determination unit 170.

<<< §7. Moving body tracking process >>
Next, details of the moving body tracking process of the moving body tracking unit 190 in the fish-eye monitoring system 100 shown in FIG. 28 will be described.

  In the moving body tracking process by the moving body tracking unit 190, after the moving body is detected by the moving body detection unit 160, the moving body tracking process for tracking the detected moving body is performed, and the tracking cut-out center point P (i) and the tracking plane inclination angle φ (i) are performed. And the tracking magnification condition m (i) are continuously given to the actual cutting condition determination unit 170 until it is determined that the tracking moving object has disappeared.

  The moving body tracking unit 190 receives information on the detected moving body (information indicating the final target area) from the moving body detection unit 160, and then sequentially reads out the distorted circular image in units of frames from the distorted circular image memory 130 to track the detected moving body. Perform the process. In general, methods based on various algorithms are known as methods for analyzing moving image data and tracking moving objects. Such a known arbitrary method can be used for tracking the moving object by the moving object tracking unit 190.

  Here, an example of such a method will be described based on the flowchart of FIG. First, in step S21, a process is performed in which the moving object (final area of interest) detected by the moving object detection unit 160 is the 0th tracking moving object on the 0th frame image.

  In step S22, the feature amount of the 0th tracking moving body is calculated. Here, the “tracking moving object feature amount” is an amount indicating the feature of the image with respect to a region (a region of interest) constituting the moving object recognized on the target frame image. A color histogram or edge direction histogram of a pixel to be used can be used as a feature amount. Specifically, a predetermined calculation for obtaining a feature amount using the pixel value of each pixel constituting the 0th tracking moving object, that is, the moving object (final target region) detected by the moving object detection unit 160. An operation based on the expression is executed.

  In the next step S23, the tracking parameter i is set to the initial value 1. Similar to the frame counter C in the moving object detection process, the tracking parameter i is a parameter indicating the frame number of the current process when performing processing in units of frames. In the case of the example shown here, the final frame that is the target of the moving object detection process is set as the 0th frame that is the target of the moving object tracking process, and the moving object tracking is performed for the subsequent first and subsequent frames. It will be.

  In the subsequent step S24, the vicinity range of the (i−1) -th tracking moving object in the i-th frame image is searched, and processing for recognizing a region having a similar feature amount as the i-th tracking moving object is performed. For example, when i = 1, the first frame image is searched for the vicinity range of the tracking moving object on the zeroth frame (that is, the vicinity range of the final target area given from the moving object detection unit 160). A search for a range having a feature amount similar to the feature amount of the 0th tracking moving body obtained in step S22 is performed. Specifically, for example, a plurality of candidate areas are defined by shifting the final target area given from the moving object detection unit 160 by several pixels vertically and horizontally, or by increasing or decreasing by several pixels vertically and horizontally. Then, a feature amount for the plurality of candidate regions may be obtained, and processing for searching for candidate regions having similar feature amounts may be performed. Then, the candidate area having the highest feature amount with the similarity equal to or higher than a predetermined reference may be recognized as the area of the tracking moving object in the i-th frame image.

  In this way, when the tracking moving object is successfully recognized, the process proceeds from step S25 to step S26, the tracking parameter i is updated by 1 and the process of step S24 is repeated. That is, the next frame image is read out from the distorted circular image memory 130, and an area having a feature amount similar to the area of the tracking moving object recognized in the previous frame image is recognized as the area of the tracking moving object. The

  On the other hand, if the tracking moving object has failed to be recognized, that is, if a candidate area having a feature amount similar to the area of the tracking moving object recognized in the previous frame image cannot be found, the process proceeds from step S25 to step S27. It is determined that the moving object has disappeared.

  When it is determined in step S25 that the tracking moving object has been successfully recognized, the moving object tracking unit 190 determines the tracking cut-out condition (P (i), φ (i), m (i) regarding the i-th frame. ) Is generated, and this is transmitted to the actual cutting condition determination unit 170. On the other hand, when it is determined that recognition of the tracking moving object has failed, the moving object tracking unit 190 performs processing for transmitting information indicating the disappearance of the moving object to the actual extraction condition determining unit 170.

  As described in §5, the tracking cut-out conditions include the tracking cut-out center point P (i) indicating the cut-out position of the tracking moving object, the tracking plane inclination angle φ (i) indicating the cutting-out direction of the tracking moving object, and the tracking moving object. And a tracking magnification m (i) indicating the cut-out size. These pieces of information can be set based on the area that constitutes the recognized tracking moving body. For example, the tracking magnification is set so that the center of gravity of the area is set as the tracking cut-out center point P (i) and the entire area is included in the tracking cut-out area based on the vertical and horizontal widths of the area. m (i) may be set. As described above, the tracking magnification m (i) and the tracking plane inclination angle φ (i) can be fixed values depending on the installation form and operation form of the video camera.

  FIG. 34 is a block diagram showing a detailed configuration of the moving object tracking unit 190 having a function of executing the moving object tracking process shown in the flowchart of FIG. The moving object tracking unit 190 starts the operation after receiving information indicating the final target region from the moving object detecting unit 160, and reads the distorted circular image for each frame from the distorted circular image memory 130 to track the detected moving object. As shown in the drawing, the moving object region holding unit 191, the candidate region extracting unit 192, the feature amount calculating unit 193, the new moving object region recognizing unit 194, and the moving object tracking determining unit 195 are configured. However, in practice, each of these components is configured by a computer program or a logic circuit on a semiconductor integrated circuit.

  The known moving object region holding unit 191 is a component that holds information of a known moving object region that has already been recognized on the (i−1) th frame image stored in the distorted circular image memory 160. When i = 1, the information on the known moving object region on the 0th frame image is given from the moving object detection unit 160 as the information on the final target region. The information on the known moving object region after i = 2 is given as information on the new moving object region from the moving object tracking determination unit 195, as will be described later.

  The candidate area extraction unit 192 performs a process of extracting a plurality of areas located in the vicinity of the known moving object area as candidate areas from the i-th frame image stored in the distorted circular image memory 130. Specifically, as described above, the known moving object region for the (i−1) th frame image held in the known moving object region holding unit 191 is shifted by several pixels vertically and horizontally, or up and down. A plurality of candidate regions may be defined by increasing or decreasing by several pixels to the left and right.

  The feature amount calculation unit 193 includes a known moving object region (moving object region recognized on the (i−1) th) frame image) held in the known moving object region holding unit 191, and a candidate region extracting unit 192. For each of a plurality of extracted candidate regions (regions defined on the i-th frame image), an operation for obtaining a feature value indicating the feature of each image is performed. Specifically, as described above, a calculation for obtaining a color histogram or edge direction histogram of a pixel constituting a region given as a calculation target as a feature amount may be performed. Note that since the feature amount of the known moving object region has already been calculated as the feature amount of the candidate region in the processing one frame before, the feature amount calculation unit 193 does not need to perform the calculation again in practice, and the processing one frame before The calculation result obtained in step 2 can be used.

  The new moving body region recognizing unit 194 compares the feature amount of the known moving body region calculated by the feature amount calculating unit 193 with the feature amount of the plurality of candidate regions, and the similarity to the feature amount of the known moving body region. The candidate area having the highest feature quantity and a predetermined reference value or more is recognized as a new moving body area on the i-th frame image. Even when a plurality of candidate areas having similarities equal to or higher than a predetermined standard are found, the accuracy of moving body tracking can be improved by recognizing the candidate area having the highest feature amount as a new moving body area. .

  When the recognition by the new moving body region recognition unit 194 is successful, the moving body tracking determination unit 195 determines the tracking cutout conditions P (i), φ (for the i-th frame image based on the recognized new moving body region. i) and m (i) are generated and transmitted to the actual cutting condition determination unit 170. In this case, the position of the center of gravity of the new moving object region recognized for the i-th frame image is set as the tracking cut-out center point P (i), and the tracking magnification m (i) is set based on the vertical and horizontal widths of the new moving object region. ) Is as described above. On the other hand, if the recognition fails, it is determined that the tracking moving object has disappeared, and information indicating the disappearance of the moving object is transmitted to the actual cutting condition determination unit 170.

  Moreover, the moving body tracking determination part 195 gives the information which shows a new moving body area | region to the existing moving body area | region holding | maintenance part 191, when recognition is successful. The existing moving object area holding unit 191 holds the area information of the detected moving object given from the moving object detection unit 160 as the information of the first known moving object area, but thereafter, the new moving object area given sequentially from the moving object tracking determination unit 195 Information is held as information on a new known moving object region. Therefore, when the candidate area extracting unit 192 extracts a candidate area for the i-th frame, the existing moving object area holding unit 191 has a moving object area recognized for the (i−1) -th frame. Will be held.

  When a fixed value is adopted as the tracking plane inclination angle φ (i), the moving object tracking determination unit 195 does not need to generate φ (i) as the tracking cutout condition. In addition, when the tracking magnification m (i) is set to a fixed value without zooming, it is not necessary to generate the tracking magnification m (i). It suffices to generate only the point P (i) as the tracking cutout condition and give it to the actual cutout condition determining unit 170.

<<< §8. Processing for determining actual cutting conditions >>>
Here, details of the actual cutting condition determination processing by the actual cutting condition determination unit 170 in the fish-eye monitoring system 100 shown in FIG. 28 will be described.

<8-1. Basic functions of actual cutting condition determination unit>
The basic function of the actual cutting condition determining unit 170 is an actual cutting formed by a cutting center point P, which is one point on the distorted circular image S, a predetermined plane inclination angle φ, and a predetermined magnification m. The condition is to be given to the image cutout conversion unit 150 in real time.

  FIG. 35 is a flowchart showing the procedure of the actual cutting condition determination process performed by the actual cutting condition determination unit 170. As shown in the figure, this determination process is performed for each of three periods, a static monitoring period W1 composed of steps S31 and 32, a transition period W2 composed of steps S33 and 34, and a dynamic monitoring period W3 composed of steps S35 and 36. Different content processing is performed.

  In the static monitoring period W1, as shown in step S31, the standard cutting condition (P0, φ0, m0) stored in the standard cutting condition storage unit 180 is used as it is as the actual cutting condition (P, φ , M), the processing given to the image cutout conversion unit 150 is performed. Then, until the moving object detection is determined in step S32, in other words, the process of step S31 is continued until the target cutting condition (Pn, φn, mn) is given from the moving object detection unit 160, and static The monitoring period W1 continues.

  On the other hand, when the moving object detection is determined in step S32, that is, when the target cutting condition (Pn, φn, mn) is given from the moving object detection unit 160, the transition period W2 starts. That is, as shown in step S33, the actual cutting conditions (P that change in stages so as to shift from the standard cutting conditions (P0, φ0, m0) to the target cutting conditions (Pn, φn, mn)). , Φ, m) is generated, and processing for giving this to the image cutout conversion unit 150 is performed. Until the transition completion is determined in step S34, in other words, until the generated actual cutting condition (P, φ, m) reaches the target cutting condition (Pn, φn, mn). The process in step S33 is continued, and the transition period W2 continues.

  Here, when it is determined in step S34 that the transition has been completed, that is, the actual cutting conditions (P, φ, m) that have been changed stepwise from the standard cutting conditions (P0, φ0, m0) are the targets. When the cutting condition (Pn, φn, mn) is reached, the dynamic monitoring period W3 starts. That is, as shown in step S35, the tracking cutting conditions (P (i), φ (i), m (i)) given from the moving body tracking unit 190 are used as they are as the actual cutting conditions (P, φ, m), the processing given to the image cut-out conversion unit 150 is performed. Then, until the determination of the disappearance of the moving object is made in step S36, in other words, the process of step S35 is continued until the report indicating the disappearance of the moving object is made from the moving object tracking unit 190, and the dynamic monitoring period W3 continues. .

  When it is determined in step S35 that the moving object has disappeared, the dynamic monitoring period W3 ends, the process returns to step S31, and the static monitoring period W1 resumes.

  In order to execute such processing, the actual cut condition determining unit 170 is, as shown in FIG. 36, a static monitoring period responsible unit 171, a transition period responsible unit 172, and a dynamic monitoring period responsible unit 173. May be configured. However, in practice, each of these components is configured by a computer program or a logic circuit on a semiconductor integrated circuit.

  Here, the static monitoring period charge unit 171 is a component responsible for the static monitoring period W1 in which no moving object is detected, and the standard stored in the standard extraction condition storage unit 180 during the period W1. It has a function of determining and outputting the cutting conditions (P0, φ0, m0) as the actual cutting conditions (P, φ, m). In addition, the transition period charge unit 172 is a component responsible for a predetermined transition period W2 after the moving object is detected, and during the period W2, target cutout is performed from the standard cutout condition (P0, φ0, m0). It has a function of generating and outputting actual cutting conditions (P, φ, m) that change stepwise so as to shift to the conditions (Pn, φn, mn). The dynamic monitoring period responsible unit 173 is a component that is in charge of the dynamic monitoring period W3 from when the transition period is completed until the disappearance is determined, and is given from the moving object tracking unit 190 during the period W3. The tracking cutout conditions (P (i), φ (i), m (i)) to be output are determined as actual cutout conditions (P, φ, m).

  In short, the respective responsible units 171, 172, and 173 shown in FIG. 36 function to output the actual cutting conditions (P, φ, m) only during their own assigned period of the periods W 1, W 2, and W 3. Therefore, the actual cutting condition (P, φ, m) output from the actual cutting condition determining unit 170 is a condition that is selectively output from any of the responsible units 171, 172, and 173.

  In addition, as already stated, it is possible to take an operation in which the plane inclination angle φ and the magnification m are fixed among the actual cutting conditions. In this case, the actual cutting condition determination unit 170 needs only to always output the fixed value for the plane inclination angle φ and the magnification m, so that the process of substantially determining only the value of the cutting center point P is performed. Just do it. Further, it is sufficient that the moving object detection unit 160 outputs only the value of the target cutting center point Pn as the target cutting condition, and the moving object tracking unit 190 sets the value of the tracking cutting center point P (i) as the tracking cutting condition. Output only.

  36, the static monitoring period responsible unit 171 may perform the process of outputting the standard extraction condition as it is, and the dynamic monitoring period responsible unit 173 performs tracking. Since the process of outputting the cutting condition as it is may be performed, the configuration is very simple. On the other hand, the transition period charge unit 172 transitions the actual cutting conditions in stages from the standard cutting conditions (P0, φ0, m0) to the target cutting conditions (Pn, φn, mn). Need to do. Therefore, a specific method of the migration process will be described below with some examples.

  The actual cutting condition is constituted by three parameters P, φ, and m, but the transition can be performed independently for each parameter. That is, the cutting center point P is shifted stepwise from P0 to Pn, the plane inclination angle φ is shifted stepwise from φ0 to φn, and the magnification m is stepped from m0 to mn. Can be migrated. These transitions may proceed simultaneously in parallel or may be performed in separate periods.

  In §4, an example in which the standard cutout area E0 is shifted to the target cutout area En in a stepwise manner on the distorted circular image S shown in FIG. In this example, the transition period W2 is divided into a first panning period and a second zooming period. In the panning period, as shown in FIG. 25, the cutting center point P is shifted from the point P0 to the point Pj. In FIG. 26, the cutout area E is shifted from the area Ej to En as shown in FIG. Here, let's consider the transition of specific actual cutting conditions when performing such a migration process.

  FIG. 37 is a plan view showing the moving state of the cut center point P in this transition process. As illustrated, the point P is along the moving path A (the direction indicated by the azimuth angle α with respect to the Y axis) from the center point P0 of the circle constituting the distorted circular image S along the radius of the circle. , P1,..., Pi,..., Pj to Pn. The points Pj to Pn are the same point. As will be described later, Ji, Jj to Jn indicated by arrows are rotation reference axes at respective positions, and are axes orthogonal to the movement path A. Here, the movement period from the points P1 to Pj is the first half of the panning period, and the period remaining at the points Pj to Pn is the second half of the zooming period.

  FIG. 38 is a table showing a specific transition of the actual cutting conditions in this migration process. The frame F column describes the frame numbers F0 to Fn of the distorted circular image to be cut out, and the cut region E column shows the cut region E0 set on each frame image. ~ En are described. The actual cutting conditions change as described in each column of the cutting center point P, the plane inclination angle φ, and the magnification m.

  First, the cut-out center point P changes from the 0th point P0 to the nth point Pn. Actually, however, the jth and subsequent points Pj˜ are all the nth point Pn. The movement of the cut-out center point P is completed at the j-th frame Fj. This means that the first panning period is completed by the j-th frame Fj. On the other hand, the magnification m changes from the 0th magnification m0 to the nth magnification mn, but actually, the magnification is changed up to the jth magnification mj and is the same as the original magnification m0. It starts from the (j + 1) -th magnification. This means that the latter zooming period starts from the j-th frame Fj.

  On the other hand, the plane inclination angle φ may always be the same value during movement along the movement path A in this example. The plane inclination angle φ is a parameter that determines the cutting direction, but is an angle determined with reference to the rotation reference axes Ji, Jj to Jn shown in FIG. φ should always be fixed to the same value. The handling of the plane inclination angle φ will be described later.

  As in the example shown here, the transition period charge unit 172 changes the cutting center point P from the standard cutting center point P0 to the target cutting center point Pn in stages during the panning period of the first half of the transition period. In the zooming period in the latter half of the transition period, when the moving condition is detected, from the viewpoint of the user, first, when the moving condition is detected, the magnification m is changed stepwise from the standard magnification m0 to the target magnification mn. After panning in which the visual field moves to the moving object position, zooming is performed to a magnification suitable for moving object observation. In this way, if the panning is performed in the first half while the magnification is fixed, the user can more intuitively grasp the positional relationship. In the latter zooming, the zoom-in operation is performed when mn> m0, but the zoom-out operation is performed when mn <m0.

  However, if panning and zooming are performed separately as in the above example, the transition period becomes longer, and if the setting like the cutout area Ej in FIG. 25 is performed, a part of the distorted circular image S There is also an adverse effect that protrudes from the area. In order to avoid such an adverse effect, the transition period charge unit 172 sets the entire transition period as a panning period and a zooming period, and the cutting center point P changes from the standard cutting center point P0 to the target cutting center point Pn. At the same time, the actual cutting condition in which the magnification m changes stepwise from the standard magnification m0 to the target magnification mn may be determined, and panning and zooming may be performed in parallel. In this case, the cut-out center point P and the magnification m gradually change in parallel from the 0th frame F0 to the nth frame Fn.

<8-2. Movement of the cutting center point P (panning)>
Next, consider a specific method of moving the cut center point P for panning. In general, constant-speed panning and unequal-speed panning are known as panning methods. For example, when performing horizontal panning using a video camera equipped with a normal lens, the camera will be rotated in the horizontal direction. However, if the rotational angular velocity is kept constant, panning will be constant and the rotational angular velocity can be changed. Panning is not uniform. However, as in the present invention, when performing processing for obtaining a planar regular image as obtained by panning shooting using a distorted circular image shot by a video camera equipped with a fisheye lens, a normal lens is used. It needs to be handled slightly differently from the panning operation of the attached video camera.

  First, consider performing constant-speed panning. When performing pseudo panning using a distorted circular image captured using a fisheye lens, the effect of constant speed panning cannot be obtained even if the cut center point P is moved at a constant speed. For example, in the example shown in FIG. 37, the extraction center point P is moved from the point P0 to the point Pj along the movement path A in the first half of the panning period, but the movement speed of the point P is kept constant. In this case, the effect of constant-speed panning by a video camera equipped with a normal lens cannot be obtained. This is because the angle of view of the subject to be photographed differs between the central portion and the peripheral portion of the distorted circular image S.

  FIG. 39 is a side view showing a specific method of constant velocity panning according to the present invention. The horizontal line in the figure indicates the XY plane on which the distorted circular image S is formed, and the semicircle indicates the virtual spherical surface H defined on the distorted circular image S. Here, let us consider taking a moving path of the cutting center point P in the direction of the arrow A pointing to the right in the figure, and performing constant-speed panning consisting of n stages from the point P0 to the point Pn in the figure. .

  As described in §2, the corresponding point Q can be defined on the phantom spherical surface H for any point P on the XY plane. In this case, the two points P and Q have a relationship that “external light incident on the point Q on the phantom spherical surface H toward the origin O reaches the point P on the XY plane”. Is used, the corresponding point Q is a point immediately above the point P (a point having the same XY coordinates). The points Q0, Q1,..., Qi,..., Qn shown in FIG. 39 are corresponding points of the points P0, P1,. The distorted circular image S on the XY plane is distorted because the spherical image on the virtual spherical surface H is projected onto the XY plane. In other words, the real image space on the virtual spherical surface H is distorted and accommodated in the fish-eye image space on the XY plane.

  Constant velocity panning using a video camera equipped with a normal lens considers a line-of-sight vector in which the real image space on the virtual spherical surface H is desired from the position of the point P0 (the position of the origin O) in FIG. This is equivalent to exercising at an angular velocity. Therefore, in order to obtain the effect of constant-speed panning by a video camera equipped with a normal lens, the motion of the corresponding point Q on the virtual spherical surface H is not constant motion but the motion of the cut-out center point P on the XY plane. It should be exercised.

  For example, as shown in the figure, when the cut center point P is moved in n stages from the point P0 to the point Pn, the movement distance ai of the i-th point Pi (along the movement path A from the movement start point P0). The effect of constant-speed panning cannot be obtained even if the distance) changes linearly with respect to time t. In order to obtain the effect of constant-speed panning, the i-th corresponding point Qi moves on the virtual spherical surface H (from the movement starting point Q0 along the moving path B on the virtual spherical surface indicated by the arrow B in the figure). In other words, the position of the i-th point Pi may be determined so that the arc Q0-Qi) changes linearly with respect to time t.

If the motion of the corresponding point Q is expressed by a mathematical expression, the relationship shown in the upper part of FIG. 39 is obtained. That is, as shown in the figure, if the zenith angles of the corresponding points Qi and Qn are βi and βn, respectively, the moving distance ai along the moving path A is the radius of the distorted circular image S (the radius of the phantom spherical surface H) R As “ai = R · sin βi”. When performing constant speed panning, it is necessary to change the angle βi from 0 ° to βn at equal intervals over n stages, so that “βi = i / n · βn”, and eventually “ai = R · sin (I / n · βn) ”is obtained. Here, the angle βn is “βn = sin ⁻¹ an”, where an final moving distance of the point P is an. After all, on the XY plane, when moving the point P along the moving path A from the point P0 to the point Pn in n stages, the moving distance ai of the i-th point Pi is
ai = R · sin (i / n · (sin ⁻¹ an))
If set to, the effect of constant speed panning can be obtained.

  FIG. 40 is a plan view and a graph showing a specific method of this constant velocity panning. The circular diagram shown in the upper part shows a fish-eye image space (distorted circular image S) defined on the XY plane, and a movement path A is taken in the right direction. The starting point of the cutting center point P is the standard cutting center point P0, and the end point is the target cutting center point Pn. The cut-out center point P moves by a distance an along the moving route A from the start point P0 to the end point Pn. The movement is performed in n stages, and the point Pi indicating the i-th stage movement position is a point separated from the start point P0 by the movement distance ai. Since this XY plane corresponds to a distorted fish-eye image space, when performing constant-speed panning, the moving speed of the point P on the XY plane does not become constant, and the speed decreases with time t as shown in the figure. I will do it.

  On the other hand, the rectangular figure shown in the middle of FIG. 40 shows the real video space defined on the virtual spherical surface H, and the movement path B is taken in the right direction. This movement path B is a path along the circumference of the phantom spherical surface H, the start point of which is the corresponding point Q0 of the point P0, and the end point thereof is the corresponding point Qn of the point Pn. The corresponding point Q moves by the distance bn along the moving route B from the start point Q0 to the end point Qn. The movement is performed in n stages, and the point Qi indicating the i-th movement position is a point separated from the starting point Q0 by the movement distance bi. A broken line connecting the upper diagram and the middle diagram indicates the correspondence between the points P0, ..., Pi, ..., Pn and the points Q0, ..., Qi, ..., Qn. Show. When performing constant speed panning, the moving speed of the corresponding point Q in the real video space may be set to a constant speed as shown in the figure.

  The graph shown at the lower left of FIG. 40 is a graph showing the movement distance b with respect to time t of the corresponding point Q moving along the movement path B on the phantom sphere toward the points Q0 to Qn, and is shown on the lower right. The graph is a graph showing the movement distance a with respect to time t of the cut center point P moving along the movement path A toward the points P0 to Pn on the XY plane. As shown in the figure, the corresponding point Q moves at a constant speed, whereas the movement of the cutting center point P does not move at a constant speed. Eventually, in the system according to the present invention, in order to obtain the same effect as constant-speed panning using a video camera equipped with a normal lens, the transition period charge unit 172 causes the movement of the corresponding point Q on the virtual sphere. What is necessary is just to make it change the position of the cutting center point P so that it may become constant velocity motion.

  As described above, if the basic principle of performing constant speed panning in the present invention is known, it is possible to perform desired inconstant speed panning using this basic principle. Invariant speed panning is a concept that literally includes all panning methods other than constant speed panning, but here, we move the line of sight so that the speed of the start and end parts of the panning period is slower than the speed of the intermediate part. Illustrates the non-uniform panning performed. Such an inconstant speed panning method matches the physical movement of the camera, and thus has the advantage of presenting a natural movement to the viewer, and is widely used for various video expressions.

  FIG. 41 is a plan view and a graph showing a specific method of inconstant speed panning according to the present invention. Similar to FIG. 40, the circular diagram shown in the upper part shows the fish-eye image space (distorted circular image S) defined on the XY plane, and the rectangular figure shown in the middle part is defined on the virtual spherical surface H. The real image space is shown, the graph shown on the left in the lower row shows the movement distance b of the corresponding point Q that moves along the movement path B on the virtual sphere, and the graph shown on the lower right shows the movement route A on the XY plane The moving distance a of the cutting center point P moving along is shown.

  Here, paying attention to the lower left graph, the motion of the corresponding point Q is set such that the speed of the motion start portion and the motion end portion is slower than the speed of the intermediate portion. Such a setting is a setting for the motion corresponding to the above-described general inconstant panning, and the corresponding point Q in the real image space is to perform the motion as shown in the middle rectangular figure. Become. Thus, if the motion of the corresponding point Q in the real video space is defined, the motion on the XY plane of the cut-out center point P corresponding to this can be determined. The lower right graph shows the movement of the cut center point P determined in this way, and the upper circular diagram shows the locus of the cut center point P that makes such a movement along the movement path A. It is shown.

Eventually, in this case, the transition period charge unit 172 determines that the motion of the corresponding point Q on the phantom spherical surface is an inconstant motion such that the speed of the start part and the end part of the panning period is slower than the speed of the intermediate part. Thus, the position of the cut-out center point P may be changed. If the movement of such a cutting center point P is expressed by a mathematical formula, as shown in the upper part of FIG.
ai = R · sin (bi / bn · (sin ⁻¹ an))
(Equation in which the coefficient “i / n” in the constant velocity panning equation is replaced with the coefficient “bi / bn”). Here, ai is the movement distance of the i-th point Pi when moving the point P along the movement path A from the point P0 to the point Pn in n stages on the XY plane, and bn 41, the distance from the point Q0 to the point Qn measured along the virtual spherical surface H as shown in the middle part of FIG. 41, and bi is the distance from the point Q0 to the point Qi measured along the virtual spherical surface H, similarly. It is. The distance bi is the moving distance at time ti given by the lower left graph in FIG. Therefore, if the lower left graph of FIG. 41 is defined, the lower right graph can be obtained by calculation based on the above equation, and the distance ai (that is, the center of extraction) at a predetermined time ti. The position of the point P) can be uniquely determined.

<8-3. Change magnification m (zooming)>
Next, consider a specific method of changing the magnification m for zooming. In general, constant speed zooming and inconstant speed zooming are also known as zooming methods. In the case of a video camera equipped with a normal lens, optical zooming is performed by an operation of moving the objective lens along the optical axis. If the moving speed of the objective lens is constant, it is constant speed zooming, and if it is not constant, it is non-constant zooming. Normally, when performing non-uniform zooming, the moving speed of the objective lens at the beginning and end of the zooming period is set to be slower than the moving speed of the intermediate portion, as in non-uniform panning. .

  In the case of the present invention, the zooming operation is performed by changing the magnification m. As described in §2, the magnification m is a parameter for determining the distance between the origin O of the three-dimensional XYZ orthogonal coordinate system and the origin G of the two-dimensional UV orthogonal coordinate system in the model shown in FIG. The distance is given by m · R, where R is the radius of the phantom spherical surface H. As shown in FIG. 10, if the magnification m is set small, a wider cutout area E1 is set, and a wide-angle image is obtained in the frame F1, and the magnification m is set large as shown in FIG. Then, a narrower cutout area E2 is set, and a telephoto image is obtained in the frame F2.

  Therefore, in the zooming operation, the transition period charge unit 172 sets all or a part of the transition period as the zooming period, and monotonically increases along the time axis during the zooming period (when mn> m0: It can be performed by changing the magnification m from the standard magnification m0 to the target magnification mn so that the zoom-in operation) or monotonously decreases (when mn <m0: zoom-out operation).

FIG. 42 is a graph showing a specific method of constant speed and inconstant speed zooming according to the present invention, wherein the upper part is a constant speed zooming graph and the lower part is a non-constant zooming graph. In either case, the horizontal axis represents time t, and the vertical axis represents the magnification m. In the case of constant speed zooming, as shown in the upper graph, the magnification m increases linearly from the standard magnification m0 to the target magnification mn during the zooming period from time t0 to tn. Therefore, when the magnification is changed in stages over n stages, the i-th stage magnification mi is
mi = m0 + i / n · (mn−m0)
Given in. Therefore, the constant speed zooming operation can be performed by the transition period charge unit 172 changing the magnification m linearly along the time axis during the zooming period.

On the other hand, in the case of non-uniform speed zooming, as shown in the lower graph, the magnification m increases non-linearly from the standard magnification m0 to the target magnification mn during the zooming period from time t0 to tn. Therefore, when the magnification is changed in stages over n stages, the i-th stage magnification mi is
mi = m0 + Ki · (mn−m0)
Given in. Here, the coefficient Ki is a proportionality coefficient unique to the i-th stage determined by the shape of the lower graph.

  Therefore, if the graph shown in the lower part of FIG. 42 is prepared in advance and the transition period charge unit 172 changes the magnification m from the standard magnification m0 to the target magnification mn in accordance with this graph, the zoom period start portion and The magnification m can be changed along the time axis so that the magnification change speed at the end portion changes at a non-uniform speed that is slower than the change speed at the intermediate portion, so that the non-uniform speed zooming can be performed. .

  FIG. 43 is a table showing specific steps of the above-described constant speed and non-uniform speed panning, and illustrates a panning operation in a total of eight steps. The left column of each table lists the step number, and the right column lists the movement distance b in the real video space (on the virtual sphere) of the corresponding point Q at the time of the step. In the constant speed panning shown in FIG. 43 (a), the distance b increases linearly, whereas in the inconstant speed panning shown in FIG. 43 (b), the distance b increases non-linearly (in the start portion and the end portion). It can be seen that the increase rate is a non-linear increase in which the increase rate is slower than the increase rate in the intermediate portion.

  Here, the value of bn varies depending on the position of the target cut-out center point Pn (that is, the position of the detected moving object), and the coefficient portion to be multiplied is the value listed in the table. Therefore, if this coefficient part is prepared as a function expression or a table, a desired panning operation can be performed according to the contents of the table.

  On the other hand, FIG. 44 is a table showing specific steps of the above-described constant speed and inconstant speed zooming, and illustrates a zooming operation by a total of eight steps. The left column of each table lists the step number, and the right column lists the actual magnification m at the time of the step. In the constant speed zooming shown in FIG. 44 (a), the magnification m increases linearly, whereas in the non-constant speed zooming shown in FIG. 44 (b), the magnification m increases nonlinearly (in the start and end portions). It can be seen that the increase rate is a non-linear increase in which the increase rate is slower than the increase rate in the intermediate portion.

  Here again, the value of the target magnification mn varies depending on the size of the detected moving object, but the coefficient portion to be multiplied by this can be fixed to the value listed in the table. If the coefficient portion is prepared as a function expression or a table, a desired zooming operation according to the contents of the table can be performed.

  FIG. 45 is a block diagram showing a detailed configuration of the transition period charge unit 172 shown in FIG. As shown in the figure, the transition period charge unit 172 includes an actual cutting center point determination unit 70, an actual plane inclination angle determination unit 80, and an actual magnification determination unit 90.

  The actual cutting center point determination unit 70 determines the value of the actual cutting center point P that changes stepwise from the point P0 to the point Pn based on the standard cutting center point P0 and the target cutting center point Pn. Execute the process to determine. The specific method is as described as the method of the above-mentioned constant speed panning or inconstant speed panning. Here, the panning speed can be controlled based on a predetermined function.

  In short, in order to perform panning at a desired speed, all or part of the transition period is set as the panning period, and during this panning period, the cutting center point P is changed from the standard cutting center point P0 to the target cutting center. When changing to the point Pn step by step, the corresponding point Q on the virtual spherical surface with respect to the cutting center point P is set on a predetermined spherical movement path set in advance as a predetermined spherical surface set as a function related to time What is necessary is just to determine the position of the cutting center point P so that it may move at a moving speed.

  For example, as shown in FIG. 46, the actual cut center point determination unit 70 is configured by a function storage unit 71 and a corresponding point conversion unit 72, and the function storage unit 71 includes a movement distance of the corresponding point Q on the virtual sphere. A function indicating the relationship between b and time t is stored, and a point on the XY plane corresponding to the “position of the corresponding point Q at time t” obtained by the corresponding point conversion unit 72 using the function is By outputting as the cut-out center point P at time t, panning can be performed at a desired speed according to the prepared function.

  More specifically, if a function expression corresponding to the graph shown in the lower left of FIG. 40 or a numerical table corresponding to the coefficient in the distance b column of FIG. Panning can be performed. Further, if the function expression corresponding to the graph shown in the lower left of FIG. 41 and the numerical value table corresponding to the coefficient in the distance b column of FIG. 43B are stored in the function storage unit 71, the start part and the end part It is possible to perform non-uniform speed panning in which the speed of is slower than the speed of the intermediate portion.

  On the other hand, the actual magnification determining unit 90 executes a process of determining the value of the actual magnification m that changes stepwise so as to shift from the magnification m0 to the magnification mn based on the standard magnification m0 and the target magnification mn. The specific method is as described as the above-mentioned method of constant speed zooming and unequal speed zooming. Here, the zooming speed can be controlled based on a predetermined function.

  For example, as shown in FIG. 47, the actual magnification determination unit 90 is configured by a function storage unit 91 and a function reference unit 92, and the function storage unit 91 stores a function indicating the relationship between the magnification m and the time t. If the function reference unit 92 outputs the magnification at the time t obtained using the function as the magnification m at the time t, zooming can be performed at a desired speed according to the prepared function. It becomes possible.

  Specifically, if a function expression corresponding to the graph shown in the upper part of FIG. 42 and a numerical table corresponding to the coefficient in the magnification m column of FIG. It becomes possible to do. Further, if the function expression corresponding to the graph shown in the lower part of FIG. 42 and the numerical value table corresponding to the coefficient in the magnification m column in FIG. It is possible to perform unequal-speed zooming in which the speed is lower than that of the intermediate portion.

<8-4. Determination of plane inclination angle φ>
Finally, consider a method for determining the plane inclination angle φ. The actual plane inclination angle determination unit 80 shown in FIG. 45 determines the value of the actual plane inclination angle φ that changes stepwise so as to shift from the angle φ0 to the angle φn based on the standard plane inclination angle φ0 and the target plane inclination angle φn. Execute the specified process. However, in practice, it is not necessary to perform the process of changing the plane inclination angle φ stepwise. The reason will be described below.

  In the fish-eye monitoring system 100 shown in FIG. 28, the image cut-out conversion unit 150 requires the plane inclination angle φ as one of the actual cut-out conditions in the image conversion formula shown in FIG. This is because φ is used as a parameter indicating the direction. As shown in FIG. 9, the plane inclination angle φ is a parameter that determines the direction of the U axis (horizontal axis of the plane regular image) of the two-dimensional UV orthogonal coordinate system. That is, the two-dimensional UV orthogonal coordinate system is defined as a coordinate system in which the origin is a point G that is a distance m · R away from the origin O on the line-of-sight vector n, and the UV plane is orthogonal to the line-of-sight vector n. A rotation factor as a rotation axis is defined as a plane inclination angle φ. Specifically, the plane inclination angle φ is defined as “the angle between the rotation reference axis J and the U axis”, and the “rotation reference axis J” is “passing through the origin G, parallel to the XY plane, and the line-of-sight vector n. Is defined as an axis orthogonal to.

  Therefore, first, consider a case where the video camera with fisheye lens 30 is attached to the side wall of the building 20 as in the example shown in FIG. In this case, the imaging plane of the camera, that is, the XY plane (the plane on which the distorted circular image S is formed) is a plane along the real world vertical plane. Here, for convenience of explanation, it is assumed that the camera is installed in such a direction that the X axis is in the vertical direction of the real world and the Y axis is in the horizontal plane direction of the real world. As a result, an external image as shown in FIG. 48 is formed on the XY plane, which is the imaging surface.

  In the captured image shown in FIG. 48, the vertical axis W in the real world at the position of the origin O is the direction directly below the figure, and coincides with the X axis. Here, an angle formed by the Y axis with respect to the vertical axis W is defined as ξ. As shown in the figure, since the vertical axis W and the Y axis are orthogonal to each other, if the angle ξ is defined in the clockwise direction with respect to the vertical axis W, in the illustrated example, ξ = −90 °. (The Y-axis is at a position obtained by rotating the vertical axis W by 90 ° counterclockwise).

  Here, let us consider that a portion of the cutout region E is cut out from the distorted circular image shown in FIG. 48 and converted into a planar regular image. The position of the cutout area E is designated by the cutout center point P, the size is designated by the magnification m, and the direction is designated by the plane inclination angle φ. As described above, when a rectangular planar regular image is obtained, the cutout region E is not a rectangle but a distorted shape, but here, for convenience of explanation, a rectangular cutout region E is drawn, and its long side The direction indicates the U-axis direction. In the illustrated example, the long side direction of the cutout region E, that is, the direction of the U axis is parallel to the Y axis. The reason for determining the cutout direction so that the U axis is parallel to the Y axis is to cut out a planar regular image in a direction in which the direction of the vertical axis W on the captured image is a downward direction. If it is cut out in such a direction, as shown in FIGS. 21 (a) and 21 (b), it is possible to always obtain a planar regular image with the direction of the vertical axis W as the downward direction.

  Of course, the orientation of the obtained planar regular image can be set arbitrarily, so it is possible to display the real world scenery upside down or diagonally on the monitor screen. An image required for typical monitoring applications will be an image with the direction of the vertical axis W in the downward direction. Therefore, in practice, the orientation when cutting out a planar regular image is “the orientation in which an image with the direction of the vertical axis W as the downward direction is obtained”.

  In this way, if the basic condition of “always cut out in such a direction that a plane regular image with the correct top-and-bottom relationship is obtained on the monitor screen” is set (in practice, such a basic condition is set. 28) In the fish-eye monitoring system shown in FIG. 28, the standard cut-out condition storage unit 180 has a standard plane inclination such that a flat regular image is cut out in a direction with the vertical axis direction of the real world in the captured image as a downward direction. The angle φ0 is set in advance, the moving object detection unit 160 sets a target plane inclination angle φn such that a plane regular image is cut out in a direction with the vertical axis direction as a downward direction, and the moving object tracking unit 190 sets the vertical axis direction The tracking plane inclination angle φ (i) may be set so that a planar regular image is cut out in a direction in which is downward. Further, the transition period charge unit 172 in the actual cut condition determination unit 170 changes the plane tilt angle φ from the standard plane tilt angle φ0 to the target plane so that the plane regular image is cut out in the direction with the vertical axis direction as the downward direction. What is necessary is just to make it change to inclination-angle (phi) n.

  Therefore, consider what value the value of “planar inclination angle φ such that a planar regular image is cut out in a direction with the vertical axis direction as the downward direction”. As described above, the plane inclination angle φ is “the angle between the rotation reference axis J and the U axis”, and the “rotation reference axis J” is “passing through the origin G, parallel to the XY plane and orthogonal to the line-of-sight vector n”. "Axis to do". Therefore, in the example shown in FIG. 48, the plane inclination angle φ is defined as the angle formed between the rotation reference axis J and the U axis, as shown. Originally, the rotation reference axis J and the U axis are defined as vectors starting from the origin G as shown in FIG. 9, but in FIG. 48, for the sake of convenience of explanation, the starting points of these vectors are extracted as the cut center point P. It shows the state of translation up to. Further, although the rotation reference axis J is an axis parallel to the XY plane, the U axis is not necessarily an axis parallel to the XY plane. Therefore, the U axis shown in FIG. 48 is not an original U axis but a cutting center. The axis corresponding to the U axis at the point P (the U axis direction when converted to a planar regular image) is indicated.

  Here, the “rotation reference axis J” is “an axis passing through the origin G, parallel to the XY plane and orthogonal to the line-of-sight vector n”, and the radial axis R shown in FIG. 48 is on the XY plane of the line-of-sight vector n. 48, the “rotation reference axis J after translation” shown in FIG. 48 is an axis orthogonal to the radial axis R. On the other hand, the radial axis R (orthographic projection of the line-of-sight vector n) is arranged in the direction indicated by the azimuth angle α with respect to the Y axis, as shown in FIG. The relationship of φ = −α + 90 ° is established. That is, no matter where the cut center point P is located on this distorted circular image, “the direction in which an image with the direction of the vertical axis W as a downward direction is obtained” (in other words, the U axis shown in the figure). In order to cut out a planar regular image in such a direction that is parallel to the Y-axis), it can be seen that the setting of the plane inclination angle φ = −α + 90 ° may be made.

  For example, when the cutting center point P is taken on the Y-axis in the figure, the azimuth angle α = 0 °, so that the plane inclination angle φ = + 90 °. With respect to the cutting center point P on the Y axis, the rotation reference axis J is defined in a direction parallel to the X axis. Therefore, if the plane inclination angle φ is set to + 90 °, the U axis shown is parallel to the Y axis. Is set to the correct direction. Similarly, when the cut center point Pn is taken on the X axis in the figure, the azimuth angle α = 90 °, so the plane inclination angle φ = 0 °. As for the cutting center point P on the X axis, the rotation reference axis J is defined in a direction parallel to the Y axis. Therefore, if the plane inclination angle φ is set to 0 °, the U axis shown in the figure is parallel to the Y axis. Is set to the correct direction.

  Thus, in the case of the example shown in FIG. 48, if the setting of the plane inclination angle φ = −α + 90 ° is performed, the plane always having the direction of the vertical axis W as the downward direction regardless of the position of the cutting center point P. A regular image is obtained. Therefore, as shown in FIG. 37, the azimuth angle α is constant even when the cutting center point P is moved from the point P0 to the point Pj along the radial axis (movement path A) during the transition period. The plane inclination angle φ may be constant (φ = −α + 90 °).

  As described above, as illustrated in FIG. 48, the case where the video camera with a fisheye lens is installed so that the imaging surface matches the vertical plane and the X axis matches the real world vertical axis W has been described. Next, consider a case where the installation is performed such that the X axis is inclined. The example shown in FIG. 49 is the same as the example shown in FIG. 48 in that the video camera is installed so that the imaging surface coincides with the vertical plane, but the X axis and the vertical axis W do not coincide with each other. This is an example. In other words, when the angle formed by the Y axis with respect to the vertical axis W in the real world is ξ, FIG. 48 is an example in which ξ = −90 °, whereas FIG. This is an example of setting an arbitrary angle.

  As apparent from FIG. 49, since the X axis and the vertical axis W do not coincide with each other, the distorted circular image obtained on the XY plane is an image arranged obliquely with respect to the X axis and the Y axis. Become. Even in this case, the radial axis R is given by the azimuth angle α, and the rotation reference axis J becomes an axis orthogonal to the radial axis R. Therefore, in order to cut out a planar regular image in the “direction in which an image with the direction of the vertical axis W as the downward direction is obtained”, φ between the plane inclination angle φ and the azimuth angle α via the angle ξ = −α−ξ is established. That is, regardless of where the cut center point P is located on the distorted circular image, the plane inclination angle φ = −α−ξ may be set. The example shown in FIG. 48 is nothing but a special example in which ξ = −90 ° in the general example shown in FIG.

  Eventually, when the video camera with a fisheye lens is installed so that the XY plane as the imaging surface coincides with the vertical plane and the Y axis is oriented in an angle ξ with respect to the vertical axis W, the plane inclination angle φ is If the setting is always such that φ = −α−ξ, a plane regular image having a correct top-to-bottom relationship is always obtained on the monitor screen.

  Next, let us consider a case where installation is performed such that the imaging surface (XY plane) of the camera is a horizontal plane. For example, the example shown in FIG. 50 is an image in which a camera is installed on the ground and a surrounding landscape is photographed. In this case, the imaging surface is a surface parallel to the ground. In the case of a general fisheye monitoring system, there will be more cases where the camera is installed on the ceiling than the case where the camera is installed on the ground or floor. The installation form is common, and the method for determining the plane inclination angle φ is also common.

  In the example shown in FIG. 50, for example, consider a case where a cutout region E is set in the vicinity of the cutout center point P to obtain a planar regular image of a part of a building. In this case, it is necessary to set the U axis in the direction shown in the figure in order to satisfy the basic condition that “a plane regular image with the correct top-and-bottom relationship is always cut out on the monitor screen”. . Here, the rotation reference axis J at the position of the cut-out center point P is an axis orthogonal to the radial axis R, and the U axis coincides with the rotation reference axis J, so that the plane inclination angle φ = 0 °.

  FIG. 51 is a plan view showing the horizontal direction in the real world of the captured image when the video camera with a fisheye lens is installed so that the imaging surface coincides with the horizontal plane. The circle C shown in the figure shows contour lines in the real world. This contour line C indicates the position of a predetermined plane parallel to the horizontal plane in the real world, and in this distorted circular image S, the horizontal direction of the external scene is the circumferential direction of the circle shown as the contour line C. Become. Therefore, for example, the horizontal direction of the scenery in the vicinity of the cutout center point Pa shown in the figure is the direction of the tangent τa at the point Pa of the circle C, and the horizontal direction of the scenery in the vicinity of the cutout center point Pb in the figure is The direction of the tangent τb at the point Pb of C. Since these tangent lines are naturally orthogonal to the radial axis, they all face the same direction as the rotation reference axis J. For this reason, in order to set the U axis in these tangential directions, the plane inclination angle φ should always be set to 0 °.

  Eventually, when the video camera with a fisheye lens is installed so that the XY plane as the imaging plane coincides with the horizontal plane, it is always set so that φ = 0 ° regardless of the position of the cut-out center point P. In this case, a planar regular image with a correct top-to-bottom relationship is always obtained on the monitor screen.

  In practical applications for fisheye monitoring systems, it is common practice to install cameras with the imaging plane aligned with either the vertical plane or the horizontal plane, and a flat regular image with the correct top-to-bottom relationship is always displayed on the monitor screen. It is common to obtain above. As long as it is used in such a general application, the plane inclination angle φ with respect to the position of the specific cut-out center point P is “φ = −α−ξ” (when the imaging surface is a vertical surface) or “φ = It can be uniquely calculated as “0 °” (when the imaging surface is a horizontal surface). In particular, when the imaging surface is a horizontal plane, the fixed value of φ = 0 ° is always set regardless of the position of the cut-out center point P. Therefore, the moving object detection unit 160 does not need to set the target plane inclination angle φn. The moving body tracking unit 190 does not need to set the tracking plane inclination angle φ (i).

  It should be noted that in design practice, when the cut-out center point P is located at the origin O of the XY plane (the center point of the distorted circular image), the value of the plane inclination angle φ is not determined. . The plane inclination angle φ is defined as “an angle formed by the rotation reference axis J and the U axis”. However, when the cut center point P is located on the origin O, the rotation reference axis J cannot be defined. That is, the rotation reference axis J is “an axis passing through the origin G, parallel to the XY plane and orthogonal to the line-of-sight vector n”, but when the cut-out center point P is located on the origin O, the line-of-sight vector n is Z It coincides with the axis, and the origin G is a point on the Z axis. For this reason, there are innumerable “axes that pass through the origin G and are parallel to the XY plane and perpendicular to the line-of-sight vector n (Z axis)”, and a specific direction cannot be defined by the rotation reference axis J. Actually, when the cut center point P is located on the origin O, the zenith angle β = 0, but the azimuth angle α and the plane inclination angle φ cannot be defined. For this reason, the conversion formula shown in FIG. 17 cannot be applied as it is, and an exceptional calculation needs to be performed.

  Therefore, it is necessary to specify the cutting direction by another method, exceptionally only when the cutting center point P is located on the origin O. For example, as in the example shown in FIG. 49, when the camera is installed so that the image pickup surface is a vertical surface, the cut-out direction with respect to the cut-out center point P is “actual at the position where the V-axis is the origin O of the captured image. What is necessary is just to set to "the cutting direction that a plane regular image is cut out so as to correspond to the world vertical axis W".

  Eventually, as in the example shown in FIG. 22, when the standard cut-out center point P0 is set on the origin O of the two-dimensional XY rectangular coordinate system, the standard cut-out condition storage unit 180 stores the V-axis in the captured image. The moving object detection unit 160 that detects the moving object 60 is configured so that a standard cutting condition indicating a cutting direction in which a planar regular image is cut out along the vertical axis in the real world at the position of the origin O is set. When the azimuth angle about the cut center point Pn is αn, the target plane inclination angle φn = −αn−ξ is set (ξ is the Y axis with respect to the vertical axis W in the real world as described above. The actual cutting condition determination unit 170 may always set the plane inclination angle φ = φn during the transition period.

  In the column of the plane inclination angle φ in the table shown in FIG. 38, the standard plane inclination angle is described as φ0 for convenience, but actually, the plane inclination angle is about the standard cutting center point P0 located on the origin O. Since it cannot be defined, it is necessary to define the cutting direction in the standard cutting condition by another method (for example, in the example shown in FIGS. 48 and 49, the angle formed by the U axis and the Y axis).

<<< §9. Various modifications >>
Here, the modification about the fish-eye monitoring system which concerns on one Embodiment of this invention described so far is described.

<9-1. Arbitrary setting of standard cutting center point>
In the embodiments described so far, the standard cut-out center point P0 is set to the position of the center point of the distorted circular image S (that is, the origin O of the XY coordinate system). , A parameter that specifies the standard position of the planar regular image displayed on the monitor device 120 during the static monitoring period W1 (while no moving object is detected within the imaging range), and an arbitrary position on the distorted circular image S Can be set.

  For example, in the example shown in FIG. 20, the standard cut-out center point P0 is set at the center of the distorted circular image S. Therefore, during the static monitoring period W1, as shown in FIG. A front image is displayed on the monitor screen. On the other hand, if it is desired to display the vicinity of the guardrail 50, the position of the standard cutting center point P0 may be corrected downward. If an arbitrary coordinate value is stored in advance in the standard cutting condition storage unit 180 shown in FIG. 28 as the position of the standard cutting center point P0, the arbitrary coordinates during the static monitoring period W1. A planar regular image cut out from the vicinity of the position indicated by the value can be displayed on the monitor screen.

  When the moving object is detected, as described above, the transition period charge unit 172 shown in FIG. 36 changes the actual cutting center point P from the standard cutting center point P0 toward the target cutting center point Pn. It is necessary to perform a panning process that moves the target. Here, when the standard cut center point P0 is set to the origin O (the center of the distorted circular image S) of the two-dimensional XY orthogonal coordinate system, as shown in FIG. The cut center point P may be moved along the radial movement path A of the distorted circular image S toward Pn. In this way, panning for moving the cut center point P in the radial direction from the center of the circle can be executed by a relatively simple arithmetic process. Actually, the constant speed panning using the formula shown in FIG. 40 and the inconstant speed panning using the formula shown in FIG. 41 do not require so complicated calculation processing.

  However, when the standard cut-out center point P0 is set at an arbitrary position other than the origin O of the two-dimensional XY orthogonal coordinate system, along the movement path directly from the arbitrary position to the target cut-out center point Pn, In order to perform panning for moving the point P, a considerably complicated calculation process is required. In order to avoid such complicated arithmetic processing, a panning path via the origin O may be taken.

  For example, as shown by a broken line path in FIG. 52, first, as the previous stage of the panning period, the cut center point P is set along the radius of the distorted circular image S from the standard cut center point P0 to the origin O. After the movement, as a subsequent stage of the panning period, the cut center point P may be moved along the radius of the distorted circular image S from the origin O toward the target cut center point Pn. The panning path indicated by a broken line in FIG. 52 is longer than the path directly from the point P0 to the point Pn, but the calculation process is simplified.

  However, if the system does not have a problem with the burden of arithmetic processing, it is preferable to take a panning path that goes directly from the point P0 to the point Pn, as shown by a broken line in FIG. In general, it is efficient to take the shortest path for the panning path in the transition period after the moving object detection is performed. That is, it is preferable to present the detection moving object as quickly as possible while clearly recognizing the spatial positional relationship between the images before and after switching. Therefore, if there is no problem in the calculation processing speed of the system, it is preferable to adopt the direct panning path as shown in FIG. 53 rather than the panning path via the origin as shown in FIG. It is preferable to do this.

  However, the “shortest path” mentioned here should not be “the shortest path in the fish-eye video space” but “the shortest path in the real video space”. For example, in the case of the example shown in FIG. 53, the “shortest path on the fisheye image space” from the point P0 to the point Pn is the “shortest path on the XY plane”, so the two points P0 and Pn are connected. The line segment becomes the shortest path. However, if the cut-out center point P is moved along such a line segment, the result of panning a video camera equipped with a normal lens along a distorted path in the real image space is obtained. From the viewpoint of the user observing the screen, it appears that an unnatural panning operation has been performed.

  In the example shown in FIG. 53, the panning path indicated by the broken line is not the shortest path on the XY plane, but the movement locus of the corresponding point on the virtual sphere is the shortest path. The result of panning along the “shortest path in the real video space” is obtained, and it appears to the user observing the monitor screen that a natural panning operation has been performed. Specifically, a virtual spherical surface H is arranged on the distorted circular image S shown in the figure, the corresponding point Q0 on the virtual spherical surface H with respect to the standard cut-out center point P0, and the virtual spherical surface H with respect to the target cut-out center point Pn. Corresponding point Qn, and the movement path of the corresponding point Q on the virtual spherical surface H with respect to the cut-out center point P on the virtual spherical surface H corresponds to the corresponding point Q0 and the corresponding point Qn on the virtual spherical surface H. What is necessary is just to make it become the shortest path | route to connect.

  Based on the above-described conditions, the path for moving the cut center point P from the point P0 to the point Pn can be uniquely obtained by geometric calculation. This arithmetic expression is a fairly complicated expression as described above, but will be briefly described below with reference to FIGS. 54 and 55.

  First, consider a three-dimensional model as shown in the upper part of FIG. This figure is a perspective view showing a state in which a virtual spherical surface H is arranged on the distorted circular image S on the XY plane. On the distorted circular image S, a point P0 and a point Pn are defined, and a path as indicated by a broken line is drawn between the two points. In order to determine the route indicated by the broken line, the coordinate value of an arbitrary point Pi on the route (i-th point when the cut center point P is moved stepwise from the point P0 to the point Pn). Find what you need.

  Therefore, first consider the corresponding points Q0 and Qn on the virtual spherical surface H for the two points P0 and Pn, respectively. As described above, in the case of the distorted circular image S photographed using the fisheye lens of the orthogonal projection method, the corresponding points Q0 and Qn are points directly above the points P0 and Pn. Next, a cut surface S3 of the phantom spherical surface H passing through the two corresponding points Q0 and Qn and the origin O is defined. Since the cut surface S3 is a surface passing through the origin O, it becomes a circular cut surface with a radius R. Then, a normal vector n that is orthogonal to the cutting plane S3 and that has the origin O as a starting point is defined.

  Subsequently, an arbitrary point Qi which is a point on the circumference of the cut surface S3 and is located between the point Q0 and the point Qn is defined, and a point Pi having the point Qi as a corresponding point is defined on the XY plane. , The coordinate value of the point Pi is determined based on the coordinate value of the corresponding point Qi. That is, in the case of the distorted circular image S photographed using the orthographic fisheye lens, the corresponding point Qi is a point directly above the point Pi, and therefore the X coordinate value and the Y coordinate value of the corresponding point Qi are not changed. It becomes the coordinate value of the point Pi. Eventually, if the coordinate value of the corresponding point Qi is obtained, the coordinate value of the target point Pi is obtained.

  Therefore, first, as shown in the lower part of FIG. 54, the coordinate value of the point Q0 is set to Q0 (x0, y0, z0), and the coordinate value of the point Qn is set to Qn (xn, yn, zn). These coordinate values are known values because they can be obtained from the coordinate values of the points P0 and Pn. Further, the coordinate value of the point Qi to be obtained is assumed to be Qi (xi, yi, zi). Further, a vector from the point O to the point Q0 is a vector V0, a vector from the point O to the point Qn is a vector Vn, a vector from the point O to the point Qi is a vector Vi, and a vector component of the normal vector n is n (nx , Ny, nz).

Then, as shown in FIG. 55, Expression (10) is derived from the outer product formula. Further, the angle θ formed by the vector V0 and the vector Vn is expressed by Expression (11) depending on the size of the outer product, and is calculated by Expression (12). Here, assuming that constant velocity panning is performed and the i-th point when the arc Q0Qn is equally divided into n is a point Qi, an angle θi formed by the vector Vi and the vector V0 is expressed by Expression (13). Is done. If the point Q0 is rotated by an angle θi along the circumference of the circular cut surface S3, the point Qi is brought to the position of the point Qi. Consequently, the coordinate value (xi, yi, zi) of the point Qi is the Rodriguez formula. Therefore, it is obtained by the equation (14). In addition, when performing non-uniform speed panning, the “i / n” portion of the equation (13) may be set to a desired value corresponding to i.

<9-2. Handling of multiple moving objects>
In the fish-eye monitoring system according to the present invention, as shown in the flowchart of FIG. 35, when the moving object is detected in the static monitoring period W1, the moving period is shifted to the dynamic monitoring period W3 through the transition period W2, and the moving object tracking is performed. Is called. When it is determined that the moving object has disappeared, processing for returning to the static monitoring period W1 is performed again. However, the basic processing procedure described so far is a procedure established on the assumption that there is only a single moving object in the imaging region, and in the specific example of the moving object detection processing described in §6, When there are a plurality of regions of interest that are candidates for moving objects, only the region of interest with a larger area is considered.

  However, if processing is performed in consideration of each of the plurality of regions of interest, the moving object detection unit 160 can detect a plurality of moving objects, and the moving object tracking unit 190 can track the plurality of moving objects. Here, how to handle a plurality of moving objects will be described based on several policies.

  The first handling policy is the simplest policy in which only the first detected moving object is considered and the second and subsequent moving objects are ignored until the moving object disappears. When adopting this policy, the moving object detection unit 160 may pause the new moving object detection process while the moving object tracking process by the moving object tracking unit 190 is being performed. FIG. 56 is a time chart showing the first handling policy.

  In each of the three bars, the horizontal axis represents the time axis, and each represents a time change of a unique event. First, the upper bar shows the transition of the moving object (that is, the moving object to be displayed on the planar regular image as the object to be extracted) included in the actual extraction region at each time point. “0” indicates that there is no moving object to be extracted, and indicates a state in which the static monitoring period W1 is maintained. “0 → A” indicates that the transition period W2 is in progress because the moving object A is detected, and “A” indicates that the dynamic monitoring period W3 for tracking the moving object A is in progress. On the other hand, the middle bar indicates whether or not the moving object detection process by the moving object detection unit 160 is performed. “Search” indicates that the moving object detection process shown in FIG. 29 is being executed, for example. "Indicates that the process is paused. The lower bar indicates the status of the tracking target by the moving object tracking unit 190, “Tracking A” indicates that the moving object A is being tracked, and “(Pause)” indicates that the process is paused. Indicates.

  The period from time t0 to t1 in the figure is a static monitoring period W1 in which no moving object detection is performed. There is no moving object to be extracted, and no moving object is displayed on the monitor screen. During this period, the moving object detection unit 160 executes a moving object detection process (search), but the moving object tracking unit 190 pauses the process.

  Here, when the moving object A is detected by the moving object detection unit 160 at the time t1, the transition period W2 is between the times t1 and t2, and a transition process is performed in which the object to be extracted is the moving object A. A panning operation is performed on the monitor screen, and the moving object A is displayed on the screen. On the other hand, when the first moving object A is detected in this way, the moving object detection unit 160 pauses the moving object detection process at that time. On the other hand, the moving object tracking unit 190 receives a report that the moving object A is detected from the moving object detection unit 160, and starts the tracking process of the moving object A.

  Eventually, when the transition process is completed at time t2, since the actual cut center point P has reached the target cut center point Pn, the moving object A is displayed on the monitor screen, and the dynamic monitoring period W3. Starts. That is, the moving object A being tracked by the moving object tracking unit 190 is continuously displayed on the monitor screen. Then, when the moving object disappearance is determined by the moving object tracking unit 190 at time t3, the process returns to the static monitoring period W1, and the same processing as that from time t0 is repeatedly executed.

  According to the first handling policy shown in FIG. 56, when the moving object detection unit 160 first detects the moving object A, the moving object detection process is suspended until the moving object A disappears. No moving object is detected. Therefore, when one moving object is detected, the user is presented with a screen considering only the moving object until the moving object disappears.

  The second handling policy is a policy that always considers the latest detected moving object and always presents the new moving object to the user when a new moving object is detected. When this policy is adopted, the moving object detection unit 160 continues the moving object detection process while the moving object tracking process is performed by the moving object tracking unit 190, and a new moving object that is different from the old moving object that is the target of the moving object tracking process. Is detected, the target cutting condition for the new moving object may be given to the actual cutting condition determination unit 170. When the target cutting condition for the new moving object is given to the actual cutting condition determining unit 170, the transition period charge unit 172 changes the target cutting condition for the new moving object from the tracking cutting condition for the old moving object. The actual cutout condition that changes step by step so as to shift to is determined, and the moving object tracking unit 190 may perform the tracking process for the new moving object. FIG. 57 is a time chart showing the second handling policy.

  The period from time t0 to t1 in FIG. 57 is a static monitoring period W1 in which no moving object detection is performed. There is no moving object to be extracted, and no moving object is displayed on the monitor screen. During this period, the moving object detection unit 160 executes a moving object detection process (search), but the moving object tracking unit 190 pauses the process.

  Here, when the moving object A is detected by the moving object detection unit 160 at the time t1, the transition period W2 is between the times t1 and t2, and a transition process is performed in which the object to be extracted is the moving object A. A panning operation is performed on the monitor screen, and the moving object A is displayed on the screen. However, the moving object detection unit 160 continues the moving object detection process (search) even after the first moving object A is detected. The moving body tracking unit 190 receives the report that the moving body A is detected from the moving body detection unit 160, and starts the tracking process of the moving body A.

  Eventually, when the transition process is completed at time t2, since the actual cut center point P has reached the target cut center point Pn, the moving object A is displayed on the monitor screen, and the dynamic monitoring period W3. Starts. That is, the moving object A being tracked by the moving object tracking unit 190 is continuously displayed on the monitor screen.

  The moving object detection unit 160 continues the moving object detection process even after time t2. In this case, the moving object A is always detected, but if the tracking result report is received from the moving object tracking unit 190, the occupied area of the moving object A on each frame image can be recognized. It is possible to determine whether the detected moving object is a new moving object or the moving object tracking unit 190 is an old moving object being tracked. When only the old moving object is detected, no report is made to the outside. However, when a new moving object is detected, the new moving object is detected by the actual cutting condition determining unit 170 and the moving object tracking unit 190. A report to that effect is made.

  FIG. 57 shows an example in which the moving object detection unit 160 detects the new moving object B at time t3. During the dynamic monitoring period W3 from time t2 to t3, the process of tracking and displaying the old moving object A was performed. When the new moving object B is detected at time t3, the actual cutting condition that has received the report is received. The determination unit 170 executes a transition process for switching the extraction target from the old moving object A to the new moving object B. Accordingly, the illustrated time t3 to t4 is the transition period W2. However, the migration process executed here is slightly different from the migration process described so far.

  That is, the transition process described so far is a process for shifting from the static monitoring period W1 to the dynamic monitoring period W3, and the actual cutting condition determination unit 170 sets the actual cutting condition as the standard cutting condition. In the transition process for switching the object to be extracted from the old moving object A to the new moving object B, the tracking cutting condition for the old moving object B (at time t3) The actual cutting conditions that change stepwise are determined so as to shift from the cutting conditions given by the moving body tracking unit 190 to the target cutting conditions for the new moving body A. In order to move the cutting center point P from an arbitrary position to another arbitrary position, for example, the method described in §9-1 may be used.

  On the other hand, when the moving body tracking unit 190 receives a report indicating that the new moving body B is detected from the moving body detection unit 160 at time t3, the moving body tracking unit 190 switches the tracking target from the old moving body A to the new moving body B. Execute the process. Eventually, when the transition process from the old moving object A to the new moving object B is completed at time t4, the actual cutting center point P has reached the target cutting center point for the new moving object B. The moving object B is displayed above, and the dynamic monitoring period W3 for the new moving object B starts. That is, the new moving object B being tracked by the moving object tracking unit 190 is displayed on the monitor screen. When the moving object tracking unit 190 determines that the moving object B disappears at time t5, the process returns to the static monitoring period W1 again, and the same processing as that from time t0 is repeatedly executed.

  According to the second handling policy shown in FIG. 57, newly detected moving objects are displayed one after another, so that the latest moving object information can always be presented to the user.

  As a variation of this second handling policy, when moving object B is detected, an area comparison with moving object A is performed, and only when the area of moving object B is larger than the area of moving object A Can be switched to the moving object B. If the area of the moving object B is not larger than the area of the moving object A, the object to be cut out is still the moving object A. By doing so, the tracking target is switched only when a moving object having a higher importance than the moving object currently being tracked (here, a moving object having a large area) appears, and the practicality increases. Become.

  When such an operation is employed, the moving object detection unit 160 continues the moving object detection process while the moving object tracking process is performed by the moving object tracking unit 190, and is different from the old moving object that is the target of the moving object tracking process. When another moving object that satisfies the condition that the area is larger than that of the old moving object is detected, the target cutting condition determination unit 170 is provided with a target cutting condition for the new moving object that satisfies the condition. You can do it. As long as the area condition is not satisfied, even if another moving object is detected, the other moving object is not handled as a new moving object.

  In the third handling policy, in addition to the second handling policy described above, the moving object tracking unit 190 executes the simultaneous tracking process of a plurality of moving objects. In the example shown in FIG. 57, when the new moving object B is detected at time t3, the moving object tracking unit 190 stops tracking the old moving object A and starts tracking the new moving object B. The moving object tracking process shown in the flowchart can be performed in parallel for a plurality of moving objects. As described above, when a plurality of moving objects are detected, an advantage of performing the simultaneous tracking process on the plurality of moving objects is that the other moving object can be extracted immediately when one moving object disappears. .

  That is, when the moving object tracking unit 190 receives a report indicating that a new moving object has been detected from the moving object detecting unit 160 while executing the tracking process for a certain moving object, the moving object tracking unit 190 performs the tracking process for the new moving object and the tracking for the old moving object. Processing is also performed. In this case, the tracking cutout condition given to the actual cutout condition determining unit 170 is a tracking cutout condition based on the new moving object. Therefore, the object to be cut out is a new moving object, and the tracking process for the old moving object is a process performed under the surface of the water. Then, when the new moving object disappears earlier than the old moving object, the moving object tracking unit 190 gives the target cutting condition for the old moving object to the actual cutting condition determining unit 170 instead of the moving object detecting unit 160. Can do. Of course, the moving object tracking unit 190 continues the tracking process for the old moving object as it is.

  When the target cutting condition for the old moving object is given from the moving object tracking unit 190 to the actual cutting condition determining unit 170, the transition period responsible unit 172 determines the old moving object from the tracking cutting condition for the new moving object. What is necessary is just to define the actual cutting condition which changes in steps so that it may transfer to target cutting conditions.

  FIG. 58 is a time chart showing the third handling policy. The difference from the time chart shown in FIG. 57 is that when a new moving object B is detected at time t3, the moving object tracking unit 190 tracks the old moving object A in parallel with the tracking of the new moving object B. It is. Then, if the new moving object B disappears at the time t5 and the old moving object A still exists, the moving object tracking unit 190 sends the new moving object B to the actual cutting condition determining unit 170. In addition to reporting the disappearance, it is possible to report the target cutting condition for making the old moving object A a cutting target. The actual cut condition determining unit 170 receives this report, and can perform a shift process for shifting the cut target from the new moving body B to the old moving body A. Therefore, as shown in the figure, the shift period W2 from time t5 to t6 is set. Then, it can transfer to the dynamic monitoring period W3 about the old moving body A. FIG.

  In the above, some handling policies when there are multiple moving objects have been described, but if the moving object to be extracted (displayed) is frequently changed in a short time, it will be very dizzying for the user Image presentation is performed, which is not preferable. Therefore, practically, when the transition period charge unit 172 performs the transition process of switching from the old moving object to the new moving object, the dynamic monitoring period W3 for the old moving object is less than the predetermined minimum reference time tmin. The process is preferably started after a waiting time until the dynamic monitoring period W3 reaches the minimum reference time tmin.

  FIG. 59 is a time chart showing an example in which such a standby time is provided in the case shown in FIG. In the time chart of FIG. 58, since the period of time t2 to t3 is relatively short, the dynamic monitoring period W3 for the moving object A cannot be sufficiently secured, and the panning operation to the position of the moving object A is completed. (Time t2) At a short time, the panning operation to the position of the new moving body B starts (time t3). If the period of time t2 to t3 is a short time such as 1 second, for example, the switching timing of the display object is too early, giving the user a very dizzy impression.

  On the other hand, in the time chart of FIG. 59, even after the time t3 when the new moving body B is detected, the object to be cut out is still maintained as the old moving body A. That is, the moving object tracking unit 190 executes the tracking operation of both the new moving object B and the old moving object A in parallel from time t3. However, the actual cut condition determining unit 170 still retains the old moving object at the timing of time t3. The dynamic monitoring period W3 for A is continued, and the tracking cutout condition for the old moving object A is adopted as the actual cutout condition.

  The actual cutting condition determination unit 170 starts the transition period W2 from the old moving body A to the new moving body B at the time t4 when the minimum reference time tmin has elapsed from the time t2, and waits at the times t3 to t4. The time has been extended. Then, since the minimum reference time tmin can be secured as the dynamic monitoring period W3 for the old moving object A, it is possible to avoid giving a dizzy impression to the user. In the case of the illustrated example, the transition period W2 from the new moving object B to the old moving object A starts at time t6 when the disappearance determination of the new moving object B is made. This is the dynamic monitoring of the new moving object B. This is because the minimum reference time tmin is secured as the period W3 (time t5 to t6). If the period from time t5 to t6 is less than the minimum reference time tmin, the start time of the transition period W2 from the new moving object B to the old moving object A is extended by a predetermined waiting time.

<9-3. Application to non-orthographic projection method>
All of the embodiments so far are examples using an orthographic fisheye lens, and the corresponding relational expression used for image conversion is also based on a distorted circular image S taken with an orthographic fisheye lens. It was. However, a commercially available fisheye lens is not necessarily an orthographic lens. Actually, various methods such as equidistant projection method, stereoscopic projection method, equisolid angle projection method are known as fisheye lens projection methods, and fisheye lenses that use these various projection methods are used depending on the application. ing. Here, a method for applying the present invention to a non-orthographic image captured by such a non-orthogonal fish-eye lens will be described.

  The optical characteristics of the orthographic fisheye lens can be explained by a model as shown in FIG. That is, with respect to an arbitrary incident point H (x, y, z) on the phantom spherical surface H, an incident light beam L1 incident from the normal direction is an XY plane as an incident light beam L2 traveling in a direction parallel to the Z axis. The characteristic is that the upper point S (x, y) is reached. FIG. 60 is a front view showing a projection state of incident light in this orthographic fisheye lens. As shown in the figure, an incident ray L1 incident from the normal direction to an incident point H (x, y, z) on the virtual sphere H having a zenith angle β is incident in a direction parallel to the Z axis. As a light ray L2, it reaches a point S (x, y) on the XY plane.

  Here, the zenith angle β of the incident point H (x, y, z) and the arrival point S (x, y) where the incident light beam L2 passing through the incident point H (x, y, z) reaches on the XY plane. In the case of an orthographic image formed by photographing using an orthographic fisheye lens, the relationship between the distance r from the origin O is expressed by the equation r = f · sinβ. Here, f is a constant specific to the fisheye lens. On the other hand, for example, in the case of an equidistant projection image formed by photographing using a fisheye lens of an equidistant projection method, the relationship between the two is expressed by the equation r = f · β.

  FIG. 61 is a perspective view showing the relationship between an orthographic projection image formed using an orthographic fisheye lens and an equidistant projection image formed using an equidistant projection fisheye lens. Concentric lines in the figure show a set of arrival points on the XY plane for incident light rays that have passed through the incident point having the same zenith angle β. In the case of the orthogonal projection image shown in FIG. 61 (a), the equation r = f · sinβ is established, and therefore the arrangement interval between adjacent concentric lines decreases from the center toward the periphery. On the other hand, in the case of the equidistant projection image shown in FIG. 61 (b), the equation r = f · β holds, and therefore concentric lines are arranged at equal intervals from the center toward the periphery.

  Thus, although the orthographic projection image and the equidistant projection image are both common in terms of a distorted circular image, the distortion state is different between the two, so of course, to convert it into a planar regular image The conversion formula used is also different. Therefore, when converting an orthographic projection image into a planar regular image, the correspondence relation for orthographic projection image shown in FIG. 17 can be used, but a non-orthogonal projection image such as an equidistant projection image is converted into a planar regular image. When doing so, it is necessary to use a dedicated correspondence relation for each.

  However, the coordinates on the orthographic image and the coordinates on the non-orthographic image can be converted into each other via a predetermined coordinate conversion formula. Therefore, in implementing the present invention, when the distorted circular image S stored in the distorted circular image memory 130 is a non-orthographic image captured by a non-orthogonal fisheye lens, What is necessary is just to use the non-orthogonal image correspondence equation obtained by correcting the orthographic image correspondence equation using a coordinate conversion equation between the coordinates of the image and the coordinates on the non-orthogonal image. Hereinafter, this method will be described by taking an example of using an equidistant projection image as a non-orthographic projection image.

As shown in FIG. 61, an arbitrary point on the orthographic projection image and a specific point on the equidistant projection image can define a one-to-one correspondence with each other (the broken line in the figure is Shows this correspondence). Specifically, if the coordinate of an arbitrary point on the equidistant projection image is (xb, yb) and the coordinate of a specific point on the orthographic image corresponding to this is (xa, ya), As shown in the upper part of FIG.
xa = xb (R / r) sin (πr / 2R) Equation (15)
ya = yb (R / r) sin (πr / 2R) Equation (16)
However, r = √ (xb ² + yb ² )
The following formula holds. Conversely, if the coordinate of an arbitrary point on the orthographic projection image is (xa, ya) and the coordinate of a specific point on the equidistant projection image corresponding to this is (xb, yb), As shown in the lower part of FIG.
xb = xa (2R / πr) sin ⁻¹ (r / R) Equation (17)
yb = ya (2R / πr) sin ⁻¹ (r / R) Equation (18)
However, r = √ (xa ² + ya ² )
Holds.

  Eventually, the above formulas (15) and (16) are the formulas for converting the coordinates (xb, yb) on the equidistant projection image into the coordinates (xa, ya) on the orthographic projection image (hereinafter referred to as the first coordinate conversion formula). The above formulas (17) and (18) are the formulas (hereinafter referred to as second formulas) for converting the coordinates (xa, ya) on the orthographic projection image into the coordinates (xb, yb) on the equidistant projection image. This is called the coordinate conversion formula.

  Therefore, when the image stored in the distorted circular image memory 130 shown in FIG. 28 is an equidistant projection image photographed by an equidistant projection fisheye lens, The corresponding coordinates (x, y) corresponding to the desired coordinates (u, v) may be calculated using the orthographic image corresponding relational expressions described so far. Since the corresponding coordinates (x, y) calculated in this way correspond to the coordinates (xa, ya) on the orthographic image, this is converted into the equidistant projection image using the second coordinate conversion formula described above. The upper coordinates (xb, yb) may be converted, and the converted coordinates (xb, yb) may be used as coordinates used when the distorted circular image memory 130 is actually accessed.

  In the models shown in FIGS. 9, 39, 54, etc., attention should be paid to the relationship between the cut-out center point P on the distorted circular image S and the corresponding point Q on the virtual sphere H corresponding thereto. is there. When the distorted circular image S stored in the distorted circular image memory 130 is an orthographic image captured by an orthographic fisheye lens as in the embodiments described so far, the model shown in FIG. If an intersection point Q (xp, yp, zp) between a straight line passing through the cutting center point P (xp, yp) and parallel to the Z axis and the virtual spherical surface H is defined, the intersection point Q becomes the cutting center point. It becomes a corresponding point of P. This is because the corresponding point Q on the phantom spherical surface H is located at a position directly above the cut-out center point P in the case of the orthogonal projection method.

  However, when the distorted circular image S stored in the distorted circular image memory 130 is a non-orthographic image captured by a non-orthogonal fish-eye lens, the corresponding point Q on the virtual sphere is the cut-out center. It is not located directly above the point P. Therefore, in this case, the coordinate of the cut-out center point P is corrected using a coordinate conversion formula between the coordinates on the orthographic image and the coordinates on the non-orthographic image, and the corrected point is passed along the Z axis. The point of intersection between the parallel straight line and the phantom spherical surface H needs to be the corresponding point Q.

  For example, when the distorted circular image S is an equidistant projection image, the coordinates of the cut-out center point P designated as one point on the distorted circular image S correspond to the coordinates (xb, yb) on the equidistant projection image. Therefore, using the first coordinate conversion formula described above, this is converted into coordinates (xa, ya) on the orthographic projection image, passing through the point indicated by the converted coordinates (xa, ya), and passing through the Z axis. The intersection point between the parallel straight line and the phantom spherical surface H may be set as the corresponding point Q.

  By adopting such a method, basically, even when the image stored in the distorted circular image memory 130 is an equidistant projection image photographed by an equidistant projection fisheye lens, basically, it has been described so far. It is possible to apply a conversion process equivalent to that of the embodiment described above. Of course, this method is not limited to application to equidistant projection images, and can be widely applied to non-orthographic projection images in general.

  After all, in general terms, the image cut-out conversion unit 150 is represented on the virtual spherical surface H by coordinates (xi, yi) on the two-dimensional XY orthogonal coordinate system according to the projection method of the fisheye lens used for photographing. A corresponding point Qi on the spherical surface corresponding to the point Si to be obtained, and on the two-dimensional UV orthogonal coordinate system of the intersection Ti of the straight line connecting the origin O and the corresponding point Qi on the spherical surface and the arrangement surface of the two-dimensional UV orthogonal coordinate system. When the coordinates of (i, vi) are (ui, vi), the distorted circular image S is converted into a planar regular image T by using a correspondence expression in which the coordinates (xi, yi) are obtained as corresponding coordinates corresponding to the coordinates (ui, vi). What is necessary is just to perform the process to convert.

  In particular, when the distorted circular image S stored in the distorted circular image memory 130 is an orthographic image captured by an orthographic fisheye lens, as described in the model shown in FIG. , Yp), a point indicated by coordinates (xp, yp, zp) given as an intersection of a straight line passing through the point P and parallel to the Z axis and the virtual spherical surface H is assumed to be virtual. As a corresponding point Q on the spherical surface H, with respect to a point Si indicated by coordinates (xi, yi), coordinates (xi, yi) given as an intersection of a straight line passing through the point Si and parallel to the Z axis and the virtual spherical surface H , Zi) and the coordinate (x, y) and the coordinate using the orthographic image corresponding relational expression (the relational expression shown in FIG. 17) that is based on the assumption that the corresponding point Qi on the virtual sphere H is the corresponding point. If you convert between (u, v) There.

On the other hand, when the distorted circular image S stored in the distorted circular image memory 130 is a non-orthographic image captured by a non-orthogonal fisheye lens, the coordinates on the orthographic image and the non-orthographic image are displayed. What is necessary is just to use the non-orthogonal image correspondence equation obtained by correcting the orthographic image correspondence equation using the coordinate conversion equation with the upper coordinates. Specifically, when the distorted circular image stored in the distorted circular image memory 130 is an equidistant projection image photographed by an equidistant projection fisheye lens, as shown in the lower part of FIG. Coordinate conversion formula for converting coordinates (xa, ya) on the orthographic projection image to coordinates (xb, yb) on the equidistant projection image xb = xa (2R / πr) sin ⁻¹ (r / R)
yb = ya (2R / πr) sin ⁻¹ (r / R)
However, r = √ (xa ² + ya ² )
May be used to correct the orthographic image correspondence equation.

<9-4. Use of coordinate system placed on curved surface>
In the embodiment described so far, as shown in the model of FIG. 9, the conversion method for obtaining the planar regular image T on the two-dimensional UV orthogonal coordinate system defined on the plane is adopted. However, for the image conversion from the distorted circular image S to the planar regular image T, it is not always necessary to use the two-dimensional UV orthogonal coordinate system defined on the plane, and the two-dimensional UV curved coordinate system defined on the curved surface is used. It is also possible to use it.

  FIG. 63 is a model using a two-dimensional UV curved coordinate system defined on a curved surface instead of the two-dimensional UV orthogonal coordinate system in the model of FIG. The only difference between the two is how to define the UV coordinate system. That is, in the model of FIG. 9, the UV coordinate system is defined on a plane passing through the point G (xg, yg, zg) and orthogonal to the line-of-sight vector n, whereas in the model of FIG. The UV orthogonal coordinate system defined above is curved along the side of the cylinder. Therefore, in the model of FIG. 9, the planar regular image T is obtained on the two-dimensional UV orthogonal coordinate system (coordinate system on the plane), whereas in the model of FIG. 63, the two-dimensional UV curved coordinate system is obtained. A curved regular image C is obtained on the (coordinate system on the cylinder side surface).

  A planar regular image T indicated by a broken line in FIG. 63 is the same as the planar regular image T indicated by a solid line in FIG. 9 and is defined on a plane that passes through the point G (xg, yg, zg) and is orthogonal to the line-of-sight vector n. It is the image on the made two-dimensional UV orthogonal coordinate system. On the other hand, the curved regular image C indicated by the solid line in FIG. 63 corresponds to an image obtained by curving the planar regular image T, and is an image on a curved surface along the cylindrical side surface.

  In short, the two-dimensional UV curved coordinate system defined by the model shown in FIG. 63 is arranged on the curved surface along the side surface of the virtual cylinder at a desired position in the space constituting the three-dimensional XYZ orthogonal coordinate system. The curved regular image C defined on the curved coordinate system is also an image curved along the side surface of the virtual cylinder.

  Since this two-dimensional UV curved coordinate system is also a two-dimensional coordinate system having a U axis and a V axis, an arbitrary point in the curved regular image C is indicated by coordinates (ui, vi). This is the same as the case of a two-dimensional coordinate system on a normal plane. However, this two-dimensional UV curved coordinate system is defined by bending a two-dimensional UV orthogonal coordinate system arranged on a plane orthogonal to the line-of-sight vector n with the point G as the origin along the side surface of the virtual cylinder. It becomes a coordinate system. The virtual cylinder used for this bending process is a cylinder having a central axis parallel to the V-axis of the two-dimensional UV orthogonal coordinate system and having a radius equal to mR (distance between points OG). Is arranged to come. Eventually, when a virtual cylinder satisfying such a condition is arranged on the three-dimensional XYZ orthogonal coordinate system, the central axis of the virtual cylinder becomes an axis V ′ passing through the origin O as shown by a one-dot chain line in FIG. The line segment connecting the two points OG constitutes the radius of this virtual cylinder.

  The principle of obtaining a curved regular image C on such a two-dimensional UV curved coordinate system is exactly the same as the principle of obtaining a planar regular image T on a two-dimensional UV orthogonal coordinate system. That is, one point Ci (ui, vi) on the curved regular image C is associated with one point Si (xi, yi) on the distorted circular image S, and the pixel value of the pixel located at the point Ci (ui, vi) May be determined based on the pixel value of a pixel located in the vicinity of the point Si (xi, yi). For this purpose, a corresponding relational expression that associates an arbitrary coordinate (u, v) on the two-dimensional UV curved coordinate system with a corresponding coordinate (x, y) on the XY plane is required. Such a corresponding relational expression is slightly different from the corresponding relational expression shown in FIG. 17, but can be obtained by considering the geometric position of the side surface of the virtual cylinder in the XYZ three-dimensional coordinate system. Details are disclosed in, for example, the specification of Japanese Patent Application No. 2008-225570 and the like, and a description of a specific correspondence relation formula is omitted here.

  Eventually, the image cut-out conversion unit 150 of the fish-eye monitoring system according to the present invention may be a two-dimensional UV coordinate system (an orthogonal coordinate system defined on a plane or a curved coordinate system defined on a curved surface). Corresponding coordinates (x, x) corresponding to coordinates (u, v) using a predetermined correspondence expression indicating the correspondence between coordinates (u, v) above and coordinates (x, y) in the two-dimensional XY coordinate system , Y) by calculating an image cut out from the distorted circular image S into a planar regular image T.

  In this case, as in the model shown in FIG. 9 or FIG. 63, a corresponding point Q that is a point on the virtual spherical surface H corresponding to the cut-out center point P is taken, and a vector passing through the corresponding point Q starting from the origin O A vector n is defined on a plane perpendicular to the line-of-sight vector n passing through the point G and having a point G on the line-of-sight vector n separated from the origin O by “product m · R of the magnification m and the radius R”. Alternatively, a two-dimensional UV coordinate system may be defined which is arranged on a curved surface obtained by bending the plane with an orientation corresponding to the plane inclination angle φ. Here, the correspondence between the point Q and the point P is that a virtual spherical surface H having a radius R with the origin O as the center in a three-dimensional XYZ orthogonal coordinate system including a two-dimensional XY orthogonal coordinate system in which the distorted circular image S is arranged. When the external light that is defined and is incident on one point on the virtual spherical surface H toward the origin O reaches one point on the XY plane by the optical action of the fisheye lens used for photographing (depending on the lens projection method). Is defined as a relationship between one point Q on the phantom spherical surface H and one point P on the XY plane.

  In the examples described so far, the center point of the obtained planar regular image coincides with the origin G of the UV coordinate system. However, the planar regular image need not necessarily be cut out with the origin G as the center. Absent.

<9-5. Transition period after moving object disappears>
In the embodiment described so far, as shown in the flowchart of FIG. 35, when it is determined that the moving object has disappeared during the tracking of the moving object, the process of returning from the dynamic monitoring period W3 to the static monitoring period W1 is performed. At the time of this return, there is no special transition period, so if it is determined on the monitor screen that the moving object has disappeared, the original standard monitoring screen (standard switching screen) will suddenly start from the screen that was tracking the moving object. The screen changes to a screen showing the periphery of the departure center point P0.

  This standard monitoring screen is a default screen that is displayed on the monitor when no moving object is detected, such as the frame F0 screen shown in FIG. 21 (a). In that case, the position is potentially recognized. Therefore, even when the switching from the dynamic monitoring period W3 to the static monitoring period W1 is suddenly performed and the monitor screen is switched to the standard monitoring screen, the recognition of the spatial positional relationship between the images before and after the switching is confused. It doesn't happen. Rather, there is a merit that an effect of explicitly informing the user of the fact that the moving object being tracked disappears can be obtained by performing a sudden switch to the standard monitoring screen.

  However, when there is a situation in which it is not preferable to perform rapid screen switching, a transition period W4 can be provided when returning from the dynamic monitoring period W3 to the static monitoring period W1. In this case, the actual cutting condition determination unit 170 determines the actual cutting condition that changes step by step so as to shift from the final tracking cutting condition in the dynamic monitoring period W3 to the standard cutting condition in the transition period W4. Just decide. The specific transition process is the same as the transition process in the transition period W2, and the display screen is gradually switched by the panning operation and the zooming operation.

<<< §10. Application to general image presentation system >>
As mentioned above, although embodiment which applied the basic technical idea of this invention to the concrete system called a fish-eye monitoring system was described, the technical idea of this invention is not limited to the application to a fish-eye monitoring system. . That is, this technical idea can be widely used not only for surveillance cameras but also for general image presentation systems. For example, in the field of entertainment such as game devices and game devices, it is also used for technologies for presenting images. Is possible.

  Of course, a video camera with a fisheye lens is not an essential component in an image presentation system used in such a field. For example, any distorted circular image S (which may be an image actually obtained by photographing using a fisheye lens or an image created by CG) is stored in a memory in advance, and an arbitrary clipping condition specified by the user In the case of an image presentation system that cuts out an image from a predetermined location in a predetermined direction at a predetermined magnification and converts the image into a planar regular image and presents the image, every time the user specifies a new extraction condition In addition, a process of shifting from the old cutting condition to the new cutting condition is required, but the panning process and zooming process described in §8 can be applied to such a shifting process.

  In short, the present invention cuts out a part of a distorted circular image obtained by photographing using a fisheye lens or a distorted circular image created by equivalent CG processing, converts it into a planar regular image, and presents it on the screen. It can be widely applied to image presentation systems.

  For example, it is composed of an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY coordinate system, has a radius R around the origin O of the coordinate system, and uses a fisheye lens. A distorted circular image memory for storing a distorted circular image obtained by photographing, and an aggregate of a large number of pixels arranged at positions indicated by coordinates (u, v) on the two-dimensional UV coordinate system. Planar regular image memory for storing a planar regular image, an image display device for displaying a planar regular image stored in the planar regular image memory on a screen, and a cutout center as one point on a distorted circular image Actual cutting condition for determining an actual cutting condition constituted by the point P, a predetermined plane inclination angle φ, and a predetermined magnification m as a condition for cutting a plane regular image from a part of a distorted circular image Decision part and this actual cutting Based on the actual cutting conditions given from the matter determining unit, the cutting direction indicated by the plane inclination angle φ from the cutting position indicated by the cut center point P of the distorted circular image stored in the distorted circular image memory In addition, an image cut-out conversion unit that cuts out an image having a cut-out size indicated by the magnification m, converts the image into a plane regular image, and stores the image in a plane regular image memory. If the panning process according to the present invention is applied, the actual cutting condition determination unit determines the target cutting point including the target cutting center point Pn from the current cutting condition including the current cutting center point P0. When changing to the extraction condition, a process of shifting the position of the extraction center point P given to the image extraction conversion unit from the point P0 to the point Pn in a stepwise manner may be performed.

  Specifically, as in the embodiments described so far, in a three-dimensional XYZ orthogonal coordinate system including a two-dimensional XY orthogonal coordinate system in which a distorted circular image is arranged, an imaginary spherical surface H having a radius R about the origin O. When the outside light incident on one point on the virtual spherical surface H toward the origin O reaches one point on the XY plane by the optical action of the fisheye lens used for photographing, the light on the virtual spherical surface H And a point on the XY plane are defined as points that correspond to each other, a corresponding point Q that is a point on the virtual spherical surface H corresponding to the cut-out center point P is taken, and the origin O is the starting point A vector passing through the point Q is defined as a line-of-sight vector n. A point G on the line-of-sight vector n that is separated from the origin O by a “product m · R of the magnification m and the radius R” is the origin, and the line of sight passes through the point G. Curved on or on a plane orthogonal to vector n And with the direction corresponding to the planar inclination angle φ on the curved surface to define the arranged two-dimensional UV coordinate system. Then, the image cut-out conversion unit uses a predetermined correspondence expression indicating the correspondence between the coordinates (u, v) on the two-dimensional UV coordinate system and the coordinates (x, y) on the two-dimensional XY orthogonal coordinate system. Thus, by calculating the corresponding coordinates (x, y) corresponding to the coordinates (u, v), processing for converting an image cut out from the distorted circular image into a planar regular image may be performed.

  At this time, the actual cutting condition determination unit determines that the corresponding point Q on the virtual spherical surface H with respect to the cutting center point P is a predetermined spherical surface set as a function related to time on a predetermined spherical movement path set in advance. If the position of the cut-out center point P is determined so as to move at a moving speed, a panning operation can be performed at a desired speed in the real image space, so that a natural panning effect is provided to the user. can do.

  If the zooming process according to the present invention is applied in addition to the panning process according to the present invention, the actual cutting condition determination unit gradually changes the position of the cutting center point P from the point P0 to the point Pn. And a process of shifting the value of the magnification m from the current magnification m0 to the target magnification mn in a stepwise manner. Also in this case, if the value of the magnification m is shifted at a predetermined magnification change speed set as a function related to time, a zooming operation at a desired speed in the real video space can be performed. A natural zooming effect can be provided.

10: road surface 20: building 30: video camera with fisheye lens 31: imaging center point 40: tree 50: guard rail 60: moving body 70: actual cutting center point determination unit 71: function storage unit 72: corresponding point conversion unit 80: actual plane Inclination angle determination unit 90: actual magnification determination unit 91: function storage unit 92: function reference unit 100: fisheye monitoring system 110: video camera with fisheye lens 115: image input unit 120: monitor device 125: image output unit 130: distorted circle Image memory 140: Planar regular image memory 150: Image extraction conversion unit 160: Moving object detection unit 161: Background image holding unit 162: Corresponding pixel comparison unit 163: Region-of-interest search unit 164: Moving object detection determination unit 170: Actual cutting Output condition determining unit 171: Static monitoring period responsible unit 172: Transition period responsible unit 173: Dynamic monitoring period responsible unit 180: Standard extraction condition storage unit 185: Manual Case setting unit 190: moving object tracking unit 191: known moving object region holding unit 192: candidate region extracting unit 193: feature amount calculating unit 194: new moving object region recognizing unit 195: moving object tracking determining unit A to F: rotation coefficient (sign in formula) )
A: Movement paths A1 and A2 on the XY plane: A continuous area composed of the pixel of interest a: Horizontal dimension of the monitor device (number of pixels in the horizontal direction)
a, ai, an: Movement distance B along movement path A: Movement path on virtual sphere b: Vertical dimension of monitor device (number of pixels in vertical direction)
b, bi, bn: moving distance along moving path B C: curved regular image on two-dimensional UV coordinate system defined on curved surface / contour line / frame counter value Ci (ui, vi): defined on curved surface Point D in the curved regular image C on the two-dimensional UV coordinate system thus made: reference dimensions E, E1, E2: cutout area E0: standard cutout areas Ei, Ej, Ek: cutout area En during transition En: target Cutout areas E (0), E (i), E (j), E (k): Tracking cutout areas F1, F2: Angles of view F0, F1, Fi, Fj, Fk, Fn: Each frame of the captured image f: Constants G, G1, G2 specific to fisheye lens: Origin G (xp, yp, zp) of two-dimensional UV coordinate system: Origin G (xg, yg, zg) of two-dimensional UV coordinate system: Two-dimensional UV coordinate system Origin H: virtual spherical surface H (x, y, z): incident point i on virtual spherical surface H: integer J, Ji, Jn: Rotation reference axis j: integer k: integer Ki: proportionality constant L1, L2: incident light m: magnification / actual magnification m0: standard magnification m1, mi, mk: magnification during transition mn: target magnification m (i): tracking magnification n: line-of-sight vector / integer n (nx, ny, nz): normal vector ni: straight line connecting O and Qi O: origin P of three-dimensional XYZ orthogonal coordinate system: cutting center point / real cutting center point P (Xp, yp): Cutting center point Pa, Pb: Cutting center point P0: Standard cutting center point Pn: Target cutting center point Pi: Cutting center point Pn during transition Pn: Target cutting center point P ( 0), P (i), P (j), P (k): tracking cut-out center points Q, Q0, Q1, Q2, Qi, Qn: corresponding points on the spherical surface Q (xp, yp, zp): on the spherical surface Corresponding point Q0 (x0, y0, z0): Spherical corresponding point Qi (xi, yi, zi): Spherical corresponding point Qn (xn, yn, z) ): Corresponding point on spherical surface R: radius of distorted circular image S (radius of virtual spherical surface H) / radial axis r: distance from center point of distorted circular image S: distorted circular image / reference area S1: inclined surface S2 : Tangent plane S3: virtual spherical cut surface S (0): background distorted circular image S (C): currently read distorted circular image S (x, y): in distorted circular image S on a two-dimensional XY orthogonal coordinate system Point S1 (x1, y1): point S2 (x2, y2) in the distorted circular image S on the two-dimensional XY orthogonal coordinate system: point Si (xi, xi) in the distorted circular image S on the two-dimensional XY orthogonal coordinate system yi): points S11 to S36 in the distorted circular image S on the two-dimensional XY orthogonal coordinate system: steps T, T1, T2 of the flowchart: planar regular image Ti (ui, vi) on the two-dimensional UV orthogonal coordinate system: Point T (0,0) in the planar regular image T on the two-dimensional UV orthogonal coordinate system: two-dimensional UV Orthogonal coordinate system origin T1 (u1, v1): point T2 (u2, v2) in the planar regular image T on the two-dimensional UV orthogonal coordinate system: point Ti in the planar regular image T on the two-dimensional UV orthogonal coordinate system (Ui, vi): points t, t0 to t7 in the planar regular image T on the two-dimensional UV orthogonal coordinate system, tn: time tmin: minimum reference time U: coordinate axis u of the two-dimensional UV coordinate system u: two-dimensional UV coordinate Coordinate value V related to coordinate axis U of system: Coordinate axes V0, Vi, Vn of two-dimensional UV coordinate system: Vector v: Coordinate value related to coordinate axis V of two-dimensional UV coordinate system W: Real world vertical axis W1: Static monitoring period W2 : Transition period W3: Dynamic monitoring period w: Numerical value X given by m × R: Coordinate axis x, xa, xb of three-dimensional XYZ orthogonal coordinate system: Coordinate value Y regarding coordinate axis X of two-dimensional XY orthogonal coordinate system Y: Three-dimensional XYZ Cartesian coordinate system coordinate axes y, ya, yb: two-dimensional Coordinate value Z regarding coordinate axis Y of Y orthogonal coordinate system Z: coordinate axis α of three-dimensional XYZ orthogonal coordinate system α: azimuth angle β, βi, βn: zenith angle θ, θi: angle between vectors φ: plane inclination angle / real plane inclination angle φ0: standard plane tilt angle φn: target plane tilt angle φ (i): tracking plane tilt angle ξ: angles τa, τb with respect to vertical axis W: tangent

Claims (44)

A fisheye monitoring system that cuts out a part of a distorted circular image obtained by shooting using a fisheye lens, converts it into a planar regular image, and displays it on a screen,
A video camera with a fisheye lens that captures images of the outside world as a distorted circular image with the attached fisheye lens,
A distorted circle having a radius R centered on the origin O of the two-dimensional XY coordinate system, which is composed of a collection of a large number of pixels arranged at positions indicated by coordinates (x, y) on the two-dimensional XY coordinate system. A distorted circular image memory storing at least one frame of an image;
An image input unit for storing a distorted circular image sequentially given as time-series data for each frame from the video camera with a fisheye lens in the distorted circular image memory;
A planar regular image memory for storing a planar regular image composed of a collection of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system;
An image output unit that reads and outputs a planar regular image stored in the planar regular image memory;
A monitor device for displaying a planar regular image output from the image output unit on a screen;
An actual cutting condition constituted by a cutting center point P that is one point on the distorted circular image, a predetermined plane inclination angle φ, and a predetermined magnification m is determined from a part of the distorted circular image as a planar regular image. An actual cutting condition determination unit that determines as a condition for cutting
An image of the cut-out size indicated by the magnification m from the cut-out position indicated by the cut-out center point P of the distorted circular image stored in the distorted circular image memory in the cut-out direction indicated by the plane inclination angle φ. An image cut-out conversion unit that converts the image into a plane regular image and stores it in the plane regular image memory;
A standard cutting condition storage unit for storing a standard cutting condition composed of a standard cutting center point P0, a standard plane inclination angle φ0, and a standard magnification m0;
When the moving object detection processing is performed by comparing the distorted circular images sequentially stored in time series in the distorted circular image memory, and the moving object is detected, the target cut-out center point Pn indicating the cut-out position of the detected moving object is obtained. And at least a target cut-out center point Pn among the target cut-out conditions formed by the target plane inclination angle φn indicating the cut-out direction of the detected moving object and the target magnification mn indicating the cut-out size of the detected moving object. A moving object detection unit to be provided to the cutting condition determination unit;
After the moving object detection, a moving object tracking process for tracking the detected moving object is performed, and a tracking cut-out center point P (i) indicating the cut-out position of the tracking moving object and a tracking plane inclination angle φ (i) indicating the cut-out direction of the tracking moving object are performed. And the tracking magnification m (i) indicating the tracking moving object cut size, and at least the tracking cut center point P (i) is determined to disappear as the tracking moving object disappears. A moving body tracking unit continuously giving to the actual cutting condition determining unit until it is made;
With
The actual cutting condition determining unit is in charge of a static monitoring period in charge of a static monitoring period in which the moving object is not detected, and a transition period in charge of in charge of a predetermined transition period after the moving object is detected And a dynamic monitoring period charge unit that takes charge of a dynamic monitoring period from the completion of the transition period until the disappearance determination is made, and the static monitoring period charge part includes the standard extraction condition Is determined as an actual cutting condition, the transition period responsible section defines an actual cutting condition that changes step by step so as to shift from the standard cutting condition to the target cutting condition, and the dynamic monitoring period responsible section The fish-eye monitoring system is characterized in that the tracking cut-out condition is determined as an actual cut-out condition. The fisheye monitoring system according to claim 1,
In a three-dimensional XYZ Cartesian coordinate system including a two-dimensional XY Cartesian coordinate system in which a distorted circular image is arranged, a virtual sphere having a radius R is defined with the origin O as the center, and one point on the virtual sphere is directed toward the origin O. When incident external light reaches one point on the XY plane by the optical action of the fisheye lens used for photographing, a point on the virtual sphere and a point on the XY plane correspond to each other A corresponding point Q that is a point on the virtual sphere corresponding to the cut-out center point P is taken, a vector passing through the corresponding point Q starting from the origin O is defined as a line-of-sight vector n, and the line-of-sight vector n A curved surface obtained by bending a point G on the plane passing through the point G and orthogonal to the line-of-sight vector n with the point G separated from the origin O by “product m · R of the magnification m and the radius R” Orientation according to the plane inclination angle φ When defining the disposed two-dimensional UV coordinate system with,
The image cut-out conversion unit uses a predetermined correspondence expression indicating the correspondence between the coordinates (u, v) on the two-dimensional UV coordinate system and the coordinates (x, y) on the two-dimensional XY orthogonal coordinate system. The fish-eye monitoring system is characterized by performing processing for converting an image cut out from a distorted circular image into a planar regular image by calculating corresponding coordinates (x, y) corresponding to the coordinates (u, v). . In the fish-eye monitoring system according to claim 1 or 2,
In the first half of the transition period, the transition period responsible section determines the actual cutting conditions in which the cutting center point P gradually changes from the standard cutting center point P0 to the target cutting center point Pn. A fish-eye monitoring system characterized in that an actual cut-out condition in which the magnification m changes stepwise from the standard magnification m0 to the target magnification mn during the zooming period. In the fish-eye monitoring system according to claim 1 or 2,
The transition period responsible section sets the entire transition period as a panning period and a zooming period, and the cutting center point P changes from the standard cutting center point P0 to the target cutting center point Pn step by step. A fish-eye monitoring system characterized by defining an actual cutting condition in which m gradually changes from a standard magnification m0 to a target magnification mn. In the fish-eye monitoring system according to claim 2,
The transition period responsible section sets all or a part of the transition period as the panning period, and the cutting center point P is gradually changed from the standard cutting center point P0 to the target cutting center point Pn during the panning period. When defining the corresponding point Q0 on the phantom spherical surface with respect to the standard cutting center point P0 and the corresponding point Qn on the phantom spherical surface with respect to the target cutting center point Pn, The movement path on the virtual sphere of the corresponding point Q on the virtual sphere with respect to the cutting center point P is the shortest path connecting the corresponding point Q0 and the corresponding point Qn on the virtual sphere. This is a fish-eye monitoring system. In the fish-eye monitoring system according to claim 5,
The standard cut-out center point P0 is set on the origin O of the two-dimensional XY orthogonal coordinate system,
A fish-eye monitoring system in which the transition period charge unit moves the cut center point P along the radius of the distorted circular image from the origin O toward the target cut center point Pn. In the fish-eye monitoring system according to claim 2,
The standard cutting center point P0 is set at a position other than the origin O of the two-dimensional XY orthogonal coordinate system,
The transition period responsible section sets all or a part of the transition period as the panning period, and the cutting center point P is gradually changed from the standard cutting center point P0 to the target cutting center point Pn during the panning period. , The cut center point P is moved along the radius of the distorted circular image from the standard cut center point P0 toward the origin O, and then moved from the origin O toward the target cut center point Pn. A fish-eye monitoring system that moves the cut-out center point P along the radius of the distorted circular image. In the fish-eye monitoring system in any one of Claims 5-7,
The fish-eye monitoring system, wherein the transition period responsible section changes the position of the cut-out center point P so that the movement of the corresponding point Q on the virtual spherical surface is a constant speed movement. In the fish-eye monitoring system in any one of Claims 5-7,
The transition period charge unit is such that the movement of the corresponding point Q on the phantom spherical surface is an inconstant speed movement such that the speed of the start part and the end part of the panning period is slower than the speed of the intermediate part. A fish-eye monitoring system characterized by changing the position of the point P. In the fish-eye monitoring system in any one of Claims 5-7,
The transition period department
A function storage for storing a function indicating the relationship between the movement distance b of the corresponding point Q on the virtual sphere and the time t;
A corresponding point conversion unit that outputs a point on the XY plane corresponding to the “position of the corresponding point Q at time t” obtained using the function as a cut-out center point P at the time t;
A fish-eye monitoring system characterized by comprising: In the fish-eye monitoring system according to claim 2,
The transition period responsible section sets all or a part of the transition period as the panning period, and the cutting center point P is gradually changed from the standard cutting center point P0 to the target cutting center point Pn during the panning period. The corresponding point Q on the virtual spherical surface with respect to the cut-out center point P is moved at a predetermined spherical moving speed set as a function related to time on a predetermined spherical moving path set in advance. And determining the position of the cut-out center point P. In the fish-eye monitoring system according to claim 2,
The transition period responsible section sets all or a part of the transition period as the zooming period, and the magnification m is changed from the standard magnification m0 so that it monotonously increases or decreases along the time axis during the zooming period. A fish-eye monitoring system characterized by changing to a target magnification mn. The fisheye monitoring system according to claim 12, wherein
The fish-eye monitoring system, wherein the transition period charge section linearly changes the magnification m along the time axis during the zooming period. The fisheye monitoring system according to claim 12, wherein
The transition period charge section changes the magnification m along the time axis so that the magnification change rate at the start and end of the zooming period changes at an inconstant speed that is slower than the change rate at the intermediate part. A fish-eye monitoring system. The fisheye monitoring system according to claim 12, wherein
The transition period department
A function storage unit for storing a function indicating the relationship between the magnification m and the time t;
A magnification determining unit that outputs a magnification at time t obtained using the function as a magnification m at time t;
A fish-eye monitoring system characterized by comprising: In the fish-eye monitoring system according to claim 2,
In the standard cutout condition storage unit, a standard plane inclination angle φ0 is set such that a plane regular image is cut out in a direction in which the vertical axis direction of the real world in the photographed image is downward.
The moving body detection unit sets a target plane inclination angle φn such that a plane regular image is cut out in a direction in which the vertical axis direction is a downward direction,
The transition period responsible unit changes the plane inclination angle φ from the standard plane inclination angle φ0 to the target plane inclination angle φn so that a plane regular image is cut out in a direction in which the vertical axis direction is the downward direction,
The fish-eye monitoring system, wherein the moving body tracking unit sets a tracking plane inclination angle φ (i) so that a planar regular image is cut out in a direction in which the vertical axis direction is a downward direction. The fisheye monitoring system according to claim 16,
The video camera with a fisheye lens is installed so that the XY plane as the imaging surface coincides with the vertical plane and the Y axis faces the direction that forms an angle ξ with respect to the vertical axis W.
The angle formed by the rotation reference axis J that passes through the origin G of the two-dimensional UV coordinate system, is parallel to the XY plane, and is orthogonal to the line-of-sight vector n, and the U axis of the two-dimensional UV coordinate system is the plane inclination angle φ. When the angle between the orthographic projection of the line-of-sight vector n on the XY plane and the Y axis is defined as the azimuth angle α, and the U axis direction is defined as the lateral direction of the planar regular image, φ = −α A fish-eye monitoring system characterized in that a setting of ξ is made. The fisheye monitoring system according to claim 17,
The standard cut-out center point P0 is set on the origin O of the two-dimensional XY orthogonal coordinate system,
In the standard cutout condition storage unit, a standard cutout condition indicating the cutout direction is set so that the planar regular image is cut out so that the V axis corresponds to the vertical axis in the real world at the position of the origin O of the photographed image. And
The moving object detection unit sets the target plane inclination angle φn = −αn−ξ when the azimuth angle with respect to the target cutting center point Pn is αn,
The fish-eye monitoring system characterized in that the transition period responsible section always sets the plane inclination angle φ = φn. The fisheye monitoring system according to claim 16,
A video camera with a fisheye lens is installed so that the XY plane as the imaging surface coincides with the horizontal plane.
The angle formed by the rotation reference axis J that passes through the origin G of the two-dimensional UV coordinate system, is parallel to the XY plane, and is orthogonal to the line-of-sight vector n, and the U axis of the two-dimensional UV coordinate system is the plane inclination angle φ. The fish-eye monitoring system is characterized in that when the U-axis direction is defined as the horizontal direction of the planar regular image, the setting is always φ = 0 °. In the fish-eye monitoring system according to claim 2,
The image cut-out conversion unit calculates the corresponding coordinates (x, y) for the coordinates (u, v) of one target pixel constituting the planar regular image, and the corresponding coordinates (x, y) in the distorted circular image memory are calculated. The pixel value of the pixel arranged in the vicinity of y) is read out, and the process of determining the pixel value of the pixel of interest based on the read pixel value is executed for each pixel constituting the planar regular image, and the planar regular A fish-eye monitoring system that performs processing for conversion to a planar regular image by writing a pixel value of each pixel in an image memory. The fisheye monitoring system according to claim 20,
When the image cut-out conversion unit determines the pixel value of the pixel of interest arranged at the position indicated by the coordinates (u, v), the distortion arranged near the position indicated by the corresponding coordinates (x, y) A fish-eye monitoring system that performs an interpolation operation on pixel values of a plurality of reference pixels on a circular image. In the fish-eye monitoring system according to claim 2,
On the virtual sphere, the image cut-out conversion unit corresponds to the point on the sphere corresponding to the point Si indicated by the coordinates (xi, yi) on the two-dimensional XY orthogonal coordinate system, according to the projection method of the fisheye lens used for shooting. When Qi is taken and the coordinates on the two-dimensional UV coordinate system of the intersection Ti between the straight line connecting the origin O and the corresponding point Qi on the spherical surface and the arrangement surface of the two-dimensional UV coordinate system are (ui, vi) In addition, a fisheye monitoring system using a corresponding relational expression in which the coordinates (xi, yi) are obtained as corresponding coordinates corresponding to the coordinates (ui, vi). The fisheye monitoring system according to claim 22,
Image cropping conversion unit
When the distorted circular image stored in the distorted circular image memory is an orthographic image captured by the orthographic fisheye lens, with respect to the cut-out center point P indicated by the coordinates (xp, yp). A point indicated by coordinates (xp, yp, zp) given as an intersection of a straight line passing through the point P and parallel to the Z axis and the virtual sphere is designated as a corresponding point Q on the sphere, and is indicated by coordinates (xi, yi). For an orthogonal projection image with respect to the point Si, a point indicated by coordinates (xi, yi, zi) given as an intersection of a straight line passing through the point Si and parallel to the Z axis and the virtual sphere is a corresponding point Qi on the sphere. Using the correspondence equation,
When the distorted circular image stored in the distorted circular image memory is a non-orthographic image captured by a non-projective fisheye lens, the coordinates on the orthographic image and the non-orthographic image A fish-eye monitoring system using a non-orthogonal image correspondence relation obtained by correcting the orthographic image correspondence relation using a coordinate conversion formula between coordinates. The fisheye monitoring system according to claim 23,
The image cut-out conversion unit defines an angle formed by the orthogonal projection of the line-of-sight vector n on the XY plane and the Y axis as an azimuth angle α, and an angle formed between the line-of-sight vector n and the Z axis positive direction as a zenith angle β.
A = cosφ cosα-sinφ sinα cosβ
B = −sinφ cosα − cosφ sinα cosβ
C = sinβ sinα
D = cosφ sinα + sinφ cosα cosβ
E = -sinφ sinα + cosφ cosα cosβ
F = -sinβ cosα
w = mR
As a correspondence relation expression for an orthographic image indicating the correspondence between coordinates (u, v) and coordinates (x, y),
x = R (uA + vB + wC) / √ (u ² + v ² + w ² )
y = R (uD + vE + wF) / √ (u ² + v ² + w ² )
A fish-eye monitoring system characterized by using the following formula. The fisheye monitoring system according to claim 24,
Image cropping conversion unit
When the distorted circular image stored in the distorted circular image memory is an equidistant projection image photographed by an equidistant projection type fisheye lens, the coordinates (xa, ya) on the orthographic projection image are equidistantly projected. Coordinate conversion formula to convert to coordinates (xb, yb) on the image xb = xa (2R / πr) sin ⁻¹ (r / R)
yb = ya (2R / πr) sin ⁻¹ (r / R)
However, r = √ (xa ² + ya ² )
A fish-eye monitoring system that corrects a correspondence relation for an orthographic image by using a projection image. In the fish-eye monitoring system in any one of Claims 1-25,
The moving object detector
A background image holding unit that holds an image stored in a distorted circular image memory at a predetermined initial time point as a background distorted circular image;
The pixel value of each pixel on the distorted circular image sequentially stored in the distorted circular image memory for each frame in time series is compared with the pixel value of the corresponding pixel on the background distorted circular image, and the difference between the two values is compared. A corresponding pixel comparison unit that extracts, as a pixel of interest, a pixel having a threshold value equal to or greater than a threshold value;
A focused area search unit that searches for a focused area that is a continuous area composed of the focused pixels and that satisfies a condition that is equal to or larger than a reference area and that has both a vertical width and a horizontal width equal to or larger than a predetermined reference dimension;
A moving object that determines that a moving object has been detected when a distorted circular image including the target area is continuously obtained for a number of frames equal to or greater than a reference number, and generates a target cut-out condition based on the final target area A detection determination unit;
A fish-eye monitoring system characterized by comprising: The fisheye monitoring system according to claim 26,
The moving object is detected when the moving object detection determination unit continuously obtains a distorted circular image including a specific region of interest that is expected to be formed due to the same moving object for the number of frames that is the reference number or more. The fish-eye monitoring system is characterized in that, when there are a plurality of regions of interest separated from each other, the region of interest having a larger area is determined as the specific region of interest. The fisheye monitoring system according to claim 26 or 27,
A fish-eye monitoring system, wherein the moving object detection determination unit sets the center of gravity position of the final target region as the target cut-out center point Pn, and sets the target magnification mn based on the vertical width and the horizontal width of the final target region. In the fish-eye monitoring system according to any one of claims 1 to 28,
The moving body tracking unit
A known moving object region holding unit for holding information of a known moving object region that has already been recognized on the (i-1) th frame image stored in the distorted circular image memory;
A candidate area extracting unit that extracts a plurality of areas located in the vicinity of the known moving body area as candidate areas from the i-th frame image stored in the distorted circular image memory;
A feature amount calculation unit that performs an operation for obtaining a feature amount indicating a feature of an image for each of the known moving object region and the plurality of candidate regions;
The feature amount of the known moving object region calculated by the feature amount calculating unit is compared with the feature amount of a plurality of candidate regions, and the similarity to the feature amount of the known moving object region is equal to or greater than a predetermined reference, and A new moving body region recognition unit for recognizing a candidate region having the highest feature amount as a new moving body region on the i-th frame image;
When the recognition by the new moving body region recognition unit is successful, a tracking cut-out condition for the i-th frame image is generated based on the new moving body region, and when the recognition fails, the tracking moving body is A moving body tracking determination unit for determining disappearance when it has disappeared,
Have
The existing moving body area holding unit holds the area information of the detected moving body given from the moving body detection unit as information of the first known moving body area, and thereafter, the information of the new moving body area as information of the new known moving body area. A fish-eye monitoring system characterized by holding. The fisheye monitoring system according to claim 29,
The moving object tracking determination unit sets the gravity center position of the new moving object region recognized for the i-th frame image as the tracking cut-out center point P (i), and the tracking magnification based on the vertical width and the horizontal width of the new moving object region A fish-eye monitoring system characterized by setting m (i). The fisheye monitoring system according to claim 29 or 30,
A fish-eye monitoring system characterized in that a feature amount calculation unit obtains a color histogram or edge direction histogram of pixels constituting an area given as a calculation target as a feature amount. In the fish-eye monitoring system in any one of Claims 1-31,
A fish-eye monitoring system, wherein the moving object detection unit pauses a new moving object detection process while the moving object tracking process is performed by the moving object tracking unit. In the fish-eye monitoring system in any one of Claims 1-31,
If the moving object detection unit continues the moving object detection process while the moving object tracking process is being performed by the moving object tracking unit, and a new moving object that is different from the old moving object that is the target of the moving object tracking process is detected, Perform processing to give the target cutting condition for the new moving object to the actual cutting condition determination unit,
When the target cutting condition for the new moving object is given to the actual cutting condition determining unit, the transition period responsible section shifts from the tracking cutting condition for the old moving object to the target cutting condition for the new moving object. The actual cutting conditions that gradually change to
A fish-eye monitoring system, wherein a moving object tracking unit performs a tracking process on the new moving object. In the fish-eye monitoring system in any one of Claims 1-31,
The moving object detection unit continues the moving object detection process while the moving object tracking process is performed by the moving object tracking unit, and is different from the old moving object that is the target of the moving object tracking process and has a larger area than the old moving object. When another moving object that satisfies the condition is detected, the target cutting condition for the new moving object that satisfies the condition is given to the actual cutting condition determining unit,
When the target cutting condition for the new moving object is given to the actual cutting condition determining unit, the transition period charge unit shifts from the tracking cutting condition for the old moving object to the target cutting condition for the new moving object. The actual cutting conditions that change step by step
A fish-eye monitoring system, wherein a moving object tracking unit performs a tracking process on the new moving object. The fisheye monitoring system according to claim 33 or 34,
When the moving object tracking unit performs the tracking process for the new moving object, also performs the tracking process for the old moving object, sets the tracking cut-out condition based on the new moving object, and the new moving object disappears earlier than the old moving object, Instead of the moving object detection unit, the target cutting condition for the old moving object is given to the actual cutting condition determination unit,
When the target cutting condition for the old moving object is given to the actual cutting condition determining unit, the transition period charge unit shifts from the tracking cutting condition for the new moving object to the target cutting condition for the old moving object. The actual cutting conditions that change step by step
The fish-eye monitoring system, wherein the moving object tracking unit continues the tracking process for the old moving object. The fisheye monitoring system according to any one of claims 33 to 35,
When the transition period charge section performs the transition process of switching from the old moving object to the new moving object, if the dynamic monitoring period for the old moving object is less than the predetermined minimum reference time, the dynamic monitoring period is the minimum A fish-eye monitoring system characterized in that processing is started after a waiting time until the reference time is reached. In the fish-eye monitoring system according to any one of claims 1 to 36,
Manual condition setting unit with a function to arbitrarily set the actual cutting condition to be determined by the actual cutting condition determination unit and the standard cutting condition stored in the standard cutting condition storage unit based on the user's setting operation The fish-eye monitoring system further comprising:   The program for functioning a computer as a component except the video camera with a fisheye lens and monitor apparatus in the fisheye monitoring system in any one of Claims 1-37.   38. A semiconductor integrated circuit in which an electronic circuit functioning as a component excluding the video camera with a fisheye lens and the monitor device in the fisheye monitoring system according to claim 1 is incorporated. An image presentation system that cuts out a part of a distorted circular image obtained by shooting using a fisheye lens, converts it into a planar regular image, and presents it on a screen,
A fish-eye lens is constituted by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY coordinate system, having a radius R around the origin O of the two-dimensional XY coordinate system. A distorted circular image memory for storing a distorted circular image obtained by photographing used;
A planar regular image memory for storing a planar regular image composed of a collection of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV coordinate system;
An image display device for displaying a planar regular image stored in the planar regular image memory on a screen;
An actual cutting condition constituted by a cutting center point P that is one point on the distorted circular image, a predetermined plane inclination angle φ, and a predetermined magnification m is determined from a part of the distorted circular image as a planar regular image. An actual cutting condition determination unit that determines as a condition for cutting
Based on the actual cutting conditions given from the actual cutting condition determining unit, the plane inclination is determined from the cutting position indicated by the cutting center point P of the distorted circular image stored in the distorted circular image memory. An image cut-out conversion unit that cuts out an image of the cut-out size indicated by the magnification m in the cut-out direction indicated by the angle φ, converts the image into a plane regular image, and stores the image in the plane regular image memory;
With
When the actual cutting condition determination unit changes the current cutting condition including the current cutting center point P0 to the target cutting condition including the target cutting center point Pn, the image cutting condition is determined. An image presentation system that performs a process of shifting the position of a cut-out center point P given to a conversion unit from a point P0 to a point Pn in a stepwise manner. The image presentation system according to claim 40, wherein
In a three-dimensional XYZ Cartesian coordinate system including a two-dimensional XY Cartesian coordinate system in which a distorted circular image is arranged, a virtual sphere having a radius R is defined with the origin O as the center, and one point on the virtual sphere is directed toward the origin O. When incident external light reaches one point on the XY plane by the optical action of the fisheye lens used for photographing, a point on the virtual sphere and a point on the XY plane correspond to each other A corresponding point Q that is a point on the virtual sphere corresponding to the cut-out center point P is taken, a vector passing through the corresponding point Q starting from the origin O is defined as a line-of-sight vector n, and the line-of-sight vector n A curved surface obtained by bending a point G on the plane passing through the point G and orthogonal to the line-of-sight vector n with the point G separated from the origin O by “product m · R of the magnification m and the radius R” Orientation according to the plane inclination angle φ When defining the disposed two-dimensional UV coordinate system with,
The image cut-out conversion unit uses a predetermined correspondence expression indicating the correspondence between the coordinates (u, v) on the two-dimensional UV coordinate system and the coordinates (x, y) on the two-dimensional XY orthogonal coordinate system. An image presentation system that performs processing for converting an image cut out from a distorted circular image into a planar regular image by calculating corresponding coordinates (x, y) corresponding to coordinates (u, v). The image presentation system according to claim 41, wherein
The actual cutting condition determination unit moves the corresponding point Q on the virtual spherical surface with respect to the cutting center point P on the predetermined spherical moving path set at a predetermined spherical moving speed set as a function related to time. The image presentation system characterized by determining the position of the cutting center point P as described above. In the image presentation system according to any one of claims 40 to 42,
The actual cutting condition determination unit performs a process of shifting the position of the cutting center point P from the point P0 to the point Pn in a stepwise manner, and changes the value of the magnification m from the current magnification m0 to the target magnification mn. An image presentation system characterized by performing a process of shifting in stages. The image presentation system according to claim 43,
An image presenting system, wherein the actual cutting condition determining unit shifts the value of the magnification m at a predetermined magnification changing speed set as a function related to time.