KR20240009094A - Camera device capable of pan-tilt-zoom operation and video surveillance system and method using the same - Google Patents

Abstract

Disclosed is a technology relating to an image surveillance system using a pan-tilt-zoom camera. The image surveillance system comprises: a pan-tilt-zoom camera that photographs a surveillance area; and a display device that outputs an image of the pan-tilt-zoom camera. The pan-tilt-zoom camera provides a fish-eye image showing an entire surveillance area on a first area of the display device, calculates a driving parameter of the pan-tilt-zoom camera based on position information of a selected part in the fish-eye image, moves the pan-tilt-zoom camera to a position of the selected part according to the driving parameter, and provides a surveillance image of the selected part photographed in the pan-tilt-zoom camera in a second area of the display device. According to the present invention, the pan-tilt-zoom camera can intuitively and rapidly be controlled using the fish-eye image, and enables rapid generation of the fish-eye image.

Description

Camera device capable of pan-tilt-zoom operation and video surveillance system and method using the same {Camera device capable of pan-tilt-zoom operation and video surveillance system and method using the same}

The present invention relates to a surveillance system that remotely monitors a surveillance area, and particularly to a surveillance system and camera device that controls a pan-tilt-zoom camera that photographs a surveillance area using fisheye images.

In an increasingly complex society, surveillance cameras are being used as the most effective tool to prevent ever-increasing crimes and to record and confirm everyday human behavior. Although there are negative aspects such as invasion of privacy caused by surveillance cameras, the necessity and importance of surveillance cameras are increasing, and accordingly, the demand for surveillance cameras is increasing dramatically.

Recently, camera devices that use fisheye lens cameras to monitor the entire surveillance target area and pan-tilt zoom cameras to precisely monitor specific, designated areas of the surveillance target area have been released. It is becoming. The fisheye lens is a familiar interface to surveillance system managers because it shows the surveillance area at a glance. However, because fisheye images are distorted, pan-tilt-zoom cameras are typically used for detailed surveillance areas.

Registered Patent No. 10-1179131, announced on September 7, 2012, relates to a surveillance system using a simultaneous surveillance camera with integrated pan/tilt/zoom functions, through a pan/tilt/zoom camera with an integrated wide-angle lens. A camera unit that captures omnidirectional (360 degrees) long-distance images of the surveillance area; an image signal processing unit that converts a distorted long-distance image signal captured by the camera unit into image data; The video signal processing unit selects an unwrapping region among the distorted image data to create a ROI (Region of Interest), and converts the required curved region of the ROI into a flat image using a lookup table to display it on the monitor. a central processing unit that displays the planar image and restores the original image through an image calibration process to form at least one surveillance screen; and an intensive monitoring unit that enlarges a specific area among the images restored through the central processing unit through the pan/tilt/zoom camera and configures and displays it as an intensive monitoring screen. However, by combining a wide-angle lens in front of the pan-tilt-zoom camera, distorted images are always obtained, so there is a problem that a dewarping process must be performed for images in the surveillance area.

Publication Patent No. 10-2017-0136904, published on December 12, 2017, displays the corrected image in a large size and displays the original image as a minimap when the user desires, allowing monitoring with more emphasis on the corrected image. It relates to a monitoring device and monitoring system, wherein the monitoring device includes a communication unit that receives the original video acquired by the camera; a storage unit that stores the original image; a screen unit that displays the original image and a corrected image obtained by dwarfing the original image; and a control unit that controls operations of the communication unit, the storage unit, and the screen unit, wherein the screen unit displays a mini-map showing the original image on a portion of the corrected image. However, since distorted images are always obtained using a fisheye camera, there is a problem that a dewarping process must be performed for images in the surveillance area.

Registered Patent No. 10-2193984, announced on December 22, 2020, relates to a PTZ camera control method using a fisheye camera when a fisheye camera and a PTZ (Pan-Tilt Zoom) camera are installed adjacent to each other. It includes the step of selecting an area of interest from the entire image of the surveillance target area captured by and adjusting the panning angle (P) and tilt angle (T) of the PTZ camera to obtain a precise image of the selected area of interest, The adjustment step includes first adjusting the panning angle (P) and tilt angle (T) of the PTZ camera to be parallel to the optical axis of the fisheye camera centered on the area of interest, and adjusting the panning angle (P) and tilt angle (T). In the first adjustment state, the panning angle (P) and tilt angle (T) of the PTZ camera are adjusted based on the distance difference between the center of the image of the area of interest in the image captured from the PTZ camera and the center of the image captured from the PTZ camera. A configuration including a secondary adjustment step is disclosed. However, although it is possible to obtain pan-tilt-zoom camera images directly, it has the disadvantage of complicating the system configuration and increasing the production cost because it includes both a fisheye camera and a pan-tilt-zoom camera.

Therefore, there is a need to develop a simple video surveillance system that can easily operate a pan-tilt-zoom camera using fisheye images and directly provides undistorted pan-tilt-zoom camera images to the surveillance area.

One purpose of the present invention is to provide a surveillance system that can intuitively, quickly, and easily control a pan-tilt-zoom camera from a fisheye image.

Another object of the present invention is to provide a surveillance system that quickly forms a fisheye image using small computational resources when forming a fisheye image from a pan-tilt-zoom camera image.

Another object of the present invention is to provide a monitoring system that can be constructed inexpensively and is easy to maintain.

According to one aspect of the proposed invention, a camera device capable of pan-tilt-zoom operation includes a memory configured to store surveillance image data, and a processor electrically coupled to the memory. The processor generates a fisheye image representing the entire surveillance area, outputs it to an external device, calculates driving parameters for pan-tilt zoom operation of the camera based on location information of the selected portion of the fisheye image, and calculates driving parameters for the camera's pan-tilt zoom operation. It is configured to output and provide surveillance video data captured from an external device.

According to an additional aspect, the fisheye image is generated by extracting predetermined pixels from a partial image captured in each partial area divided by a preset method among the entire surveillance area.

According to an additional aspect, the fisheye image is updated at preset intervals using image data of subareas re-photographed for each subarea divided in a preset manner among the entire surveillance area.

According to an additional aspect, the processor is further configured to display a preset area in the surveillance area in which a location to be captured is preset on the fisheye image.

According to an additional aspect, the driving parameter includes a pan angle and a tilt angle for photographing the center of the selected portion, and a zoom factor set by a ratio of the angle of view of the selected portion and the angle of view of the camera.

According to another aspect of the proposed invention, a video surveillance system using a camera capable of pan-tilt-zoom operation includes a camera capable of pan-tilt-zoom operation for photographing a surveillance area, a display for outputting images from the camera, and the camera. It includes a control unit that operates and transmits the image captured by the camera to the display. The control unit includes a fisheye image providing unit that provides a fisheye image representing the entire surveillance area in a first area of the display, and calculates driving parameters for a pan-tilt zoom operation of the camera based on location information of a selected portion of the fisheye image. It includes a driving parameter calculation unit that calculates a driving parameter, and a surveillance image providing unit that provides a surveillance image captured for the selected portion in a second area of the display.

According to an additional aspect, the fisheye image providing unit provides a fisheye image generated by combining and distorting a plurality of partial images captured by dividing the entire surveillance area into preset partial regions.

According to an additional aspect, the fisheye image providing unit divides the entire surveillance area into preset partial areas and provides a fisheye image generated by extracting predetermined pixels from each partial image captured.

According to an additional aspect, the control unit further includes a preset map providing unit that displays a preset area in which a location to be captured is preset among the surveillance area on the fisheye image.

According to an additional aspect, the driving parameter calculation unit calculates a pan angle and a tilt angle that directs the camera toward the center of the selected portion, and determines the zoom magnification of the pan-tilt zoom camera based on the ratio of the angle of view of the selected portion and the camera angle of view. Set it.

According to another aspect of the proposed invention, a video surveillance method using a camera capable of pan-tilt-zoom operation includes providing a fisheye image representing the entire surveillance area, and using the camera based on location information of a selected portion of the fisheye image. calculating driving parameters for a pan-tilt-zoom operation, moving a camera toward a selected part using the driving parameters, and providing a surveillance image of the selected part captured by a pan-tilt-zoom camera. Includes.

According to an additional aspect, the step of providing the fisheye image includes dividing the entire surveillance area into preset partial areas, shooting a partial image in each partial area, and extracting predetermined pixels from each captured partial image. It includes the step of generating a fisheye image.

According to an additional aspect, the video surveillance method further includes a preset map providing step of displaying a preset area in which a location to be captured is preset among the surveillance area on the fisheye image.

According to an additional aspect, calculating the driving parameters may include calculating a pan angle and a tilt angle that directs the camera toward the center of the selected portion, and calculating the angle of view of the camera based on the ratio of the angle of view of the selected portion and the angle of view of the camera. It includes the step of setting the zoom factor.

According to the proposed invention, it is possible to provide a surveillance system that can intuitively, quickly, and easily control the pan-tilt-zoom camera from the fisheye image by simultaneously providing fisheye images and pan-tilt-zoom camera images.

Furthermore, the proposed invention can save computational resources and quickly form a fisheye image by using a method of mapping pixels required for a fisheye image from a pan-tilt-zoom camera image.

Furthermore, the proposed invention allows a surveillance system to be constructed simply and inexpensively by forming a fisheye image from a pan-tilt-zoom camera, and is also easy to maintain.

Figure 1 is a configuration diagram showing the overall configuration of a video surveillance system using a camera capable of pan-tilt-zoom operation according to an embodiment.
Figure 2 is a conceptual diagram showing the configuration of a screen displayed on a display of a video surveillance system using a camera capable of pan-tilt-zoom operation according to an embodiment.
Figure 3 is a configuration diagram showing the main configuration of the control unit of a video surveillance system using a camera capable of pan-tilt-zoom operation according to an embodiment.
Figure 4 is a configuration diagram showing the additional configuration of the control unit of a video surveillance system using a pan-tilt-zoom camera according to an embodiment.
Figure 5 is a conceptual diagram showing camera control and preset maps using fisheye images in a video surveillance system using a pan-tilt-zoom camera according to an embodiment.
Figure 6 is a conceptual diagram showing a method of generating a fisheye image by distorting a pan-tilt-zoom image in a video surveillance system using a pan-tilt-zoom camera according to an embodiment.
Figure 7 is a conceptual diagram showing a method of generating a fisheye image by matching pixels of the pan-tilt-zoom image to the fisheye image in a video surveillance system using a pan-tilt-zoom camera according to an embodiment.
Figure 8 is a flowchart showing a video surveillance method using a pan-tilt-zoom camera according to an embodiment.
Figure 9 is a flowchart showing the steps of providing a fisheye image in an image surveillance method using a pan-tilt-zoom camera according to an embodiment.

The foregoing and additional aspects are embodied through embodiments described with reference to the accompanying drawings. It is understood that the components of each embodiment can be combined in various ways within the embodiment or with components of other embodiments as long as there is no other mention or contradiction between them. Based on the principle that the inventor can appropriately define the concept of terms in order to explain his or her invention in the best way, the terms used in this specification and claims have meanings that correspond to the description or proposed technical idea. It must be interpreted as a concept. In this specification, a module or part may be a set of program instructions stored in a memory to be executable on a computer or processor, or may be implemented using a set of electronic components or circuits such as ASIC or FPGA to execute these instructions. Additionally, the operation of each module or part may be performed by one or more processors or devices.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a configuration diagram showing the overall configuration of a video surveillance system using a camera capable of pan-tilt-zoom operation according to an embodiment.

Referring to FIG. 1, according to one embodiment, a video surveillance system 100 includes one or more cameras 110 for photographing a surveillance area, and clients 130 and 140 connected to the cameras and a network 120. do. The camera 110 may use a pan-tilt-zoom camera that can adjust the pan angle, tilt angle, and zoom ratio to increase the captured surveillance area. Multiple cameras (111, 112, ??) can be used to monitor an entire area divided into multiple areas, such as a building. The network 120 connecting the camera and the client may be implemented as a wired communication network or a wireless communication network. Clients 130 and 140 can be used as PCs, tablets, laptops, mobile phones, etc. The client includes displays 135 and 145 to show captured content of the surveillance area. In this specification, 'display' is used as a concept that includes not only the displays 135 and 145, but also the clients 130 and 140 that are coupled with the displays 135 and 145 and drive the displays. In Figure 1, the display is connected to the camera and a network, but it can also be connected directly to the camera or manufactured as a built-in configuration in the camera.

According to one aspect of the proposed invention, a video surveillance system 100 using a camera capable of pan-tilt-zoom operation includes a camera, a display, and a control unit. The camera 110 is capable of pan-tilt-zoom operation and photographs the surveillance area. The displays 135 and 145 are devices that output images from cameras. In this specification, the control unit drives the camera and transmits the image captured by the camera to the display.

The control unit can be built into each camera (111, 112, ??) or installed on the client. When controlling multiple cameras, the burden on the client increases, and in the case of mobile devices, computational resources are scarce, so it is desirable to embed the control unit in each camera (111, 112, ??).

Figure 2 is a conceptual diagram showing the configuration of a screen displayed on a display of a video surveillance system using a camera capable of pan-tilt-zoom operation according to an embodiment.

As the displays 135 and 145, a monitor, PDA, tablet, mobile phone, etc. having a display unit capable of outputting images such as LCD or OLED can be used. When a touch panel is used for the displays 135 and 145, input may be received from the user through the display. The display portions of the displays 135 and 145 include a first area 210 and a second area 220 that output different images. For example, a fisheye image can be output to the first area 210, and an image captured by a pan-tilt-zoom camera can be output to the second area 220. Referring to FIG. 2 , the pan-tilt-zoom camera image corresponding to the portion 215 indicated in light green among the fisheye images in the first area 210 is output to the second area 220.

The first area 210 may overlap the second area 220 . For example, the fisheye image of the first area 210 may be output to a portion of the second area 220. The display device 150 may additionally include a third area 230. For example, the third area 230 may display preset setting information or commands to control the camera.

Figure 3 is a configuration diagram showing the main configuration of the control unit of a video surveillance system using a camera capable of pan-tilt-zoom operation according to an embodiment.

In the embodiment of Figure 3(a), the control unit is built into the pan-tilt-zoom camera, but in other embodiments, the control unit may be installed in a separate device. The imaging unit 310 includes various lenses and image sensors to capture images of the surveillance area. The driving unit 330 adjusts the pan angle, tilt angle, and zoom ratio of the camera to point the imaging unit of the camera in the direction in which photography is desired. The control unit 320 includes a memory and a processor, stores images captured by the imaging unit in memory or transfers them to externally connected output devices, and controls the driving unit to direct the camera to a location requested by the external input device.

According to another aspect of the proposed invention, the camera device 110 capable of pan-tilt-zoom operation includes a memory and a processor. The memory is configured to store surveillance video data. The processor is electrically coupled to the memory and controls camera operation.

Figure 3(b) is a configuration diagram showing the main configuration of the control unit 320. The control unit 320 includes a fisheye image providing unit, a driving parameter calculating unit, and a surveillance image providing unit. The fisheye image providing unit 360 provides a fisheye image representing the entire surveillance area to the first area of the display. The driving parameter calculation unit 370 calculates driving parameters for the pan-tilt-zoom operation of the camera based on the location information of the selected part of the fisheye image. The surveillance image providing unit 380 provides a surveillance image captured for a selected portion in the second area of the display.

These configurations may be performed by a processor in the control unit. That is, the processor generates a fisheye image representing the entire surveillance area, outputs it to an external device, calculates driving parameters for the pan-tilt zoom operation of the camera based on the location information of the selected part of the fisheye image, and calculates the driving parameters for the camera's pan-tilt zoom operation. It may be configured to output and provide surveillance video data captured from the device to an external device.

Figure 4 is a configuration diagram showing the additional configuration of the control unit of a video surveillance system using a pan-tilt-zoom camera according to an embodiment.

According to an additional aspect, the control unit 320 further includes a camera operation control unit 470. The camera driving control unit 470 performs a pan-tilt-zoom operation of the camera in accordance with the driving parameters to move the camera to a shooting position. When a pan-tilt-zoom operation is performed, the camera is pointed at the location to be photographed, and an image is captured through the imaging unit, inputted, and stored in memory.

According to an additional aspect, the control unit 320 further includes an event monitoring unit 450. The event monitoring unit 450 monitors whether there is an input from the user and whether an abnormality occurs in the monitoring area. When an event is detected, a predetermined procedure is performed in response to each event. For example, when information is received from the user that a specific part of the fisheye image has been selected, the pan-tilt-zoom camera can be moved to the center of the specific part, the screen can be enlarged to fit that area, and the captured video can be saved or transmitted to the display. there is. If an abnormality is detected in a specific surveillance area, the camera can be moved to the center of the area and an alarm can be raised.

According to an additional aspect, the control unit 320 further includes a mapping data storage unit 490. The mapping data storage unit 490 stores mapping data used in the fisheye image in a storage device such as memory for quick calculation when generating the fisheye image. Mapping data is described in detail in the descriptions of FIGS. 7 and 9.

Figure 5 is a conceptual diagram showing camera control and preset maps using fisheye images in a video surveillance system using a pan-tilt-zoom camera according to an embodiment.

In Figure 5(a), the user's action of changing the surveillance area of the pan-tilt-zoom camera using an input device such as a touch panel or mouse in the fisheye image is indicated in orange. By dragging the currently filming surveillance area 215, shown in light green, the pan-tilt-zoom camera can be adjusted to capture and monitor a new area 510. User input is transmitted to the control unit 320 through a touch panel or mouse, and the control unit moves the center position of the pan-tilt-zoom camera to the center of the new area 510. The image of the new area 510 captured by the pan-tilt-zoom camera is output to the second area 220 of the display.

The user can click a random location 520 in the fisheye image to have the pan-tilt-zoom camera monitor a new area. In addition, by dragging an arbitrary area, the camera can capture the dragged area 530. At this time, the pan-tilt-zoom camera can adjust the pan angle and tilt angle to point toward the center of the dragged area 530, and adjust the zoom magnification to include the dragged area.

According to an additional aspect, the control unit 320 further includes a preset setting unit 430 that sets a preset area in which the location to be captured is preset among the surveillance area, and a preset map providing unit 410 that displays the preset area on the fisheye image. do. That is, the processor of the control unit 320 is further configured to set a preset area in which the location to be captured is preset among the surveillance area and display the set preset area on the fisheye image.

The preset setting unit 430 receives location information for a specific surveillance area from the user and stores it. Location information can be a specific location input from a fisheye image, or the location of a dragged area in a fisheye image. When setting a preset, you can set the driving parameters of the pan-tilt-zoom operation using the center point location and size of the location to be photographed. The set preset area can be saved to a storage device, non-volatile memory, etc.

Figure 5(b) shows an example of a preset map displayed in blue in a fisheye image. Users can set frequently moved surveillance areas as preset areas in advance. The preset map displays preset preset areas on a fisheye image. When a specific surveillance area is input from the user, the preset setting unit 430 can obtain the central pan angle and tilt angle, and calculate and store the zoom ratio for photographing the area. Referring to FIG. 5(b), three preset areas 560, 570, and 580 are displayed on the preset map. In a fisheye image, the small preset area 580 has a larger zoom factor than the large preset map 560. Meanwhile, a separate identifier can be added to the preset map. For example, the identifier 565 of 'warehouse' is displayed in the upper preset area 560.

Figure 6 is a conceptual diagram showing a method of generating a fisheye image by distorting a pan-tilt-zoom image in a video surveillance system using a pan-tilt-zoom camera according to an embodiment.

According to an additional aspect, fisheye image data may be generated by synthesizing and distorting a plurality of partial image data for the entire surveillance area captured with a pan-tilt-zoom camera.

Figure 6(a) is a conceptual diagram showing an embodiment of converting a pan-tilt-zoom camera image into a fisheye image. The left rectangular area 610 in FIG. 6(a) represents data synthesized from a plurality of partial images of the entire surveillance area captured by a pan-tilt-zoom camera. Using the (i, j) coordinates of the horizontal W and vertical H composite images, the distance r _u and the fan angle θ from the center of the fisheye image 620 of radius R can be obtained using the following equation 1.

[Equation 1]

θ = i / W * 360

r _u = R ( 1 - j / H )

If the total pixels of the fisheye image are F * F, the (i', j') coordinates of the fisheye image can be obtained using Equation 2 below.

[Equation 2]

i' = F/2 + r cos θ

j' = F/2 - r sin θ

Figure 6(b) is a conceptual diagram showing another embodiment of converting a pan-tilt-zoom camera image into a fisheye image. Pan-tilt-zoom camera images can be converted to fisheye images using the FOV (Field of View) model. The position where the external point (A) forms an image on the sensor surface is located on the straight line 680 that passes through the external point (A) and the camera center (O). In the pinhole model, point A forms an image at point B where this straight line 680 and the image plane 670 at the focal distance R meet. That is, the distance r _u from the center to the undistorted image on the image sensor 650 corresponds to the distance from the center (O) to the point (B) on the plane or the point (B') projected on the sensor.

In the case of a fisheye lens in the FOV model, an image is formed at the point C' where the straight line 680 intersects the spherical surface 660 of radius R and is projected onto the image sensor 650. At this time, the distance r _d from the center O to the spherically distorted image C' on the image sensor 650 can be obtained using Equation 3 below.

[Equation 3]

Here, ω is a distortion coefficient that indicates the degree of distortion. When using a fisheye lens, the distortion coefficient ω can be estimated by minimizing the error function for the distortion coefficient ω. When using a pan-tilt-zoom camera, a tilt angle of view can be used for the distortion coefficient ω of the fisheye image.

High-resolution image synthesis requires a lot of computational resources. For example, when using a pan-tilt-zoom camera with a horizontal angle of view of 56 degrees and a vertical angle of view of 35 degrees, the pan angle interval is 45 degrees and the tilt angle interval is 45 degrees to capture the entire surveillance area without gaps. You can shoot from 25 degrees to 100 degrees. The entire area has a pan angle of 360 degrees, so 8 images are needed horizontally, and 4 images are needed vertically with a tilt angle of 117.5 degrees (= 100 degrees + 35 degrees / 2). In other words, approximately 32 images (=8 * 4) are needed for the entire surveillance range. In the case of a 2M (1920*1080) image, a large number of operations on the order of 2M * 32 = 64 M are required for image synthesis and conversion.

To reduce the amount of computation and memory usage in generating a fisheye image, a method can be used to match pixels of the fisheye image with pixels of an image captured by a pan-tilt-zoom camera. Since the pixels of a 1000 * 1000 fisheye image are 785 k ( = π * 500 ² ), not only can computational resources be saved by reducing the computation to about 0.8 M, but the fisheye image creation time can also be significantly reduced.

Figure 7 is a conceptual diagram showing a method of generating a fisheye image by matching pixels of the pan-tilt-zoom image to the fisheye image in a video surveillance system using a pan-tilt-zoom camera according to an embodiment.

According to an additional aspect, the fisheye image providing unit provides a fisheye image generated by extracting predetermined pixels from a plurality of partial images of the entire surveillance area captured with a pan-tilt-zoom camera.

Referring to FIG. 7, the coordinates of point B of the image 720 of the pan-tilt-zoom camera with tilt angle T _n can be matched with the coordinates of point A at the tilt angle T _f in the fisheye image center (O). there is. To find the coordinates of point B, consider a spherical surface whose radius is the focal length R of the pan-tilt-zoom camera. If the reference point (D) at the center of the pan-tilt-zoom camera image 710, which has a tilt angle of 0 degrees, is rotated by the tilt angle T _f of the fisheye image, point C on the spherical surface facing point A can be obtained. If point C is reversely rotated by the tilt angle T _n of the pan-tilt-zoom camera image (720), the coordinates of point C' can be obtained from the pan-tilt-zoom camera image, which can be converted into the pan-tilt-zoom camera image (710) with a tilt angle of 0 degrees. If you project it to , you can get the B' coordinates. Since the B' coordinate is the same in the image 720 of the pan-tilt-zoom camera with the tilt angle T _n , at point A at the tilt angle T _f in the fisheye image, the image 720 of the pan-tilt-zoom camera with the tilt angle T _n is B' coordinates can be matched or mapped. By mapping the image of the pan-tilt-zoom camera to all points of the fisheye image, the fisheye image can be formed at high speed while using less computational resources. Such mapping data can be stored using the mapping data storage unit 490.

Below, rotation conversion considering both pan angle and tilt angle will be described. First, in a reference coordinate system in which both the pan angle and the tilt angle are 0, the coordinates (R, 0, 0) of the reference point A in the center of the camera image can be obtained using the focal length R in Equation 4 below.

[Equation 4]

R = ( H / 2 ) tan ( ω / 2 )

Here, ω is the angle of view of the pan-tilt-zoom camera, and H represents the image length in the direction of the angle of view.

In the fisheye coordinate system with the center of the fisheye image as the origin, if the coordinates of the point representing point A are F _x pixels on the horizontal axis and F _y pixels on the vertical axis, the pan angle P _f and tilt angle T _f of point A are expressed in the following equations It can be obtained as 5.

[Equation 5]

P _f = arctan ( F _y / F _x )

T _f = T _m * ( d / R _f )

Here, T _m represents the tilt range, d is the pixel distance from the origin to point P, and R _f represents the pixel distance corresponding to the radius of the fisheye image. The tilt range T _m can be obtained by adding half the angle of view to the maximum angle of view of a fisheye camera or the maximum tilt value of a pan-tilt-zoom camera.

Meanwhile, the rotation transformation matrix W for rotation with the pan angle P _f and the tilt angle T _f can be obtained using Equation 6 below.

[Equation 6]

The coordinates of point C, where the reference point D (R, 0, 0) is rotated by the pan angle P _f and tilt angle T _f of point A, can be obtained by applying the matrix W of Equation 6. That is, the coordinates (x, y, z) of point C after rotation transformation can be calculated as in Equation 7 below.

[Equation 7]

Since pan-tilt-zoom camera images are captured with a certain preset pan and tilt angle range, it is common for them to not match the pan and tilt angles of point A. Therefore, from the plurality of images of the pan-tilt-zoom camera, the nth image with the pan angle P _f and tilt angle T _f closest to the pan angle P _f and tilt _angle T f of point A is found. To obtain the coordinates corresponding to point A in this image, the coordinates (x, y, z) of point C are rotated inversely by the pan angle P _n and tilt angle T _n . Since the reverse rotation matrix C _R is the transposed matrix of Equation 6, it is equivalent to the following Equation 8.

[Equation 8]

The coordinates (x', y', z') of point C', which is the reverse rotation of point C (x, y, z), can be obtained using Equation 9 below.

[Equation 9]

The image position (i, j) in the image of a pan-tilt-zoom camera with a pan angle Pn and a tilt angle Tn can be obtained using Equation 10 below.

[Equation 10]

In summary, in _the fisheye _{image ,} at _the point ( _{F y} , _F And the (i, j) coordinates obtained by Equation 10 above can be mapped in the nth image with the tilt angle T _n .

According to a further aspect, the entire surveillance area may be divided into partial surveillance areas with a pan angle smaller than the horizontal field of view (FOV) of the pan-tilt-zoom camera and a tilt angle smaller than the vertical field of view. If you change the pan angle and tilt angle to point toward the center of each divided partial surveillance area (720) and shoot with a pan-tilt-zoom camera, you can obtain multiple images (710, 720) for the entire surveillance area with overlapping parts. You can. At this time, areas with a small tilt angle corresponding to the center of the fisheye image have a large overlap, so the number of captured images can be reduced by increasing the pan angle interval.

According to an additional aspect, the fisheye image data is updated at preset intervals using a plurality of partial image data obtained by shooting the entire surveillance area with a pan-tilt-zoom camera. Fisheye image data can be provided using image data captured during initial setup, and the fisheye image can be updated periodically or at the user's request by shooting divided areas according to a preset division method.

According to an additional aspect, the driving parameter calculation unit 370 calculates a pan angle and a tilt angle that directs the pan-tilt-zoom camera toward the center of the selected portion, and performs the pan-tilt-zoom by the ratio of the angle of view of the selected portion and the preset angle of view. Set the camera zoom ratio.

If the selected area is one point 520, the pan angle P _f and the tilt angle T _f can be obtained using Equation 5. Set this value as the camera driving parameter.

If the selected area includes a range (510, 530), the pan angle P _f and the tilt angle T _f can be obtained using the position of the center point, as in the case of a single point (520). The location of the center point can be used by calculating the average of the pan and tilt angles of the corner points. If there is an area, the zoom factor can be adjusted to include the pan angle range and tilt angle range of the area where the camera view angle is set. In the case of preset maps, movement can be done immediately using preset driving parameters.

Figure 8 is a flowchart showing a video surveillance method using a pan-tilt-zoom camera according to an embodiment.

The video surveillance method using a pan-tilt-zoom camera includes the following steps. In order to provide a fisheye image, multiple images of the entire surveillance area are captured while changing the pan and tilt angles of the pan-tilt-zoom camera according to predetermined standards (S810). At this time, the images are moved at an angular interval smaller than the angle of view so that they overlap to prevent omission of the surveillance area in multiple images.

A fisheye image representing the entire surveillance area is provided using pan-tilt-zoom camera images (S820). A fisheye image may be provided in a first area of the display.

Events are monitored to see if input from the user and other notifications occur (S830). When an event occurs (S840), the event is processed according to established procedures. When a signal to change the camera shooting location is input from the user, the driving parameters of the pan-tilt-zoom camera for moving the camera to the corresponding location are calculated based on the location information of the selected part of the fisheye image (S850). The shooting area of the pan-tilt-zoom camera is moved using the calculated driving parameters (S860). Provides surveillance video of selected areas captured by a pan-tilt-zoom camera (S870). Surveillance video may be provided in a second area of the display.

Meanwhile, you can set a preset, which is the area to be captured in the fisheye image in advance (S880). The driving parameters for the corresponding area can be saved in the preset. If a preset is set, a preset map indicating the shooting location in the fisheye image can be provided (S890).

According to an additional aspect, in the step of calculating driving parameters (S850), the pan and tilt angles that direct the pan-tilt-zoom camera toward the center of the selected portion are calculated, and the ratio of the angle of view of the selected portion to the preset angle of view is calculated. Set the zoom ratio of the pan-tilt-zoom camera.

Figure 9 is a flowchart showing the steps of providing a fisheye image in an image surveillance method using a pan-tilt-zoom camera according to an embodiment.

According to an additional aspect, in the step of providing a fisheye image (S820), the fisheye image is updated using a plurality of partial image data obtained by shooting the entire surveillance area with a pan-tilt-zoom camera at preset intervals.

According to an additional aspect, in the step of providing a fisheye image (S820), a plurality of partial image data for the entire surveillance area captured with a pan-tilt-zoom camera may be synthesized and distorted to be generated. In this case, distorted data can be generated using Equations 1 to 3.

According to an additional aspect, in the step of providing a fisheye image (S820), a fisheye image may be generated by extracting predetermined pixels from a plurality of partial images of the entire surveillance area captured with a pan-tilt-zoom camera. This method is described in detail below.

First, in a reference coordinate system where both the pan angle and the tilt angle are 0, the coordinates (R, 0, 0) of the reference point D at the center of the camera image are obtained (S900). In a pan-tilt-zoom camera, if the angle of view is ω and the image length in the direction of the angle of view is H, R corresponds to the focal length and can be obtained using Equation 4.

[Equation 4]

R = ( H / 2 ) tan ( ω / 2 )

Next, in the fisheye coordinate system with the center of the fisheye image as the origin, a point A (F _x , F _y ) located at a distance of F _x pixels on the horizontal axis and F _y pixels on the vertical axis is selected (S910). And the pan angle P _f and tilt angle T _f of point A are obtained using Equation 5 (S920).

[Equation 5]

P _f = arctan ( F _y / F _x )

T _f = T _m * ( d / R _f )

Here, T _m represents the tilt range, d is the pixel distance from the origin to point F in the fisheye image, and R _f represents the pixel distance corresponding to the radius of the fisheye image. The tilt range T _m can be obtained by adding half the angle of view to the maximum angle of view of a fisheye camera or the maximum tilt value of a pan-tilt-zoom camera.

Next, calculate the coordinates of point C by rotating the reference point D by the pan angle and tilt angle of point A (S930). The rotation transformation matrix W for the pan angle P _f and the tilt angle T _f can be obtained using Equation 6.

[Equation 6]

Therefore, by applying the matrix W of Equation 3 to the reference point D (R, 0, 0), the coordinates (x, y, z) of point C after rotation can be calculated as in Equation 7.

[Equation 7]

Meanwhile, in the multiple pan-tilt-zoom camera images taken while changing the pan angle and tilt angle for the entire surveillance area, the pan angle P _f and tilt angle P _n and tilt angle close to the tilt angle T _f of point A of the fisheye image. Select the video image Img with T _n (S940). By subtracting the pan angle and tilt angle of point A from the pan angle and tilt angle of the surveillance area image, the image with the minimum value can be selected as the adjacent video image Img.

Find the point C' by rotating the point C obtained previously by the pan angle P _n and tilt angle T _n of the close-up image Img (S950). The reverse rotation matrix C _R is the transposed matrix of Equation 6, so it is equivalent to Equation 8.

[Equation 8]

Therefore, the coordinates (x', y', z') of point C', which is the reverse rotation of point C (x, y, z), can be obtained using Equation 9 below.

[Equation 9]

The image position (i, j) of point B' in the image of a pan-tilt-zoom camera with a pan angle P _n and a tilt angle T _n can be obtained using Equation 10 (S960).

[Equation 10]

_The image position _{of point A ( F} Copy the pixels (i, j) (S970). If the fisheye image formation is not completed (S980), the process of selecting a point of the fisheye image is returned and repeated until the fisheye image formation is completed (S910). When the fisheye image is completed, pixel information extracted from the fisheye image and the pan-tilt-zoom image can be stored as a mapping table (S990).

In the above, the present invention has been described through embodiments with reference to the accompanying drawings, but it is not limited thereto, and should be interpreted to encompass various modifications that can be easily derived by those skilled in the art. The claims are intended to cover these variations.

110: Camera 120: Network
130: Client 135: Display
140: Client 145: Display
310: imaging unit 320: control unit
330: driving unit
360: Fisheye image providing unit 370: Driving parameter calculation unit
380: Surveillance video provision unit
410: Preset map providing unit 430: Preset setting unit
450: Event monitoring unit 470: Camera operation control unit
490: Mapping data storage unit

Claims (14)

In a camera device capable of pan-tilt-zoom operation,
a memory configured to store surveillance video data; and
Including a processor electrically coupled to the memory,
The processor,
Generate a fisheye image representing the entire surveillance area,
Based on the location information of the selected part of the fisheye image, driving parameters for pan-tilt-zoom operation are calculated,
To output surveillance video data captured from the selected part
A camera device capable of pan-tilt-zoom operation. In claim 1,
The fisheye image is generated by extracting predetermined pixels from partial images taken in each partial area divided by a preset method among the entire surveillance area,
A camera device capable of pan-tilt-zoom operation. In claim 1,
The fisheye image is updated at a preset period using image data of the partial area re-photographed for each partial area divided in a preset manner among the entire surveillance area,
A camera device capable of pan-tilt-zoom operation. In claim 1,
The processor,
To display a preset area in the surveillance area where the location to be captured is preset on the fisheye image.
A camera device capable of pan-tilt-zoom operation is further configured. In claim 1,
The driving parameters are,
Pan angle and tilt angle for photographing the center of the selected part; and
A zoom factor set by the ratio of the angle of view of the selected portion and the angle of view of the camera;
A camera device capable of pan-tilt-zoom operation, including a. In a video surveillance system using a camera capable of pan-tilt-zoom operation,
A camera capable of pan-tilt-zoom operation to film the surveillance area;
a display that outputs images from the camera; and
A control unit that drives the camera and transmits the image captured by the camera to the display,
The control unit,
a fisheye image providing unit providing a fisheye image representing the entire surveillance area in a first area of the display;
a driving parameter calculation unit that calculates driving parameters for a pan-tilt-zoom operation of a camera based on location information of a selected portion of the fisheye image; and
a surveillance image providing unit that provides surveillance images captured for a selected portion in a second area of the display;
A video surveillance system using a camera capable of pan-tilt-zoom operation, including. In claim 6,
The fisheye image providing unit provides a fisheye image generated by combining and distorting a plurality of partial images taken by dividing the entire surveillance area into preset partial regions,
A video surveillance system using a camera capable of pan-tilt-zoom operation. In claim 6,
The fisheye image providing unit provides a fisheye image generated by dividing the entire surveillance area into preset partial areas and extracting predetermined pixels from each partial image.
A video surveillance system using a camera capable of pan-tilt-zoom operation. In claim 6,
The control unit,
a preset map providing unit that displays a preset area in which a location to be captured is preset among the surveillance area on the fisheye image;
A video surveillance system using a camera capable of pan-tilt-zoom operation, further comprising: In claim 6,
The driving parameter calculation unit,
Calculate the pan angle and tilt angle that directs the camera toward the center of the selected portion,
Setting the zoom factor based on the ratio of the angle of view of the selected part and the angle of view of the camera,
A video surveillance system using a camera capable of pan-tilt-zoom operation. In a video surveillance method using a camera capable of pan-tilt-zoom operation,
Providing a fisheye image representing the entire surveillance area;
calculating driving parameters for a pan-tilt-zoom operation of a camera based on location information of a selected portion of the fisheye image;
moving the camera toward the selected part using the driving parameters; and
providing surveillance images of the selected portion captured by a camera;
A video surveillance method using a camera capable of pan-tilt-zoom operation, including. In claim 11,
The step of providing the fisheye image is,
Dividing the entire surveillance area into preset partial areas and shooting partial images in each partial area; and
generating a fisheye image by extracting predetermined pixels from each captured partial image;
A video surveillance method using a camera capable of pan-tilt-zoom operation, including. In claim 11,
The above method is,
A preset map providing step of displaying a preset area in which a location to be captured is preset among the surveillance area on the fisheye image;
A video surveillance method using a camera capable of pan-tilt-zoom operation, further comprising: In claim 11,
The step of calculating the driving parameters is,
Calculating pan angles and tilt angles to direct the camera toward the center of the selected portion; and
setting the zoom magnification of the camera based on the ratio of the angle of view of the selected portion and the angle of view of the camera;
A video surveillance method using a camera capable of pan-tilt-zoom operation, including.

Images