KR102249380B1 - System for generating spatial information of CCTV device using reference image information - Google Patents

Abstract

A spatial information generation system of a CCTV apparatus using reference image information comprises: a reference image database unit which stores reference image data in link with reference geometry information including reference position data and first external geometry data having first posture data defining a view direction set in generating a reference image; a search and selection unit which obtains CCTV geometry information including second external geometry data with at least two dimensional position data in relation to the image data obtained from the CCTV apparatus and which searches for and selects the image data that can be matched with the reference image data by referring to at least the reference position data and the two dimensional position data; a matching unit which mutually matches the reference image data with the selected image data; a geometric transformation unit which generates correction CCTV geometry information including position data for correction with higher accuracy than the two dimensional position data by geometrically transforming the CCTV geometric information of the selected image data by means of geometric transformation modeling referring to the reference image data; and an image information generating unit which generates CCTV image information from the image data of the CCTV apparatus based on the correction CCTV geometry and matching information. The present invention can transform the image data, obtained by the CCTV apparatus, into spatial information with convenience and high precision.

Description

System for generating spatial information of CCTV device using reference image information

The present invention relates to a system and method for generating spatial information of a CCTV device using pre-stored reference image information as reference data.

With the development of communication technology, infrastructure technology, and camera technology, CCTV has been installed and used in many places, and the national government and local governments are also installing and using CCTV for various purposes such as traffic management, disaster preparedness, and crime prevention.

Accordingly, a large number of CCTVs are installed and used in many areas, and the situation at the corresponding location can be monitored by managing CCTV address information or longitude and latitude information together.

In general, in order to access the CCTV, the CCTV URL is accessed in real time through field values in the table such as address values or location values in a database arranged in a table format and analyzed in real time.

Most of the conventional analysis technologies have many research results on technologies for recognizing and analyzing objects and situations existing in CCTV by applying deep learning and machine learning technologies based on image information.

CCTV infrastructure combined with such deep learning and machine learning technologies are being attempted as part of a smart city project.

However, there are cases in which incorrect information is input or data is lost due to insufficient data management for CCTV-related information.

Typical CCTV standard data often involve a location error of several meters to tens of meters in the case of location information, and some CCTV data are often missing location information or the location is described in an ambiguous manner.

In addition, for precise spatial information analysis, direction information of CCTV is required, but there is a problem in that there is no direction information, so there is a limit to service in the form of a map in connection with spatial information.

Such limitations are limited in the case of organically analyzing the target area locally or globally. For this analysis, personnel who monitor screen images in real time, such as a CCTV control center, are required, and infrastructure for analyzing images. I need a cost. As a result, it is acting as an obstacle to the realization of a smart city.

On the other hand, in order to provide spatial information service through CCTV images, the location and attitude information of the CCTV is important. Location and posture information is important data for grasping the location of objects displayed in an image on a map.

In order to overcome this limitation, it is necessary to construct reference data to correct the position and posture of the CCTV, and to extract spatial information of objects detected in the image through the corrected information. However, because CCTVs are distributed sporadically, collecting reference data also requires a lot of manpower and cost. In addition, in order to connect the collected reference data with CCTV image information, the image coordinate position of the reference data must be registered in the CCTV image. However, this operation also consumes a lot of manpower and cost, and it has the disadvantage that it is difficult to determine the correct position and posture value when the zoom, pan, and tilt functions are activated on the CCTV.

The technical problem to be achieved by the present invention is to match the image data of a CCTV device having geometric data related to position and posture that is incorrect or missing in some areas with the reference image information of the reference image data generated from a multi-sensor platform composed of an observation sensor and a position sensor. The object of the present invention is to provide a system and method for generating spatial information of a CCTV device using reference image information for generating convenient and high-precision spatial information from image data acquired by a CCTV device by performing geometric transformation and the like.

The object of the present invention is not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention for achieving the above technical problem, a system for generating spatial information of a CCTV device using reference image information includes reference position data and first posture data defining a viewing direction set to generate a reference image. 1 CCTV including a reference image database unit that stores reference image data in association with reference geometric information including external geometric data, and second external geometric data having at least two-dimensional position data in relation to the image data acquired from a CCTV device. A search selector for acquiring geometric information and searching for and selecting the image data matchable with the reference image data by referring to at least the reference position data and the 2D position data; and the reference image data and the selected image data A corrected CCTV including a matching unit that matches each other and the position data for correction with higher accuracy than the two-dimensional position data by geometrically converting the CCTV geometric information of the selected image data by geometric transformation modeling referring to the reference image data And a geometric conversion unit that generates geometric information, and an image information generation unit that generates CCTV image information from image data of the CCTV device, based on the correction CCTV geometric information and matching information.

In another embodiment, the two-dimensional position data includes a two-dimensional coordinate of the CCTV device or a name of a space in which the CCTV device is installed, and the reference position data is more accurate than the two-dimensional position data of the first external geometric data. When it is composed of high 2D position data or 3D position data, and the reference position data is 2D position data, the CCTV image information includes 2D position coordinates, and the reference position data is 3D position data. In some cases, the CCTV image information may include 3D position coordinates.

In another embodiment, the second external geometric data further includes second posture data defining a viewing direction set to generate the image, and the search selection unit is based on the reference position data and the 2D position data. A candidate group of the image data that can be contrasted with the reference image data may be searched, and the image data having the highest degree of matching with the reference image data may be selected from among the candidate groups by referring to the first and second posture data.

Here, the reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor set to generate the reference image, and the CCTV geometric information is a geometric parameter defined in an image sensor set to generate the image. It further includes second internal geometric data based on, and the selection of the image data having the highest degree of matching by the search selection unit and correction of the CCTV geometric information in the geometric conversion unit are performed by the first and second Internal geometric data may be further included for reference.

In another embodiment, the reference image data is obtained from an external mobile platform having at least one of an image capturing sensor and a laser scanning sensor and a positioning sensor, through a geometric model between the sensors. It may be external image data that is matched and corrected to be fused, or corrected image data that has been corrected with the CCTV geometric information for correction and has already been stored as CCTV image information.

In another embodiment, when at least one of the reference image and the image data is obtained from a wide area image sensor that generates a distortion image, the reference image and the image data in which the distortion image is generated are the distorted image When planar image data converted into a planar image is used and the reference image uses the planar image data, the first posture data may be defined as a viewing direction of the planar image data.

In another embodiment, when the reference image originates from a laser scanning sensor, the reference image uses virtual image data converted from a point cloud obtained from the laser scanning sensor, and the reference position data is It is defined as position data of the virtual image data, and the first posture data may be defined as a viewing direction of the virtual image data.

In another embodiment, the matching unit extracts and compares feature geometry from the selected image data and the reference image data, and matches the commonly estimated feature geometry to each other, or at least similar to a template designated in the reference image data. The selected image data may be searched and matched according to a feature correlation analysis between the templates, and the matching information may be the feature geometry or the template, and the unique feature geometry or the template may be stored in a reference image database.

In another embodiment, by randomly sampling the sample feature geometries of the image data to which the correction CCTV geometric information is linked, the distance between the sample feature geometries and the corresponding feature geometries of the reference data is minimized. An optimum value estimating unit for estimating the geometry as a matching optimum value; And a geometric information estimating unit for estimating the corrected CCTV geometric information geometrically converted in relation to the feature geometry estimated as the matching optimal value as final corrected CCTV geometric information, wherein the information based on the generation of the CCTV image information As for the correction CCTV geometric information, the final corrected CCTV geometric information may be used.

In another embodiment, based on an object information extraction unit for extracting an object of a specific type from the CCTV image information, and the object of the same reference image data as the extracted object, the correction CCTV geometric information and the reference image data A spatial information generation unit that generates new spatial information including the object information related to the stored object in the CCTV image information, and the new spatial information is stored in the spatial information database unit, and the existing information already stored in the spatial information database unit When spatial information exists, it is determined whether the new spatial information is different from the existing spatial information in object information of the same location, and as a result of the determination, the existing spatial information is converted to the new spatial information for different object information. It may further include a spatial information update unit to update to.

In another embodiment, the reference geometric information is composed of 3D position data as the reference position data and the first external geometric data having the first posture data, and is related to a partial position of the reference image data. When the precision of the geometric information is lower than the threshold value, the image information generator converts the corrected image data of the CCTV image information corresponding to the reference image data with low precision into a two-dimensional geometric model, and then the two-dimensional converted image Estimate the height of the object through pixel data analysis in the data, analyze the object height by assuming the height of the image sensor of the CCTV device as a specific height, or specify the attitude data related to the viewing direction of the plurality of image sensors of the CCTV device. It is defined as a value, and the object height is estimated using a trigonometric function model, and the 3D position data is estimated by reflecting the estimated object height to the corrected CCTV geometric information, thereby generating the corresponding CCTV image information. .

Details of other embodiments are included in the detailed description and drawings.

According to the present invention, the image data of a CCTV device having geometric data related to position and posture that is incorrect or missing in some areas is matched with the reference image information of the reference image data generated from a multi-sensor platform composed of an observation sensor and a positioning sensor, By performing conversion correction or the like, image data acquired by a CCTV device can be generated as spatial information with convenience and high precision.

In addition, by linking and analyzing the corrected position and posture information and image information acquired from CCTV, object information expressed in the image is converted into spatial information, and linked with a map service.

In addition, by processing observation data acquired from low-cost sensors excluding expensive sensors such as lidar when constructing the reference image data, it is possible to achieve a reduction in construction cost.

In addition, by storing the corrected geometric information such as the position and posture of the CCTV on the reference image database and using it when generating 3D spatial information of the image data of other CCTVs in a similar area, various types of reference image data can be used. Can be utilized.

In addition, effects derived through configurations that can be grasped by those skilled in the art through the present specification are not excluded.

1 is a block diagram of a system for generating spatial information of a CCTV device using reference image information according to an embodiment of the present invention, a CCTV device and a CCTV controller communicating therewith.
2 is a flowchart illustrating a method of generating CCTV image information for image data of a CCTV device implemented in a spatial information system of a CCTV device using reference image information according to an embodiment of the present invention.
3 is a diagram illustrating an example of reference image information stored in a reference database unit.
4 is a diagram illustrating types of data stored in first external geometric data and first internal geometric data when reference image information stored in a reference database unit is obtained from an image capturing sensor.
5 is a diagram illustrating a process of converting a geometric shape to search for and extract a feature geometry when reference image information stored in a reference database unit is obtained from aerial observation data.
6 is a flowchart of a first process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data.
7 and 8 are diagrams schematically illustrating a first process of image distortion removal and planar image data conversion.
9 is a flowchart illustrating a second process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data.
10 is a diagram schematically illustrating a second process of image distortion removal and planar image data conversion.
11 is a diagram illustrating a form in which virtual image data converted from a point cloud obtained from a laser scanning sensor in a reference image database unit is stored.
12 is a flowchart of another embodiment of search selection in a method of generating spatial information for video data of a CCTV device.
13 is a diagram illustrating a matching process using feature geometry in a matching unit.
14 is a flowchart showing a matching process using a characteristic correlation analysis between templates in the matching unit.
15 is a diagram illustrating a process of calculating 3D position data of reference image data corresponding to 2D position data of image data.
16 is a flowchart illustrating a process of estimating the height of an object of CCTV image data by an image information generator.
17 is a diagram illustrating a process of estimating an object height through pixel data analysis in CCTV image data.
18 is a flowchart illustrating a method of generating and updating spatial information implemented in a spatial information system of a CCTV device using reference image information according to an embodiment of the present invention.
19 is a diagram illustrating an example of generating new spatial information including final corrected CCTV geometric information and object information in CCTV image information.
20 is a diagram for describing a process of updating spatial information.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings and the following description. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed contents may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art. The same reference numbers throughout the specification indicate the same elements. Meanwhile, terms used in the present specification are for explaining embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the recited component, step, operation and/or element is Or does not preclude additions.

In addition, "sub" to "module" generally refer to components such as software (computer, program) and hardware that can be logically separated. Accordingly, the module in the present embodiment refers not only to a module in a computer or a program, but also a module in a hardware configuration. Therefore, the present embodiment is a computer program for making them function as modules (a program for executing each step in a computer, a program for making a computer function as each means, a program for realizing each function in a computer) ), also describes the system and method. However, for convenience of explanation, phrases equivalent to "save" and "save" are used, but these words are stored in a memory device or stored in a memory device when the embodiment is a computer program. It means to control like that. In addition, the "sub" to the module may correspond to the function one-to-one, but in the implementation, one module may be composed of one program, a plurality of modules may be composed of one program, and conversely, one module may be composed of a plurality of programs. do. Further, a plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers by a computer in a distributed or parallel environment. In addition, another module may be included in one module. In addition, hereinafter, "connection" is adopted in the case of logical connection (data exchange, instruction, reference relationship between data, etc.) in addition to physical connection. The term "predetermined" means that it is determined before the target processing, as well as before the processing according to the present embodiment is started, even after the processing according to the present embodiment is started, and before the target processing, the It includes the meaning of what is determined according to the situation and state of the time, or the situation and state up to that time.

In addition, a system or device is configured by connecting a plurality of computers, hardware, devices, etc. by means of communication such as a network (including a one-to-one communication connection), and a single computer, hardware, device, etc. This includes cases where it is realized. The terms "device" and "system" are used as terms of mutual agreement. Of course, the "system" does not include anything but an artificial decision, a social "mechanism" (social system).

In addition, when processing by each unit or module or a plurality of processing within each unit or module, the target information is read and input from the storage device for each processing, and after the processing is performed, the processing result is displayed. It is to write to the storage device. Therefore, the description of the read input from the storage device before processing and the writing to the storage device after processing may be omitted in some cases. Further, the storage device herein may include a hard disk, a random access memory (RAM), an external storage medium, a storage device through a communication line, a register in a CPU (Central Processing Unit), and the like.

Hereinafter, referring to FIG. 1, corrected image data of a CCTV device generated by performing a predetermined process by matching image data of two-dimensional position information acquired from a CCTV device according to an embodiment of the present invention with reference image data of a reference image database unit. A system for generating spatial information of a CCTV device for generating CCTV image information based on will be described.

1 is a block diagram of a system for generating spatial information of a CCTV device using reference image information according to an embodiment of the present invention, a CCTV device and a CCTV controller communicating therewith.

The system 100 transmits the two-dimensional position data of the CCTV device 102 and/or the image data included in the image data and attribute information of spatial objects such as terrain and features captured by the CCTV device 102 to the CCTV controller ( 104). As attribute information, the CCTV controller 104 receives information such as a viewing direction of the image sensor 106 of the CCTV device 102 estimated at the time of photographing, and a geometric parameter unique to the image sensor 106.

First, when describing the CCTV device 102 and the CCTV controller 104, the CCTV device 102 may include an image sensor 106, a communication unit 108, and a control unit 110 for controlling and managing them, and The file 104 may include an attribute information management unit 111, an image storage unit 112 and a control/communication unit 114.

The CCTV device 102 transmits the image data photographed through the image sensor 106 to the CCTV controller 104 by the communication unit 108. The CCTV controller 104 is a device that transmits information in which the two-dimensional position data of the CCTV stored as attribute information is combined with the received image data to the system 100 through a wired or wireless network. As described above, the attribute information may further include a photographing time of the image sensor 106, a viewing direction of the image sensor 106, and a unique geometric parameter, and image data in which the viewing direction and the geometric parameter are further combined It is sent to the system 100.

The image sensor 106 may be equipped with a plurality of sensors that are mounted on the CCTV device 102 to acquire image data and time information for an image at the time of acquisition of the image data by taking images of surrounding objects, such as terrain and features. have. The image sensor 106 may be a flat image sensor, a wide area image sensor having a wider field of view than the planar image sensor, and a surveying or non-surveying camera, but is not limited thereto. The wide-area image sensor generates a stereo image in which images having different viewing directions are combined, or a panoramic image in which images from a camera disposed within a predetermined viewing angle are combined, and for example, a camera that has a fisheye lens to generate a fisheye lens image Alternatively, it may be a multi-camera system that outputs omnidirectional images by multiple cameras using a plurality of cameras. The module in which the image sensor 106 is mounted may further include zoom, pan, and tilt functions.

In the case where the image data originates from a wide area image sensor generating a distorted image, the image data is processed by the search selection unit 118 to be described later. Planar image data generated by converting a distorted image to planar image data can be used, and a detailed conversion process will be described later.

The attribute information management unit 111 of the CCTV controller 104 manages the attribute information of the plurality of CCTV devices 102 in the form of a table recorded in numerical or text, and the attribute information is transferred to the image data by the control/communication unit 114. Combine.

The attribute information includes two-dimensional position data of the CCTV device 102, and the two-dimensional position data may be the two-dimensional coordinates of the CCTV device 102, or the name of the space where the CCTV device is installed as schematic information other than coordinates. have. The attribute information may additionally have a photographing time of the image sensor 106. In some cases, the attribute information may additionally include a photographing direction of the image sensor 106, a viewing direction such as an angle, a viewing angle, and a geometric parameter defined in the image sensor 106, but typical attribute information is generally two-dimensional. Since only the location data is included, a process of selecting the image data by distinguishing between the case of having only the 2D location data and the case including the viewing direction and even geometric parameters will be described later.

Viewing direction, posture data having a viewing angle (hereinafter referred to as'second attitude data'), two-dimensional position data, and shooting time can constitute external geometric data (hereinafter referred to as'second external geometric data'). , The geometric parameter defined in the image sensor 106 may constitute internal geometric data (hereinafter, referred to as “second internal geometric data”).

The second external geometric data is an error of observation data calculated based on a parameter value that varies each time image information of the image sensor 106 is acquired due to the position and posture of the image sensor 106, and in this case, a predetermined It can be calculated through the equation or modeling of. When the image data originates from a wide-area image sensor, the 2D position data utilizes 2D position data of the planar image data, and the second posture data may be defined as a viewing direction and a viewing angle of the planar image data.

The second internal geometric data is an intrinsic value of the image sensor 106 itself, and is a geometric parameter related to an error of the observed data of the image sensor 106. The geometric parameter of the image sensor 106 may include any one or more of a focal length, a main point position, a lens distortion parameter, and a sensor format size.

The control/communication unit 114 controls the image data acquired from the image sensor 106 to generate CCTV geometric information including at least two-dimensional position data and external geometric data having a photographing time and transmit it to the system 100 , It is possible to perform necessary processing by receiving specific data from the system 100.

The system 100 includes a reference image database unit 116, a search selection unit 118, a matching unit 120, a geometric conversion unit 122, a matching optimal value estimating unit 124, a CCTV geometric information estimating unit 126. , An image information generating unit 128, an object information extracting unit 130, a spatial information generating unit 132, a spatial information updating unit 134, and a spatial information database unit 136.

The reference image database unit 116 includes the reference position data of the reference image, the acquisition time of the reference image, the viewing direction set to generate the reference image, and first external geometric data having first posture data defining the viewing angle. The reference image data is stored in connection with the geometric information.

The reference position data may be composed of two-dimensional position data or three-dimensional position data with higher accuracy than the two-dimensional position data of the first external geometric data, and is not schematic information of the two-dimensional position data of the CCTV device 102, It may be 2D position data of an XY coordinate system or 3D position data of an XYZ coordinate system. The viewing direction and viewing angle are substantially the same as those described for the image sensor 106. In addition, the reference geometric information may further include first internal geometric data based on a geometric parameter defined in a sensor set to generate a reference image.

The reference image is obtained from the outside, is an image including at least reference position data, and is processed into reference image data that can be contrasted with the image data obtained from the CCTV device 102. The first sensor used to generate the reference image may be at least one of an image acquisition device having higher positioning accuracy and image realization performance than the CCTV device 102, and a laser scanning sensor (eg, Lidar). The reference geometric information linked to the reference image includes color information such as RGB data, first external geometric data, and first internal geometric data for each pixel coordinate, and the reference image combined with the reference geometric information is stored as reference image data. Can be.

The reference image data matches and corrects image information and point cloud information acquired from an external mobile platform having at least one of an image capturing sensor (image acquisition device) and a laser scanning sensor and a positioning sensor through a geometric model between the sensors. It may be the fused external image data, or the corrected image data of the existing CCTV device already stored in the image information generation unit 128 and/or the spatial information database unit 136 with CCTV image information corrected with the corrected CCTV geometric information. have.

When the first sensor used to generate the reference image is an image capturing sensor, the first internal geometric data is substantially the same as the image sensor 106 described in the second internal geometric data. In the case of the image sensor, the projection center and direction used to project the three-dimensional shape of the object in two dimensions are related to the reference position data and the viewing direction of the reference image, so the projection center and direction are together with the first external geometric data. It may be stored as reference image data. The reference geometric information of the reference image data generated from the image capturing sensor may include 2D position data or 3D position data as reference position data.

When the reference image is aerial observation data, since it has a different geometry from ground observation image data such as the CCTV device 102, the aerial observation data may be converted into a view form in a specific direction. For example, each pixel of aerial observation data may be converted into an orthogonal image form having a horizontal geometry and used as a reference image.

When the reference image originates from a wide-area image sensor that generates a distorted image, the reference image uses flat image data generated by converting the distorted image into flat image data, and the reference position data uses the position data of the flat image data. In addition, the first posture data may be defined as a viewing direction and a viewing angle of the planar image data. The detailed conversion process will be described later.

When the reference image originates from the laser scanning sensor, the reference image may use virtual image data converted from a point cloud obtained from the laser scanning sensor, and the 3D position data is a 3D position of the virtual image data. It is defined as data, and the first posture data may be defined as a viewing direction of the virtual image data. Transformation of the point cloud may be performed by virtual camera modeling taking into account the reflection intensity and altitude value of the point cloud. When an image sensor is additionally involved in generating the reference image, color data acquired by the image sensor at the time of conversion of the point cloud may be included in the virtual image data.

When the first sensor is a laser scanning sensor, the first internal geometric data may be, for example, at least one of an incident angle of each laser, a distance scale, a distance direction offset, and an axial offset. The point cloud information acquired by the laser scanning sensor may be expressed as a set of points including all or part of 3D coordinate information, laser intensity information, color information, class information, time information, and sensor observation information.

When the reference image is acquired from an external mobile platform equipped with multi-sensors such as an image photographing sensor, a laser scanning sensor, and a positioning sensor, the registration and correction of the loading model between the sensors are performed by the image sensor and the laser scanning sensor (LIDAR). ), an external geometric model of a lidar for combining data with a positioning sensor, and an external geometric model of an image capturing sensor for combining data with a positioning sensor. Each geometric model may be defined by a predetermined equation. Accordingly, each image information and point cloud information obtained from the image capturing sensor and the laser scanning sensor may be combined and fused, and the point cloud information may have color information, or the image information may have three-dimensional position coordinates for each pixel. I can have it. The point cloud information is converted into virtual image data, as described above, and this data may be employed as reference image data.

In addition, the reference image database unit 116 connects and stores information related to a specific type of object, that is, object information in the reference image data. For example, the object may correspond to a building that needs to be displayed on a map, a traffic sign that needs to be recognized for autonomous driving, a traffic light, a lane, and the like. In this case, the object information may include the type of the object, the overall shape, a specific geometric shape such as a line, surface, circle, or sphere constituting at least a part of the object, a color, and texture-related properties.

Meanwhile, the search selection unit 118 acquires CCTV geometric information in relation to the image data acquired from the CCTV device 102, and searches for image data that can be matched with the reference image data by referring to at least the reference position data and the two-dimensional position data. And select it. The search selection unit 118 searches for two-dimensional coordinates of the reference position data when the two-dimensional position data has coordinates, and the name of the installed space of the CCTV device 102 in which the two-dimensional position data is rough data rather than coordinates. In this case, the reference image database unit 116 searches for registered reference position data related to the name of the space.

The reference position data may be composed of 2D position data or 3D position data having higher accuracy than the 2D position data of the first external geometric data. When the reference position data is two-dimensional position data, the search selection unit 118 searches for CCTV geometric information and second-dimensional position data of the reference geometric information. When the reference position data is 3D position data, 2D position data of CCTV geometric information and 2D position data of XY coordinates extracted from 3D position data are searched.

Unlike the above embodiments, when the CCTV geometric information includes all of the second posture data and the second internal geometric data, the search selection unit 118 includes the reference position data, the two-dimensional position data, and the first posture data. And a candidate group of image data that can be matched with the reference image data by referring to the second external geometric data and the first and second internal geometric data, and among the candidate groups, image data having the highest degree of matching with the reference image data is selected.

Specifically, the search selector 118 searches for a candidate group of image data that can be contrasted with the reference image data based on the reference position data and the two-dimensional position data, and refers to the first and second posture data to determine the reference image data among the candidate groups. You can select the image data with the highest degree of matching with.

When the image data acquired from the CCTV device 102 is originated from a wide area image sensor, the search selection unit 118 provides a process of removing distortion of the wide area image and converting it into flat image data. This can be done before the data selection process.

The matching unit 120 matches the selected image data and the reference image data with each other. The matching may be implemented by a method by feature geometry extraction or by a method by a template feature specified in an image.

First, the method by feature geometry extraction is to match the reference image data and the selected image data based on a predetermined type of feature geometry that is common between the reference image data and the selected image data, and refer to the feature geometry to determine the geometry such as a collinear conditional expression. A plurality of selected image data may be combined through the model.

The feature geometry of a predetermined shape may be extracted based on the selected image data estimated to be the same location and the specific geometric shape of the object expressed in the reference image data. The object may be randomly selected as an object from which feature geometry such as points, lines, surfaces, pillars, etc. can be easily extracted from the image data, such as road signs, unusually shaped objects, and building boundaries. In addition, feature geometry extraction may be performed based on a color change in image data. Such a feature geometry adopts a predetermined shape with a high probability of being estimated to be the same object even though the attitude data between the selected image data and the reference image data are slightly different, and may be, for example, a lane displayed on a road surface, a road sign, a building outline, or the like.

Here, the image data may be used as a descriptor having a characteristic capable of representing a specific object as well as data in a matrix form having color information. In the case of a descriptor having a characteristic representing a specific object, SIFT (Scale Invariant Feature Transform) can be used as an image comparison algorithm.

In order to calculate a descriptor according to SIFT, scale-space extrema, key-point localization & filtering, orientation assignment, and descriptor generation may be performed in the order of scale-space extrema.

Specifically, if a simple boundary is detected in the image data, a lot of edges including noise are also detected. Therefore, first, a Gaussian filter is applied at various levels (σ value is adjusted) to blur the image data, and then the image data When the difference between the and the reference image data is calculated, a characteristic robust to noise is obtained (Difference of Gauassian: DoG). When the work on the current scale is finished, the same work is performed while changing the size of the image so that it can respond to the scale change, and extrema, which is a strong feature at different sizes, is obtained.

Even though a relatively strong extrema is extracted through DoG, it still contains a lot of unnecessary information, so only meaningful information is extracted through keypoint localization and filtering. Next, a histogram according to the direction of the gradient is calculated.

An N×N×m-dimensional descriptor is generated in m directions in an N×N gradient window, and the image matching is determined by comparing the descriptors.

According to the method according to the template characteristics, the matching unit 120 searches for the selected image data that is at least similar to a template designated in the reference image data, and matches the selected image data according to a characteristic correlation between the templates. The template may be, for example, an object occupying a significant area in image data such as a pre-designated sign or sign. In the characteristic correlation analysis, in order to improve the matching speed, a method of matching a similar characteristic region by converting a spatial characteristic of an image into a frequency characteristic through Fourier transform can be used. The feature geometry or template constitutes matching information, and the matching information may be stored in the reference image database unit 116.

The geometric conversion unit 122 geometrically converts the CCTV geometric information of the selected image data by geometric transformation modeling that refers to the reference image data in common feature geometries or templates, so that the accuracy is higher than that of the 2D position data. Compensated CCTV geometry information including location data is generated.

For geometric transformation modeling, a collinear conditional expression such as Equation 1 may be used, or alternative modeling may be used for one of perspective transformation, affine transformation, and homography transformation that may reflect the geometry of image data in a matrix form.

[Equation 1]

Image coordinates are two-dimensional position data of image data related to the same feature geometry or template, and object coordinates are three-dimensional position data of reference image data with respect to the same feature geometry or template. The camera position and posture are first external geometric data of the reference image data, and the main point position, focal length, and lens distortion are first internal geometric data. In the above, an example in which the reference position data is 3D position data is described as an example in which the reference image data is expressed as 3D position data. However, when the reference position data is 2D position data, the reference image data is 2D. It can be expressed as two-dimensional position data of coordinates.

The matching optimal value estimating unit 124 randomly samples the feature geometries of the sample image data to which the correction CCTV geometric information is linked, so that the mutually spaced amount of the sampled feature geometries and the corresponding reference data feature geometries is minimized. The feature geometry for the sample can be estimated as a matching optimal value.

Specifically, a plurality of image data sharing the same estimated feature geometry for a sample are sampled. These sampling sets are acquired in plurality for different standard feature geometries. At the same time, reference image data having a feature geometry that is estimated to be the same as the sample feature geometry is interlocked to be selected. Subsequently, when the image data sharing the same estimated sample feature geometry and the reference image data are arranged according to the CCTV geometry information for correction, the amount of mutual separation between the sample feature geometry of the image data and the feature geometry of the reference image data The sample feature geometry of the image data judged to be the minimum is estimated as a matching optimal value.

The CCTV geometric information estimating unit 126 may estimate the geometrically converted CCTV geometric information for correction in relation to the feature geometry estimated as the matching optimal value as the final corrected CCTV geometric information.

The image information generation unit 128 generates CCTV image information from image data of the CCTV device 102 based on the final corrected CCTV geometry information and matching information for the feature geometry.

CCTV image information, like the reference image data of the reference image database unit 116, the image data corrected by geometric transformation, the position data associated therewith, the corrected second external geometric data including the corrected second posture data, It may include final corrected CCTV geometric information including the corrected second internal geometric data, color data, time data, and the like.

Position data associated with the corrected image data may be composed of two-dimensional position coordinates when the reference position data is two-dimensional position data, and may be composed of three-dimensional position coordinates when the reference position data is three-dimensional position data.

On the other hand, the reference geometric information is composed of 3D position data as reference position data and first external geometric data having first posture data, and the accuracy of the reference geometric information of the reference image data related to some positions of the image data is lower than the threshold. In this case, the image information generation unit 128 may convert the 3D image information of the image data of the CCTV device 102 corresponding to the reference image data with low precision into a 2D geometric model.

The image information generator 128 estimates the height of the object through pixel data analysis in the 2D converted image data, or analyzes the height of the object by assuming the height of the image sensor 106 of the CCTV device 102 as a specific height. Alternatively, the second posture data related to the viewing direction of the plurality of image sensors of the CCTV device 102 may be defined as a specific value, and the height of the object may be estimated using a trigonometric function model. Accordingly, the 3D image information of the corresponding image data may be generated by estimating 3D position data by reflecting the estimated object height to the final corrected CCTV geometric information. This can be applied when the reference image data has low precision of reference geometric information, such as an aerial image or a street view (street view).

The object information extraction unit 130 may extract a specific type of object from CCTV image information resulting from the image data.

Specifically, the object information extraction unit 130 may extract object information related to an attribute of the extracted object based on data included in the CCTV image information.

The object information extraction unit 130 detects candidate group data related to the object identified from image data included in CCTV image information, final corrected CCTV geometric information, color data, time data, etc., using a machine learning technique, and uses the detected candidate group data. The information of the object can be recognized by sequentially applying additional machine learning models.

Machine learning is performed in two stages: object object detection and object object recognition, and in the object object detection stage, candidate groups are detected in the data based on properties such as the shape, color, and texture of the object object through machine learning. And, by sequentially applying an additional machine learning model to the detected candidate group, the information represented by the target object is recognized and classified.

The spatial information generator 132 includes a predetermined coordinate system including, in CCTV image information, the final corrected CCTV geometric information and object information related to the object stored in the reference image data, based on the object of the same reference image data as the extracted object. New spatial information according to the coordinate system is generated, and may be transmitted to and stored in the spatial information database unit 136. The spatial information is spatial data to be described later, and detailed information will be described later.

The spatial information database unit 136 organizes and stores data previously generated for drawing as spatial data based on observation data and observation data previously acquired from heterogeneous sensors according to a predetermined coordinate system, for example, an absolute coordinate system. Spatial data is data generated in the corresponding coordinates of a predetermined coordinate system, such as observation data, internal geometry, external geometry, intrinsic data, time information, acquisition environment information, and the amount of separation, accuracy, and precision due to the positional error of sensor data between heterogeneous sensors. It may include related probability information, point cloud-related information, image data-related information, object information, and map-related information. In the present embodiment, the spatial data is described as including all of the above-described data or information, but is not limited thereto, and may be set to include only some of the above-described data/information according to design specifications.

When the existing spatial information already stored in the spatial information database unit 136 exists, the spatial information update unit 134 determines whether the new spatial information is different from the existing spatial information in the object information at the same location, As a result of the determination, for different object information, existing spatial information may be updated with new spatial information.

According to the present embodiment, the image data of the CCTV device 102 having geometric data related to the position and posture that is incorrect or missing in some areas is reference image information of the reference image data generated from a multi-sensor platform consisting of an observation sensor and a positioning sensor. By matching with and performing geometric transformation correction, etc., image data acquired from a CCTV device can be generated as spatial information with convenience and high precision. Specifically, when the two-dimensional position data linked to the image data of the CCTV device 102 has low accuracy, is missing from the beginning, or is recorded with a rough space name, the CCTV image corrected by the reference geometric information The information may have location data composed of coordinates with higher accuracy than the original image data.

In addition, by linking and analyzing the corrected position and posture information and image information acquired from CCTV, object information expressed in the image is converted into spatial information, and linked with a map service.

In addition, by processing observation data acquired from low-cost sensors excluding expensive sensors such as lidar when constructing the reference image data, it is possible to achieve a reduction in construction cost.

In addition, in this embodiment, the final corrected CCTV geometric information, etc., is generated with the feature geometries with the minimum mutual separation between the feature geometries estimated to be the same through the matching optimal value estimating unit 124, but the second external geometric data is In the case of data capable of satisfactory matching between image data, the matching optimal value estimating unit 124 may be omitted if the condition in which the mutually spaced amount between the feature geometries is less than or equal to the threshold is satisfied.

Hereinafter, with reference to FIGS. 2 to 16, FIG. 2 is a method of generating CCTV image information for image data of a CCTV device implemented in a spatial information system of a CCTV device using 3D image information according to an embodiment of the present invention. I will explain. In this embodiment, the function of the matching optimal value estimating unit 124 is performed. However, as described above, this can be omitted.

2 is a flowchart illustrating a method of generating 3D image information for image data of a CCTV device implemented in a spatial information system of a CCTV device using CCTV image information according to an embodiment of the present invention.

First, the search selection unit 118 acquires CCTV geometric information including second external geometric data composed of two-dimensional position data related to the image data acquired from the CCTV device 102 (S205).

Next, the search selector 118 includes at least the reference position data and the image data among the reference geometric information of the reference image data including the first external geometric data and the first internal geometric data having the reference position data and the first posture data. A candidate group of image data is searched and selected based on the 2D position data (S210).

The two-dimensional location data may be a two-dimensional coordinate of the CCTV device 102, or a name of a space in which the CCTV device is installed as schematic information rather than coordinates. When the two-dimensional position data has coordinates, the two-dimensional coordinates of the three-dimensional position data are searched, and if the two-dimensional position data is the name of the installed space of the CCTV device 102, which is rough data rather than coordinates, the reference image database unit At (116), the registered 3D location data related to the name of the space is searched.

The reference position data may be composed of 2D position data or 3D position data having higher accuracy than the 2D position data of the first external geometric data. When the reference position data is two-dimensional position data, the search selection unit 118 searches for CCTV geometric information and second-dimensional position data of the reference geometric information. When the reference position data is 3D position data, 2D position data of CCTV geometric information and 2D position data of XY coordinates extracted from 3D position data are searched.

The reference image data is stored in the reference image database unit 116, and in the case of the reference image data linked to the 3D position data, as shown in FIG. 3, the above-described data including 3D position data in the pixels displayed in dark color are Is saved. The 3D location data may be 2D location data, that is, XYZ coordinates obtained by adding an altitude value Z from the XY coordinates.

In this way, the first external geometric data includes the above-described first posture data together with the reference position data, and the first internal geometric data is based on a geometric parameter defined in a sensor set to generate the reference image.

The reference image is an image that is obtained from the outside and includes at least reference position data, and is processed into reference image data that can be contrasted with the image data obtained from the CCTV device 102. The first sensor used to generate the reference image may be at least one of an image acquisition device having higher positioning accuracy and image realization performance than the CCTV device 102, and a laser scanning sensor (eg, Lidar). The reference geometric information associated with the reference image includes color information such as RGB data, first external geometric data, and first internal geometric data for each pixel coordinate, and the reference image combined with the reference geometric information is stored as reference image data. Can be.

The reference image data matches and corrects image information and point cloud information acquired from an external mobile platform having at least one of an image capture sensor (image acquisition device) and a laser scanning sensor and a positioning sensor through a geometric model between the sensors. Thus, it may be fused external image data, or image data of an existing CCTV device that is corrected with corrected CCTV geometric information and already stored in the image information generation unit 128 and/or the spatial information database unit 136 as image information.

When the first sensor used to generate the reference image is an image capturing sensor, the first internal geometric data is substantially the same as the image sensor 106 described in the second internal geometric data. In the case of the image capture sensor, as shown in FIG. 4, the projection center and direction used to project the three-dimensional shape of the object in two dimensions are related to the three-dimensional position data and the viewing direction of the reference image. 1 Can be stored as reference image data together with external geometric data. The reference geometric information of the reference image data generated from the image capturing sensor may include 2D position data or 3D position data as reference position data.

When the reference image is aerial observation data, since it has a different geometry from the ground observation image data such as the CCTV device 102, the aerial observation data is in a view form in a specific direction, for example, a top view shape, as shown in FIG. 5. Can be converted. For example, each pixel of aerial observation data may be converted into an orthogonal image form having a horizontal geometry and used as a reference image.

When the reference image originates from a wide-area image sensor that generates a distorted image, the reference image uses flat image data generated by converting the distorted image into flat image data, and the reference position data uses the position data of the flat image data. In addition, the first posture data may be defined as a viewing direction and a viewing angle of the planar image data.

In this regard, a process of removing distortion and generating planar image data performed by the reference image database unit 116 will be described with reference to FIGS. 6 to 10.

6 is a flowchart illustrating a first process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data, and FIGS. 7 and 8 schematically illustrate a first process of removing image distortion and converting planar image data. It is a drawing shown.

9 is a flowchart illustrating a second process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data, and FIG. 10 is a diagram schematically showing a second process of removing image distortion and converting planar image data to be.

Referring to FIG. 6 related to the first process, when image data for a fisheye lens distorted by a wide-area image sensor having a fisheye lens is input (S605), the reference image database unit 116 is as shown in FIG. , Each pixel is given an index to define a correspondence relationship between the distorted image data from the image data and the reference plane image data (S610).

Next, the reference image database unit 116 decomposes the distorted image data to which the index is assigned into preset sections (S615).

Subsequently, the reference image database unit 116 generates flat image data by removing distortion of the distorted image data based on the distortion correction model of the decomposed section by referring to the reference plane image data (S620).

The distortion correction model may be modeling based on an equation or index of the decomposed section.

If the omnidirectional image data is input, the reference image database unit 116 decomposes the omnidirectional image data into preset sections, as in the second step of FIG. 8.

Next, as in the third step of FIG. 8, the reference image database unit 116 re-decomposes the decomposed omnidirectional image into preset sections. Subsequently, the reference image database unit 116 removes the distortion based on a distortion correction model of the re-decomposed section, for example, an equation or an index, and generates flat image data as shown in the right image of FIG. 8.

This is substantially the same process even when the image data received from the CCTV device 102 contains a distorted image due to the wide area image sensor, and this process can be performed in the search selection unit 118 for inputting the image data. have.

Meanwhile, referring to FIG. 9 related to the second process, when omnidirectional image data distorted by a wide area image sensor generating an omnidirectional image is input to the reference image database unit 116 (S905), the reference image database unit ( 116) reconstructs a virtual 3D shape from the distorted image data based on the camera geometry based on the distorted image data from the image data (S910).

Next, the reference image database unit 116 creates planar image data by defining a virtual camera geometry and projecting it onto a virtual plane camera through the geometry (S915).

In contrast, when the reference image originates from a laser scanning sensor, the reference image may use virtual image data converted from a point cloud obtained from the laser scanning sensor, and the reference position data is 3 of the virtual image data. It is defined as dimensional position data, and the first posture data may be defined as a viewing direction of the virtual image data.

The transformation of the point cloud may be performed by modeling a virtual camera taking into account the reflection intensity, altitude, and the like of the point cloud located below the drawing, as shown in FIG. 11. As a result, virtual image data may be generated as in the example above of FIG. 11. When an image sensor is additionally involved in generating the reference image, color data acquired by the image sensor at the time of conversion of the point cloud may be included in the virtual image data.

In step S210, the search selector 118 selects image data by referring only to the reference position data and the 2D position data, but when the CCTV geometric information includes all of the second posture data and the second internal geometric data, the search line The government 118 may proceed with the process of FIG. 12. 12 is a flowchart of another embodiment of search selection in a method of generating spatial information for video data of a CCTV device.

First, the search selection unit 118 stores CCTV geometric information including second external geometric data and second internal geometric data having two-dimensional position data and second attitude data in relation to the image data acquired from the CCTV device 102. Acquire (S1205)

The second external geometric data is based on a parameter value that changes every time image information of the image sensor 106 is acquired due to the position and posture of the CCTV device 102 in motion, that is, the position and posture of the image sensor 106 It is an error of the observed data calculated by doing so. The second internal geometric data is an inherent value of the sensors 104 to 108 itself, and is an error in observed data for each sensor 104 to 108 due to a parameter maintained regardless of whether the CCTV device 102 is moved or not. The geometric parameters for each of the sensors 104 to 108 are omitted in detail.

Next, the search selector 118 includes at least the reference position data and the image data among the reference geometric information of the reference image data including the first external geometric data and the first internal geometric data having the reference position data and the first posture data. A candidate group of image data is searched based on the 2D position data (S1210).

Here, the image data may be used as a descriptor having a characteristic capable of representing a specific object as well as data in a matrix form having color information. In the case of a descriptor having a characteristic representing a specific object, SIFT (Scale Invariant Feature Transform) can be used as an image comparison algorithm. This will be described in detail in the description of the system 100 and omitted.

Next, the search selector 118 selects the image data having the highest degree of matching with the reference image data from among the candidate groups by referring to the 2D position data, the 3D position data, and the first and second posture data (S1215).

Specifically, if the first and second posture data of the first and second external geometric data are different according to the viewing direction and viewing angle of the image data having the same or very similar two-dimensional position data as the reference image data, due to the image geometry The same feature geometry is recognized as a different geometry, making it difficult to match the reference image data and the image data. Accordingly, the accuracy and processing speed of the subsequent matching step may be improved by selecting only image data that is similar to the first and second posture data within a predetermined range from the same position from the candidate group.

Referring to FIG. 2 again, the matching unit 120 matches the selected image data and the reference image data (S215).

Matching may be implemented by a method of extracting feature geometry according to FIG. 13 or a method according to a template characteristic specified in the image according to FIG. 14.

FIG. 13 is a diagram illustrating a matching process using feature geometry in the matching unit, and FIG. 14 is a flowchart showing a matching process using characteristic correlation analysis between templates in the matching unit.

In the method by feature geometry extraction as shown in FIG. 13, the reference image data and the selected image data are matched based on feature geometries of a predetermined type that are common between the reference image data and the selected image data, and a collinear conditional expression, etc., is performed by referring to the feature geometry. It is possible to combine a plurality of selected image data through the geometric model of.

Here, the image data may be used as a descriptor having a characteristic capable of representing a specific object as well as data in a matrix form having color information. In the case of a descriptor having a characteristic representing a specific object, SIFT can be used as an image comparison algorithm.

Referring to FIG. 14 for a method based on a template characteristic, the matching unit 120 searches for the selected image data that is at least similar to a template designated in the reference image data (S1405).

Next, the characteristic correlations between the templates of the reference image data and the selected image data are also matched according to analysis (S1410). The template may be, for example, an object occupying a significant area in image data such as a pre-designated sign or sign. In the characteristic correlation analysis, in order to improve the matching speed, a method of matching a similar characteristic region by converting a spatial characteristic of an image into a frequency characteristic through Fourier transform can be used.

Subsequently, the reference image data and the selected image data are matched with templates having a characteristic correlation (S1415).

The feature geometry described in FIG. 13 or the template described through FIG. 14 constitute matching information, and the matching information may be stored in the reference image database unit 116.

Referring again to FIG. 2, the geometric conversion unit 122 geometrically converts CCTV geometric information of the selected image data by geometric transformation modeling that refers to the reference image data in common feature geometries or templates. Compensated CCTV geometric information including position data for correction with higher accuracy is generated (S220).

For geometric transformation modeling, a collinear conditional expression such as Equation 1 above may be used, or alternative modeling may be used for one of perspective transformation, affine transformation, and homography transformation capable of reflecting the geometry of image data in a matrix form.

15 is a diagram illustrating a process of calculating reference position data of reference image data corresponding to 2D position data of image data. In the case of using the collinear conditional equation, in the same feature geometry, the image data is geometrically transformed into a three-dimensional positional coordinate of the reference image data and corrected.

Next, the matching optimal value estimating unit 124 randomly samples the feature geometries of the image data to which the CCTV geometric information for correction is linked, so that the sampled feature geometries and the feature geometries of the corresponding reference data have a minimum distance from each other. The feature geometry of the image data is estimated as a matching optimal value (S225). The detailed process of the matching optimal value will be omitted in detail.

Subsequently, the CCTV geometric information estimating unit 126 estimates the geometrically transformed 3D CCTV geometric information in relation to the feature geometry estimated as the matching optimal value as the final corrected CCTV geometric information (S230).

Next, the image information generation unit 128 generates CCTV image information for the image data of the CCTV device 102 based on the final corrected CCTV geometry information and matching information for the feature geometry (S235).

CCTV image information, like the reference image data of the reference image database unit 116, the image data corrected by geometric transformation, the position data associated therewith, the corrected second external geometric data including the corrected second posture data, It may include final 3D CCTV geometric information including the corrected second internal geometric data, color data, time data, and the like.

Position data linked to the corrected image data may be composed of two-dimensional position coordinates when the reference position data is two-dimensional position data, and may be composed of three-dimensional position coordinates when the reference position data is three-dimensional position data. .

16 and 17, when the reference geometric information is composed of 3 dimensional original position data as reference position data and first external geometric data having first attitude data, and the accuracy of reference geometric information belonging to the reference image data is low In the following, an object height estimation process performed by the image information generation unit 128 will be described.

16 is a flowchart illustrating a process of estimating the height of an object of CCTV image data in a 3D image information generator, and FIG. 17 is a diagram illustrating a process of estimating the height of an object through pixel data analysis in CCTV image data. .

The process of FIG. 16 is performed when the precision of reference geometric information related to a partial position of the reference image data having 3D position data is lower than a threshold value.

First, the image information generation unit 128 converts the 3D image information of the image data of the CCTV device 102 corresponding to the reference image data with low precision into a 2D geometric model (S1605).

Next, the image information generation unit 128 estimates the height of the object through pixel data analysis in the 2D converted image data, or analyzes the height of the object by assuming the height of the image sensor of the CCTV device as a specific height, or The second posture data related to the viewing direction of the plurality of image sensors of the device is defined as a specific value, and the height of the object is estimated using a trigonometric function model (S1610).

In the case of pixel data analysis, as shown in FIG. 17, by analyzing pixels in the image data for each horizontal and vertical pixel, the height of the image sensor 106, the angle h between the horizontal line and the highest point of the object from the image sensor 106, and the image sensor. The angle (a) between the extension line from the center focal point (f) of 106 and the horizontal line is calculated, and the height of the object 202 is estimated in consideration of the number of pixels and vanishing points based on this. Accordingly, 3D image information is collected.

Subsequently, the image information generation unit 128 generates the 3D image information of the image data of the corresponding CCTV device 102 by estimating 3D position data by reflecting the estimated object height to 3D CCTV geometric information. Do (S1615).

According to the present embodiment, 3D image information of reference image data generated from a multi-sensor platform consisting of an observation sensor and a positioning sensor for image data of the CCTV device 102 having position and posture-related geometric data that are inaccurate or missing in some areas. By matching with and performing geometric transformation correction or the like, the image data acquired from the CCTV device 102 can be generated as 3D spatial information with convenience and high precision.

In addition, by processing observation data acquired from low-cost sensors excluding expensive sensors such as lidar when constructing the reference image data, it is possible to achieve a reduction in construction cost.

Hereinafter, a method of generating and updating spatial information implemented in a spatial information system of a CCTV device using 3D image information according to an embodiment of the present invention will be described with reference to FIGS. 18 to 20.

18 is a flowchart illustrating a method of generating and updating spatial information implemented in a spatial information system of a CCTV device using CCTV image information according to an embodiment of the present invention. 19 is a diagram illustrating an example of generating new spatial information including final corrected CCTV geometric information and object information in CCTV image information, and FIG. 20 is a diagram for explaining a process of updating spatial information.

First, the object information extraction unit 130 extracts an object of a specific type from CCTV image information resulting from the image data (S1805).

Specifically, the object information extraction unit 130 may extract object information related to an attribute of the extracted object based on data included in the CCTV image information.

The object information extraction unit 130 detects candidate group data related to the object identified from image data included in CCTV image information, final corrected CCTV geometric information, color data, time data, etc., using a machine learning technique, and uses the detected candidate group data. The information of the object can be recognized by sequentially applying additional machine learning models.

Next, the spatial information generation unit 132 includes, in the CCTV image information, the final corrected CCTV geometric information and object information related to the object stored in the reference image data, based on the object of the same reference image data as the extracted object. , For example, new spatial information according to the absolute coordinate system is generated (S1810).

The spatial information generation unit 132 may transmit and store new spatial information to the spatial information database unit 136. Spatial information is data generated in the corresponding coordinates of a predetermined coordinate system, such as observation data, internal geometry, external geometry, unique data, time information, acquisition environment information, and the amount of separation, accuracy, and precision due to the positional error of sensor data between heterogeneous sensors. It may include related probability information, point cloud related information, image data related information, object information, and map related information. In the present embodiment, the spatial information is described as including all of the above-described data or information, but is not limited thereto, and may be set to include only some of the above-described data/information according to design specifications.

As shown in FIG. 19, new spatial information may be generated by including final corrected CCTV geometric information and object information in CCTV image information, and based on this, through a drawing process, a map on the right side may be produced.

Next, when the existing spatial information already stored in the spatial information database unit 136 exists, the spatial information update unit 134 determines whether the new spatial information is different from the existing spatial information in the object information at the same location. As a result of the determination (S1815), for different object information as a result of the determination, the existing spatial information is updated with new spatial information as shown in FIG. As a result of the determination, when the object information is the same, the existing spatial information may be maintained (S1820).

Components constituting the apparatus shown in Fig. 1 or steps according to the embodiments shown in Figs. 2, 6, 9, 12, 14, 16, 18 are computer-readable recording media in the form of a program that realizes the function Can be written on. Here, the computer-readable recording medium refers to a recording medium that can be read by a computer by storing information such as data or programs by electrical, magnetic, optical, mechanical, or chemical action. Examples of such recording media that can be separated from a computer include portable storage, flexible disks, magneto-optical disks, CD-ROMs, CD-R/Ws, DVDs, DATs, and memory cards. In addition, as a recording medium fixed to a CCTV device and a computer, there is a solid state disk (SSD), a hard disk, a ROM, and the like.

Further, in the above, even if all the constituent elements constituting the embodiments of the present invention have been described as being combined into one operation, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all the constituent elements may be selectively combined and operated in one or more. In addition, all of the components may be implemented as one independent hardware, but some or all of the components are selectively combined to provide a program module that performs some or all of the combined functions in one or a plurality of hardware. It may be implemented as a computer program having.

Although the present invention has been described in detail through exemplary embodiments above, a person of ordinary skill in the art to which the present invention pertains will realize that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. I will understand. Therefore, the scope of the present invention is limited to the described embodiments and should not be determined, and should be determined by all changes or modifications derived from the claims and the concept of equality as well as the claims to be described later.

Claims (10)

A reference image database unit for storing reference image data in association with reference geometric information including reference position data and first external geometric data having first posture data defining a viewing direction set to generate the reference image;
Acquiring CCTV geometric information including second external geometric data having at least two-dimensional position data in relation to the image data acquired from the CCTV device, and referring to at least the reference position data and the two-dimensional position data to obtain the reference image data and A search selector for searching and selecting the matchable image data;
A matching unit for matching the reference image data and the selected image data;
A geometric conversion unit that geometrically converts the CCTV geometric information of the selected image data by geometric transformation modeling referring to the reference image data to generate corrected CCTV geometric information including correction position data with higher accuracy than the two-dimensional position data. ;
By randomly sampling the sample feature geometries of the image data to which the corrected CCTV geometric information is linked, the optimal value matching the feature geometry at which the distance between the sample feature geometries and the corresponding feature geometries of the reference image data is minimum An optimal value estimating unit that estimates by;
A geometric information estimating unit for estimating the corrected CCTV geometric information geometrically transformed in relation to the feature geometry estimated as the matching optimal value as final corrected CCTV geometric information; And
A system for generating spatial information of a CCTV device using reference image information, comprising an image information generator for generating CCTV image information from image data of the CCTV device, based on the final corrected CCTV geometric information and matching information. The method of claim 1,
The two-dimensional position data includes two-dimensional coordinates of the CCTV device or a name of a space in which the CCTV device is installed, and the reference position data is two-dimensional position data with higher accuracy than the two-dimensional position data of the first external geometric data. Or, when it is composed of 3D location data, and the reference location data is 2D location data, the CCTV image information includes 2D location coordinates, and when the reference location data is 3D location data, the CCTV The image information includes three-dimensional position coordinates, a spatial information generation system of a CCTV device using reference image information. The method of claim 1,
The second external geometric data further includes second posture data defining a viewing direction set to generate the image,
The search selection unit searches for a candidate group of the image data that can be contrasted with the reference image data based on the reference position data and the 2D position data, and refers to the first and second posture data to determine the reference image among the candidate groups. A system for generating spatial information of a CCTV device using reference image information to select the image data having the highest degree of matching with the data. The method of claim 1,
The reference image data is fused by matching and correcting image information and point cloud information obtained from an external mobile platform having a positioning sensor and at least one of an image capturing sensor and a laser scanning sensor through a geometric model between the sensors. A system for generating spatial information of a CCTV device using external image data or reference image information, which is corrected image data that has been corrected with the correction CCTV geometric information and has already been stored as CCTV image information. The method of claim 1,
When at least one of the reference image and the image data is obtained from a wide-area image sensor that generates a distortion image, the reference image and the image data in which the distortion image is generated are a plane in which the distortion image is converted into a flat image. When the image data is used and the reference image uses the planar image data, the first posture data is defined as a viewing direction of the planar image data. A system for generating spatial information of a CCTV device using reference image information. The method of claim 1,
When the reference image originates from a laser scanning sensor, the reference image uses virtual image data converted from a point cloud obtained from the laser scanning sensor, and the reference position data is of the virtual image data. A system for generating spatial information of a CCTV device using reference image information, which is defined as location data, and the first attitude data is defined as a viewing direction of the virtual image data. The method of claim 1,
The matching unit extracts and compares feature geometry from the selected image data and the reference image data, and matches the commonly estimated feature geometry to each other, or searches for the selected image data that is at least similar to a template specified in the reference image data. Thus, the matching according to the analysis of the characteristic correlation between the templates, the matching information is the feature geometry or the template, and the feature geometry or the template is stored in the reference image database unit, the CCTV device using the reference image information Spatial information generation system. delete The method of claim 1,
An object information extraction unit for extracting an object of a specific type from the CCTV image information;
Spatial information for generating new spatial information including, in the CCTV image information, the correction CCTV geometric information and the object information related to the object stored in the reference image data, based on an object of the same reference image data as the extracted object Generation unit; And
If the new spatial information is stored in the spatial information database unit, and there is existing spatial information already stored in the spatial information database unit, whether the new spatial information is different from the existing spatial information in the object information at the same location. A system for generating spatial information of a CCTV device using reference image information, further comprising a spatial information update unit that determines whether or not, and updates the existing spatial information with the new spatial information for different object information as a result of the determination. The method of claim 1,
The reference geometric information is composed of 3D position data as the reference position data and the first external geometric data having the first posture data, and the accuracy of the reference geometric information is related to a partial position of the reference image data. In the lower case,
The image information generator converts the corrected image data of the CCTV image information corresponding to the low-precision reference image data into a two-dimensional geometric model, and then calculates an object height through pixel data analysis in the two-dimensional converted image data. Estimate or analyze the object height by assuming the height of the image sensor of the CCTV device as a specific height, or define the attitude data related to the viewing direction of the plurality of image sensors of the CCTV device as a specific value, and use a trigonometric function model, A system for generating spatial information of a CCTV device using reference image information, which generates the corresponding CCTV image information by estimating the height of an object and estimating 3D position data by reflecting the estimated height of the object to the corrected CCTV geometric information .

Images