KR102836813B1 - Method and apparatus for positioning of moving object - Google Patents

Abstract

A method and device for determining a position of a moving object are provided. According to one embodiment, a method for determining a position of an object may include the steps of receiving an image frame from a camera, performing deep learning-based object recognition to determine object recognition information of an object included in the image frame, performing view control of the camera based on the object recognition information to track the object, and determining object GPS position information of the object based on view state information indicating a degree to which the camera is adjusted due to the view control.

Description

Method and apparatus for positioning of moving object {Method and apparatus for positioning of moving object}

The following embodiments relate to a method and device for determining the position of a moving object.

Object tracking is a technique for continuously tracking the position and movement of a specific object in an image sequence. Object tracking is a subfield of computer vision and can be used in various applications. For example, object tracking can be used in self-driving cars, surveillance systems, sports analysis, augmented reality, and video editing. Deep learning-based artificial intelligence (AI) can be used for object tracking. AI can greatly improve the performance of object tracking.

When a surveillance device is fixed, if the object to be monitored leaves the surveillance device's field of view or hides behind an obstacle, surveillance accuracy may decrease and it may be difficult to maintain surveillance.

According to one embodiment, a method for determining an object position includes the steps of receiving an image frame from a camera, performing deep learning-based object recognition to determine object recognition information of an object included in the image frame, performing view control of a camera based on the object recognition information to track the object, and determining object GPS position information of the object based on view state information indicating a degree to which the camera is adjusted due to the view control.

According to one embodiment, a computer-readable storage medium stores one or more programs including instructions for performing a method for determining an object position.

According to one embodiment, a control device for performing object position determination includes one or more processors and a memory including instructions executable by the one or more processors, wherein when the instructions are executed by the one or more processors, the instructions cause the control device to receive an image frame from a camera, perform deep learning-based object recognition to determine object recognition information of an object included in the image frame, perform view control of the camera based on the object recognition information to track the object, and determine object GPS position information of the object based on view state information indicating a degree to which the camera is adjusted due to the view control.

According to embodiments, even if an object to be monitored leaves the field of view of the surveillance device or hides behind an obstacle, surveillance accuracy can be maintained high and continuous surveillance can be possible.

According to the embodiments, aerial surveillance and reconnaissance can be performed in conjunction with mobile mission devices such as drone systems. In addition, the embodiments can be applied to various surveillance and reconnaissance applications including military, such as ground-mobile vehicles. Such surveillance and reconnaissance can enable continuous surveillance with high accuracy.

According to embodiments, in a surveillance target area, an object can be tracked and object GPS location information can be continuously determined through a system of simple configurations such as a camera, a view controller, and an object recognition deep learning device.

FIG. 1 is a drawing exemplarily showing the configuration of devices for determining the position of a moving object according to one embodiment.
FIG. 2 is a drawing exemplarily showing a specific configuration of devices for determining the position of a moving object according to one embodiment.
FIG. 3 is a drawing exemplarily showing PTZ control operations among view controls of an object tracking system according to one embodiment.
FIG. 4 is a diagram exemplarily showing an operation of an object GPS location information determination system determining image center GPS location information based on view state information according to PTZ control, according to one embodiment.
FIG. 5 is a diagram exemplarily showing an operation of an object GPS location information determination system determining object GPS location information by applying shift values to image center GPS location information according to one embodiment.
FIG. 6 is a flow chart illustrating a method for determining the position of a moving object according to one embodiment.
FIG. 7 is a block diagram illustrating a configuration of an electronic device for determining the position of a moving object according to one embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present disclosure includes modifications, equivalents, or alternatives included in the technical idea described in the embodiments.

Although the terms first or second may be used to describe various components, such terms should be construed only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, the terms "comprises" or "has" and the like are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In describing with reference to the attached drawings, identical components are given the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

FIG. 1 is a drawing exemplarily showing the configuration of devices for determining the position of a moving object according to one embodiment. Referring to FIG. 1, a control device (130) receives an image (121) of an object (102) from a surveillance device (110), performs view control of the surveillance device (110) based on the image (121) captured by the surveillance device (110), tracks the object (102), and determines the position of the object (102) based on the degree to which the surveillance device (110) is adjusted due to the view control. The object (102) may be a moving object having mobility. The image (121) of the object (102) captured may correspond to a plurality of image frames. The position of the object (102) determined by the control device (130) may be used to control the flight of a reconnaissance device such as a determined drone (140).

The surveillance device (110) can capture an object (102) included in a specific scene (101). The surveillance device (110) can include a camera set (111) and a view controller (112) for controlling a view of the camera set (111). The camera set (111) can include one or more cameras. For example, the camera set (111) can include, but is not limited to, a visible light camera (e.g., an EO (electro-optical) camera) for recognizing an object during the day using a visible light band and an IR (infrared) camera for recognizing an object at night using infrared rays. For example, the camera set (111) can be an EO/IR camera that combines an EO camera and an IR camera.

The view controller (112) can perform view control to adjust the camera set (111). The view control can be performed to continuously track the object (102) in the scene (101) captured by the camera set (111). For example, if the object (102) continuously moves to the left, the view controller (112) can perform view control to rotate the camera set (111) to the left so that the object (102) can be included in the scene (101) and captured. The view controller (112) can receive a view control signal from the control device (130). For example, if the object (102) continuously moves to the left in the image (121), the view controller (112) can receive a view control signal to rotate the camera set (111) to the left from the control device (130).

In order for the control device (130) to perform a mission by allowing a reconnaissance device such as a drone (140) to approach the object (102), accurate positioning of the object (102) may be essential. If the accurate positioning of the object (102) is not determined, the reconnaissance device may not be able to efficiently approach the target point. For example, in the case of the drone (140), a lot of time may be consumed in searching for the object (102), and the mission may not be completed due to battery limitations. In addition, since the object (102) (e.g., a person, a drone, an animal, a vehicle, a tank, etc.) is mobile, continuous object tracking may be required for a reconnaissance device such as a drone (140) to perform its mission. For example, in order to track and determine the position of an object moving at a relatively high speed, such as a vehicle or a drone, precise view control may be required so that the object does not leave the image of the surveillance device (110). The control device (130) can track the object (102) and determine the position of the object (102) by adjusting the monitoring device (110) to perform precise view control.

The control device (130) can receive an image (121) from the surveillance device (110) and determine information on an object (102) included in the image (121). Based on the information on the object (102), the control device (130) can perform object tracking so that the object (102) does not leave the scene (101) being captured by the surveillance device (110). The control device (130) transmits a view control signal to a view controller (112) of the surveillance device (110) to perform object tracking, and the view controller (112) that receives the view control signal can perform view control.

The control device (130) can determine the degree to which the monitoring device (110) has been adjusted due to the performance of the view control. In one embodiment, the control device (130) can determine the degree to which the monitoring device (110) has been adjusted based on view state information provided by the view controller (112) of the monitoring device (110). In another embodiment, the control device (110) can determine the degree to which the monitoring device (110) has been adjusted based on a view control signal transmitted to the view controller (112) of the monitoring device (110). When the control device (130) receives the image (121) and view state information from the monitoring device (110) and the control device (130) transmits the view control signal to the monitoring device (110), a communication link (120) can be utilized. The communication link (120) can be a wired communication link, a wireless communication link, or a combination thereof. The control device (130) can determine the location of the object (102) based on the degree to which the monitoring device (110) is adjusted. The determined location information of the object (102) can include GPS location information. This GPS location information can be referred to as object GPS location information.

Since the surveillance device (110) photographs the object (102) from a fixed position, there may be limitations in tracking the object (102) in situations such as when the object (102) moves far away from the surveillance device (110) or when the object (102) hides behind an obstacle that blocks the view of the surveillance device (110). The object GPS location information can be used to supplement the tracking limitations due to the fixed position of the surveillance device (110). For example, the object GPS location information can be provided to a reconnaissance device such as a drone (140), reconnaissance equipment used by humans with the same reconnaissance mission, or another surveillance device that is far away from the surveillance device (110), but is not limited thereto.

For example, a reconnaissance device such as a drone (140) may perform an autonomous mission flight by receiving a flight mission instruction from a control device (130). An autonomous mission flight may mean an operation in which a drone (140) autonomously flies with minimal intervention of a user to reach a destination. The drone (140) may include a set of cameras and a view controller, such as a surveillance device (110). The drone (140) may communicate with the control device (130) to provide information about the object (102) or receive information about the object (102). In one embodiment, the flight mission of the drone (140) may be a surveillance mission that assists the surveillance of the surveillance device (110). For example, the drone (140) may fly to the location of the object (102) before the object (102) leaves an area that the surveillance device (110) can capture, and perform a surveillance mission of the object (102) together with the surveillance device (110). In another embodiment, the flight mission of the drone (140) may be a warning mission linked to the surveillance of the surveillance device (110). For example, when a person, vehicle, etc. is captured by the surveillance device (110), a warning mission such as not to approach and enter the object (102) may be performed.

FIG. 2 is a drawing exemplarily showing a specific configuration of devices for determining a position of an object according to one embodiment. Referring to FIG. 2, the position of an object (201) may be determined using at least one of a monitoring device (210), a deep learning-based object recognition system (220), an object tracking system (230), a view control signal conversion system (231), an object GPS position information determination system, a flight control system (250), and an attitude control system (270). The monitoring device (210), the deep learning-based object recognition system (220), the object tracking system (230), the view control signal conversion system (231), the object GPS position information determination system, the flight control system (250), and the attitude control system (270) may be referred to as an object position determination system.

The object (201) may be a mobile object having mobility. One or more of the deep learning-based object recognition system (220), the object tracking system (230), the view control signal conversion system (231), the object GPS position information determination system, the flight control system (250), and the attitude control system (270) of FIG. 2 may be included in the control device (130) of FIG. 1.

The surveillance device (210) may include a camera set (211), a view controller (212), and an inertial measurement device (213), and the surveillance device (210) may correspond to the surveillance device (110) of FIG. 1. The view controller (212) may be a controller that controls a view of the camera set (211). In one embodiment, the view controller (212) may be a PTZ controller that controls PTZ (pan, tilt, zoom) of the camera set (211). In one embodiment, the inertial measurement device (213) may include sensors (e.g., a gyroscope, an accelerometer, a geomagnetic sensor, etc.) for orienting the horizontal and true north directions of the camera set (211).

The deep learning-based object recognition system (220) can recognize an object (201) in an image (214) captured by a surveillance device (210). The deep learning-based object recognition system (220) can perform object recognition using artificial intelligence (AI). For example, artificial intelligence using a neural network can be used. The neural network can be trained to recognize a given object from an input image through deep learning.

The deep learning-based object recognition system (220) can receive an image (214) from a surveillance device (210) via a communication link (202). The image (214) received by the deep learning-based object recognition system can be a video including a plurality of continuous image frames (e.g., a sequence of image frames). The communication link (202) can be a wired communication link, a wireless communication link, or a combination thereof. The deep learning-based object recognition system (220) can recognize an object (201) in the received image (214) based on deep learning. In one embodiment, the deep learning-based object recognition system (220) can recognize an object (201) in each image frame of the plurality of image frames.

The deep learning-based object recognition system (220) can determine object recognition information (221) based on the recognized object (201). The object recognition information (221) can include information used to track the object (201) and determine the location of the object. For example, the object recognition information (221) can include information about an object bounding box of the object (201) (e.g., x-coordinate, y-coordinate, and box size of the bounding box, etc.), information about an actual size of the object (e.g., average size of the recognized object, etc.), and image resolution, etc. In one embodiment, the deep learning-based object recognition system (220) can determine object recognition information (221) in each image frame of a plurality of image frames. The deep learning-based object recognition system (220) can transmit the object recognition information (221) to the object tracking system (230) and the object GPS location information determination system (240).

The object tracking system (230) can track the object (201) by adjusting the surveillance device (210) based on the object recognition information (221). The object tracking system (230) can track the object (201) by allowing the view controller (212) of the surveillance device (210) to perform view control. The object tracking system (230) can track the object (201) according to the object tracking criterion (236). For example, the object tracking criterion (236) may be, but is not limited to, adjusting the camera set (211) so that the object (201) is located at the center of the image (214) and adjusting the zoom so that the size of the object bounding box is maintained at a predetermined size. For example, the object tracking system (230) can adjust the pan and tilt of the camera set (211) so that the object (201) is positioned at the center of the image (214) according to the object tracking criterion (236), and can adjust the zoom of the camera set (211) so that the box size of the object bounding box remains constant. Below, a PTZ control, which is an embodiment of the view control, is described in detail with respect to FIG. 3.

Even if the object tracking system (230) performs view control (e.g., PTZ control) corresponding to the first image frame, the object tracking criterion (236) may not be satisfied within the second image frame as the object (201) moves. Accordingly, the object tracking system (230) may need to re-perform view control corresponding to the second image frame. For example, the object tracking system may re-perform view control of the camera based on object recognition information (221) corresponding to the second image frame and view state information (233) corresponding to the first image frame, such that the object (201) is located at the center of the second image frame and the box size of the object bounding box within the second image frame becomes a predetermined size. The object recognition information (221) corresponding to the second image frame may be referred to as second object recognition information. In one embodiment, the view state information (233) corresponding to the second image frame may be determined by updating a relative value indicating the degree to which view control corresponding to the second image frame has been performed to the view state information (233) corresponding to the first image frame. In another embodiment, the view state information (233) corresponding to the second image frame may be determined based on an absolute value indicating the degree to which view control corresponding to the second image frame has been performed.

The view control signal conversion system (231) can receive view control information (232) from the object tracking system (230). The view control information (232) can include information for adjusting the camera set (211) for object tracking. The view control signal conversion system (231) can generate a view control signal (234) recognizable by the view controller based on the view control information (232). The view control signal conversion system (231) can transmit the view control signal (234) to the view controller (212). The view control signal conversion system (231) can receive a view status signal (234) generated by the view controller (212). The view control signal conversion system (231) can convert the view status signal (234) to generate view status information (233). The view status information (233) can indicate the degree to which view control of the camera set (211) has been performed.

In one embodiment, the view state information (233) may be PTZ state information and the view control information (232) may be PTZ control information. For example, the view state information (233) may include a value indicating the degree to which the camera set (211) is currently panned, tilted, or zoomed. Also, for example, the view control information (232) may include a value indicating the degree to which the camera set (211) should be panned, tilted, or zoomed, and the degree to which the camera set (211) should be panned, tilted, or zoomed may be an absolute value or a relative value based on the view state information (233).

In one embodiment, the view control signal conversion system (231) may be included in the object tracking system (230). In another embodiment, signal transmission between the view control signal conversion system (231) and the view controller (212) may be via a communication link (203). The communication link (203) may be a wired communication link, a wireless communication link, or a combination thereof, such as the communication link (202). The view control signal conversion system (231) may receive view control information (232) from the attitude control system (270) and transmit view state information (233) to the attitude control system (270).

The object GPS location information determination system (240) can determine the location of the object (201). The object GPS location information determination system (240) can receive view state information (233) and field of view information (237) from the object tracking system (230). In one embodiment, the view state information (233) can include field of view information (237). In one embodiment, the field of view information (237) can be determined based on the degree to which the camera set (211) is zoomed. In one embodiment, the object GPS location information determination system can determine the object GPS location information (241) of the object (201) by determining the GPS location information of the center of the image (214). For example, if the object tracking system (230) performs object tracking so that the object (201) is located at the center of the image (201), the determined GPS location information of the center of the image (214) can be identical to the object GPS location information (241). In another embodiment, the position of the object (201) (e.g., object GPS position information) can be determined by determining image center position information (e.g., image center GPS position information) at the center of the image (214) and performing a GPS shift based on the image center position information. Below, with reference to FIGS. 4 and 5, an operation of determining object GPS position information based on view state information according to PTZ control is specifically described.

Despite object tracking by the object tracking system (230), the image center GPS location information and the object GPS location information (241) at the center of the image (214) may not be the same. In one embodiment, the view state information (233) and the field of view information received by the GPS location information determination system (240) may be the view state information and the field of view information corresponding to an initial image frame (which may be referred to as a first image frame), and the object recognition information (221) received by the object GPS location information determination system (240) may be the object recognition information corresponding to the n-th image frame (n may have a value greater than 1) due to a time difference. For example, if the object recognition information (221) received by the object GPS location information determination system (240) due to a time difference corresponds to the second image frame and the view state information (233) corresponds to the first image frame, the object (201) in the second image frame may not be located at the center of the second image frame despite the object tracking of the object tracking system (230) corresponding to the first image frame. The time difference may occur due to the time required to capture the first image frame and the second image frame and/or the time for view control to be performed by the view controller (212).

The view state information (233) may indicate the degree to which the camera set (211) of the surveillance device (210) is adjusted based on a specific reference point. The attitude control system (270) may set a reference point for the degree to which the camera set (211) is adjusted. The attitude control system (270) may receive camera attitude information (271) from the inertial measurement device (213) of the surveillance device (210) before tracking the object (201), adjust the position of the camera set (211), and perform view control. For example, the attitude control system (270) may perform PTZ control by adjusting the camera set (211) until the horizontal and true north orientation of the camera set (211) is completed. In addition, for example, when the horizontal and true north orientation of the camera set (211) is completed, the attitude control system may set the current pan and tilt values to 0 using the current position as a reference point.

The flight control system (250) is an exemplary application system based on the position determined by the object GPS position information determination system (240). The flight control system (250) can receive object GPS position information (241) and position reliability information (242) from the object GPS position information determination system (240). The flight control system (250) can determine object position information (251) based on the object GPS position information (241) and the position reliability information (242). For example, the object position information (251) can be determined as the sum of errors based on the object GPS position information (241) and the position reliability information (242). The drone system (260) can include the drone (140) of FIG. 1.

The flight control system can transmit object location information (251) to the drone system (260) via a wireless communication link. The drone system (260) can perform a flight to the location of the object (201) based on the object location information (251). The drone system (260) can include a location transmission module (261) and a view controller (262). The drone system (260) can photograph the object (201) through a camera attached to the drone system (260). The view controller (262) can perform view control of the camera attached to the drone system (260). In one embodiment, the view controller (262) of the drone system (260) can be controlled by the object tracking system (230).

Drone GPS information (263) including information about an object (201) may be transmitted to a flight control system (250) by a location transmission module (261). In one embodiment, the flight control system (250) may compare the drone GPS information (263) received from the drone system (260) with the object GPS location information (241) to generate object location information (251). The flight control system (250) may transmit the object location information (251) generated based on the drone GPS information (263) to the drone system (260) to control the flight.

FIG. 3 is a drawing exemplarily showing a PTZ control operation among view controls of an object tracking system according to one embodiment. Referring to FIG. 3, the object tracking system adjusts the pan and tilt of a camera through a PTZ controller of a surveillance device so that an object center (311) of an object (310) is located at the center (304) of an image. The object center (311) may represent the center of a bounding box of the object (310). The PTZ controller may correspond to the view controller (112) of FIG. 1 and the view controller (212) of FIG. 2. In one embodiment, the object tracking system may track the object (310) by adjusting the pan and tilt through the PTZ controller so as to match the center of the object bounding box with the center of the image (301). Additionally, for example, the object tracking system can track the object (310) by adjusting the zoom via the PTZ controller so that the box size of the object bounding box becomes a predetermined size.

The pan angle (A) and tilt angle (B), which are the degrees by which the object tracking system adjusts the pan and tilt of the camera so that the center of the object bounding box coincides with the center of the image (301), can be determined using the following mathematical expressions 1 and 2 based on the current angle of view after zoom adjustment.

A may represent a pan angle, and B may represent a tilt angle. P and T may represent a pan pixel count (321) and a tilt pixel count (322) from the object center (311) of the object (310) to the image center (304), respectively. image_size_h and image_size_w may represent an image vertical pixel count (303) and an image horizontal pixel count (302), respectively. The image vertical pixel count (303) and the image horizontal pixel count (302) may be included in the object recognition information (221) of FIG. 2. a and b may be values dynamically assigned within the object tracking system in consideration of the speed and movement direction of the object (310). For example, when the object (310) moves quickly to the right from the image center (304), a may be assigned a positive value and b may be assigned a value of 0.

fov_w may represent a horizontal angle of view of the camera, and fow_h may represent a vertical angle of view of the camera. In one embodiment, the angle of view information (237) of FIG. 2 may include a horizontal angle of view and a vertical angle of view. In one embodiment, one or more cameras constituting the camera set of the surveillance device may provide a current zoom value and a focal length. In one embodiment, the angle of view information (237) may be determined according to the degree to which the view controller (212) has performed a zoom. In one embodiment, if the focal length is provided, the horizontal angle of view and the vertical angle of view may be determined through the following mathematical expressions 3 and 4.

fl and sensor_size_w may represent the focal length of the camera and the horizontal size of the image sensor of the camera. In another embodiment, if the focal length is not provided, the focal length may be determined directly by the following mathematical expression 5.

fl_max and fl_min can represent the maximum and minimum focal length, respectively. zoom_max and zoom_min can represent the maximum and minimum zoom values, respectively. zoom can represent the current zoom value.

Even if the object tracking system performs PTZ control corresponding to the first image frame, the object center (311) and the image center (304) within the second image frame may not coincide as the object (310) moves. Accordingly, the object tracking system may need to re-perform PTZ control corresponding to the second image frame. For example, the object tracking system may re-perform PTZ control of the camera based on object recognition information corresponding to the second image frame and PTZ status information indicating the degree to which the camera is PTZ-ed corresponding to the first image frame, such that the object (310) is located at the center of the second image frame and the box size of the object bounding box within the second image frame becomes a predetermined size.

FIG. 4 is a diagram exemplarily showing an operation of an object GPS location information determination system determining an image center GPS location based on view state information according to PTZ control, according to one embodiment. Referring to FIG. 4, the object GPS location information determination system determines an image center GPS location (412) by applying offsets (431, 432, 433) to a camera GPS location (411). The object GPS location information determination system can determine information on the camera GPS location (411) and an object distance (420). The object GPS location information determination system can determine the offsets (431, 432, 433) based on the object distance (420), the pan angle (421), and the tilt angle (422).

In one embodiment, the object GPS location information determination system can predetermine the camera GPS location (411). The object GPS location information determination system can receive a pan angle (A) (421) and a tilt angle (B) (422), which are degrees to which the pan and tilt of the camera are adjusted according to the PTZ control, as view state information. The pan angle (421) and the tilt angle (422) can be measured along a predetermined direction (402, 403) from a reference (401, 423) determined by the attitude control system (270) of FIG. 2, etc. The object GPS location information determination system can use the following mathematical expressions 6 and/or 7 to determine the object distance (420), which is used to determine the offsets (431, 432, 433).

distance can represent the object distance (420). object_size_w and object_size_h can represent the number of horizontal pixels and the number of vertical pixels of the actual size of the object included in the object recognition information, respectively. bb_size_w and bb_size_h can represent the number of horizontal pixels and the number of vertical pixels of the box size of the object bounding box of the moving object, respectively.

The latitude offset (431) can be determined by multiplying the object distance (420) by cos(B), cos(A), and (the latitude difference per object distance unit, diff_LAT). The longitude offset (432) can be determined by multiplying the object distance (420) by cos(B), sin(A), and (the longitude difference per object distance unit, diff_LON). The elevation offset (433) can be determined by multiplying the object distance (420) by sin(B) and (the elevation difference per object distance unit, diff_ALT). The object GPS positioning system can determine the image center GPS position (412) by adding the determined latitude, longitude, and elevation offsets (431, 432, 433) to the latitude, longitude, and elevation of the camera GPS position (411), respectively.

FIG. 5 is a diagram exemplarily showing an operation of an object GPS location information determination system determining object GPS location information by applying shift values to image center GPS location information according to one embodiment. Referring to FIG. 5, the object GPS location information determination system can determine shift values (541, 542) based on a distance between an image center (520) and an object center (540) of a moving object (530). The shift values (541, 542) can indicate a degree to which the object center (540) deviates from the image center (520). The object GPS location information determination system can determine object GPS location information of a moving object (530) by applying the shift values (541, 542) to the image center GPS location information.

In one embodiment, the view state information and the object recognition information received by the GPS position information determination system may not correspond to the same image frame due to a time difference. For example, the object recognition information received by the object GPS position information determination system may correspond to the second image frame, and the view state information may correspond to the first image frame. Accordingly, even if the PTZ control is performed, the object center (540) of the object (530) in the image corresponding to the second image frame, etc. may not coincide with the image center (520).

In one embodiment, the shift values (541, 542) may be determined as the coordinates of the object center (x, y) (540) after setting the coordinates of the image center (520) to (0,0). In one embodiment, if the current direction to which the pan of the camera is adjusted is very close to the due north or due south direction, the shift values (541, 542) may correspond to the longitude shift and the altitude shift. In another embodiment, if the current direction to which the pan of the camera is adjusted is very close to the due east or due west direction, the shift values (541, 542) may correspond to the latitude shift and the altitude shift. FIG. 5 shows an embodiment in which the direction of the camera is close to due north, and thus the shift values (541, 542) correspond to the longitude shift (541) and the altitude shift (542). In one embodiment, if the current direction of the camera is very close to the north, the object GPS location information (LAT, LON, ALT) is obtained by applying the shift values (x, y) (541, 542) to the image center GPS location information (x, y) (541, 542) using the following mathematical expression 8. , , h) can be determined. is the latitude of the object GPS, can represent the longitude of the object GPS, and h can represent the altitude of the object GPS.

FIG. 6 is a flow chart illustrating a method for determining a position of a moving object according to one embodiment. Referring to FIG. 6, in step (601), a control device receives an image frame from a camera. The control device may receive a second image frame from the camera. The control device may control the flight of the drone system based on object GPS position information. The control device may control the flight of the drone system based on drone GPS information acquired from the drone system.

In step (602), the control device performs deep learning-based object recognition to determine object recognition information of an object included in an image frame. The object recognition information may include information about an object bounding box of the object. The control device may perform deep learning-based object recognition to determine second object recognition information of an object included in a second image frame. The second object recognition information may include information about a second object bounding box of the object.

In step (603), the control device performs view control of the camera based on object recognition information to track the object. The control device can perform view control of the camera so that the object is located at the center of the image frame and the box size of the object bounding box has a predetermined size. The control device can adjust the pan and tilt of the camera based on the distance between the center of the image frame and the center of the object bounding box and the angle of view of the camera to position the object at the center of the image frame. The control device can re-perform view control based on the view state information so that the object is located at the center of the second image frame and the second box size of the second object bounding box has a predetermined size. The control device can generate a view control signal that can be recognized by a view controller that performs view control of the camera. The view state information can be generated by converting a view state signal generated by the view controller.

In step (604), the control device determines object GPS location information of the object based on view state information indicating the degree to which the camera is adjusted due to the view control. The control device can determine the object GPS location information based on the second object recognition information. The control device can determine image center GPS location information of the center of the second image frame based on the view state information and the second object recognition information. The control device can determine the object GPS location information based on the second object recognition information and the image center GPS location information. The control device can determine the object distance between the camera and the object. The control device can determine offsets based on the object distance. The control device can determine the image center GPS location information of the center of the second image frame by applying the offsets to the camera GPS location information of the camera. The control device can determine shift values based on the distance between the center of the second image frame and the center of the second object bounding box. The control device can determine the object GPS location information by applying the shift values to the image center GPS location information.

In addition, the description of Figs. 1 to 5 can be applied to the object positioning method.

FIG. 7 is a block diagram illustrating a configuration of an electronic device for determining a position of a moving object according to one embodiment. Referring to FIG. 7, an electronic device (700) may include one or more processors (710), memory (720), storage (730), an I/O (input/output) device (740), and a network interface (750), which may communicate with each other through a communication bus (760).

If the control device of Fig. 1 is implemented as a single configuration, the electronic device (700) can correspond to the control device. If the control device of Fig. 1 is implemented as multiple configurations, such as the deep learning-based object recognition system (220), the object tracking system (230), the view control signal conversion system (231), the object GPS location information determination system, the flight control system (250), and the attitude control system (270) of Fig. 2, the electronic device (700) can correspond to each of the multiple configurations.

One or more processors (710) can execute instructions stored in memory (720) or storage (730). The instructions, when executed by one or more processors (710), can cause the electronic device (700) to perform operations described with reference to FIGS. 1 to 6. The memory (720) can include a computer-readable storage medium or a computer-readable storage device. The memory (720) can store instructions to be executed by one or more processors (710) and can store related information while software and/or applications are executed by the electronic device (700). The memory (720) can store a control program (721) that performs position determination of a moving object in one embodiment. With at least a portion of the control program (721) stored in the memory (720), the electronic device (700) can perform operations described with reference to FIGS. 1 to 6.

When the control device of FIG. 1 is implemented as a single configuration, the control program (721) can be stored in the memory (720) of the electronic device (700) corresponding to the control device, and when the control device of FIG. 1 is implemented as multiple configurations, the control program (721) can be distributed and stored in the memory (720) of the electronic device (700) corresponding to each of the multiple configurations.

Storage (730) can include a computer-readable storage medium or a computer-readable storage device. Storage (730) can store a larger amount of information than memory (720) and can store information for longer periods of time. For example, storage (730) can include a magnetic hard disk, an optical disk, flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

The I/O device (740) can receive input from a user via traditional input methods such as a keyboard and mouse, and newer input methods such as touch input, voice input, and image input. For example, the I/O device (740) can include a keyboard, a mouse, a touch screen, a microphone, or any other device that can detect input from a user and transmit the detected input to the electronic device (700). The I/O device (740) can provide output of the electronic device (700) to the user via visual, auditory, or tactile channels. The I/O device (740) can include, for example, a display, a touch screen, a speaker, a vibration generating device, or any other device that can provide output to the user. The network interface (750) can communicate with an external device via a wired or wireless network.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the OS. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on them. For example, even if the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

Claims (20)

A step of receiving a video frame from a first camera;
A step of performing deep learning-based object recognition to determine object recognition information of an object included in the image frame;
A step of tracking the object by performing view control of the first camera based on the object recognition information;
A step of determining first object GPS location information of the object based on view state information indicating the degree to which the first camera is adjusted due to the view control;
A step of controlling the flight of a drone system including a second camera based on the first object GPS location information;
A step of receiving object surveillance information determined using the second camera from the drone system;
A step of generating second object GPS location information based on the first object GPS location information and the object surveillance information; and
A step for controlling the flight of the drone system based on the second object GPS location information.
A method for determining an object position, comprising: In the first paragraph,
The above object recognition information
Contains information about the object bounding box of the above object,
The steps to track the above object are:
A step of performing view control of the first camera so that the object is positioned at the center of the image frame and the box size of the object bounding box becomes a predetermined size,
How to determine the position of an object. In the second paragraph,
The steps to perform the above view control are:
A step of positioning the object at the center of the image frame by adjusting the pan and tilt of the first camera based on the distance between the center of the image frame and the center of the object bounding box and the angle of view of the first camera,
How to determine the position of an object. In the second paragraph,
A step of receiving a second image frame following the image frame from the first camera; and
A step of performing the object recognition based on the deep learning described above to determine second object recognition information of the object included in the second image frame.
Including more,
The above second object recognition information
Contains information about the second object bounding box of the above object,
The steps to track the above object are:
A step of re-performing the view control of the first camera based on the view state information so that the object is located at the center of the second image frame and the size of the second box of the second object bounding box becomes the predetermined size.
A method for determining an object position, the method further comprising: In the first paragraph,
The steps to track the above object are:
A step of generating a view control signal recognizable by a view controller that performs the view control of the first camera,
The above view state information is
Generated by converting the view state signal generated by the above view controller.
How to determine the position of an object. In the first paragraph,
A step of receiving a second image frame following the image frame from the first camera; and
A step of performing the object recognition based on the deep learning described above to determine second object recognition information of the object included in the second image frame.
Including more,
The step of determining the first object GPS location information is
Further performed based on the second object recognition information,
How to determine the position of an object. In Article 6,
The step of determining the first object GPS location information is
A step of determining image center GPS location information of the center of the second image frame based on the view state information and the second object recognition information; and
A step of determining the first object GPS location information based on the second object recognition information and the image center GPS location information.
A method for determining an object position, comprising: In Article 7,
The step of determining the above image center GPS location information is
A step of determining an object distance between the first camera and the object;
a step of determining offsets based on the object distance; and
A step of applying the offsets to the camera GPS location information of the first camera to determine the image center GPS location information of the center of the second image frame.
A method for determining an object position, comprising: In Article 7,
The above second object recognition information
Contains information about the second object bounding box of the above object,
The step of determining the first object GPS location information based on the image center GPS location information
A step of determining shift values based on the distance between the center of the second image frame and the center of the second object bounding box; and
A step of determining the first object GPS location information by applying the shift values to the image center GPS location information.
A method for determining an object position, comprising: delete delete A computer-readable storage medium storing one or more programs including commands for performing the method of any one of claims 1 to 9. In a control device that performs object positioning,
one or more processors; and
Memory containing instructions executable by one or more of said processors
Including,
When the instructions are executed by the one or more processors, the instructions cause the control device to:
Receive video frames from the first camera,
Perform deep learning-based object recognition to determine object recognition information of the object included in the image frame,
Tracking the object by performing view control of the first camera based on the object recognition information,
Determining first object GPS location information of the object based on view state information indicating the degree to which the first camera is adjusted due to the view control;
Controlling the flight of a drone system including a second camera based on the first object GPS location information,
Receive object surveillance information determined using the second camera from the drone system,
Generate second object GPS location information based on the first object GPS location information and the object surveillance information,
Controlling the flight of the drone system based on the second object GPS location information;
controller. In Article 13,
The above object recognition information
Contains information about the object bounding box of the above object,
When the instructions are executed by the one or more processors, the instructions cause the control device to: track the object;
To perform the view control of the first camera so that the object is located at the center of the image frame and the box size of the object bounding box is a predetermined size.
controller. In Article 14,
When the instructions are executed by the one or more processors, the instructions cause the control device to perform the view control.
Adjusting the pan and tilt of the first camera based on the distance between the center of the image frame and the center of the object bounding box and the angle of view of the first camera to position the object at the center of the image frame.
controller. In Article 13,
When the instructions are executed by the one or more processors, the instructions cause the control device to:
Receive a second video frame following the video frame from the first camera,
Performing the object recognition based on the deep learning described above to determine second object recognition information of the object included in the second image frame,
To determine the first object GPS location information based further on the second object recognition information;
controller. In Article 16,
When the instructions are executed by the one or more processors, the instructions cause the control device to determine the first object GPS location information.
Determine the image center GPS location information of the center of the second image frame based on the view state information and the second object recognition information,
Determine the first object GPS location information based on the second object recognition information and the image center GPS location information.
controller. In Article 17,
When the instructions are executed by the one or more processors, the instructions cause the control device to determine the image center GPS location information.
Determining the object distance between the first camera and the object,
Determine the offsets based on the above object distances,
Applying the offsets to the camera GPS location information of the first camera to determine the image center GPS location information of the center of the second image frame.
controller. In Article 17,
The above second object recognition information
Contains information about the second object bounding box of the above object,
When the instructions are executed by the one or more processors, the instructions cause the control device to determine the first object GPS location information based on the image center GPS location information.
Determine shift values based on the distance between the center of the second image frame and the center of the second object bounding box,
To determine the first object GPS location information by applying the shift values to the above image center GPS location information,
controller. delete