WO2018211396A1

WO2018211396A1 - Detection of powerlines in aerial images

Info

Publication number: WO2018211396A1
Application number: PCT/IB2018/053330
Authority: WO
Inventors: S. N. Omkar; J. Senthilnath; K. N. Ramesh; Anurag Jain; Abhishek Kumar; Gautham ANAND
Original assignee: Indian Institute of Science IISC
Current assignee: Indian Institute of Science IISC
Priority date: 2017-05-13
Filing date: 2018-05-14
Publication date: 2018-11-22
Anticipated expiration: 2019-11-13

Abstract

A system and method for detection of powerlines in aerial images is disclosed. The system for detection of a powerline in an aerial image comprises an image retrieval module 202 to obtain at least one aerial image including the powerline, a binary classified image generation module 204 to subject said at least one aerial image to a pixel -by-pixel analysis, a skeletonization module 206 configured to effect skeletonization of said binary classified image to reduce objects of said at least one aerial image to a median line, a region growing module 208 configured to apply region growing to restore continuity of the objects lost during the skeletonization, a geometric feature determination module 210 to calculate shape index and density index for the powerline in the skeletonized image, and a powerline detection module 212 configured to remove any non-powerline feature from the at least one aerial image.

Description

DETECTION OF POWERLINES IN AERIAL IMAGES

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of powerline surveillance. More specifically, the present disclosure relates to a system and method for detection of powerlines in aerial images.

BACKGROUND

[0002] Background description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.

[0003] The growth of emerging economies such as India and China is fuelled by the demand of increasing investments in power infrastructure, which includes increased powerline capacity additions and subsequent maintenance and support of transmission lines.

[0004] Increased transmission infrastructure directly increases cost of maintaining the transmission line infrastructure since the current practices involve manual inspection. Surveillance and maintenance of electrical infrastructure is a significant cost component for companies in the power sector. Regular inspection of powerlines is regulatory and mandatory for proper functioning of power transmission. Increased capacity of generation and transmission infrastructure increases the cost and effort in ground inspection since they are manually inspected. Currently ground inspection of powerlines is mandatory and challenging too, as in remote hilly areas and rural hinterland where physical access to powerlines is an issue. Traditionally, on-site manual examination is time consuming, expensive and often shows inaccurate results.

[0005] The constraints and challenges of manual inspection is addressed by researchers by using advanced air borne powerline surveillance systems based on manned helicopters and Unmanned Aerial Vehicles. Although the flight and data acquisition systems are advanced and evolved data processing of the images for powerline monitoring uses traditional image processing methods. There is scope for innovation in using advanced machine learning based image processing methods for powerline extraction.

[0006] Several researchers have worked on extraction of powerlines in UAV images. Zhengrong et al {Image and Vision Computing New Zealand, 23rd International Conference, pp. 1-6, IEEE, 2008)discloses Radon transform to extract line segments of the powerlines, followed by a grouping method to link each segment, and a Kalman filter was finally applied to connect the segments into an entire line. Song et al (Neurocomputing, \29 (2014): 350- 361)discloses application of edge detection method to suppress non powerline part and employing graph theory to group line segments into whole lines. McLaughlin (Geoscience and Remote Sensing Letters, IEEE 3, no. 2 (2006): 222-226) discloses two stage algorithm, wherein in the first stage data was classified into three categories- powerline lines, vegetation and surface. In second stage, algorithm identifies each individual data point of powerlines. Huang {Cognitive Computation 6, no. 3 (2014): 376-390 discloses ELM having a single hidden layer feed forward network (SLFN), which is advantageous over back propagation (BP) learning algorithms due to the factors like the training is independent of the training samples, learning time is faster since learning is non-iterative approach and no issues of local minima.

[0007] Other ways in handling this application is with the help of Airborne LIDAR based system. LIDAR sensors are however costly and bulky thereby increasing the size of the UAV and the subsequent increased cost of data acquisition. LIDARs also suffer from issues in quality of data acquisition. Areas such as dense canopies cannot be detected by LIDAR.

[0008] A person skilled in the art would immediately realize that each of thesesystem(s) and method(s) carry out powerline extraction using image processing methods such as Houghline transforms, Kalman filtering, graph theory and the likes that primarily focus on the underlying fact that powerlines are linear structures. However, these assumptions are not always true,as other linear structures such as periphery fences, road stretches and the likes can potentially be extracted as linear structures leading to incorrect detection of powerlines. Accordingly, the conventional system(s) and method(s) cannot be reliably used for detection of powerlines in the aerial images.

[0009] There is, therefore,a need to provide a system that can offer reliable detection of powerlines in aerial images. Need is also felt for the method of detection of powerlines in the aerial images. The present disclosure satisfies the existing needs, interalia, others and overcomes the one or more shortcomings associated with the conventional system and method for detection of powerlines in the aerial images.

OBJECTS OF THE INVENTION

[0010] It is a general object of the present disclosure to provide a system and method to detect presence of powerlines in aerial images. [0011] It is an object of the present disclosure to provide a system and method that effectively point out powerlines in an image captured by a camera device coupled with an aerial vehicle, such as an unmanned aerial vehicle (UAV) or a manned vehicle, for instance, a helicopter.

[0012] It is another object of the present disclosure to provide a system and method that envisages to eliminate manual monitoring and inspection of powerlines by utilizing advanced machine learning and image processing methods of powerline detection in an image.

[0013] It is still another object of the present disclosure to provide a system and method that overcomes deficiencies associated with conventional powerline monitoring and detection techniques.

SUMMARY

[0014] The present disclosurerelates generally tothe field of powerline surveillance. More specifically, the present disclosure relates to a system and method fordetection of powerlines in aerial images, more particularly, UAV based remote sensed images.

[0015] An aspect of the present disclosure relates to a system for detection of a powerline in an aerial image, the system including a non-transitory storage device having embodied therein one or more routines operable to detect the powerline in the aerial image, andone or more processors coupled to the non-transitory storage device, wherein the one or more routines include an image retrieval module, which when executed by the one or more processors, obtains at least one aerial image including the powerline, a binary classified image generation module, which when executed by the one or more processors, subjects said at least one aerial image to a pixel-by-pixel analysis, a skeletonization module, which when executed by the one or more processors, enables skeletonization of said binary classified image to reduce one or more objects of said at least one aerial image to a median line, a geometric feature determination module, which when executed by the one or more processors, calculates any or a combination of a shape index and a density index for the powerline in the skeletonized image based on one or more dimensional characteristics of the powerline in the skeletonized image, anda powerline detection module, which when executed by the one or more processors, detects the powerline in the aerial image by removing any non-powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a pre-determined set of shape index and density index of the powerline. [0016] In an embodiment, the aerial image is an aerial vehicle based remote sensed image acquired by an image capturing device operatively coupled with the aerial vehicle.

[0017] In an embodiment, the system further includes a region growing module that subjects the skeletonized image to region growing segmentation to restore continuity of the one or more objects lost during the skeletonization.

[0018] In an embodiment, the binary classified image generation module is trained using a plurality of RGB spectral values to differentiate between pixels corresponding to powerline features and pixels corresponding to non-powerline features.

[0019] In an embodiment, the powerline detection module is configured to classify the one or more objects in the aerial image as non-powerline feature in case the calculated shape index pertaining to the powerline in the aerial image is below a pre-determined limit.

[0020] In an embodiment, the powerline detection module is configured to classify the one or more objects in the aerial image as non-powerline feature in case the calculated density index pertaining to the powerline in the aerial image is above a pre-determined limit.

[0021] In an embodiment, the skeletonization module is further configured to decrease density of the powerline in the binary classified image.

[0022] Another aspect of the present disclosure relates to a method for detection of a powerline in an aerial image, the method including the steps of (i) obtaining, by a computing device, at least one aerial image including the powerline, (ii) obtaining, by the computing device, a binary classified image by subjecting said at least one aerial image to a pixel -by- pixel analysis, (iii) effecting, by the computing device, skeletonization of said binary classified image to reduce one or more objects of said at least one aerial image to a median line, (iv) calculating, by the computing device, any or a combination of a shape index and a density index for the powerline in the skeletonized image, and (v) detecting, by the computing device, the powerline in the aerial image by removing any non-powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a pre-determined set of shape index and density index of the powerline.

[0023] In an embodiment, the aerial image is an aerial vehicle based remote sensed image acquired by an image capturing device operatively coupled with the aerial vehicle.

[0024] In an embodiment, the method further includes a step of subjecting the skeletonized image to region growing segmentation to restore continuity of the one or more objects lost during the skeletonization step. [0025] In an embodiment, the step of removing any non-powerline feature to effect detection of the powerline in the aerial image further includes the steps ofclassifying the one or more objects in the aerial image as non-powerline feature in case the calculated shape index pertaining to the powerline in the aerial image is below a pre-determined limit, andclassifying the one or more objects in the aerial image as non-powerline feature in case the calculated density index pertaining to the powerline in the aerial image is above a predetermined limit.

[0026] Various objects, features, aspects and advantages of the inventive subject matter will become more apparent from the following detailed description of preferred embodiments, along with the accompanying drawing figures in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] FIG. 1A through FIG. IF illustrate aerial images including powerlines in accordance with an embodiment of the present disclosure.

[0029] FIG. 2 illustrates an exemplary representation depicting functional modules of the system for detection of a powerline in an aerial imagein accordance withembodiments of the present disclosure.

[0030] FIG. 3 illustrates an exemplary UAV and payload in accordance with an embodiment of the present disclosure.

[0031] FIG.4 illustrates an exemplary block diagram depicting controller of the UAV system in accordance with an embodiment of the present disclosure.

[0032] FIG. 5illustrates an exemplary flowchart depicting a method of detection of a powerline in an aerial imagein accordance with an embodiment of the present disclosure.

[0033] FIG. 6A illustrates an exemplary aerial image including powerlines; FIG. 6B illustrates an exemplary output image from the binary classified image generation module; FIG. 6C illustrates an exemplary output image from the skeletonization module; FIG. 6D illustrates an exemplary output image from theregion growing module; FIG. 6E illustrates an exemplary output image from the geometric image classifying module; and FIG. 6F illustrates an exemplaryoverlapped Extreme Learning Machines (ELM) segmented image with original input image.

[0034] FIG. 7 A through FIG. 7F illustrate output for six different images in accordance with embodiments of the present disclosure.

[0035] FIG. 8 illustrates a graphical representation of performance parameters (measures) with different number of hidden neurons in accordance with an embodiment of the present disclosure.

[0036] FIG. 9 illustrates the statistical visualization of performance of the method in comparison with ground reference data in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0037] The following is a detailed description of embodiments of the disclosure depicted in the accompanying drawings. The embodiments are in such detail as to clearly communicate the disclosure. However, the amount of detail offered is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0038] Each of the appended claims defines a separate invention, which for infringement purposes is recognized as including equivalents to the various elements or limitations specified in the claims. Depending on the context, all references below to the "invention" may in some cases refer to certain specific embodiments only. In other cases it will be recognized that references to the "invention" will refer to subject matter recited in one or more, but not necessarily all, of the claims.

[0039] As used in the description herein and throughout the claims that follow, the meaning of "a," "an," and "the" includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise.

[0040] All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.The use of any and all examples, or exemplary language (e.g. "such as") provided with respect to certain embodiments herein is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention otherwise claimed.No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.

[0041] Various terms as used herein are shown below. To the extent a term used in a claim is not defined below, it should be given the broadest definition persons in the pertinent art have given that term as reflected in printed publications and issued patents at the time of filing.

[0042] The present disclosurerelates to a system and method of detection of powerlines in aerial images. An aspect of the present disclosure relates to a system for detection of a powerline in an aerial image, the system including a non-transitory storage device having embodied therein one or more routines operable to detect the powerline in the aerial image, andone or more processors coupled to the non-transitory storage device, wherein the one or more routines include an image retrieval module, which when executed by the one or more processors, obtains at least one aerial image including the powerline, a binary classified image generation module, which when executed by the one or more processors, subjects said at least one aerial image to a pixel-by-pixel analysis, a skeletonization module, which when executed by the one or more processors, enables skeletonization of said binary classified image to reduce one or more objects of said at least one aerial image to a median line, a geometric feature determination module, which when executed by the one or more processors, calculates any or a combination of a shape index and a density index for the powerline in the skeletonized image based on one or more dimensional characteristics of the powerline in the skeletonized image, anda powerline detection module, which when executed by the one or more processors, detects the powerline in the aerial image by removing any non-powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a pre-determined set of shape index and density index of the powerline.

[0043] In an embodiment, the aerial image is an aerial vehicle based remote sensed image acquired by an image capturing device operatively coupled with the aerial vehicle.

[0044] In an embodiment, the system further includes a region growing module that subjects the skeletonized image to region growing segmentation to restore continuity of the one or more objects lost during the skeletonization.

[0045] In an embodiment, the binary classified image generation module is trained using a plurality of RGB spectral values to differentiate between pixels corresponding to powerline features and pixels corresponding to non-powerline features. [0046] In an embodiment, the powerline detection module is configured to classify the one or more objects in the aerial image as non-powerline feature in case the calculated shape index pertaining to the powerline in the aerial image is below a pre-determined limit.

[0047] In an embodiment, the powerline detection module is configured to classify the one or more objects in the aerial image as non-powerline feature in case the calculated density index pertaining to the powerline in the aerial image is above a pre-determined limit.

[0048] In an embodiment, the skeletonization module is further configured to decrease density of the powerline in the binary classified image.

[0049] Another aspect of the present disclosure relates to a method for detection of a powerline in an aerial image, the method including the steps ofobtaining, by a computing device, at least one aerial image including the powerline, obtaining, by the computing device, a binary classified image by subjecting said at least one aerial image to a pixel -by-pixel analysis, effecting, by the computing device, skeletonization of said binary classified image to reduce one or more objects of said at least one aerial image to a median line, calculating, by the computing device, any or a combination of a shape index and a density index for the powerline in the skeletonized image, anddetecting, by the computing device, the powerline in the aerial image by removing any non-powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a pre-determined set of shape index and density index of the powerline.

[0050] In an embodiment, the aerial image is an aerial vehicle based remote sensed image acquired by an image capturing device operatively coupled with the aerial vehicle.

[0051] In an embodiment, the method further includes a step of subjecting the skeletonized image to region growing segmentation to restore continuity of the one or more objects lost during the skeletonization step.

[0052] In an embodiment, the step of removing any non-powerline feature to effect detection of the powerline in the aerial image further includes the steps ofclassifying the one or more objects in the aerial image as non-powerline feature in case the calculated shape index pertaining to the powerline in the aerial image is below a pre-determined limit, andclassifying the one or more objects in the aerial image as non-powerline feature in case the calculated density index pertaining to the powerline in the aerial image is above a predetermined limit.

[0053] FIG. 1 A through FIG. IF illustrate aerial images including powerlines acquired at varying altitudes and angles in accordance with an embodiment of the present disclosure to better represent the challenges faced while detecting the powerlines in the aerial images. As can be seen from the highlighted area of FIG. 1 A, when the image capturing device is at certain distance, the RGB values of far-away point of powerline may get mismatched with other non powerline features, which shows spectral-classification challenge. FIG. IBillustrates an exemplary image captured when the image capturing device is at different angle and highlighted area showing a light pole with almost same RGB values as of powerline and hence, face the same above mentioned problem. FIG. 1C illustrates an exemplary image captured at a relatively higher altitude, wherein it can be seen that the middle portion of powerline is not clearly visible and highlighted area shows existence of the same problem as presented hereinabove. Highlighted portion of FIG. ID illustrates a street light having almost same RGB values as of powerline and hence, may create problem in classification. FIG. IE illustrates the overlapped powerlines, and FIG. IF illustrates the overlapping of mud patches with powerline pixels. The proposed system and method realized in accordance with embodiments of the present disclosure can help overcome such inaccuracies amongst others and can find utility in surveillance of powerline(s). Accordingly, the present disclosure can envisage autonomous surveillance of powerlines to detect any physical fault therein.

[0054] FIG. 2 illustrates an exemplary representation depicting functional modules of the system for detection of a powerline in an aerial image in accordance withembodiments of the present disclosure. The system includes a non-transitory storage device having embodied therein one or more routines operable to detect the powerline in the aerial image, andone or more processors coupled to the non-transitory storage device. The one or more routines include an image retrieval module 202, which when executed by the one or more processors, obtains at least one aerial image including the powerline, a binary classified image generation module 204, which when executed by the one or more processors, subjects said at least one aerial image to a pixel-by-pixel analysis, a skeletonization module 206, which when executed by the one or more processors, enables skeletonization of said binary classified image to reduce objects of said at least one aerial image to a median line and to decrease density of the powerline, a region growing module 208, which when executed by the one or more processors, applies region growing to restore continuity of the objects lost during the skeletonization, a geometric feature determination module 210, which when executed by the one or more processors, calculates any or a combination of a shape index and a density index for the powerline, and a powerline detection module212, which when executed by the one or more processors, detects the powerline in the aerial image by removing any non-powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a pre-determined set of shape index and density index of the powerline.

[0055] In an embodiment, an aerial vehicle/UAV (unmanned aerial vehicle) configured with an image/video capturing deviceis used to capture video and/or still images including the powerline(s) as illustrated in FIG. 3. As illustrated, the UAV 302 can be configured with an image/video capturing device 304 (shown separately from the UAV for better clarity) among other payload (not shown) to capture aerial images and/or video including the powerline(s) 306. In an embodiment, the UAV 302 is further configured with other instruments that can help in the surveillance of the powerline. In an exemplary embodiment, the UAV is configured with a thermal camera that can help detect leakage of current in the powerlines. However, a person skilled in the art would realize that any other instruments, as known to or appreciated by a person skilled in the art, can be configured with the UAV or aerial image/video capturing device without departing from the scope and spirit of the present disclosure.

[0056] In an embodiment, the UAV used for remote sensing for powerline detection can be flown at low altitude and can be used for remote sensing. Remote sensing from a UAV platform is commonly known as low altitude remote sensing (LARS). For powerline detection, fixed-wing or rotary wing can be used. In an embodiment, small UAV is utilized along with small payload, which is more feasible and cost effective solution for powerline detection. In an embodiment, quad-copter UAV is used, which is a stable platformand suitable for aerial imaging. Their ability to hover and vertical takeoff can make them suitable for powerline monitoring applications.

[0057] In an embodiment, the UAV may be a quad-copter built using off the shelf components (viz. remote receive 310, flight stabilizer 312, battery 314, brushless motor speed controller 316, brushless motor 318 and the likes) and modified to carry the payload and camera requirements. In an embodiment, the UAV belonging to class of multi-rotor aircrafts and powered by batteries is utilized to serve its intended purpose as laid down in disclosure of the present disclosure. However, a person skilled in the art would appreciate that any other UAV can be utilized without departing from the scope and spirit of the present disclosure.

[0058] In an embodiment, UAV with the flight endurance of around 20 minutes is utilized. In an embodiment, the flight maneuver of UAV is controlled remotely by a pilot using a remote control308. In an embodiment, the range of remote control is around 500 meters. In an embodiment, the weight of the quad-copter UAV is about 1.4kg including the camera payload. In an embodiment, the UAV that can be utilized to serve its intended purpose as laid down in embodiments of the present disclosure is with the specifications as shown in Table 1.

Table 1: UAV Specification

[0059] In an embodiment, the camera 304 that can be utilized to serve its intended purpose as laid down in embodiments of the present disclosure is with the specifications as shown in Table 2.

Table 2: Camera Specification

[0060] In an embodiment, the UAV includes 4 brushless DC motors affixed with propellers mounted on a quad-copter frame. In an embodiment, speeds of the motors can be governed by on-board electronic speed controllers (ESC) which can be powered by a special Lithium-Polymer battery.

[0061] FIG. 4 illustrates an exemplary block diagram depicting controller of the UAV in accordance with an embodiment of the present disclosure. Speed of the motors can help control the angular and linear velocity of UAV. In an embodiment, communication between UAV and remote control is of 2.4 GHz wireless receiver-transmitter pair. In an embodiment, pilot sends the control command through the control joysticks of the remote control 308, which encodes the signals and transmits it via radio waves. The transmitted signals can be received by the on-board receiver 310 and decoded. These commands can then be interpreted by the on-board flight stabilized 12. In an embodiment, the on-board flight stabilizer 312 is an Arduino based multiwii multi-copter firmware. In an embodiment, the flight stabilizer includes ATmega AVR 2560 microcontroller and MEMS based motion sensors for measuring the angular velocity and linear velocity of the quad-copter. In an embodiment, the MEMS based sensors are any or a combination of 3-axis gyroscope, 3-axis accelerometer, 3- axis magnetometer and 1-axis barometer. In an embodiment, the MEMS based sensors are 3- axis gyroscope, 3-axis accelerometer, 3-axis magnetometer and 1-axis barometer. In an embodiment, the flight stabilizer can obtain the angular velocity and linear velocity of UAV via the MEMS sensors and compares it with the desired state based on the commands sent by the pilot. The flight stabilizer can then decide the speed of motors required to achieve the desired state of the UAV. In an embodiment, this comparison step is updated about every 3ms to achieve stable flight of UAV. However, a person skilled in the art would appreciate that the comparison step can be performed continuously or at any interval to serve its purpose without departing from the scope and spirit of the present disclosure.

[0062] In an embodiment, data acquisition is done with GoPro HD Hero2 (Go Pro, California, United States) camera304. In an embodiment, the camera measures about 98x58x34mm and weighs about 98grams. In a preferred embodiment, the camera has the specifications asshown in Table 2. In an embodiment, focal length of the camera is fixed. In an embodiment, camera has a SD card slot onto which the recorded video can be stored. In an embodiment, Field of View of camera can be varied depending on the usage. In an embodiment, for acquiring the video, camera was set to the video resolution of 1080p at 30fps. In an embodiment, Field of view was set to 127° in order to capture the powerlines.

[0063] In an embodiment, UAV can be flown with the payload (camera) at a height of 25-30 meters above ground over the powerline to capture the video and/or still image(s).In an embodiment, camera 304 is fixed at the front end of the UAV 302 at an angle of 15 degrees in order to obtain the front view. Front view can allow the camera to capture the larger length of the powerline. In an embodiment, spatial resolution of the video varies with the altitude of the UAV. In an embodiment, the variation of altitude can be up to 5 meters, which can allow acquisition of images at varying altitudes. In an embodiment, the acquired video and/or still image(s) is analyzed to select the test images with good aerial view of the powerlines at different altitudes, angles and background.

[0064] In an embodiment, the captured video or image(s) extracted therefrom and/or still image(s) viz. aerial images including powerline(s) are transmitted to the image retrieval module202. In an embodiment, the image retrieval module 202 receive/acquires/obtains the aerial images wirelessly from the image capturing device. Alternatively, the captured video or image(s) extracted therefrom and/or still image(s) viz. aerial images including powerline(s) are provided as an input to the system 200 using methods as known to a person skilled in the art, for example, by operatively coupling the SD card of the UAV with the system 200. In an exemplary embodiment, the image retrieval module 202 is configured to communicate with the image/video capturing device associated with the UAV so as to receive the images therefrom.

[0065] In an embodiment, the image retrieval module 202 is operatively coupled with the binary classified image generation module 204 to communicate aerial image(s) including powerline(s) thereto. In an embodiment, the binary classified image generation module 204 is subjected to Machine Learning. In an embodiment, the binary classified image generation module 204 may be an Extreme Learning Machine (ELM) thatis trained using dataset containing RGB values of the powerline and non-powerline classes. In an embodiment, each training sample may be of the form I={Xi, Yi}, where Χ_ί={χι_;Γ, xi_;g, Xi_;t,} represents the RGB value of each sample respectively and Yi ε{0, 1 } represents the class labels where i = 1,2, ... , n. If training data has n samples then the training matrix can be of the form

[0066] The input samples have 3 -dimensional features; hence, there are 3 input nodes in the input layer and the number of output nodes in output layer is 2 as this is a binary classification. An optimal number of hidden nodes can be chosen for hidden layer. Let the number of hidden nodes be H. Based on the number of hidden nodes, input weight matrix and bias matrix can be generated. Input weight matrix can be generated randomly based on the number of input nodes and hidden nodes. Dimensions of input weight matrix is Hx3. Bias matrix can also be generated randomly having dimensions Hxl.

[0067] Input to the hidden nodes can be given as lh = __, ^Wjk ^xik + bjW ere j = 1, 2 ,3 ... H and i = 1, 2, 3 ... N (2) k=l

[0068] Inputs to each hidden nodes can be calculated for the entire set of training data. The inputs can then be passed through the activation function to get the output of each hidden neuron. Output of hidden neuron can be given by

wherein, G(.) is the activation function for the hidden nodes

[0069] Activation function used for this problem is logistic function given by,

G = ¹ 4

Wherein, I_h is the input applied to the logistic function.

[0070] Output matrix of hidden layer is given by

[0071] Output weight matrix can be computed analytically from equation 6 below where H^"1 is the pseudo-inverse of hidden layer output matrix and T is the target matrix.

W₀ = R¹ * T (6)

[0072] The input and output weight matrices generated during training can be used to classify the image into powerline and non-powerline class.

[0073] Spectral value of each pixel (Pi) of test image can be applied as the input to the trained model.

Pi = {Vi.r . Vi,g- Vi,b} ) wherein, P _r , P_{l g}, and /^denotes the red, blue and green pixel value of each sample.

[0074] Input matrix I_H to the hidden nodes is computed using the equation where w is the input weight matrix and b is the bias of the hidden node

[0075] The activation function applied to matrix I_H results in matrix H₀. It should be noted that activation function is same as the function (equation 4) used for training the model.

[0076] The hidden layer output matrix (H₀)is multiplied with output weight matrix (Wo) to determine the output labels corresponding to powerline and non-powerline class.

Y,= H₀ * W₀ (10)

where Yi is the resultant class type (powerline or non-powerline label) of the i^thpixel.

[0077] In an embodiment, the skeletonization module 206 is configured to effect skeletonization of said binary classified image to reduce objects of said at least one aerial image to a median line and to decrease density of the powerline. Skeletonizationcan preserve the topological features and the original size of the objects. Skeletonizationcan reduce the objects into a median line thereby reducing the density of the objects. Skeletonization S(A) of an object can be given by

S_k = (A®B) - (ΑΘΒ)οΒ (11)

[0078] where A is the connected object in the binary image and B is the structuring element. S_k is the skeleton point. Object A is eroded k times until it is reduced to single pixel and each erosion is followed by opening the eroded image. The difference between the eroded image and opened image gives the skeleton points. Union of all skeleton points gives the skeleton of the object A. Skeletonizationcan affect the powerline features causing discontinuity in the powerlines (including other objects).

[0079] In an embodiment, loss of continuity is restored by the region growing module 108. In an embodiment, the region growing module 208 is configured to apply region growing to restore continuity of the objects lost during the skeletonization. In an embodiment, to retrieve the lost information, iterative region growing is applied after Skeletonization. In an embodiment, in case of 1 pixel discontinuity in the images, region growing is applied 1 time. In an embodiment, if the discontinuity in powerlines is more than 1 pixel, region growing is applied for a maximum of 3 times.

[0080] Powerlines are the continuous linear structures, whereas non-powerline features are the not continuous. Accordingly, geometrical featurescan be taken into consideration and Shape Index and/or Density Index can be determined/calculated/applied to bring out clear distinction between the two features (powerline and non-powerline). In an embodiment, the geometric feature determination module 210 is configured to calculate any or a combination of a shape index and a density index for the powerline (or any other object). In an embodiment, the geometric feature determination module 210 is configured to calculate both of a shape index and a density index for the powerline (or any other object). Shape Index (SI) and Density Index (DI) of an object can be given by

A

DI = (14)

1+ JVar{X)+Var(Y)

where A is the area of the object, P is the perimeter of the object and Var(X) and Var(Y) represents variance in X and Y coordinates, which approximately represent the radius of the object.

[0081] In an embodiment, the powerline detection module212 is configured to detect powerline in the aerial image and to remove any non-powerline feature from the aerial image. In an embodiment, the powerline detection module212 is configured to classify the objects in the aerial image based on the pre-determined shape index and/or density index. In an embodiment, high threshold of SI is set to remove the non-powerline feature due to the fact thatpowerline being thin and long, has larger perimeter and lesser area and hence, ratio of perimeter to area is very high. Accordingly, the powerline detection module212 is configured to classify the objects in the aerial image as non-powerline feature in case the SI is below a pre-determined limit. In an embodiment, low threshold of DI is set to remove the non- powerline feature as the same has less radius. Accordingly, the powerline detection module212 is configured to classify the objects in the aerial image as non-powerline feature in case theDI is above a pre-determined limit.

[0082] FIG. 5 illustrates an exemplary flowchart depicting a method of detection of a powerline in an aerial image. As shown at step 502, at least one aerial image including the powerline is obtained by computing device followed by subjecting said at least one aerial image to a pixel-by-pixel analysis to obtain a binary classified image therefrom as shown at step 504. Skeletonization of said binary classified image is done by a skeletonization module to reduce objects of said at least one aerial image to a median line and to decrease density of the powerline as shown at step 506.Region growing is then applied by a region growing module to restore continuity of the objects lost during the skeletonization step as shown at step 508 followed by calculating any or a combination of a shape index and a density index for the powerline by a geometric feature determination module as shown at step 510. Any non-powerline feature is removed by a powerline detection module to effect detection of the powerline in the aerial image as shown at the step 512.

[0083] FIG. 6A through FIG. 6F illustrates transformation of the aerial image to detect powerlines therefrom in accordance with an embodiment of the present disclosure. An aerial image (for example, UAV remote sensed input image), as illustrated in FIG. 6A is subjected to the trained binary classified image generation module, output of which, i.e., binary classified image is illustrated in FIG. 6B. It can be observed that the binary classified image predominantly contains the powerline features. However, some misclassification can be observed as well. The binary classified image, as shown in FIG. 6B,is subjected to morphological operation (i.e., skeletonization), output from which is as shown in FIG. 6C. In an embodiment, the skeletonized image is subjected to region growing operation, output from which is as shown in FIG. 6D.In an embodiment, the threshold value for shape index (SI) and density index(DI) may befixed at greater than 0.8 and less than 0.65, respectively. As can be observed from FIG. 6E, there is a marked improvement in classification after the application of shape indexand density index.FIG. 6F illustrates an exemplary overlapped ELM segmented image with original input image.

[0084] In accordance with an embodiment of the present disclosure, Table 3 below exemplifies the threshold values for shape and density indices utilized for segmentation of image for final extraction of powerlines for different input aerial images as illustrated in FIG. 1 A through FIG. IF. Accordingly, a person skilled in the art would appreciate that any other threshold values for shape and density indices can be fixed/utilized to serve its intended purpose as laid down in embodiments of the present disclosure.

Table-3: Threshold values of shape and density indices for different images for ELM-SEG model

4 (Fig. 3d) 0.8 0.65

5 (Fig. 3e) 0.6 0.52

6 (Fig. 3f) 0.5 0.55

[0085] FIG. 7 A through FIG. 7F illustrates output for six different images (extracted and/or analyzed images overlapped on the real images) in accordance with embodiments of the present disclosure. In an embodiment, the performance parameters (measures) viz. completeness, correctness and quality are analyzed. FIG. 8 illustrates various performance measures for trained binary classified image generation module (ELM model) with different number of hidden neurons after performing spectral-spatial measures. In accordance with an embodiment of the present disclosure, results indicate that with increase in number of hidden neurons in trained binary classified image generation module, the performance measures (parameters) gradually increase. FIG. 8 illustrates the performance measure with different number of hidden neurons, as these are increasing gradually but at a certain level they are consistent. Accordingly, when number of hidden neurons exceeds 100, the variation in the results is very minimal. In an embodiment, the optimum number of neurons in the hidden layer is 100. Table 4 below depicts the performance parameters (measures) of the segmented images utilizing system and method in accordance with embodiments of the present disclosure.

Table 4: Performance measures of segmented images

[0086] FIG. 9 illustrates the statistical visualization of performance of ELM-SEG in comparison with ground reference data in accordance with an embodiment of the present disclosure. The R2 value is found to be 0.817, which indicates a good fit.

[0087] Based on the comparison of image output from conventional trained ELM method (as illustrated in FIG. 6B) and that from the methodin accordance with embodiments of the present disclosure (as illustrated in FIG. 6F), it would be apparent that although, the conventional method is able to classify powerline properly between two features, there are several mismatch (errors in classification) due to mismatch of RGB values. Accordingly, the system and method as realized and embodied hereinabove, is superior in comparison to the conventional methods for UAV based detection/supervision of powerlines.

[0088] While the foregoing describes various embodiments of the invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof. The scope of the invention is determined by the claims that follow. The invention is not limited to the described embodiments, versions or examples, which are included to enable a person having ordinary skill in the art to make and use the invention when combined with information and knowledge available to the person having ordinary skill in the art.

ADVANTAGES OF THE INVENTION

[0089] The present disclosure provides a system and method to detect presence of powerlines in aerial images.

[0090] The present disclosure provides a system and method that effectively point out powerlines in an image captured by a camera device coupled with an aerial vehicle, such as an unmanned aerial vehicle (UAV) or a manned vehicle, for instance, a helicopter.

[0091] The present disclosure provides a system and method that envisages to eliminate manual monitoring and inspection of powerlines by utilizing advanced machine learning and image processing methods of powerline detection in an image.

[0092] The present disclosure provides a system and method that overcomes deficiencies associated with conventional powerline monitoring and detection techniques.

Claims

We Claim:

1. A system for detection of a powerline in an aerial image, the system comprising:

a non-transitory storage device having embodied therein one or more routines operable to detect the powerline in the aerial image; and

one or more processors coupled to the non-transitory storage device, wherein theone or more routines comprise:

an image retrieval module, which when executed by the one or more processors, obtains at least one aerial image including the powerline;

a binary classified image generation module, which when executed by the one or more processors, subjects said at least one aerial image to a pixel -by-pixel analysis;

a skeletonization module, which when executed by the one or more processors, enables skeletonization of said binary classified image to reduce one or more objects of said at least one aerial image to a median line;

a geometric feature determination module, which when executed by the one or more processors, calculates any or a combination of a shape index and a density index for the powerline in the skeletonized image based on one or more dimensional characteristics of the powerline in the skeletonized image; and

a powerline detection module, which when executed by the one or more processors, detectsthe powerline in the aerial image by removing any non- powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a pre-determined set of shape index and density index of the powerline.

2. The system as claimed in claim 1, wherein the aerial image is an aerial vehicle basedremote sensed image acquired by an image capturing device operatively coupled with the aerial vehicle.

3. The system as claimed in claim 1, further comprisinga region growing module that subjects the skeletonized image to region growing segmentation to restore continuity of the one or more objects lost during the skeletonization.

4. The system as claimed in claim 1, wherein the binary classified image generation module is trainedusing a plurality of RGB spectral values to differentiate between pixels corresponding to powerline features and pixels corresponding to non-powerline features.

5. The system as claimed in claim 1, wherein the powerline detection module is configured to classify the one or more objects in the aerial image as non-powerline feature in case the calculated shape index pertaining to the powerline in the aerial image is below a predetermined limit.

6. The system as claimed in claim 1, wherein the powerline detection module is configured to classify the one or more objects in the aerial image as non-powerline feature in case thecalculated density index pertaining to the powerline in the aerial image is above a predetermined limit.

7. A method for detection of a powerline in an aerial image, the method comprising the steps of:

obtaining, by a computing device, at least one aerial image including the powerline;

obtaining, by the computing device,a binary classified image by subjecting said at least one aerial image to a pixel-by-pixel analysis;

effecting, by the computing device, skeletonization of said binary classified image to reduce one or more objects of said at least one aerial image to a median line;

calculating, by the computing device,any or a combination of a shape index and a density index for the powerline in the skeletonized image; and

detecting, by the computing device, the powerline in the aerial image by removing any non-powerline feature from the aerial image based on comparison of any or a combination of the calculated shape index and the calculated density index with a predetermined set of shape index and density index of the powerline.

8. The method as claimed in claim 7, wherein the aerial image is an aerial vehicle basedremote sensed image acquired by an image capturing device operatively coupled with the aerial vehicle.

9. The method as claimed in claim 7, further comprising a step of subjecting the skeletonized image to region growing segmentation to restore continuity of the one or more objects lost during the skeletonization step.

10. The method as claimed in claim 7, wherein the step of removing any non-powerline feature to effect detection of the powerline in the aerial image further comprises the steps of:

classifying the one or more objects in the aerial image as non-powerline feature in case the calculated shape index pertaining to the powerline in the aerial image is below a pre-determined limit; and classifying the one or more objects in the aerial image as non-powerline feature in case thecalculated density index pertaining to the powerline in the aerial image is above a pre-determined limit.