WO2025188376A2 - Platform for analyzing the condition of infrastructure using automated defect and change detection - Google Patents

Abstract

A platform for automatically detecting and analyzing defects in infrastructure assets uses aerial imagery, machine learning models and an interactive geographic information system (GIS) interface. Drone or aircraft images are processed by deep neural networks to identify and quantify surface and subsurface defects. Defects are mapped onto 3D models with precise geospatial coordinates. A web platform allows engineers to visualize defects from individual or multiple time periods, filter and query based on attributes, analyze density heatmaps, highlight changes, and perform side-by-side comparative analysis. The unique interface enables rapid condition assessments to support proactive maintenance.

Description

Attorney Docket No.: NIRI-2024003pct

TITLE

PLATFORM FOR ANALYZING THE CONDITION OF INFRASTRUCTURE USING AUTOMATED DEFECT AND CHANGE DETECTION

CROSS-REFERENCE TO RELATED APPLICATION^]

[0001] This application claims priority to US Provisional Patent Application Serial No. 63/547,283, filed on November 3, 2023, which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

[0002] The present invention relates generally to inspection systems for civil infrastructure, and more particularly to a software platform, which we call AUTOSPEX, for analyzing infrastructure assets using automated defect detection and change analysis.

BACKGROUND

[0003] Aging civil infrastructure assets like dams, bridges and buildings are subject to harsher conditions due to climate change, requiring more detailed and frequent inspections. However, manual inspections are hazardous, time-consuming and subjective.

SUMMARY

[0004] Disclosed herein is a platform, referred to herein as AUTOSPEX platform, that leverages drone imagery, machine learning for automated defect detection, and data visualization tools to enable faster, cheaper and more quantitative condition assessments of infrastructure assets. The AUTOSPEX platform provides a system for automatically detecting defects in images of infrastructure assets using machine learning, quantifying changes in defect parameters over time, and visualizing the defect data in an interactive web-based geographic information system (GIS). The platform ingests drone or aircraft imagery which is processed by deep neural networks to classify surface defects like cracks, spalling, and delamination. These defects are extracted and their properties like length, width, area, depth and volume are quantified. The defects and imagery are mapped to precise geographic coordinates on 3D models and/or a set of 2D orthomosaic images of the asset structure. Engineers can then use the AUTOSPEX interface to visualize defects for one or multiple time periods, view and filter the defects based on attributes, analyze defect density heatmaps, quantify and highlight regions of change, and compare synchronized views of multiple surveys side-by-side. The unique AUTOSPEX interface enables rapid condition assessments and change analysis to support, proactive, preventative maintenance of infrastructure.

[0005] The AUTOSPEX platform described herein includes inventive system, method, and dataset elements. These may be summarized as follows:

[0006] A system for analyzing civil infrastructure assets comprises: an image collection module configured to control an unmanned aerial vehicle to capture imagery of the infrastructure asset; a photogrammetry module configured to process the captured imagery into an orthomosaic and 3D digital surface model; an ingestion module configured to upload the orthomosaic, 3D model, and image metadata into a cloudbased storage server; a machine learning module comprising a trained convolutional neural network configured to analyze the orthomosaic to detect and classify defects in the imagery; a feature extraction module configured to quantify attributes of the detected defects and defected areas; a geospatial registration module configured to map the defects onto precise geographic coordinates on the orthomosaic and 3D model based on the image metadata; a database configured to store the defects and asset models in a relational structure; and a web-based user interface configured to visualize the orthomosaic, models, defects, and analytics, provide interactive filtering based on attributes, enable comparative analysis of multiple surveys, and output condition assessment reports.

[0007]A computer-implemented method for assessing infrastructure assets comprises: collecting high resolution aerial imagery of the asset using drones or aircraft; generating an orthomosaic and 3D model from the aerial imagery using photogrammetry routines; storing the imagery, models, and metadata in a cloud-based structured database; applying machine learning algorithms to analyze the orthomosaic and detect defects in the imagery, resulting in classification labels, segmentation masks, and optionally bounding boxes for each defect; quantifying attributes of the detected defects including type, size, shape, and location; mapping the defects onto the 3D model using the image metadata and model coordinates; accumulating and relating the defects, models, and metadata in a relational database with spatial extensions; providing a web-based interface allowing users to visualize the defects and models on interactive maps; enabling users to filter visible defects based on attributes using the interface; and generating comparative visualizations and analytics to identify defect changes across multiple surveys of an asset.

[0008] A labeled dataset for training machine learning models comprises: a plurality of labeled aerial images of civil infrastructure assets with pixel-level annotations delineating cracks, spalls, corrosion, and other defects; wherein the plurality of images covers diversity in defect type, defect severity, image perspective, lighting conditions, background surfaces, occlusion levels, and imagery capture platforms; wherein the plurality of images are augmented through transformations including flips, rotations, crops, and color shifts, resulting in an enhanced training dataset; and wherein machine learning models trained on the diverse, representative, and augmented labeled dataset improve generalization and accuracy in detecting concrete and infrastructure defects compared to models trained without this enhanced dataset.

[0009]As will be seen, the benefits offered by our innovative platform include:

• Faster and cheaper condition assessments compared to manual visual inspections

• More quantitative and objective defect evaluation using Al and geospatial mapping

• Improved accuracy of defect detection using deep learning versus human inspectors

• Ability to identify emerging defects early through frequent surveys and change analysis

• Focus engineering efforts on highest priority defects and areas using interactive filtering

• Enhanced data accessibility with secure access to historical surveys from any location

• Prevent catastrophic failures through proactive repair planning based on actionable insights

• Higher frequency surveys and regular monitoring at lower cost than manual methods

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 is a block diagram of the overall AUTOSPEX system architecture. [0011] FIG. 2 is a flow chart of the defect detection and analysis workflow.

[0012] FIG. 3 is a screenshot of an example AUTOSPEX platform graphical user interface (GUI) showing a user’s assets.

[0013] FIG.4 is a screenshot of an example asset card.

[0014] FIG. 5 is a screenshot of an example AUTOSPEX platform GUI that includes an example asset with multiple structures.

[0015] FIG. 6 is a screenshot of an example AUTOSPEX platform GUI that includes an example asset dashboard.

[0016] FIG. 7 is a screenshot of an example AUTOSPEX platform GUI that includes an example AUTOSPEX map layout.

[0017] FIG. 8 is a screenshot of an example AUTOSPEX platform GUI that includes an example AUTOSPEX map components.

[0018] FIG. 9 is a screenshot of an example AUTOSPEX platform GUI for displaying map layers.

[0019] FIG. 10 is a screenshot of an example AUTOSPEX platform GUI that includes an example defect overview.

[0020] FIG. 11 is a screenshot of an example AUTOSPEX platform GUI that includes an example defect listing.

[0021] FIG. 12 is a screenshot of an example defect card.

[0022] FIG. 13 is a screenshot of an example AUTOSPEX platform GUI that includes an example image showing cracking and highlighting a defect

[0023] FIG. 14 is a screenshot of an example AUTOSPEX platform GUI that includes an example image showing spalling and highlighting a defect.

[0024] FIG. 15 is a screenshot of an example AUTOSPEX platform GUI for defect filter drawer.

[0025] FIG. 16 is a screenshot of an example AUTOSPEX platform GUI for adjusting a density grid heatmap and listing.

[0026] FIG. 17 is a screenshot of an example AUTOSPEX platform GUI for adjusting a change grid heatmap and listing.

[0027] FIG. 18 is a screenshot of an example AUTOSPEX platform GUI for change grid filter drawer. [0028] FIG. 19 is a screenshot of an example AUTOSPEX platform GUI for changing grid settings.

[0029] FIG. 20 is a screenshot of an example p AUTOSPEX platform GUI that includes a split screen during a comparison mode.

[0030] FIG. 21 is a screenshot of an example AUTOSPEX platform GUI that includes a split screen that makes area comparison with quantified defect values.

[0031] FIG. 22 is a screenshot of an example AUTOSPEX platform GUI that includes a split screen that makes defect comparison with quantification metrics.

DETAILED DESCRIPTION

[0032] The AUTOSPEX platform is a cloud-hosted solution enabling centralized storage, processing, analysis and visualization of infrastructure survey data. As shown in the block diagram in FIG. 1 , the platform 100 includes a data ingestion module 10 (e.g., data ingestion tools), a machine learning module 20 (e.g., machine learning components) that further includes a feature extraction module 22, a geospatial registration module 25, data servers 30, and a web-based user interface 40.

[0033] DATA INGESTION MODULE 10

[0034] In the data ingestion phase, aerial survey imagery is collected via drones, planes, helicopters, satellites, or other suitable means based on user-defined flight plans and requirements. Survey metadata is captured like GPS coordinates and camera angles. The imagery is processed, by the data ingestion module 10, through stitching and orthorectification routines to create 2D orthomosaic photo maps and 3D digital surface models of the asset structure. These base maps provide precise geospatial reference layers for defect mapping.

[0035] The first step in the AUTOSPEX workflow is data ingestion, which involves collecting aerial survey imagery and processing it into analysis-ready formats. In a presently preferred embodiment of the platform, the following steps are employed: [0036] Surveys are performed by flying drones, manned aircraft, or helicopters over the infrastructure asset based on pre-planned flight paths defined in the mission planning software. Flight parameters like altitude, speed, overlap, and camera angles are configured to ensure complete coverage with high resolution imaging. [0037] The aerial imagery is captured by RGB cameras and optionally multispectral, thermal, or lidar sensors mounted on the aircraft. In addition, implementations of the platform may employ proprietary acoustic sensors for collecting acoustic points (sounds and vibration) useful to detect/classify delaminations. Imagery and other data from each flyover are stored on onboard devices. Survey metadata like GPS coordinates, altitude, camera intrinsics, and orientation is logged by onboard avionics equipment.

[0038] After a survey flight is complete, the imagery is downloaded and fed into photogrammetry software suites, which may be one of the tools of the data ingestion module 10, for processing. The individual aerial photos are combined into a single stitched 2D orthomosaic image that covers the full asset structure using stitching algorithms.

[0039] The photogrammetry software applies camera calibration and aerotriangulation routines to correct lens distortion effects and align images into a unified coordinate reference system. This results in a high-resolution composite photo map with uniform scale and geometry. As described below, the overall process may be summarized as: 1 . Collect raw imagery, 2. Stitch images to create 3D model, 3. Extract 2D orthomosaic (Orthos), 4. Apply ML/AI on Orthos to detect defects, 5. QC (check results), 6. Publishment/lnsights.

[0040] Next, digital surface model (DSM) generation algorithms, which may be another tool of the data ingestion module 10, analyze pixel shifts between overlapping images to reconstruct a 3D point cloud representation of the asset surface. This 3D model encodes elevation data for the structure.

[0041] The orthomosaic and DSM provide core geospatially-referenced data layers for downstream defect analysis. They are exported in standard raster and vector formats and uploaded to cloud storage servers. Metadata like GPS coordinates, point cloud, and camera angles are stored in databases to enable precise geospatial mapping of defects. [0042] By automating the process of converting raw survey imagery into analysis-ready orthomosaics, 3D models, and georeferenced metadata, the AUTOSPEX data ingestion pipeline enables rapid generation of high-fidelity base maps for defect detection and visualization.

[0043] MACHINE LEARNING MODULE 20 [0044] The orthomosaic imagery is then analyzed by deep neural network models of the machine learning module 20 to detect and classify surface defects like cracks, spalls, efflorescence based on visual patterns. These models are convolutional neural networks pre-trained on diverse defect image datasets. Key defect attributes like width, length, area, depth and volume are extracted by, for example, a feature extraction module 22, through image processing routines like boundary tracing. The defects are registered to real-world geographic coordinates by projecting onto the 3D model of the structure using the image metadata and reference points.

[0045] The orthomosaic imagery collected for each survey is analyzed by deep neural networks to detect and classify different types of defects on the asset structure. In the presently preferred embodiment of the platform, the following machine learning techniques are employed for analyzing the aerial survey imagery:

[0046]AUTOSPEX employs state-of-the-art convolutional neural network architectures like ResNet and VGGNet pre-trained on diverse datasets of concrete crack, spall, corrosion, and damage images. These networks excel at learning meaningful visual patterns from pixel data.

[0047] The models are fine-tuned through supervised training on labeled defect datasets specific to the infrastructure vertical. Labeled training images are collected covering crack, spall, delamination, efflorescence, rust, and domain-specific defects across various surface types, lighting conditions, defect widths, orientations, and severity levels.

[0048] Data augmentation techniques like rotations, flips, crops, and color shifts are applied to expand the diversity of training examples. The deep learning models converge after extensive iterative training to accurately classify defect types, while generalizing across assets.

[0049] For inference on new survey data, the pre-trained models are run across the orthomosaic using sliding window object detection and image segmentation routines to identify defect locations on the asset surface.

[0050] Post-processing filters like non-maximum suppression are applied to merge overlapping detections. The models output classified defect labels, segmentation masks, and detection confidence scores for each identified defect instance. [0051] Key defect attributes like length, width, area, depth, orientation, and volume are extracted by, for example, the geospatial registration module 25, using image processing techniques like boundary tracing, contour analysis, shape fitting, and pixel counts. Each detected defect is parameterized for quantitative analysis.

[0052] By leveraging deep neural networks trained on infrastructure-specific defect datasets, AUTOSPEX automates the laborious process of manually identifying and assessing defects in survey imagery. This enables rapid quantitative condition assessments.

[0053] DATA SERVERS 30

[0054] All the imagery, models, and defect data are stored in a cloud server 30, such as one or more cloud servers and SQL databases for centralized access. Data from historical surveys is also archived for temporal analysis. Users access the data through the AUTOSPEX web platform using a standard browser.

[0055] Here is a more detailed overview of the data servers and data models leveraged in the AUTOSPEX platform architecture:

[0056] SERVERS

[0057]AUTOSPEX employs cloud-hosted servers running on infrastructure like Amazon Web Services (AWS) to provide flexible scalability, security, and easy access. The imagery files, 3D models, and defect data require significant storage, so distributed object storage services like S3 are used. Compute-optimized instances like EC2 provide processing power for on-demand photogrammetry, model inference, and data analysis. The servers are fully managed, allowing focus on application logic rather than infrastructure maintenance. Cloud hosting also aids collaboration, with asset owners and engineering teams worldwide able to securely access the platform.

[0058] DATA MODELS

[0059] To organize the myriad data types like imagery, models, defect lists, user data, and metadata generated in the AUTOSPEX workflow, the platform utilizes relational databases. Examples include SQL and NoSQL databases. PostgreSQL provides an enterprise relational database to interrelate entities like assets, surveys, users, and defects via foreign keys and join tables. This enables complex query capabilities across these entities. MongoDB supplies a document-based NoSQL database to store unstructured defect data like pixel masks, bounding boxes, classifications, and image patches. The combined data model provides structured storage for core system entities as well as flexible schemas for new defect data sources.

[0060] The database servers are synergistically designed around spatial data types, geospatial queries, and geospatial relationships. PostGIS adds geographic object support to PostgreSQL, allowing it to store asset coordinates, basemap layers, and defect locations while retaining spatial relationships. This enables location-based filtering and mapping. MongoDB has geospatial indexing and querying as well, optimizing location-driven analysis like finding defects within specified basemap boundaries.

[0061] By leveraging robust cloud infrastructure and combining the strengths of relational and NoSQL databases for interlinked storage of all system data from users to defects to models, AUTOSPEX provides a scalable and flexible data backend tailored for defect analysis of infrastructure assets.

[0062] USER INTERFACE 40

[0063] The AUTOSPEX user interface provides a suite of data visualization, filtering and analytical tools tailored for infrastructure condition assessments. As shown in the workflow in FIG. 2, the interface allows users to select a specific asset and survey period to display the associated orthomosaic map, digital surface model, defect layers and base metadata in an interactive geographic information system (GIS) map.

[0064]As shown in FIG. 2, the workflow comprises the following steps:

1 . Plan infrastructure asset survey flight using drones or aircraft

2. Capture high resolution aerial imagery of asset

3. Process imagery into orthomosaic and 3D digital surface model

4. Upload orthomosaic, 3D model, and metadata to cloud servers

5. Apply deep learning defect detection model to orthomosaic

6. Classify and localize defects in imagery through object detection

7. Extract defect attributes like size, type, shape metrics

8. Map and register defects to geographic coordinates on 3D model

9. Accumulate defects, models, metadata in spatial databases

10. Provide interactive web platform for users to access data 11 . Visualize orthomosaic and defect layers on interactive map

12. Filter visible defects based on attributes and locations

13. Generate defect density heatmaps and change analysis

14. Compare defect maps between multiple survey periods

15. Digitally annotate areas of concern on map

16. Export condition assessment reports and findings

17. Share data with asset owners and engineering teams

18. Monitor assets over time and plan maintenance actions

[0065] We will now provide a more detailed description of key user interface elements and functions in the AUTOSPEX platform.

[0066] The AUTOSPEX web interface provides users a centralized portal to access and interact with their infrastructure survey data. The interface contains the following key components:

Dashboard - Displays summary information on assets, surveys, defects, and provides access to analysis tools. Users can filter and search for assets of interest.

Interactive Map - Displays the basemap, model, defects, and layers for a selected survey. Provides native GIS tools like zoom, pan, measure, and geospatial queries. Defects are overlaid as points, lines, and polygons.

Side-by-Side View - Displays synchronized split-screen map views enabling comparison of the same location across different survey periods. Linked navigation and highlighting.

Layers Panel - Contains layer selector, opacity sliders, and legend for toggling imagery, models, defects, and density grids on/off.

Filters Panel (e.g., filter drawer) - Provides filters to constrain visible defects by attributes like type, size, date. Linkable to other views.

Inspector Panel - Displays properties and attributes for selected defects. Links to external asset databases.

Analytics - Utilities like defect density heatmaps, change analysis, and summary statistics to quantify defects. Annotation Tools - Allows users to draw digital markers, lines, and polygons on the map to denote areas of interest or additional defects.

Reporting - Generate reports and exportable files for survey data and findings. Account Tools - Manage user accounts, access privileges, API keys, and notifications.

[0067] The responsive web interface enables intuitive exploration of survey data on any device. The key differentiator is tight integration of location-based analytics, comparative visualizations, and interactive filtering tailored for infrastructure condition assessments. [0068] As illustrated in FIGS 3-22, the interface enables engineers to visualize different defect types like cracks, spalls, and delamination as separate layers on the map. They can filter the defects based on attributes like width, length or severity. A heatmap visualization shows defect density distribution. Engineers can digitize additional defects or areas of concern using drawing tools. The interface also allows comparison of multiple survey periods for the same asset using a split-screen view with synchronized navigation and highlighting. This enables rapid analysis of defect changes over time. [0069] By providing automated defect detection integrated with interactive visualization and change analysis tools tailored for infrastructure assets, the AUTOSPEX platform enables more rapid, quantitative, accurate and proactive condition assessments compared to manual survey methods. The system outputs actionable insights for preventing catastrophic failures and planning timely repairs.

[0070] KEY FEATURES AND SCREENSHOTS

[0071] In this section, we will show details of several key features of a presently preferred implementation of the AUTOSPEX platform. These key features are designed to make the platform particularly beneficial to users responsible for inspection and maintenance of large structures.

[0072] MY ASSETS

[0073] Users can view all their organization assets in one single view with key asset information such as asset type, site name, structure name and location tags in My Assets Listing page, as shown in FIG. 3.

[0074] DEFINITIONS

[0075] Assets - are defined as Site and Structures in AUTOSPEX. [0076] Sites - are defined as an accumulation of different structure types in the same geographical location (i.e. New Bullards Bar is a Site that includes a Penstock, Arch Dam, and a Concrete Spillway).

[0077] Structures - are defined as a single component of a site where an individual/routine survey(s) may be required.

[0078] FIG. 4 shows an Asset Card.

1. Manage Assets Users can view all their organization assets in one single view with key asset information such as asset type, site name, structure name and location tags (FIG.l My Assets Listing).

2. User can search for a specific asset by typing site, structure, and location name (FIG.l My Assets Listing).

3. User can sort the assets by name, asset type, survey dates and last updated time.

4. User can access to survey results and related data of an Asset Structure.

[0079] FIG. 5 shows an asset with Multiple Structures.

[0080]ASSET DASHBOARD

[0081] Asset Dashboard provides key information and all survey results for the selected asset.

[0082] FIG. 6 shows an Asset Dashboard.

1. User can access to key asset information.

2. User can access all survey results for the asset in different widgets. a. Base Layers Widget: Shows base layers such as Ortho imagery, 3D models, thermal imagery, and depth layer if available for the selected survey. b. Defect Widget: Shows available defect layers as well as the total defect area in cm² as a quick insight for the selected survey. c. Download Center Widget: Provide all survey report and data for the asset team to access if they have permission. d. Insights Widget: Interactive charts, aggregated survey results and insights on condition of the asset structure to view, collaborate and download based on their permission. e. AUTOSPEX Map Widget: Users can access the defect map to visualize survey results and collaborate on condition assessment of the asset structure.

3. User can view or update the asset team access settings. a. User can see the team members for the asset to collaborate. Admins can set certain access level and download permission for each asset team member.

[0083] AUTOSPEX MAP - LAYOUT

[0084]AUTOSPEX Map is a collaborative GIS map, as illustrated for example, in FIG. 7, to visualize high resolution base and Al-generated surface and subsurface defect layers, find areas to concern and compare multi-year surveys to support making informed- decisions on the condition assessment of the asset structure.

[0085] MAP LAYERS

[0086] User can visualize multi-layer base imagery, defect layers and system -created layers to help them find areas-to-concern about the integrity of their structure for the selected survey. See, for example, FIG. 9.

[0087] Base Ortho: 2D imagery of the asset structure

[0088] Defect Layers: Al-detected defect layers

[0089] Density Grid: Shows a density heatmap by defected area percentages within the predefined areas in grid sizes such as 1 m by 1 m or 5m by 5m areas.

[0090] Heatmap: System generated heatmap by the number of defects on the asset structure. See, for example, FIG. 16.

[0091] Depth: Depth of the spalls and/or offset areas

[0092] Thermal: Thermal imagery of the asset structure

[0093] DEFECT LAYERS - OVERVIEW

[0094] Defect Overview, such as illustrated n FIG. 10, provides a comprehensive view of the defect result summary for the selected survey. AUTOSPEX provide defect mapping for: [0095] Mapping surface defects: Cracking, Spalling, Vegetation, Efflorescence, Honeycomb. See, for example, FIGS. 13 and !4 highlighting cracking and spalling defects.

[0096] Mapping subsurface classification: Delamination

1. User can see all available defect layers with user-defined legends based on a quantification metric for the selected survey.

2. User can turn on/off any defect layers.

3. User can see the total active defects break down by defect type and legend bin.

4. User can download the defect list for active selected defects.

5. User can click on any legend bin to see the defect list under each legend bin to find defects to concern for the selected survey.

[0097] DEFECT LAYERS - DEFECT LISTING

[0098] Defect listing, as illustrated, for example, in FIG. 11 , provides all defects under a user-defined legend and/or threshold to help user find the widest, longest, largest defects easily.

1. User can see the defect list of a defect type and/or threshold.

2. User can sort the defects by a quantification metric Z-A.

3. User can see all key information legend, tags, quantification information of each defects in the list.

4. User can flag/unflag any defects in the list for a quick access and further analysis later.

5. User can click on any defect cards to highlight and locate on the map. See, for example, FIG. 12, which illustrates a screenshot of an example defect card.

[0099] DEFECT FILTERING

[0100] Defect filtering helps user to filter out defects by using relevant quantification metrics and defect attributes. See, for example, FIG. 15, which illustrates a defect filter drawer for selectively filtering defects.

1. User can choose flag/unflag to focus on what is important to them.

2. User can see the min/max values of each quantification metric and use the slider to define a defect filter (e.g., filter drawer). 3. User can load existing filters that were created by their team or themselves.

4. User can define and save a defect filter which can be accessible by their team members.

[0101] DENSITY GRID (LAYER)

[0102] Density grids show a heatmap by the defected area in selected grid size and defect type which helps the structural engineers to identify areas of concern efficiently.

1. User can change the grid size from predefined grid sizes (lm x lm, 2m x 2m, 3m x 3m, 5m x 5m, 10m x 10m). See, for example, FIG. 17, which shows an example GUI for changing a grid.

2. User can change the primary defect type to see the heatmap generated by the defected area.

3. User can see each grid tile and all the defect types that the grid tile includes with their defected area percentage and absolute values.

4. User can flag/unflag the grid areas to access later and make further analysis.

5. User can click on any grid tiles to highlight and see on the map.

6. User can download the current grid tile data if they have permission.

[0103] CHANGE GRID (LAYER)

[0104] Change grids show a heatmap by the change percentage between two surveys. The algorithm finds the defected area in each survey for the same grid tile coordinates then calculate the change. Structural engineers can identify areas where defected areas have increased, decreased, appeared in new area, or disappeared in some areas between surveys. Grid tiles are color-coded that represents the percentage of change, with accompanying legends for clear interpretation.

1. User can change the grid size from predefined grid sizes (lm x lm, 2m x 2m, 3m x 3m, 5m x 5m, 10m x 10m) to track the change. See, for example, FIG. 19.

2. User can change the primary defect type to see the change heaap generated by the defected area.

3. User can see change statistics with absolute values and percentage of change on grid tile cards including any defect type that was detected within that grid tile.

4. User can flag/unflag the grid areas to access later and make further analysis. 5. User can click on any grid tiles to highlight and see on the map.

6. User can download the current change grid tile data if they have permission.

7. Users can filter areas based on total changed area enabling focused analysis with Change Filter so that they can eliminate the small changes, using sliders or text boxes and focus on areas to concern. See, for example, FIG. 18.

8. Users can customize settings to compare surveys, including selecting surveys for comparison, choosing map views, specifying grid sizes, and applying changes.

[0105] COMPARISON MODE (SPLIT SCREEN)

[0106] With Comparison mode, user can compare two surveys visually and with quantification data in the same screen. See, for example, FIGS. 20, 21 , and 22 showing results when in comparison mode.

1. Monitoring Area Changes Over Time: Users can track changes in specific areas over multiple surveys, compare side-by-side visuals, and access quantification history.

2. Analyzing New Major Defects: Engineers can track and analyze newly appearing major defects in the latest survey.

3. Monitoring Repaired Areas: Users can assess areas that have undergone repairs and compare them before and after.

4. Flagging Areas: Users can access all the flagged areas to track the condition over time with synchronized displays and change cards.

5. Comparing Surveys for Specific Defects: Engineers can visually compare different surveys for specific defects and access quantification history.

[0107] CONCLUSION

[0108] Some of the key innovative features of the AUTOSPEX platform include:

• Automated defect detection using deep learning algorithms to identify cracks, spalls, delamination, efflorescence from imagery

• Precise geospatial mapping of defects onto 3D digital surface models using aerial image metadata

• Interactive comparison view with synchronized side-by-side defect map visualization for two time periods • Customizable defect density heatmaps based on grid cell size and defect type

• Defect change analysis through grid-based quantification of defect evolution between surveys

• Interactive filtering of defects by attributes like width, length, severity

• Drawing tools for annotating areas of concern on the defect map

• Cloud-hosted system for centralized storage, processing and access to imagery, models and defect data

[0109] In summary, AUTOSPEX provides infrastructure asset owners with a unique defect visualization and analytics platform to enable proactive, data-driven maintenance.

Claims

CLAIMS WHAT IS CLAIMED IS:

1 . A system for analyzing infrastructure assets, comprising: a data ingestion module (10) configured to process aerial imagery of an infrastructure asset into a 3D digital surface model and an orthomosaic; a machine learning module (20) comprising a trained convolutional neural network configured to analyze the orthomosaic to detect and classify defects in the imagery, the machine learning module including a feature extraction module (22) configured to quantify attributes of the detected defects; a geospatial registration module (25) configured to map the defects onto geographic coordinates on the orthomosaic and 3D digital surface model; a data server (30) configured to store the defects, orthomosaic, and 3D digital surface model; and a user interface module (40) configured to: display an interactive map comprising the orthomosaic and detected defects; provide a comparison mode with a synchronized split-screen view for visualizing defects from two different time periods side-by-side; and generate a change grid overlay showing quantified changes in defect parameters between the two time periods.

2. The system of claim 1 , wherein the data ingestion module is further configured to process acoustic data associated with the infrastructure asset.

3. The system of claim 1 , wherein the user interface module is further configured to provide interactive filtering of visible defects based on defect attributes.

4. The system of claim 1 , wherein the user interface module is further configured to generate a defect density heatmap based on a user-selectable grid cell size and defect type.

5. The system of claim 1 , wherein the user interface module is further configured to provide drawing tools for digitally annotating areas of concern on the interactive map.

6. The system of claim 1 , wherein the machine learning module is configured to detect surface defects including, but not limited to, cracks, spalling, and efflorescence, and subsurface defects including delamination.

7. The system of claim 1 , wherein the feature extraction module is configured to quantify defect attributes including, but not limited to, length, width, area, depth, and volume.

8. The system of claim 1 , further comprising an image collection module configured to control an unmanned aerial vehicle to capture the aerial imagery of the infrastructure asset.

9. The system of claim 1 , wherein the data server includes a cloud-based storage system providing centralized access to the defects, orthomosaic, and 3D digital surface model.

10. The system of claim 1 , wherein the user interface module is further configured to generate condition assessment reports based on the detected defects and quantified attributes.

11 . The system of claim 1 , wherein the change grid overlay comprises color-coded grid cells representing percentage changes in defect parameters between the two time periods.

12. The system of claim 1 , wherein the imagery includes optical and thermal imagery.

13. A method for analyzing infrastructure assets, comprising: processing aerial imagery of an infrastructure asset into 3D digital surface model and an orthomosaic; analyzing the orthomosaic using a trained convolutional neural network to detect and classify defects in the imagery; quantifying attributes of the detected defects; mapping the defects onto geographic coordinates on the orthomosaic and 3D digital surface model; storing the defects, orthomosaic, and 3D digital surface model in a data server; displaying an interactive map comprising the orthomosaic and detected defects; providing a comparison mode with a synchronized split-screen view for visualizing defects from two different time periods side-by-side; and generating a change grid overlay showing quantified changes in defect parameters between the two time periods.

14. The method of claim 13, further comprising processing acoustic data associated with the infrastructure asset to provide a 3D digital surface model and orthomosaic.

15. The method of claim 13, further comprising providing interactive filtering of visible defects based on defect attributes.

16. The method of claim 13, further comprising generating a defect density heatmap based on a user-selectable grid cell size and defect type.

17. The method of claim 3, further comprising providing drawing tools for digitally annotating areas of concern on the interactive map.

18. The method of claim 13, wherein analyzing the orthomosaic comprises detecting surface defects including cracks, spalling, and efflorescence, and subsurface defects including delamination.

19. The method of claim 13, wherein quantifying attributes of the detected defects comprises quantifying length, width, area, depth, and volume.

20. The method of claim 13, further comprising controlling an unmanned aerial vehicle to capture the aerial imagery of the infrastructure asset.

21 . The method of claim 13, wherein storing the defects, orthomosaic, and 3D digital surface model comprises storing them in a cloud-based storage system providing centralized access.

22. The method of claim 13, further comprising generating condition assessment reports based on the detected defects and quantified attributes.

23. The method of claim 13, wherein generating the change grid overlay comprises generating color-coded grid cells representing percentage changes in defect parameters between the two time periods.

24. The method of claim 13, wherein the imagery includes optical and thermal imagery.