CN104766086B - The monitoring and managing method and system of a kind of way mark - Google Patents

Abstract

A method and system for supervising highway markings, the method comprising: firstly, establishing a preprocessing standard model by marking a single sequence from existing highway standard GPS and image data; secondly, obtaining the latest real-time GPS and Image data, establish a real-time model by marking the double sequence; then, compare the real-time model data with the standard model data through double-sequence matching search, and match the real-time best image obtained from the real-time model with the standard image in the standard model ; Furthermore, use the image matching algorithm to compare the image data of the standard model, so as to determine the specific type of mark damage; finally, update the standard model according to the recognition result and maintenance status. According to the internal relationship between the position of the road mark and the image, the invention periodically collects in real time, automatically identifies and locates the damage and the type of the road mark, improves the detection efficiency of the road assets, and reduces the maintenance workload.

Description

Supervision method and system for highway signage

technical field

The invention relates to the field of computer information technology, in particular to a method and a system for supervising road signs by using an information system.

Background technique

As one of the basic constructions, roads are an important part of public assets. After road construction is completed, real-time supervision and maintenance are required to ensure the normal use of roads. At the same time, real-time supervision and maintenance are also one of the most basic means to protect road assets. At present, the systems used for road supervision and maintenance mainly include pavement management system and bridge management system, etc., and these systems are only used to detect and maintain the geology and road surface, and cannot realize the supervision of road signs.

Highway signs are usually set on both sides of the highway or on the top of the highway. When monitoring them, staff need to collect and input data on the spot along the way, which not only consumes a lot of manpower, material resources, time and resources, but is prone to missed inspections and missing records. , and because the detection cycle takes a long time, the damaged road signs will not be found and replaced in time, which will affect the specification, appearance and safety index of the road, and have a negative impact on people's travel instructions; in addition, due to human judgment Differences can easily lead to problems such as inconsistent standards. It is self-evident that the integrity of the road signs on the expressway is important to the safe driving of drivers. For example, the lack of road signs on the expressway can easily lead to serious traffic accidents such as speeding of high-speed vehicles; The long distance of the road and the excessive speed of the vehicle will also affect the working efficiency of the staff, and more importantly, it will also endanger the safety of the staff.

Contents of the invention

The technical problem to be solved by the present invention is to provide a monitoring method and a monitoring system that can monitor road signs in real time and accurately locate fault signs without the need for staff to go to the scene.

In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

A method for supervising highway signs, characterized in that it specifically comprises the following steps:

S101. Establishing a preprocessing standard model of road signs;

S102. Acquiring real-time information of highway signs, and building a real-time model;

S103. Obtain the best real-time image of the road sign in the real-time information collected in step S102 through an expert algorithm;

S104. The real-time optimal image obtained in step S103 is matched with the standard image of the preprocessed standard model in step S101 through sequence search, and the specific classification of actual road sign damage is obtained by expert algorithm, and stored;

S105. Determine whether maintenance is required according to the recognition result, and determine whether the standard image in the pre-processing standard model needs to be updated according to the feedback maintenance result;

S106. Establish a database to manage the data in steps S101 to S104;

S107. Design the host computer interface, display the data in an intuitive way, and realize people's actual needs for highway sign supervision.

Step S101 of the present invention specifically includes the following content:

S1011. Determine the model of the camera, and calculate according to parameters such as the height and position of the road sign, and install the camera at a suitable position on the inspection vehicle;

S1012. Collecting standard images of road signs;

S1013. Collect standard GPS positioning information of road signs;

S1014. According to the results collected in steps S1012 and S1013, a preprocessing standard model is established by labeling a single sequence.

Step S102 of the present invention specifically includes the following content:

S1021. Obtain real-time image information of road signs by using the video mode of the camera;

S1022. Carry out real-time GPS positioning on the road signs, and obtain real-time GPS information of the road signs;

S1023. Match the real-time GPS information with the corresponding real-time image, and establish a real-time model by marking the double sequence.

Step S1023 of the present invention specifically includes the following content:

Firstly, collect the real-time GPS information of road signs one by one and mark them sequentially;

Secondly, calculate the sampling frequency of the real-time image information of the road sign, and perform double-sequence marking on the sampled image;

Then, it is judged whether the real-time image information of the highway sign matches the first sequence of the real-time GPS information, and if they match, a real-time model is established.

Step S103 of the present invention specifically includes the following content:

S1031. Preliminary image processing;

S1032. Select the best real-time image containing road signs.

In the step S1031 of the present invention, an expert algorithm is used to preliminarily process the image, and the processing content includes the processing steps of preliminary denoising, filtering, restoration, tilt correction, and motion blur elimination of the image collected in S102.

Step S1032 of the present invention specifically includes the following content:

First, detect n frames of images obtained after preliminary processing of real-time images (n=1...5);

Secondly, according to the particularity of the road sign color, convert RGB to HSV color space;

Then, use the HSV color space to determine whether the current image contains a special color;

Again, the image is further segmented by the color region;

Finally, compare the size of the segmented regions between different images, and select the one with the largest segmented region as the real-time best image.

Step S104 of the present invention specifically includes the following content:

S1041, calculating a threshold to match the real-time GPS information with the standard GPS latitude and longitude information;

S1042, saving the real-time GPS information within the threshold range, and saving the matched GPS single sequence of the preprocessing standard model and the GPS double sequence in the real-time model;

S1043, according to the threshold requirement, match the matched real-time GPS information with the standard image corresponding to the pre-processing standard model and the corresponding real-time best image in the real-time model;

S1044. Obtain the specific classification of the real-time best road sign damage through expert algorithm, and store it.

A system for supervising highway signs, the system comprising:

An image collection module 101, configured to collect images of road signs;

GPS information collection module 102, used for collecting the GPS positioning information of road signs;

Preprocessing standard model and real-time model building module 103 are used to carry out synchronous sequence marking respectively to standard image information of fixed-point road signs and standard GPS longitude and latitude information by image acquisition module 101 and GPS information acquisition module 102, to establish standard image information and standard GPS One-to-one correspondence of latitude and longitude information sequence road sign preprocessing standard model;

The real-time GPS information and real-time images of the road signs are collected by the image acquisition module 101 and the GPS information acquisition module 102, the real-time GPS information of the road signs and the image information corresponding to the real-time GPS information are extracted, and the road signs are established by image sampling and double-sequence marking real-time model of

The two-model matching module 104 extracts key images through sequence matching and threshold value deletion in the pre-processing standard model and real-time model building module 103, and controls the selected key images through parameter calculation and expert algorithm control, and the road signs corresponding to fixed-point GPS process the key image and get the real-time best image from it;

The image processing and recognition module 105 is used to compare the real-time optimal image and the pre-processing standard model obtained by the expert algorithm statistics in the two model matching modules 104 with the standard image information in the real-time model building module 103 Pattern matching, through expert algorithm matching training, to obtain the damage type of the road sign after the actual matching, and classify and store;

The database module 106 is used to carry out orderly and effective supervision on the data information in the system, and establish a database mechanism;

The upper computer control module 107 is linked with the database in the database module to visually display the processed data.

The information displayed by the control module of the upper computer mainly includes the GPS latitude and longitude information of the fixed-point road sign, the image of the fixed-point road sign, the specific category after image recognition, the damage status information of the road sign and whether the maintenance is required. Completed identification information.

Due to the adoption of the above technical solutions, the technological progress achieved by the present invention is as follows.

The invention integrates the functions of acquiring, detecting, identifying and locating highway sign asset information, can automatically detect whether the highway sign is damaged, the type of damage and the location of the damage, shortens the detection period of the road asset, improves the detection efficiency, and reduces the safety. Hidden dangers have effectively ensured the driving safety of the highway.

The application of the present invention provides a convenient collection path for the asset situation collection of the expressway, provides a simple channel for the asset cleaning of the expressway, and brings great convenience for the staff to maintain the driving safety of the expressway, and is beneficial to the reduction of road traffic. Driving safety accidents have certain positive effects. Compared with the traditional asset clearing system, it is more efficient and saves manpower, financial and material resources. The GPS positioning system of the present invention can locate and count the position of the fault point of the road mark, realize intelligent positioning, can accurately obtain the specific position of the damaged road mark, is practical for any road section covered by the GPS, and has a wide application range. The image processing process uses advanced expert algorithms to ensure the accuracy of the processing results; at the same time, various possible faults are extracted and calculated, and can identify a variety of road marking faults caused by different factors, covering a wide range , high recognition performance. The invention forms the integrated information of GPS and images and various types of data after identification into a database, and can share, restore, unify and completely control the data. The invention also provides a reasonable interactive interface, so that the staff can directly understand the specific conditions of the road signs on the interactive interface and perform related operations, which is convenient and practical.

Description of drawings

Fig. 1 is the overall realization block diagram of the present invention;

Fig. 2 is the overall structural block diagram of system of the present invention;

Fig. 3 is a flowchart of the method of the present invention;

Fig. 4 is a schematic diagram of the installation position of the camera in the present embodiment;

Fig. 5 is a schematic diagram of the travel range of the inspection vehicle in this embodiment;

Fig. 6 is the flow chart that the present invention realizes extracting key frame from ingested image in real-time operation;

Fig. 7 is the flow chart that the present invention realizes the overall image processing;

Fig. 8 is the realization block diagram of real-time GPS information and real-time image matching of the present invention;

Fig. 9 is a realization block diagram of real-time best image and standard image matching of the present invention;

Fig. 10 is a flow chart of the present invention to realize preliminary processing of real-time images;

Fig. 11 is a flow chart of the present invention selecting the real-time best image containing road signs;

Fig. 12 is a real-time image recognition flowchart of the present invention taking point pattern matching as an example;

Fig. 13 is a structural block diagram of the database of the present invention;

Fig. 14 is a schematic diagram of the interactive interface of the upper computer control module of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

A monitoring system for highway signs, its structural block diagram is shown in Figure 2, including an image acquisition module 101, a GPS information acquisition module 102, a preprocessing standard model and real-time model building module 103, a two-model matching module 104, image processing and recognition Module 105, database module 106 and host computer control module 107. The functions of each module and the transmission of information between modules are as follows.

The image collection module 101 is configured to collect images of road signs. The image collection module 101 includes two parts: one is to collect and store standard images of fixed-point road signs through the vehicle-mounted industrial camera, and the other is to collect real-time images of road signs during the traveling process.

In this embodiment, the Z30M industrial camera is used to collect images, and the Z30M camera supports two modes of video and photo. Therefore, when the preprocessing standard model is established to collect standard images of fixed-point road signs, the photo mode can be used for manual collection. In the real-time collection of highway signs, the video mode can be used to collect the road signs along the way as a whole.

The GPS information collection module 102 is configured to collect GPS positioning information of road signs.

The GPS information acquisition module 102 includes two parts: the one is to collect, mark and store the standard GPS positioning information of the highway identification in the preprocessing standard model to form the standard GPS longitude and latitude information of the highway identification; The identified real-time GPS information is collected.

The preprocessing standard model and real-time model building module 103 are used to carry out statistics on the standard image information and standard GPS longitude and latitude information of fixed-point highway signs through the image acquisition module 101 and GPS information acquisition module 102, and perform synchronous single-sequence marking respectively to establish highways The preprocessing standard model of the sign is used as a realistic basis to match it with the real-time statistical information of the road sign.

Collect real-time GPS information and real-time images of highway signs by image acquisition module 101 and GPS information acquisition module 102, extract real-time GPS information of highway signs and image information corresponding to real-time GPS information, and through image sampling and double-sequence marking, both The first sequence is one-to-one correspondence, and on this basis, a real-time model of the road sign is established.

The two model matching modules 104, according to the GPS threshold value range calculated by the parameters collected by the GPS information collection module 102, compare the standard GPS longitude and latitude information of the highway identification in the preprocessing standard model, save the effective real-time GPS information in the threshold range, and Sequence labeling and storage are also performed to prepare for the matching of real-time images and real-time GPS information.

In the present invention, the two model matching modules 104 can compare the standard GPS longitude and latitude information of the highway identification in the pre-processing standard model with the GPS threshold range calculated according to the parameters collected by the GPS information collection module 102, and save the effective real-time GPS within the threshold range. information strip. Extract images with road signs in the minimum range, reduce the storage space and processing quantity, improve the efficiency of real-time matching and the accuracy of image extraction for road sections that may have road signs in the real-time acquisition process.

Key images can be extracted from the real-time information in the preprocessing standard model and real-time model building module 103 through sequence matching and threshold value deletion, and the selected key images can be controlled by parameter calculation and expert algorithm. The image is processed and the best image in real time is derived from it.

Image processing and recognition module 105, the processing includes preliminary processing such as frame labeling, restoration, and denoising of the extracted image information, detection and segmentation of images with road identification information, etc., to obtain roads that can be compared with the pre-processing standard model Identify the real-time best image for matching against a standard image.

The real-time optimal image containing the road sign obtained after the statistics of the expert algorithm in the two model matching modules 104 and the preprocessing standard model and the statistical standard image information in the real-time model building module 103 are pattern-matched, and trained by expert algorithm matching, The damage type of the actually matched road sign is obtained and classified and stored.

The database module 106 is used to conduct orderly and effective supervision on the data information in the system, establish a database mechanism, and facilitate data extraction and viewing.

The upper computer control module 107 is linked with the database in the database module to visually display the overall system of the present invention, which is convenient for staff to operate. The host computer control module uses a friendly interactive interface to make the processed data more intuitive for relevant personnel to use. The displayed information mainly includes GPS latitude and longitude information of fixed-point road signs, images of fixed-point road signs, specific categories after image recognition, and road signs. Information such as the damage status and the identification information of whether the repair is completed, so that the staff can take corresponding replacement or maintenance measures in time.

The overall flow chart of the image processing of the present invention is shown in Figure 7, specifically: 1) collect the standard image of the road sign and the standard GPS positioning information to form pre-processing integrated information, and then obtain the road standard from the pre-processing integrated information 2) Set thresholds through standard GPS longitude and latitude information; 3) Obtain real-time image information and real-time GPS information of highway signs, and select matching real-time images and real-time GPS information according to thresholds, and form real-time integrated information; 4) After a series of operations such as preprocessing, detection, and segmentation, the obtained matched real-time images are subjected to a series of operations to obtain the real-time best images that meet the requirements; 5) The real-time best images are compared and identified with the standard images in the pre-processing standard model, Get the real-time status information of road signs and store them in categories.

A method for supervising highway signs, the flow chart of which is shown in Figure 1. The method includes: firstly, establishing a preprocessing standard model by labeling a single sequence from the existing highway standard GPS and image data; secondly, determining the optimal sampling Frequency and automatically obtain the latest GPS and image data through highway inspection operations, and establish a real-time model by marking the double sequence; The best image is matched with the standard image; then the point pattern image matching algorithm is used to compare the standard data to determine the specific type of road sign damage; finally, the preprocessing standard model is updated according to the recognition result and maintenance status.

The concrete flowchart of this method is shown in Figure 3, comprises the following steps:

S101. Establish a preprocessing standard model of road signs.

The image of the standard road sign is collected through the selected tool, and the GPS positioning information of the current image is obtained synchronously, the obtained data is stored, and the preprocessing standard model is established. The establishment of the preprocessing standard model specifically includes the following steps.

S1011. Determine the camera model, and install the camera at a proper position on the inspection vehicle according to the height of the road sign.

First determine the camera model, so as to determine the focal length, frame rate and other parameters, and obtain the corresponding actual distance according to the video frame captured in the actual unit time, and then determine the speed of the inspection vehicle to make it match, and at the same time avoid the same road section. Repeat the judgment; install the camera at a specific position of the inspection vehicle. Under standard conditions, the height and position of the road signs have certain regulations. Therefore, according to the height position, the installation position and installation angle of the camera on the inspection vehicle can be determined, and the inspection The car runs on the track so that the camera can capture the best real-time image of the road markings.

In this embodiment, the selection of the specific camera model and the calculation of the installation position are as follows.

In this embodiment, the camera adopts a Z30M color camera with 360,000 pixels, and the resolution can provide 752*480 (54 frames/second), 640×480 (60 frames/second) or 320×240 (60 frames/second), and the frame The rate can be set according to needs; the sensor of the camera lens is 1/3CMOS (black and white/color), the sensor size is 4510um*2880um (width*height), and the default mount of Z30M/Z30C is CS model.

According to the maximum installation height of the standard road sign is 10m, the maximum size of the road sign is 4*3m, the height of the inspection vehicle is 1.5m, and the speed of the inspection vehicle is 14m/s, in order to ensure that the road sign can be within the shooting range, In this embodiment, 14m away from the road sign is selected as the shooting distance of the camera.

1. The upward viewing angle of camera installation is calculated according to the following formula.

After calculation, tanθ=0.82 can be obtained, and then θ≈40°, that is, the elevation angle setting of the camera is about 40° when it is installed. The schematic diagram of the installation position of the camera in this embodiment is shown in FIG. 4 .

2. The wide-angle range of the camera lens is required to be calculated according to the following formula.

Then θ ₃ =θ ₁ -θ ₂ =49°-31°=18°, that is, the wide-angle range of the lens must reach at least 18°.

3. The choice of camera lens model.

According to the sensor model 1/3COMS of the selected lens, the sensor size is 4510um*2880um (width*height), and the vertical focal length and horizontal focal length of the lens are calculated by the following formula:

Among them, f _w is the horizontal focal length, f _h is the horizontal focal length, w is the width of the sensor, L is the distance from the subject to the lens, W is the width of the subject, h is the height of the sensor, and H is the height of the subject.

In this embodiment, the maximum width of the subject, namely the road sign, is 4m, and the maximum height of the subject is 3m, then:

In this embodiment, it is first assumed that the minimum distance between the subject and the lens needs to be 14m, but the actual shooting process may be much larger than this, so the minimum range of available focal length is 13.44mm-15.785mm. According to the minimum viewing angle and the minimum focal length range of the selected lens, the lens model selected in this embodiment is 0660MV.

Z30M industrial camera can provide one or more USB high-speed interfaces to connect with PC, simple wiring, convenient data transmission, no need to add additional connecting lines, easy to use, plug and play, support hot swap, aluminum alloy shell, strong shock resistance , has a strong ability to resist electromagnetic interference, and the parameters meet the requirements of the image acquisition of the present invention. The focal length of the camera lens can be adjusted between 6mm and 60mm, with a wide range of adjustment, and the horizontal field of view Angle of View: 44.5~4.6, which meets the needs of multiple angles, and the Z30M industrial camera can switch between taking photos and videos , which can satisfy the establishment of the preprocessing standard model in the present invention and the video acquisition in the subsequent real-time detection.

The expressway road is spacious and there are many vehicles. It is impossible for the inspection vehicle to travel on a fixed lane, but there is a certain limit in the range of image capture. Therefore, the present invention needs to determine the travel range of the inspection vehicle. The determination method of is as follows.

Suppose the size of the road sign is l*w, the field of view angle of the selected industrial camera is θ ₁ , and the optimal distance is L, as shown in Figure 5. Since the road sign should be installed on the right side of the road according to the national standard, l ₁ is the distance from the center line of the camera to the left line of the sign, then:

From this, it can be obtained that the inspection vehicle can deviate from the specific range of the traffic sign on the edge of the road.

After determining the optimal position range for image capture, the image can be actually collected. However, in the actual fixed-point collection, how to determine the fixed-point position also needs certain auxiliary equipment. The present invention utilizes an infrared sensor to measure the distance S between the actual camera and the road sign and the operating range _l of the vehicle. When the set value is reached , you can press the camera shutter through the control handle in the inspection vehicle to obtain the current image.

S1012. Collecting standard images of road signs.

The standard image of the road sign is collected in the form of fixed-point photography using the camera's photography mode. In the present invention, it only needs to be collected once, and the collected standard image of the road sign is used to establish a preprocessing standard model.

S1013. Collect standard GPS positioning information of road signs.

Carry out GPS positioning on road signs, and obtain standard GPS positioning information of road signs. The present invention only needs to be collected once, and the collected standard GPS positioning information of road signs is used to establish a preprocessing standard model.

S1014. According to the results collected in steps S1012 and S1013, a preprocessing standard model is established by labeling a single sequence. The single-sequence marking method is to mark the synchronously collected standard images and GPS information successively according to the collection order.

The standard GPS and standard images in the preprocessing standard model cannot be automatically collected, and manual operations are required by the staff. The present invention determines the best collection point of the image with the cooperation of relevant sensors. When the inspection vehicle runs to the best collection point When within the range, the standard GPS and images are manually collected after the staff confirms, and a single sequence is marked for the two respectively, and the images are sequentially marked through the fixed-point collection of the standard images, and the standard GPS positioning information is obtained synchronously and the sequence is marked sequentially , because the standard image and the standard GPS positioning information are synchronized during this process, the marking sequence is also consistent. Through the matching of the serial number, the standard image information and the standard GPS positioning information are integrated and fused to establish a preprocessing standard model.

S102. Acquiring real-time information of highway signs, and building a real-time model.

Obtain the real-time video information and real-time GPS information of the road sign during the inspection vehicle's travel, determine the GPS threshold through parameter calculation, obtain the road section that needs to be positioned, obtain the real-time image of the road sign for the positioning section, and ensure the extracted image Information includes road signs. The acquisition of real-time information specifically includes the following steps.

S1021. Acquire the real-time image information of the road sign by using the video mode of the camera.

The collection of real-time images of highway signs adopts the video mode of the camera, which can be carried out at any time according to the needs of the actual supervision plan. The video acquisition is connected with the PC through the industrial digital camera, and the information is directly transmitted to the PC through the USB interface for storage. Need to consider the installation position of camera during video collection, the calculation of each parameter of camera, the selection of camera lens and Sensor (sensor); Need to consider the impact of the change of speed on video image when vehicle inspection simultaneously, so inspection vehicle adopts in the present invention The most important thing is the constant speed inspection. When there is an accident such as overtaking, you can manually operate the brakes or shift gears, and then perform the constant speed inspection when the situation is resolved.

S1022. Perform real-time GPS positioning on the road sign, and obtain real-time GPS information of the road sign.

The collection of real-time GPS information on highway signs can be carried out at any time according to the needs of the actual supervision plan. In the process of real-time collection of real-time GPS information of highway signs, if there is a standard highway sign at the current GPS position, it is sufficient to perform sequence matching marking.

Due to the long-distance nature of highway sections, the GPS positioning of the present invention adopts a GPS positioning system with an error within a certain range, uses expert algorithms to ensure the integrity and reliability of GPS positioning information acquisition, and stores real-time GPS information to corresponding locations.

S1023. Match real-time GPS information (including time, longitude and latitude) with corresponding real-time images, and establish a real-time model by marking double sequences. The double-sequence marking method is to mark the real-time GPS information with the collection time of the real-time information as a sequence; firstly carry out the first sequence mark in real-time time for the real-time images collected according to a certain sampling frequency, and secondly mark the multi-frames with the same real-time time sequence number The real-time images are marked in sequence according to the sampling order, so as to establish a real-time model.

How to extract the image containing the road sign corresponding to the real-time GPS information from the image captured in real time will be described below with reference to FIG. 6 .

First of all, the real-time GPS information of road signs is collected one by one, and double-sequence marking is carried out. When the real-time GPS information is serially marked one by one, the marked sequence is the real-time acquisition time of GPS: year, month, day, hour, minute, second, that is, a sequence field with a length of 14 bits.

In this embodiment, the GPS frequency and the image frame rate are marked, and an expert algorithm is designed to achieve a one-to-one matching effect between the two. The GPS is connected to the PC through the USB serial port, and the implementation process of receiving and extracting the GPS latitude and longitude information through VB is as follows:

// Get latitude and longitude information

//Determine when to start receiving GPS information

If Strnum＝vbLf Then' when a carriage return is encountered

If(StrArray(0)="$")Then'Encounter the first character "$" at the beginning of the string

g=g+1//to count the number of GPS information collected

MyY1=TStrGps(5)' extract the longitude in the string

MyY=""&Mid(MyY1,1,2)&"'"&Mid(MyY1,3,3)&"""&Mid(MyY1,7,4)

Text4.Text＝g&MyY//mark the extracted latitude information and store it in Text4.text

Open "C:\GPS\Latitude"&d&".txt"For Append As#4

Print#4, Text3. Text

Close#4//Save the stored latitude information to the corresponding location on the computer

MyX1=TStrGps(3)' extract the latitude in the string

MyX=""&Mid(MyX1,1,2)&"'"&Mid(MyX1,3,2)&"""&Mid(MyX1,6,4)

Text3.Text＝g&MyX//Mark the extracted longitude information and store it in Text3.text

Open "C:\GPS\Longitude"&d&".txt"For Append As#5

Print#5, Text4. Text

Close#5//Save the file with longitude information to the corresponding location on the computer.

Secondly, calculate the sampling frequency of the real-time image information of the highway sign, and perform double-sequence marking on the sampled image; It is divided into the first sequence and the second sequence. The first sequence is the real-time time of image frame acquisition: year, month, day, hour, minute, second, and the length is a sequence field of 14 bits. The second sequence is the multi-frame image under the same first sequence number. Sampling order.

In the real-time model, the image acquisition is continuously collected in the form of video. Due to the high acquisition frequency of the camera, if each frame of image is processed, higher requirements are placed on the processing speed and disk storage. Based on this, the present invention samples the real-time image by calculating the sampling frequency, reduces the amount of image processing, and guarantees the processing result at the same time. Assuming that the camera sampling frequency is X frame/S, and the vehicle travel speed is Vm/s, then the sampling frequency F can be set as follows:

According to the shooting frequency of the Z30M camera of the present invention X=60 frames/s, V=14m/s, the calculation can be obtained: F=5 frames/m.

Then, from the real-time GPS information and image acquisition methods, it can be known that the acquisition time of the picture information and real-time GPS information is consistent, and the establishment of a real-time model is realized through time series matching.

S103. Obtain the best real-time image of the road sign in the real-time information collected in step S102 through an expert algorithm.

Preliminary processing is performed on the real-time images collected in step S102, and the best real-time image information containing road signs is acquired through an expert algorithm, marked and stored. The preliminary processing of the image includes processing steps such as preliminary denoising, filtering, restoration, tilt correction, and motion blur elimination on the image collected in S102, so as to obtain a road image that contains road signs and can be matched with the standard image in the pre-processing standard model. Best images in real time. Specific steps are as follows.

S1031. Preliminary image processing.

In practical applications, primary image problems caused by noise and motion blur can easily affect subsequent image processing, so preliminary image processing is required first. The flow chart of the present invention for realizing preliminary image processing is shown in FIG. 10 .

First, calculate according to parameters such as the speed of the inspection vehicle, the frame rate of the shooting camera, and the frequency of GPS positioning to obtain the key frame matching parameters; secondly, convert the collected video images into JPG format images; then, perform image denoising on the JPG format images , filtering, motion blur removal, tilt correction and other processing. Image denoising adopts improved median filter algorithm with good denoising effect to eliminate noise influence, motion blur elimination uses Wiener filter and improved algorithm for processing, and tilt correction uses expert algorithm to correct tilted pictures.

In this embodiment, the saved real-time GPS information corresponds to the acquired real-time image. According to the parameters of Z30M, the saved image corresponding to GPS is 5 frames, and in the subsequent specific identification, the present invention only needs one of the real-time most to achieve the best images.

S1032. Select the best real-time image containing road signs

The present invention selects the flow chart that contains the real-time best image of the road sign as shown in Figure 11, and the specific implementation process is:

First, detect the 5 frames of images obtained after preliminary processing of real-time images.

Second, convert RGB to HSV color space according to the particularity of road sign colors.

Then, use the HSV color space to determine whether the current image contains a special color. The special colors refer to the colors of red, blue, and yellow corresponding to the HSV space; if one of them is included, it is considered that the current image contains the ingestion of road signs, and the image is marked and stored.

Again, the image is further segmented by color regions.

Finally, compare the size of the segmented regions between different images, and select the one with the largest segmented region as the real-time best image.

In practical applications, due to the influence of shooting distance and other factors, the effects of road sign shooting are not the same. The present invention determines that the best real-time image is the one with the largest segmented area, because the larger the segmented area, the closer the camera is to the actual road sign, and the larger and clearer the area occupied by the road sign in the captured image. That is, the closer the image effect is to the ideal real-time optimal image of the present invention.

S104. Match the real-time optimal image obtained in step S103 with the standard image of the preprocessed standard model in step S101, and obtain a specific classification of road sign damage through an expert algorithm, and store it.

S1041. Calculate a threshold, so that real-time GPS information is matched with standard standard GPS latitude and longitude information.

As can be seen from the foregoing, when the present invention extracts GPS information from the preprocessing standard model and the real-time model, both of them are sequentially marked, and the threshold range within which the two can achieve matching can be obtained through parameter calculation. The specific calculation process is as follows.

Knowing the longitude and latitude information of point A and point B, based on the knowledge of circle and triangle, the formula for the distance on the surface of the earth is as follows:

C=sin(MLatA)*sin(MLatB)*cos(MLonA-MLonB)+cos(MLatA)*cos(MLatB)

Distance＝R*Arccos(C)*π/180

It can be obtained from the above formula: Arccos(C)=Distance*180/(R*π)

Among them, the units of R and Distance are the same.

In the present embodiment, the inspection speed of the highway sign is 14m/s, and the GPS positioning frequency is 1S/time, so the present invention selects the threshold value earth surface distance as 15m, that is, Dis tan ce=15(m) can be calculated to obtain Arccos(C) =1.34×10 ^-4 , and it can be further obtained that C=cos(1.34×10 ^-4 )≈1.

And it can be seen from the above formula that when point A and point B are at the same longitude:

C=sin(MLatA)*sin(MLatB)*cos(0°)+cos(MLatA)*cos(MLatB)

C=sin(MLatA)*sin(MLatB)*1+cos(MLatA)*cos(MLatB)

According to C≈1 and the theorem of mathematical trigonometric functions, when MLatA≈MLatB,

C≈sin ² (MLatA)+cos ² (MLatA)=1,

Therefore, when the latitude of point A and point B is also at the same position, that is, when the latitude threshold reaches 0, the ground distance threshold of 15m can be met, but such an accurate threshold range cannot be found in the actual application process. Therefore, to make

C＝sin(MLatA)*sin(MLatB)*1+cos(MLatA)*cos(MLatB)≈

sin ² (MLatA)+cos ² (MLatA)＝1

That is, sin(MLatA)≈sin(MLatB), cos(MLatA)≈cos(MLatB)

According to the detection of sine and cosine and experimental data, when the longitude is consistent, it is assumed that taking two decimal places is an effective calculation. After consulting the data sheet, sin(θ)=0.0044, taking two decimal places is valid, sin(θ)≈0, and At this time θ≈0.25°=15", therefore, setting the latitude threshold range at 0.25° can obtain effective positioning. Similarly, when two points are at the same latitude, the longitude threshold can also be determined by the above method. After calculation The longitude threshold was also determined to be 0.25°.

S1042. Save the real-time GPS information within the threshold range, and save the matched GPS single sequence of the preprocessed standard model and the GPS double sequence in the real-time model.

When the matching between the two is within the threshold range, the currently extracted real-time GPS information is saved, and at the same time, the GPS sequence in the pre-processed standard model matched with it is stored correspondingly; and the saved real-time GPS information corresponds to Road signs must exist within the ground extent of . Therefore, according to the initial marking sequence interface, it is easy to find the real-time image containing the road sign corresponding to the retained real-time GPS information, and complete the acquisition of real-time information.

S1043. Match the matched real-time GPS information with the standard image corresponding to the pre-processed standard model and the corresponding real-time best image in the real-time model according to the threshold requirement.

When the first sequence of the real-time image information of the highway sign matches the first sequence of the real-time GPS information, the real-time image information matching the first sequence of the real-time GPS positioning information is defined as a key frame image and saved. These images must contain roads At the same time, the sequence of standard GPS positioning information in the pre-processing standard model is correspondingly stored again, so that the image acquisition containing road signs at fixed-point GPS locations in real-time operation is realized.

In this embodiment, a flow chart for realizing one-to-one matching of real-time GPS information and real-time images is shown in FIG. 8 . Firstly, the extracted real-time image of the road sign is marked frame by frame in the form of a double sequence, that is, the shape is as follows: ij.jpg (i is a 14-bit real-time time sequence, j=1...5). The corresponding real-time GPS information is marked as k.GSA.GSV (k is a 14-bit real-time time sequence) in sequence, where GSA represents longitude and GSV represents latitude. Extract the real-time image and real-time GPS tag information, and match the first character. If it matches, the current picture corresponds to the matched GPS information, thus realizing the pairing of real-time image and real-time GPS information.

In this embodiment, the process of sorting images in two sequences using C language is as follows:

When real-time GPS information and real-time image realize matching among the present invention: promptly i=g, then carry out matching between the two, and realization process is:

ifstream in("5.txt");//Read the txt file containing longitude information

string string;

while(getline(in,string))//Read data in line and store in string until all data is read

char c1=string[0];//read the first character of the string string

int g＝atoi(c1);//Convert the first character c1 into a digital value

If(i==g)//Judge whether the first character of GPS matches the first character of the picture name.

The flow chart of pattern matching between the real-time optimal image extracted after the above-mentioned real-time acquisition and the corresponding image in the preprocessing standard model is shown in FIG. 9 . The present invention performs pattern matching on the extracted real-time optimal image after real-time acquisition and the corresponding image in the pre-processing standard model to identify the damage type of the road sign, so how to ensure the matching between the two is particularly important.

When the present invention establishes a preprocessing standard model and processes real-time information, it marks the standard GPS positioning information, standard image, real-time GPS information and real-time image respectively, wherein the standard image is consistent with the real-time GPS information, and the real-time image is consistent with the real-time The GPS information corresponds to the same. Therefore, when the real-time GPS information is matched with the standard GPS positioning information, the correspondence between the real-time GPS information and the standard image can be realized, and the real-time GPS information is consistent with the real-time image sequence, so that the real-time optimal image and the standard image can be extracted in real time Realize the correspondence and provide the basis for subsequent pattern recognition.

S1044. Obtain the specific classification of the real-time best road sign damage through expert algorithm, and store it.

The real-time optimal image obtained in step S103 is consistent with the sequence label of the standard image of the preprocessed standard model in step S101, so the two images with the same sequence label are subjected to pattern matching under the control of the expert algorithm to obtain the actual recognition result, and Classified storage.

See Figure 12 for the flow chart of image recognition under real-time image pattern matching. The actual road signs may be damaged in various forms. In this embodiment, point pattern matching is used as an example for illustration.

The standard corner points of the rectangular road sign are four corner points. First, the corner points of the standard road sign are extracted by SUSAN algorithm to establish a standard model; secondly, the corner points of the real-time image are extracted from the processed real-time best image; and then , the two are trained through point pattern matching to get the final matching result; if the number and position of the corner points of the two are consistent, it is determined that the current real-time road sign is normal; , combined with GPS location, classified as abnormal. In this embodiment, the specific implementation method is as follows.

srcImage＝cvLoadImage("image storage location", 1);//load image

grayImage = cvCreateImage(cvGetSize(srcImage), IPL_DEPTH_8U, 1);

// grayscale the original image

cvCvtColor(srcImage, grayImage, CV_BGR2GRAY);

//Create a temporary image with the same size as the original image

corners1 = cvCreateImage(cvGetSize(srcImage), IPL_DEPTH_32F, 1);

corners2 = cvCreateImage(cvGetSize(srcImage), IPL_DEPTH_32F, 1);

// corner detection

cvGoodFeaturesToTrack(grayImage, corners1, corners2, corners,

&cornerCount,qualityLevel,minDistance,0);

printf("num corners found:%d\n", cornerCount);

//Mark the corner points in the original image

The actual road sign edge corners can be obtained after the calculation of the standard image by the above algorithm, and the real-time edge corner points can also be obtained for the real-time image, and then the number of corner points of the two is counted. If they are consistent, the standard image is retained and the road is judged. The identification is a normal identification, if the number of corner points is inconsistent, the road identification is abnormal.

S105. Determine whether maintenance is required according to the recognition result obtained in step S104, and determine whether it is necessary to update the standard image in the pre-processing standard model according to the feedback maintenance result;

Whether the original image is updated or not is judged by whether the maintenance is completed by the background staff. If the maintenance is not completed, no processing is performed. If the maintenance is completed, the current complete image after maintenance is used to update the standard image in the pre-processing standard model.

S106. Establish a database to manage the data in steps S101-S104.

Due to the huge amount of data in image processing, simple storage cannot meet people's needs for data. The construction of the database provides guarantees for relevant personnel to make correct decisions and efficiently process data.

The present invention integrates the collected information and processed data into the database for management. The information imported into the database includes: standard images of road signs, standard GPS positioning information of road signs, real-time image and video information of road signs, real-time GPS information of road signs, GPS and Image matching integration information, the best real-time image frame containing road sign images calculated by expert algorithms, metadata database (that is, the standard image library used to train and generate classifiers), and road asset image integration information with obstacles after classification and recognition Wait.

The structural block diagram of the database in the present invention is shown in Figure 13. The database includes five sub-databases, namely meta database, real-time database, image management database, background personnel management database and time management database. The functions of each sub-database are as follows.

Metadata base: store the tag sequence, storage location, and specific image stored in the database in binary format of the standard image of the fixed-point road sign in the preprocessing standard model; the tag sequence of the standard GPS positioning information and the corresponding longitude and latitude; One corresponding to the generation time of the image and GPS integration information, etc.

Real-time database: manage the video segments and periodic GPS information collected in the real-time model, processed key video frames and marker sequences, sampled positioning information (longitude, latitude) and marker sequences, and key frames and sampled positioning information according to the previous article It can be seen from the extraction that a good match between the two is directly related to the final processing result, so the management of the real-time model cannot be wrong.

Image management library: There are images in both the pre-processing standard model and the real-time model. After the images in the real-time model find the corresponding images in the pre-processing standard model according to the sequence, under the control of the expert algorithm, it will be get a final result. The image management function of the present invention is to further manage the image library according to the recognition result. If the recognition result is normal, the image in the preprocessing standard model will not be processed, and the standard image will be reserved; The images in the model are updated. And this update is linked to whether it is repaired or not. After the repair is completed, the image will be extracted from the repaired sign, and the pre-processed image will be updated. If there is no repair, it will remain pending.

A background worker manages the library. The background staff includes: pre-processing standard model collection personnel, real-time inspection personnel, statistics of fault points and fault types and intermediate transition personnel for contact maintenance, maintenance personnel, and check personnel for confirming maintenance results based on the feedback results of maintenance personnel. After the abnormal signs are repaired by the maintenance personnel, the maintenance results are determined, and the abnormal signs that have been repaired are marked, so as to determine the update of the standard image in the pre-processing standard model to ensure the real-time follow-up of the metadata database.

Time management library: Since the normality of road signs is seriously related to people's personal and property safety, time is particularly important for work efficiency. The time includes: preprocessing standard model establishment time, real-time inspection time, result identification time, maintenance personnel contact time and maintenance completion time, etc. In the present invention, the real-time inspection time can be set according to the inspection cycle based on experience, thereby setting the inspection date, and a warning can be given if the inspection is not timely when it expires; the time limit for maintenance completion can be limited, and a certain penalty will be imposed if it is overdue.

S107. Design the interface of the upper computer control module, display the data in an intuitive manner, realize people's actual needs for highway sign supervision, and embody the humanization and intelligence of the present invention.

In the present invention, the hardware connection between the upper computer control module and the database and the formulation of the communication protocol through the serial port ensure that the stored data can be accurately uploaded to the upper computer control module interaction interface, which provides convenience for people's query and report printing.

The interface reference diagram of the host computer control module in this embodiment is shown in Figure 14, which can display real-time images of road signs, real-time GPS information, and actual image classification results, so that staff can compare information to maintain road assets.

Claims (10)

1. A method for supervising highway signs, characterized in that it specifically comprises the following steps: S101. Establishing a preprocessing standard model of road signs; S102. Acquiring real-time information of highway signs, and building a real-time model; S103. Obtain the best real-time image of the road sign in the real-time information collected in step S102 through an expert algorithm; S104. The real-time optimal image obtained in step S103 is matched with the standard image of the preprocessed standard model in step S101 through sequence search, and the specific classification of actual road sign damage is obtained by expert algorithm, and stored; S105. Determine whether maintenance is required according to the recognition result, and determine whether the standard image in the pre-processing standard model needs to be updated according to the feedback maintenance result; S106. Establish a database to manage the data in steps S101 to S104; S107. Design the host computer interface, display the data in an intuitive way, and realize people's actual needs for highway sign supervision. 2. A method for supervising highway signs according to claim 1, characterized in that step S101 specifically includes the following: S1011. Determine the camera model, and calculate according to the height position of the road sign, and install the camera at a suitable position on the inspection vehicle; S1012. Collecting standard images of road signs; S1013. Collect standard GPS positioning information of road signs; S1014. According to the results collected in steps S1012 and S1013, establish a preprocessing standard model by marking a single sequence; the single sequence marking method is to mark the synchronously collected standard images and GPS information sequentially according to the collection order. 3. A method for supervising highway signs according to claim 1, characterized in that step S102 specifically includes the following: S1021. Obtain real-time image information of road signs by using the video mode of the camera; S1022. Carry out real-time GPS positioning on the road signs, and obtain real-time GPS information of the road signs; S1023. Match the real-time GPS information with the corresponding real-time image, and establish a real-time model by marking the double-sequence method; the double-sequence marking method is to mark the real-time GPS information with the collection time of the real-time information as a sequence; collect according to a certain sampling frequency The real-time images of the real-time images are firstly marked in the order of real-time time, and then the multi-frame real-time images with the same real-time time sequence number are marked in the second sequence according to the sampling order. 4. A method for supervising highway signs according to claim 3, characterized in that step S1023 specifically includes the following: Firstly, collect the real-time GPS information of road signs one by one and mark them sequentially; Secondly, calculate the sampling frequency of the real-time image information of the road sign, and perform double-sequence marking on the sampled image; Then, it is judged whether the real-time image information of the highway sign matches the first sequence of the real-time GPS information, and if they match, a real-time model is established. 5. A method for supervising highway signs according to claim 1, characterized in that step S103 specifically includes the following: S1031. Preliminary image processing; S1032. Select the best real-time image containing road signs. 6. A method for supervising highway signs according to claim 5, characterized in that: in step S1031, an expert algorithm is used to perform preliminary processing on the image, and the processing content includes preliminary denoising and filtering of the image collected in S102 , restoration, tilt correction, motion blur removal processing steps. 7. A method for supervising highway signs according to claim 5, characterized in that step S1032 specifically includes the following: First, detect n frames of images obtained after preliminary processing of real-time images; Secondly, convert RGB to HSV color space according to the particularity of road sign color; Then, use the HSV color space to determine whether the current image contains a special color; Again, the image is further segmented by the color region; Finally, compare the size of the segmented regions between different images, and select the one with the largest segmented region as the real-time best image. 8. A method for supervising highway signs according to claim 1, characterized in that step S104 specifically includes the following: S1041, calculating a threshold to match the real-time GPS information with the standard GPS latitude and longitude information; S1042, saving the real-time GPS information within the threshold range, and saving the matched GPS single sequence of the preprocessing standard model and the GPS double sequence in the real-time model; S1043, according to the threshold requirement, match the matched real-time GPS information with the standard image corresponding to the pre-processing standard model and the corresponding real-time best image in the real-time model; S1044. Obtain the specific classification of the real-time best road sign damage through expert algorithm, and store it. 9. A supervision system for highway signs, characterized in that: the system includes An image collection module 101, configured to collect images of road signs; GPS information collection module 102, used for collecting the GPS positioning information of road signs; The preprocessing standard model and real-time model building module 103 are used to carry out synchronous single-sequence marking on the standard image information of fixed-point road signs and the standard GPS longitude and latitude information through the image collection module 101 and the GPS information collection module 102, and establish a preprocessing standard model ; The single-sequence marking method is to mark the synchronously collected standard images and GPS information successively according to the collection order; Collect real-time GPS information and real-time images of highway signs through image acquisition module 101 and GPS information acquisition module 102, extract real-time GPS information of highway signs and image information corresponding to real-time GPS information, and establish real-time model; the double-sequence marking method is to mark the real-time GPS information with the collection time of real-time information as a sequence; to the real-time images collected according to a certain sampling frequency, the first sequence is marked in order of real-time time; Multi-frame real-time images are marked sequentially according to the sampling order; The two-model matching module 104 extracts key images through sequence matching and threshold value deletion for the real-time information in the preprocessing standard model and real-time model building module 103, and controls the selected key images through parameter calculation and expert algorithm. Process the key images containing road signs, and obtain the best real-time images through processing; The image processing and recognition module 105 performs pattern matching on the real-time optimal image containing road signs obtained in the two model matching modules 104 and the preprocessing standard model and the statistical standard image information in the real-time model building module 103, and is matched by an expert algorithm. Training to obtain the damage type of the road sign after the actual matching, and classify and store it; The database module 106 is used to carry out orderly and effective supervision on the data information in the system, and establish a database mechanism; The upper computer control module 107 is linked with the database in the database module to visually display the processed data. 10. The supervision system of a road sign according to claim 9, characterized in that: the information displayed by the host computer control module mainly includes the GPS latitude and longitude information of the fixed-point road sign, the image of the fixed-point road sign, and the image after image recognition. The specific category, information on the damage status of road signs, and information on whether repairs are complete.