CN113625264B - Method and system for parallel processing of railway detection big data - Google Patents

Abstract

The invention provides a method and a system for parallel processing of railway detection big data, wherein the method divides collected detection data and stores the divided detection data according to a set storage strategy; the method comprises the steps of creating a combination of a plurality of signal processing algorithms needed to be performed on detection data as a conventional class workflow and an iterative class workflow according to engineering property requirements and interpretation target requirements of the detection data, further utilizing a designed parallel node architecture to access the stored detection data in parallel, and respectively realizing parallel processing based on conventional class operation and iterative class operation according to a scheduling strategy of a matched workflow. By adopting the scheme, the problem of unbalanced operation load and memory of an exclusive operation module in the existing data processing technology is solved, the load balancing performance and cluster resource utilization rate in the processing process are improved by combining different formulated scheduling strategies based on a special parallel node architecture, and the processing requirements of current and future large-scale detection data can be met.

Description

A method and system for parallel processing of railway inspection big data

Technical Field

The present invention relates to the technical field of cluster resource processing and optimization, and in particular to a method and system for parallel processing of railway inspection big data.

Background Art

Railway transportation brings many conveniences and support to people's lives and work, but ensuring the safe operation of railways is also a core direction that must always be paid attention to now and in the future. With the rapid growth of my country's railway operating mileage, how to conduct fast, accurate and non-destructive detection of the infrastructure of the increasingly large railway network and timely grasp the health status of the infrastructure has become a major issue that needs to be solved urgently.

Ground Penetrating Radar (GPR) is a geophysical method that uses antennas to transmit and receive high-frequency electromagnetic waves to detect the material properties and distribution patterns inside a medium. By studying the changes in the polarization of radar waves, information related to the physical properties of underground media can be obtained. When applied, ground penetrating radar can be used for detection in a variety of ways. The types, complexity and scale of the corresponding detection data obtained are large, and set strategies need to be used for processing and analysis.

In the prior art, research has been conducted on the processing methods of ground penetrating radar detection data, but most of these technologies are based on a single computing node, and the algorithms involved are often highly serialized and only applicable to the processing of small-scale GPR data sets. With the rapid expansion of the current ground penetrating radar data scale, some technicians have also conducted optimization research on existing data processing technologies. Based on the above technology, they use the massive high-speed parallel processing capabilities of big data cloud computing technology to improve the ability to quickly process large-scale data. They manage data based on Hadoop's distributed clusters, specifically using distributed file systems HDFS and MySQL's relational database clusters combined with Hbase to solve the massive storage and efficient access of structured data. Although this ensures the scalability and portability of cluster resources and can store data in multiple formats, there are problems of repeated reading and writing in the process of preprocessing or subsequent processing of radar data, and it is easy to cause load balancing problems due to the large difference in the amount of calculation of different parallel tasks. When the algorithm steps in the workflow are complex, it may cause each parallel computing node to fail to operate normally and fail to meet the processing requirements of large-scale ground penetrating radar detection data. Therefore, it is urgent to provide an efficient and reasonable data parallel processing method that can meet application requirements.

The information disclosed in the background technology section of the present invention is only intended to deepen the understanding of the general background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art known to those skilled in the art.

The rapid development of a new generation of high-performance hardware architecture systems has created new opportunities for the rapid processing of massive GPR data. Large amounts of high-precision, large-area GPR detection data can be processed using parallel technology to greatly improve processing efficiency.

Summary of the invention

To solve the above problems, the present invention provides a method for parallel processing of railway detection big data. In one embodiment, the method includes:

Data storage step, dividing the collected detection data, and storing the divided detection data according to the set storage strategy;

The algorithm combination formulation step creates a corresponding operation workflow according to the engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of multiple signal processing algorithms required for the detection data, including conventional workflows and iterative workflows;

The parallel processing step is to retrieve the stored detection data in parallel based on the pre-created parallel node architecture, and implement parallel processing based on conventional operations and iterative operations respectively according to the scheduling strategy matching the operation workflow.

Preferably, in the data storage step, the railway detection data is divided according to the collection area and the type of collection equipment to obtain detection data of different levels with different processing priorities.

Furthermore, in one embodiment, when storing the divided detection data, multiple storage methods such as relational database and distributed database are used, and priority information is combined to realize parallel storage of detection data in multiple formats;

The data body of the detection data file includes header data and data content. The header data of each detection data is associated with the file header and stored in a relational database, and the data content of each detection data is stored in a distributed database.

Furthermore, in a preferred embodiment, the method further includes: a node architecture creation step, setting a master node as a task scheduling node, which together with a read-write node and N computing nodes constitutes a parallel node architecture.

Furthermore, in one embodiment, in the parallel processing step, the parallel processing based on conventional class operations is implemented according to the following steps:

The task scheduling node determines whether there is any test data that needs to be processed. If so, it counts the total number of test data x that needs to be processed and starts the read-write node and the computing node.

The following steps are executed in a loop until all pending detection data operations are completed and the read/write nodes and computing nodes are released:

Based on the preset scheduling base K, the read/write node reads x/K channels as the data for this round of processing; the task scheduling node divides the data for this round of processing into N parts evenly and distributes them to each computing node;

Each computing node performs operations in parallel according to the conventional class workflow;

The parallel scheduling node confirms whether the processed detection data has been received. If received, it transmits it to the read-write node, which integrates it according to the corresponding spatial structure and writes it into the storage area;

Wherein, K is a positive integer, 1≤K≤N, or further optimized in combination with the number of channels S corresponding to the unit mark data to determine the scheduling base K.

Furthermore, in one embodiment, parallel processing based on iterative operations is implemented according to the following steps:

The task scheduling node determines whether there is any test data that needs to be processed. If so, it counts the total number of test data x that needs to be processed and starts the read-write node and the computing node.

The scheduling base K is set according to the hardware resources, x, and the computing unit size of the iterative algorithm. The read/write node reads x/K channels as the first round of processing data. The task scheduling node divides the processing data of this round into N parts and distributes them to each computing node.

Then, the following steps are executed in a loop until all pending detection data operations are completed and the read/write nodes and computing nodes are released:

The task scheduling node counts the number of idle computing nodes n in real time.

Read-write node reads The data is divided into n parts evenly and distributed to each idle computing node;

Each computing node executes operations in parallel according to the iterative workflow and feeds back the operation status information to the task scheduling node in real time;

The parallel scheduling node confirms whether the processed detection data has been received. If received, it will be transmitted to the read-write node, which will integrate it according to the corresponding spatial structure and write it into the storage area.

Based on the method for parallel processing of railway detection big data described in any one or more of the above embodiments, the present invention also provides a storage medium storing program codes that can implement the method described in any one or more of the above embodiments.

Based on other aspects of the method described in any one or more of the above embodiments, the present invention also provides a system for parallel processing of railway detection big data, the system comprising:

A data storage module, configured to divide the collected detection data and store the divided detection data according to a set storage strategy;

An algorithm combination formulation module is configured to create a corresponding operation workflow according to the engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of multiple signal processing algorithms required for the detection data, including conventional workflows and iterative workflows;

The parallel processing module is configured to retrieve the stored detection data in parallel based on a pre-created parallel node architecture, and implement parallel processing based on conventional operations and iterative operations according to a scheduling strategy that matches the operation workflow.

Furthermore, in one embodiment, the system also includes: a node architecture creation module, which is configured to set a master node as a task scheduling node, which together with a read-write node and N computing nodes constitutes a parallel node architecture.

Compared with the closest prior art, the present invention also has the following beneficial effects:

The present invention provides a method and system for parallel processing of railway detection big data, which divides the collected detection data to ensure that data with urgent processing needs can obtain processing results in the first time, which is conducive to avoiding accidents or impacts caused by untimely data processing; further, the data division and storage method proposed in the present invention performs multi-granularity segmentation operations based on the principle of data processing algorithm, and realizes parallel processing based on the parallel node architecture according to different scheduling strategies for conventional algorithms and iterative algorithms, which improves the data processing rate from two levels of load balancing performance and cluster resource utilization, can effectively meet the current and future large-scale data processing needs, and provide reliable assistance for the subsequent application of railway detection data.

Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

FIG1 is a schematic flow chart of a method for parallel processing of railway inspection big data provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the structure of a system for parallel processing of railway inspection big data provided by another embodiment of the present invention.

DETAILED DESCRIPTION

The following will describe the implementation methods of the present invention in detail in conjunction with the accompanying drawings and embodiments, so that the implementers of the present invention can fully understand how the present invention applies technical means to solve technical problems and achieve the implementation process of technical effects and implement the present invention specifically according to the above implementation process. It should be noted that as long as there is no conflict, the various embodiments and various features of the embodiments in the present invention can be combined with each other, and the formed technical solutions are all within the protection scope of the present invention.

Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently, or simultaneously. The order of the operations may be rearranged. A process may be terminated when its operations are complete, but may also have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc.

Computer devices include user devices and network devices. Among them, user devices or clients include but are not limited to computers, smart phones, PDAs, etc.; network devices include but are not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Computer devices can be run alone to implement the present invention, or they can be connected to the network and implement the present invention through interactive operations with other computer devices in the network. The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, VPN network, etc.

At the end of 2019, the operating mileage of railways in China reached more than 139,000 kilometers; among them, the operating mileage of high-speed railways exceeded 35,000 kilometers, ranking first in the world. According to the Medium- and Long-Term Railway Network Plan (No. 1536 of the Development and Reform Foundation [2016]), by 2025, the scale of the railway network will reach about 175,000 kilometers, of which about 38,000 kilometers will be high-speed railways. The network coverage will be further expanded, the network structure will be more optimized, the backbone role will be more significant, and the railway will better play its role in safeguarding economic and social development. The scale of the conventional railway network will reach about 131,000 kilometers, and it is planned to implement the capacity expansion and reconstruction of existing lines of about 20,000 kilometers. With the rapid growth of my country's railway operating mileage, how to quickly, accurately and non-destructively detect the infrastructure of the future huge railway network and timely grasp the health status of the infrastructure has become a major issue that needs to be solved urgently.

In the prior art, GPR (ground penetrating radar) data processing technology is relatively mature in processing small and medium-scale data sets. However, most of these technologies are based on single computing nodes, and the algorithms involved are often highly serialized, which cannot be effectively applied to the processing of large-scale railway detection data. With the rapid expansion of the current ground penetrating radar data scale, some technicians have also conducted optimization research on existing data processing technologies. Based on the above technology, it uses the massive high-speed parallel processing capability of big data cloud computing technology to improve the ability to quickly process large-scale data. It manages data based on Hadoop's distributed cluster, and specifically uses the distributed file system HDFS and MySQL's relational database cluster combined with Hbase to solve the massive storage and efficient access of structured data. Although this ensures the scalability and portability of cluster resources and can store data in multiple formats, it does not provide a clear data storage strategy, and there is a problem of repeated reading and writing in the process of pre-processing or subsequent processing of radar data. It is easy to cause load balancing problems due to the large difference in the amount of calculation of different parallel tasks. When the algorithm steps in the workflow are complicated, it may cause each parallel computing node to fail to operate normally, and it cannot meet the processing requirements of large-scale ground penetrating radar detection data. Therefore, it is urgent to provide an efficient and reasonable data parallel processing method that can meet application requirements.

The rapid development of a new generation of high-performance hardware architecture systems has created new opportunities for the rapid processing of massive GPR data. High-precision, large-area GPR detection data can be processed using parallel technology to greatly improve processing efficiency. However, in the process of parallel processing, it is difficult to balance the amount of computation of each parallel operation and overcome the impact of excessive load differences, which can easily lead to uneven computing resources and insufficient consumption of intermediate memory resources, making it difficult to guarantee processing efficiency.

To solve the above problems, the present invention provides a method and system for parallel processing of railway inspection big data. The method establishes a hybrid parallel computing architecture of "data parallelism + algorithm parallelism", divides radar signal data processing algorithms into two categories, and formulates applicable scheduling and computing strategies respectively, so that the entire workflow processing mode can achieve load balancing effect and maximize the utilization efficiency of cluster resources.

Next, the detailed process of the method of the embodiment of the present invention is described in detail based on the accompanying drawings. The steps shown in the flowchart of the accompanying drawings can be executed in a computer system including a set of computer executable instructions. Although the logical order of each step is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

Embodiment 1

FIG1 shows a flow chart of a method for parallel processing of railway detection big data provided by Embodiment 1 of the present invention. Referring to FIG1 , it can be seen that the method includes the following steps.

Data storage step, dividing the collected detection data, and storing the divided detection data according to the set storage strategy;

The algorithm combination formulation step creates a corresponding operation workflow according to the engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of multiple signal processing algorithms required for the detection data, including conventional workflows and iterative workflows;

The parallel processing step is to retrieve the stored detection data in parallel based on the pre-created parallel node architecture, and implement parallel processing based on conventional operations and iterative operations respectively according to the scheduling strategy matching the operation workflow.

In the actual application process in the field of railway data detection, the ground penetrating radar data acquisition device used includes a ground penetrating radar host, transmitting and receiving antennas, and a measuring point and line spatial coordinate determination and recording system; the ground penetrating radar host is respectively connected to and controls the transmitting and receiving antennas, the measuring point and line spatial coordinate determination and recording system; the measuring point and line spatial coordinate determination and recording system is used to determine and record the specific position of each ground penetrating radar data acquisition point.

The collected original detection data is also read by a ground penetrating radar data reader and a data converter of a corresponding model and converted into a standard format that satisfies the parallel processing of the present invention.

The researchers of the present invention have considered that the detection data obtained by different types of ground penetrating radar equipment and ground penetrating radar equipment set in different areas have different processing requirements. For example, if an earthquake occurs in a certain area, the real-time detection data of the ground penetrating radar related to the railway in the area has urgent processing requirements. Therefore, in one embodiment, in the data storage step, the railway detection data is divided according to the acquisition area and the type of acquisition equipment to obtain detection data of different levels with different processing priorities.

Specifically, priority tags can be added to each batch of detection data packets, and based on the priority tags, they can be directly stored in batches in sub-storage areas corresponding to different priority levels. In this way, the priority tags are avoided from being located in the detection data file header, affecting the reading rate and scheduling operation rate in the subsequent parallel processing process. Accordingly, the storage area with the highest processing priority has the highest scheduling priority and reading priority for the stored data.

In actual application scenarios, the data sets obtained from each detection are often multiple files, and the data sets need to be distributed to multiple parallel computing nodes for simultaneous processing. The granularity of the split scheduling operation directly affects the data processing volume and processing efficiency of each parallel computing node, and has an important impact on load balancing.

The existing technology uses file-based segmentation, for example, 10 files and 5 nodes in total, with 2 files assigned to each node. However, sometimes a single file is very large, resulting in a single node's computational workload being too large to occupy all memory, and other nodes cannot run, completely failing to take advantage of parallel processing. In layman's terms, there are multiple production lines, but they cannot produce at the same time.

Therefore, in order to perform fine-grained splitting, it is necessary to decompose the radar file. The researchers of the present invention took into account the particularity of the ground penetrating radar detection data file format and failed to decompose it directly. They combined the specific content of the file format to realize parallel partition storage, so as to realize flexible segmentation of the detection data.

Specifically, the file format of the detection data is: file header + signal data (such as the first channel data + the second channel data...); wherein the file header is a plurality of acquisition parameters, and the format is different for different radar devices.

Furthermore, each data in the signal data contains two parts: header + radar signal (data content, such as 4 numbers + 512 numbers for a certain brand);

Since the subsequent signal processing algorithm needs to use the parameters in the file header, the researchers of this patent store the file header in a relational database, such as a MySQL database. Technical personnel in the field can choose any reasonable relational database according to their needs; the radar signal data content is stored in a distributed database, such as an HDFS database. When applied, technical personnel in the field can choose any reasonable distributed database according to their needs; based on this, the signal data can be flexibly and evenly divided, and the smallest granularity is a single-channel signal. It can be seen that the technical concept of the present invention can make the data processing granularity of each parallel processing node smaller and more balanced, from 1 file (usually a single file for roadbed detection has hundreds of thousands of data) to a smaller set of data (such as dozens or hundreds of data) or even 1 data.

In addition, the trace header will indicate whether this trace is marked data, which is also useful for subsequent processing and is also stored in the relational database. The marked data can be used to match the real spatial position of the radar data. It is inserted by the acquisition software during acquisition, generally at equal distances, or at locations where the geological characteristics of the detection section have changed.

Specifically, considering that the ground penetrating radar data and its attribute characteristic values involve multiple different data formats, including time domain root mean square amplitude, time domain coherence, frequency domain -3dB bandwidth average frequency, frequency domain -3dB bandwidth average phase, time-frequency domain low frequency increase area, time-frequency domain high frequency attenuation area and other multi-dimensional attribute data. Therefore, in one embodiment, for each priority storage area, when storing the divided detection data, multiple storage methods such as MySQL relational database and distributed database are used, and priority information is combined to realize parallel partition storage of detection data in multiple formats;

The detection data includes a file header and a data body. The file header includes the total number of channels, the number of sampling points, the antenna frequency, the channel spacing and the time window parameters corresponding to each file during data collection. The file header of each detection data is stored in a relational database.

The data body of the detection data file includes header data and data content. The header data of each detection data is associated with the file header and stored in a relational database, and the data content of each detection data is stored in a distributed database.

In specific implementation, the attribute parameters corresponding to the data collection in the file header can be stored in the MySQL relational database, such as the total number of channels, the number of sampling points, the antenna frequency, the channel spacing, the time window and other parameters.

The data body is stored in blocks in HDFS. If the header prompts a channel number mark, the channel number mark is also associated and stored in the database. With this storage method, radar data can be divided into multiples of any single channel data size.

In actual application, the database storage format can be as follows:

Example of file attribute parameter table:

file name Property 1 Property 2 Property 3 … File 1 1 2 3 File 2 1 2 3 File 3 1 2 3

Example of file tag table:

file name Mark road number File 1 Mark 1 File 1 Mark 2 File 1 Mark 3 File 2 Mark 1 File 2 Mark 2 … …

The input and output of traditional algorithms are one or more radar files. The minimum data size of a single task during parallel computing is generally large, which can easily cause unbalanced parallel computing load and insufficient memory in the computing module. In severe cases, it can even affect the normal operation of all parallel computing.

Furthermore, even if the minimum computing unit of parallel computing is set to a radar data file in the prior art, since it may need to perform iterative processing and the number of iterative operations is uneven, it may still cause the problem of uneven computing load. In the prior art, if there are unfinished calculations in the batch of calculations, other parallel computing channels are in a worthless waiting state, which greatly affects the timeliness of the computing processing.

Therefore, in one embodiment, in the algorithm combination formulation step, a corresponding operation workflow is created according to the engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of multiple signal processing algorithms required for the detection data, including conventional workflows and iterative workflows. In the present invention, the operation workflow is divided into conventional workflows and iterative workflows according to different operation principles and whether it is an iterative algorithm.

It should be noted that for a certain radar file, it may involve computational processing with an execution order. Based on this, the computational processing workflow of each radar file is not limited to one conventional workflow and one iterative workflow. Researchers can reasonably plan the computational workflows of different stages of each radar data file according to their computational requirements. Each stage of the computational workflow includes at least one algorithm and is uniformly planned and stored. In practical applications, the computational workflows corresponding to the same type of batch radar data are consistent.

Furthermore, in order to ensure the load balance and utilization rate of each parallel computing channel, the researchers of the present invention set the input and output units and the minimum computing unit of the radar signal processing algorithm to a single channel or a channel set.

By determining the minimum computational unit e of the algorithm, independent computations that are not related to each other can be carried out, with the minimum computational unit as input and output, such as a single channel or a channel set, and the channel set size can be specified.

Taking the linear gain algorithm as an example, the calculation result for each data is reflected by multiplying each sampling point by a different coefficient:

P(i)=aiΔt+bexp ^ciΔt , i=1, 2,...,N ₁

Where: N ₁ is the number of sampling points of each signal (can be read in the file header);

P(i) is the weighting factor corresponding to the i-th sampling point;

Δt is the sampling time interval (can be read in the file header);

a, b, c are coefficients, and the values of the coefficients can be set by the user and input in the interface;

Therefore, for the above linear gain algorithm, the minimum calculation unit is set to single-channel data.

In combination with practical applications, in one embodiment, the parallel processing method provided by the present invention further includes: a node architecture creation step, setting a master node A as a task scheduling node, which together with a read/write node B and N computing nodes C _{1 -CN} constitute a parallel node architecture.

Furthermore, in the parallel processing step, multiple algorithms in the computational workflow are processed one by one:

If it is a conventional algorithm, a static scheduling method is used. Node A divides the data to be processed into N parts and evenly distributes them to nodes C ₁ to C _N. No task scheduling will be performed during the parallel execution. Nodes C ₁ to C _N respectively call the set conventional algorithm for parallel processing and feed back the calculation results to node A after processing.

Therefore, in one embodiment, the following steps are followed to implement parallel processing based on conventional class operations:

The task scheduling node determines whether there is any test data that needs to be processed. If so, it counts the total number of test data x that needs to be processed and starts the read-write node and the computing node.

The following steps are executed in a loop until all pending detection data operations are completed and the read/write nodes and computing nodes are released:

Based on the preset scheduling base K, the read/write node reads x/K channels as the data for this round of processing; the task scheduling node divides the data for this round of processing into N parts evenly and distributes them to each computing node; therefore, the setting of K during the operation determines the size of the scheduling granularity, which refers to the amount of data calculated by each computing node each time; in this scenario, the scheduling granularity = x/NK;

Each computing node performs operations in parallel according to the conventional class workflow;

The parallel scheduling node confirms whether the processed detection data has been received. If received, it transmits it to the read-write node, which integrates it according to the corresponding spatial structure and writes it into the storage area;

Wherein, K is a positive integer, 1≤K≤N, or further optimized in combination with the number of channels S corresponding to the unit mark data to determine the scheduling base K.

In actual application, since the calculation process of each parallel channel of conventional operation is consistent, the calculation time difference will not be too large. Therefore, the next batch of data retrieval and operation will be started after the batch is completed. However, it should be noted that once the parallel scheduling node recognizes that the processed detection data has been received and the calculation has been completed, it can immediately transmit it to the read-write node, and the read-write node will integrate it according to the corresponding spatial structure and write it to the storage area. This does not conflict with the start of the next batch of reading and operation after the above-mentioned batch is completed, and can effectively avoid data reading, writing and storage congestion.

The value of the calculation unit size e is set according to the total number of data channels x to be processed. Generally, the larger the total number of data channels to be processed, the larger the corresponding calculation unit size. The minimum e value is 1, indicating that a single channel is the minimum calculation unit.

Furthermore, considering the marker data used to match the real spatial position of the radar data, in an optional embodiment, when determining the scheduling base number for segmenting the information content, the position of the marker data is combined for calculation. Specifically, the unit scheduling channel number is kept as a single or multiple times of the channel number S corresponding to the unit marker data, so that the data within each 2 marker ranges can use the same signal processing algorithm to avoid too much difference in the calculation signal of the corresponding data unit. For example, if the marker is one per 1,000 channels, the unit scheduling channel number during data processing will also refer to the marker granularity channel number (the channel number corresponding to the unit marker data), and be equal to it or a multiple of it.

Specifically, in actual application, the value of the target call base can be determined according to the following strategy:

If the value of the unit scheduling channel number N*e obtained based on the initial scheduling cardinality K' is less than or equal to S of the corresponding data unit, the value of the selected target scheduling cardinality K is equal to S;

If the value of the unit scheduling channel number N*e is greater than the corresponding data unit S, then based on The value of is rounded up to get the target multiple m, and m*S is selected as the target value of the granularity K.

Furthermore, if it is an iterative algorithm, a dynamic scheduling method of cyclic allocation is adopted:

Set the corresponding parallel scheduling base K, where K is an integer.

The amount of data initially allocated by node A accounts for 1/K of the total amount of data, and the data is divided into N parts and evenly distributed to nodes C ₁ to C _N. C ₁ to C _N respectively call the specified algorithm for parallel processing and feed back the calculation results to node A after processing.

Node A receives feedback information from nodes _C1 ~ _CN , and counts the number of computing nodes in idle state in real time. The researchers of the present invention consider that for the algorithm combination in the iterative workflow, since the number of iterations is unknown, for the algorithm with many iterations, the single operation time difference of different computing nodes will increase with the number of iterations, forming a significant time difference, resulting in significantly different time consumption of each computing node); from the remaining computing amount, continue to evenly distribute the data amount (1/Kn) to each idle computing node;

The above steps are executed repeatedly until all data are processed.

Therefore, in one embodiment, in the parallel processing step, the parallel processing based on iterative operations is implemented according to the following steps:

The task scheduling node determines whether there is any test data that needs to be processed. If so, it counts the total number of test data x that needs to be processed and starts the read-write node and the computing node.

The scheduling base K is set according to the hardware resources, x, and the computing unit size of the iterative algorithm. The read/write node reads x/K channels as the first round of processing data. The task scheduling node divides the processing data of this round into N parts and distributes them to each computing node.

Then, the following steps are executed in a loop until all pending detection data operations are completed and the read/write nodes and computing nodes are released:

The task scheduling node counts the number of idle computing nodes n in real time.

Read-write node reads The data is divided into n parts evenly and distributed to each idle computing node;

Each computing node executes operations in parallel according to the iterative workflow and feeds back the operation status information to the task scheduling node in real time;

The parallel scheduling node confirms whether the processed detection data has been received. If received, it will be transmitted to the read-write node, which will integrate it according to the corresponding spatial structure and write it into the storage area.

For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the order of the actions described, because according to the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

It should be pointed out that, in other embodiments of the present invention, the method can also obtain a new data parallel processing method by combining one or several of the above embodiments to achieve efficient analysis and calculation of massive detection data.

The scheme described in any one or more of the above embodiments of the present invention is adopted to realize parallel processing of railway detection data. On the basis of achieving fast and accurate calculations, the load balancing of each parallel computing channel is further improved, and the utilization efficiency of parallel computing resources is guaranteed to be maximized, which helps to provide reliable data support for subsequent application research.

It should be noted that, based on the method in any one or more of the above-mentioned embodiments of the present invention, the present invention also provides a storage medium, on which is stored a program code that can implement the method as described in any one or more of the above-mentioned embodiments, and when the code is executed by the operating system, the method for parallel processing of railway detection big data as described above can be implemented.

Embodiment 2

The method is described in detail in the embodiments disclosed in the present invention. The method of the present invention can be implemented by various forms of devices or systems. Therefore, based on other aspects of the method described in any one or more embodiments, the present invention also provides a system for parallel processing of railway detection big data, which is used to execute the method for parallel processing of railway detection big data described in any one or more embodiments. Specific embodiments are given below for detailed description.

Specifically, FIG2 shows a schematic diagram of the structure of a system for parallel processing of railway detection big data provided in an embodiment of the present invention. As shown in FIG2 , the system includes:

A data storage module, configured to divide the collected detection data and store the divided detection data according to a set storage strategy;

An algorithm combination formulation module is configured to create a corresponding operation workflow according to the engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of multiple signal processing algorithms required for the detection data, including conventional workflows and iterative workflows;

The parallel processing module is configured to retrieve the stored detection data in parallel based on a pre-created parallel node architecture, and implement parallel processing based on conventional operations and iterative operations according to a scheduling strategy that matches the operation workflow.

In one embodiment, the data storage module includes a data partitioning unit configured to partition the railway detection data according to the collection area and the type of collection equipment to obtain detection data of different levels with different processing priorities.

Further, in one embodiment, when storing the divided detection data, the data storage unit of the data storage module adopts a MySQL relational database, HDFS and multiple storage methods, and combines priority information to realize partition storage of detection data in multiple formats;

The detection data includes a file header and a data body, and the file header includes the total number of channels, the number of sampling points, the antenna frequency, the channel spacing and the time window parameters during data collection.

The data body of the detection data file includes header data and data content. The header data of each detection data is associated with the file header and stored in a relational database, and the data content of each detection data is stored in a distributed database.

Furthermore, in one embodiment, the system also includes: a node architecture creation step, setting a master node as a task scheduling node, which together with a read-write node and N computing nodes constitutes a parallel node architecture.

Preferably, in one embodiment, the parallel processing module includes a conventional operation processing unit, which is configured to implement parallel processing based on conventional operations according to the following steps:

The task scheduling node determines whether there is any test data that needs to be processed. If so, it counts the total number of test data x that needs to be processed and starts the read-write node and the computing node.

The following steps are executed in a loop until all pending detection data operations are completed and the read/write nodes and computing nodes are released:

Based on the preset scheduling base K, the read/write node reads x/K channels as the data for this round of processing; the task scheduling node divides the data for this round of processing into N parts evenly and distributes them to each computing node;

Each computing node performs operations in parallel according to the conventional class workflow;

The parallel scheduling node confirms whether the processed detection data has been received. If received, it transmits it to the read-write node, which integrates it according to the corresponding spatial structure and writes it into the storage area;

Wherein, K is a positive integer, 1≤K≤N, or further optimized in combination with the number of channels S corresponding to the unit mark data to determine the scheduling base K.

In one embodiment, the parallel processing module includes an iterative operation processing unit, which is configured to implement parallel processing based on iterative operations according to the following steps:

The task scheduling node determines whether there is any test data that needs to be processed. If so, it counts the total number of test data x that needs to be processed and starts the read-write node and the computing node.

The scheduling base K is set according to the hardware resources, x, and the computing unit size of the iterative algorithm. The read/write node reads x/K channels as the first round of processing data. The task scheduling node divides the processing data of this round into N parts and distributes them to each computing node.

Then, the following steps are executed in a loop until all pending detection data operations are completed and the read/write nodes and computing nodes are released:

The task scheduling node counts the number of idle computing nodes n in real time.

Read-write node reads The data is divided into n parts evenly and distributed to each idle computing node;

Each computing node executes operations in parallel according to the iterative workflow and feeds back the operation status information to the task scheduling node in real time;

The parallel scheduling node confirms whether the processed detection data has been received. If received, it will be transmitted to the read-write node, which will integrate it according to the corresponding spatial structure and write it into the storage area.

In the system for parallel processing of railway inspection big data provided by the embodiment of the present invention, each module or unit structure can operate independently or in combination according to actual analysis and computing requirements to achieve corresponding technical effects.

It should be understood that the embodiments disclosed in the present invention are not limited to the specific structures, processing steps or materials disclosed herein, but should be extended to equivalent substitutions of these features understood by ordinary technicians in the relevant field. It should also be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not meant to be limiting.

The "one embodiment" mentioned in the specification means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. Therefore, the phrase "one embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

Although the embodiments disclosed in the present invention are as above, the contents described are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims (8)

1. A method of parallel processing of railway test big data, the method comprising:

And a data storage step: dividing the acquired detection data, and storing the divided detection data according to a set storage strategy;

the algorithm combination making step: creating a corresponding operation workflow according to engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of a plurality of signal processing algorithms required to be performed by the detection data, and comprises a conventional workflow and an iterative workflow;

Parallel processing: parallel retrieving stored detection data based on a pre-established parallel node architecture, and respectively realizing parallel processing based on conventional class operation and iterative class operation according to a scheduling strategy matched with the operation workflow;

In the data storage step, railway detection data are divided according to the acquisition area and the type of acquisition equipment, so that detection data of different levels with different processing priorities are obtained;

when the divided detection data is stored, a plurality of storage modes of a relational database and a distributed database are adopted, and parallel storage of the detection data in a plurality of formats is realized by combining priority information;

The data body of the detection data file comprises header data and data content, the header data of each detection data and the header are stored in a relational database in an associated mode, and the data content of each detection data is stored in a distributed database;

in the parallel processing step, parallel processing based on conventional class operation is realized according to the following steps:

The task scheduling node judges whether detection data needing to be processed exist or not, if so, the total channel number x of the detection data needing to be processed is counted, and the read-write node and the computing node are started;

The following steps are sequentially and circularly executed until all the detection data to be processed are operated, and the read-write node and the computing node are released:

Based on a preset scheduling base number K, reading an x/K channel by a read-write node as the current round of processing data; the task scheduling node averagely divides the round of processing data into N parts and distributes the N parts to each computing node;

Each computing node performs operation in parallel according to the conventional class workflow;

The parallel dispatching node confirms whether the processed detection data after calculation is received or not, if so, the processed detection data is transmitted to the read-write node, and the read-write node integrates the processed detection data according to the corresponding space structure and writes the processed detection data into the storage area;

wherein K is a positive integer, and K is more than or equal to 1 and less than or equal to N.

2. The method according to claim 1, wherein the method further comprises: in the parallel processing step, the number S of tracks corresponding to the unit marking data is further combined for optimization, and the scheduling base number K is determined.

3. The method according to claim 1, wherein the method further comprises: and a node architecture creation step, namely setting a master node as a task scheduling node, and forming a parallel node architecture together with a read-write node and N computing nodes.

4. The method according to claim 1, characterized in that in the parallel processing step, parallel processing based on iterative class operations is implemented according to the following steps:

The task scheduling node judges whether detection data needing to be processed exist or not, if so, the total channel number x of the detection data needing to be processed is counted, and the read-write node and the computing node are started;

Setting a scheduling base number K according to hardware resources, x and the calculation unit size of the iterative operation, and reading x/K channels by a read-write node as first-round processing data; the task scheduling node averagely divides the round of processing data into N parts and distributes the N parts to each computing node;

And then the following steps are sequentially and circularly executed until all the detection data to be processed are operated, and the read-write node and the computing node are released:

the task scheduling node counts the number n of idle computing nodes in real time,

Read-write node readingThe channel data is divided into n parts averagely and distributed to each idle computing node;

Each computing node performs operation in parallel according to the iterative workflow, and feeds back operation state information to the task scheduling node in real time;

And the parallel scheduling node confirms whether the processed detection data after calculation is received or not, if so, the processed detection data is transmitted to the read-write node, and the read-write node integrates the processed detection data according to the corresponding space structure and writes the processed detection data into the storage area.

5. A storage medium having stored thereon program code for implementing the method of any of claims 1-4.

6. A system for parallel processing of railway test big data, the system comprising:

The data storage module is configured to divide the acquired detection data and store the divided detection data according to a set storage strategy;

The algorithm combination making module is configured to create a corresponding operation workflow according to engineering property requirements and interpretation target requirements of the detection data, wherein the operation workflow is a combination of a plurality of signal processing algorithms required to be performed by the detection data, and comprises a conventional workflow and an iterative workflow;

The parallel processing module is configured to access stored detection data in parallel based on a pre-established parallel node architecture and respectively realize parallel processing based on conventional class operation and iterative class operation according to a scheduling strategy matched with the operation workflow;

the data storage module is configured to: dividing railway detection data according to the types of the acquisition areas and the acquisition equipment to obtain detection data of all levels with different processing priorities;

When the divided detection data is stored, a plurality of storage modes of a relational database and a distributed database are adopted, and parallel storage of the detection data in a plurality of formats is realized by combining priority information; the data body of the detection data file comprises header data and data content, the header data of each detection data and the header are stored in a relational database in an associated mode, and the data content of each detection data is stored in a distributed database;

the parallel processing module realizes parallel processing based on conventional class operation according to the following operations:

The task scheduling node judges whether detection data needing to be processed exist or not, if so, the total channel number x of the detection data needing to be processed is counted, and the read-write node and the computing node are started;

The following steps are sequentially and circularly executed until all the detection data to be processed are operated, and the read-write node and the computing node are released:

Based on a preset scheduling base number K, reading an x/K channel by a read-write node as the current round of processing data; the task scheduling node averagely divides the round of processing data into N parts and distributes the N parts to each computing node;

Each computing node performs operation in parallel according to the conventional class workflow;

The parallel dispatching node confirms whether the processed detection data after calculation is received or not, if so, the processed detection data is transmitted to the read-write node, and the read-write node integrates the processed detection data according to the corresponding space structure and writes the processed detection data into the storage area;

wherein K is a positive integer, and K is more than or equal to 1 and less than or equal to N.

7. The system of claim 6, wherein the parallel processing module further optimizes the number of lanes S corresponding to the unit marking data to determine the scheduling radix K when implementing parallel processing based on a conventional class operation.

8. The system of claim 6, wherein the system further comprises: the node architecture creation module is configured to set a master node as a task scheduling node, and forms a parallel node architecture together with a read-write node and N computing nodes.