CN116701456A - A data analysis method and related equipment - Google Patents

Abstract

This specification provides a data analysis method and related equipment, which are applied to a cloud service platform, and the cloud service platform is respectively connected to multiple data sources. The method includes: obtaining a sample data set corresponding to any target data source among the plurality of data sources; analyzing the data structure of the sample data in the sample data set, and generating a data set corresponding to the target data source based on the analysis result. The data parsing rules; the data parsing rules are used to indicate the data structure of each data in the target data source; based on the data parsing rules, perform data structure parsing on the target data to be parsed in the target data source .

Description

A data analysis method and related equipment

technical field

One or more embodiments of this specification relate to the technical field of data processing, and in particular, to a data parsing method and related equipment.

Background technique

The cloud service platform can connect to multiple data sources outside the cloud, and perform unified data management on the data of the multiple data sources. However, the data formats of various data sources are often different, and there is no unified standard, which makes it impossible for the cloud service platform to perform accurate and effective data management when accessing the data of these data sources.

Therefore, when the cloud service platform accesses the data of each data source, it often needs to manually configure the corresponding data analysis rules for the data of each data source, so that the cloud service platform can analyze each data source based on these manually configured data analysis rules. Analyze the data to obtain structured standard data for subsequent management. However, manually configuring a large number of data parsing rules is time-consuming, labor-intensive and error-prone, which greatly reduces the data access efficiency and data management efficiency of the cloud service platform.

Contents of the invention

In view of this, one or more embodiments of this specification provide a data parsing method and related equipment.

In the first aspect, this specification provides a data analysis method, which is applied to a cloud service platform, and the cloud service platform is respectively connected to multiple data sources; the method includes:

Obtain a sample data set corresponding to any target data source among the plurality of data sources;

Analyzing the data structure of the sample data in the sample data set, and generating a data parsing rule corresponding to the target data source based on the analysis result; the data parsing rule is used to indicate the data of each data in the target data source structure;

Based on the data parsing rules, perform data structure parsing on the target data to be parsed in the target data source.

In the second aspect, this specification provides a data parsing device, which is applied to a cloud service platform, and the cloud service platform is respectively connected to multiple data sources; the device includes:

an acquisition unit, configured to acquire a sample data set corresponding to any target data source among the plurality of data sources;

The first parsing rule generation unit is configured to analyze the data structure of the sample data in the sample data set, and generate a data parsing rule corresponding to the target data source based on the analysis result; the data parsing rule is used to indicate the The data structure of each data in the target data source;

A parsing unit, configured to analyze the data structure of the target data to be parsed in the target data source based on the data parsing rules.

Correspondingly, this specification also provides a server, which is applied to a cloud service platform, the server includes a memory and a processor; the memory stores a computer program that can be run by the processor; the processor runs the In the case of a computer program, the data analysis methods described in the above-mentioned embodiments are executed.

Correspondingly, this specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the data analysis method described in the above-mentioned embodiments is executed.

To sum up, the cloud service platform in this application can first obtain the sample data in each data source, then analyze the data structure of the sample data in each data source, and automatically generate the data corresponding to the data in each data source based on the analysis results. Corresponding data parsing rules. Further, the cloud service platform can quickly and accurately analyze the data structure of the data to be analyzed in each data source based on the automatically generated data analysis rules, so that the subsequent cloud service platform can analyze the data in each data source based on the obtained analysis results. accurate and reliable data management. In this way, this application can automatically generate corresponding data analysis rules by analyzing the sample data in the data source, thereby greatly improving the data analysis efficiency and data management efficiency of the cloud service platform, and further improving the service quality of the cloud service platform. Meet the actual needs of customers.

Description of drawings

FIG. 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

Fig. 2 is a schematic flow chart of a data parsing method provided by an exemplary embodiment;

Fig. 3 is a schematic structural diagram of a data parsing device provided by an exemplary embodiment;

Fig. 4 is a schematic structural diagram of a server provided by an exemplary embodiment.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. Implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with aspects of one or more embodiments of the present specification as recited in the appended claims.

It should be noted that in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or less steps than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments describe.

The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are authorized by the user or Information and data that have been fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse.

First of all, some terms in this specification are explained to facilitate the understanding of those skilled in the art.

(1) Structured data, also known as quantitative data, is information that can be represented by data or a unified structure. If a piece of data can clearly know which information columns and information values it contains, then this piece of data is structured data, such as data in xml format and json format. The storage and management of structured data is generally a relational database, and structured query language or SQL language can be used to query the structured data stored in the database.

Correspondingly, unstructured data is essentially everything other than structured data that does not conform to any predefined model. Unstructured data may be text or non-text. For example, an ordinary string text containing various information contents, whose meaning cannot be automatically understood by the system, is unstructured data. The storage and management of unstructured data are generally non-relational databases, and NoSQL language can be used to query the unstructured data stored in the database.

(2) Standardized data is data that has a unified and standardized standard description for each information. If multiple different data have the same field name and field type for the same information (such as source ip, target ip, etc.), then these data are standardized data, otherwise, they are non-standardized data.

In a hybrid cloud scenario, customers will purchase and use products from multiple vendors, and they hope to have a unified platform (such as a cloud service platform) to integrate data from multiple vendors for unified and secure data management. Exemplarily, these products may be network security products of various manufacturers, such as firewall products, etc., which are not specifically limited in this specification.

In an illustrated embodiment, the cloud service platform can connect to data sources of multiple manufacturers, and respectively access data in multiple data sources for unified data management. However, since the data formats of data sources of various manufacturers are often different, and the data formats of data related to different products in the same manufacturer will also be different, even the data formats corresponding to different versions of the same product will be different, or they may be different from the same The data formats of different types of product-related data (such as event logs, traffic logs, domain name logs, etc.) are not uniform, and so on. As a result, the cloud service platform often does not know how to analyze the data when accessing data from various data sources, and cannot accurately obtain the actual information in the data, making it difficult for the cloud service platform to perform effective and reliable data analysis on the data from each data source. data management.

Based on this, in some possible implementations, it is often necessary to manually configure a large number of corresponding data parsing rules for various types of data related to different products in each data source, so that the cloud service platform can The parsing rules parse the data of each data source to obtain structured standard data for subsequent management. However, manually configuring a large number of data parsing rules is time-consuming, labor-intensive and error-prone, causing the cloud service platform to fail to quickly and accurately access data from multiple data sources, and it is easier to lose data or generate data format disorder, etc. This greatly reduces the data management efficiency of the cloud service platform.

In other possible implementations, when the cloud service platform accesses the data of each data source, it may not perform structural analysis on the data, but first access and store the data, and then manually configure the analysis when querying and using it. rule. In this way, although the data access efficiency can be improved, it will affect the query performance, and it also has high special requirements for data storage and search engines, which cannot meet the actual application requirements.

Based on this, this specification provides a technical solution for automatically generating corresponding data parsing rules based on the sample data in the data source, and then performing efficient and accurate data structure parsing for the data to be parsed in the data source based on the data parsing rules .

During implementation, the cloud service platform can obtain a sample data set corresponding to any target data source among multiple data sources connected to it. Then, analyze the data structure of the sample data in the sample data set, and generate a data parsing rule corresponding to the target data source based on the analysis result. Wherein, the data parsing rule may be used to indicate the data structure of each data in the target data source. Then, the cloud service platform can analyze the data structure of the target data to be parsed in the target data source based on the above data parsing rules, and perform data management on the target data based on the parsing result.

In the above technical solution, the cloud service platform in this application can first obtain the sample data in each data source, then analyze the data structure of the sample data in each data source, and automatically generate data related to each data source based on the analysis results The corresponding data parsing rules. Further, the cloud service platform can quickly and accurately analyze the data structure of the data to be analyzed in each data source based on the automatically generated data analysis rules, so that the subsequent cloud service platform can analyze the data in each data source based on the obtained analysis results. accurate and reliable data management. In this way, this application can automatically generate corresponding data analysis rules by analyzing the sample data in the data source, thereby greatly improving the data analysis efficiency and data management efficiency of the cloud service platform, and further improving the service quality of the cloud service platform. Meet the actual needs of customers.

Please refer to FIG. 1 . FIG. 1 is a schematic diagram of a system architecture provided by an exemplary embodiment. One or more embodiments provided in this specification may be specifically implemented in the system architecture shown in FIG. 1 or a similar system architecture. As shown in FIG. 1 , the system may include a cloud service platform and multiple data sources outside the cloud, such as a data source 100a, a data source 100b, and a data source 100c, and so on. In an illustrated embodiment, the cloud service platform can interface with the data source 100a, the data source 100b, and the data source 100c. In an illustrated embodiment, the cloud service platform may establish a communication connection with the data source 100a, the data source 100b, and the data source 100c through a wireless network or the like.

In an illustrated embodiment, data source 100a, data source 100b, and data source 100c may be data sources from different vendors. Exemplarily, the data source 100a may be a data source of a vendor A, and the data source 100a stores data (such as log data) of a plurality of products held by the vendor A. Exemplarily, the data source 100b may be a data source of the manufacturer B, and the data source 100b stores data of multiple products held by the manufacturer B. Exemplarily, the data source 100c may be a data source of a manufacturer C, and the data source 100c stores data of a plurality of products held by the manufacturer C.

In an illustrated embodiment, the products owned by the above-mentioned manufacturers may be software products, such as firewalls and other products used to protect network security, etc., which are not specifically limited in this specification. In some possible implementation manners, the products owned by each manufacturer may also be audio-visual entertainment software products, which is not specifically limited in this specification.

In an illustrated embodiment, the data source 100a, the data source 100b, and the data source 100c can send the data therein to the cloud service platform, so that the cloud service platform can perform unified data management. Exemplarily, the data source 100a, the data source 100b, and the data source 100c may send data through syslog (system log), that is, the data sent to the cloud service platform may be log data. Exemplarily, the log data may include various types of log data such as event logs, traffic logs, and domain name logs.

It can be understood that, as mentioned above, the data formats in various data sources are often different, and the data formats of data related to different products in the same data source will also be different, and even the data formats of different types of data related to the same product are also different. will be different, and so on.

Exemplarily, taking the data source 100a as an example, the data source 100a can send log data to the cloud service platform, where the formats of various types of log data such as event logs, traffic logs, and domain name logs can be as follows.

event log: src_ip='10.20.3.4' dst_ip='10.20.3.5' module='syslog' severity='debug' time='1495093983'

Traffic log: srcIp:10.20.3.4desIp:10.20.3.5severity:info time:1495093983

Domain name log: src:'10.20.3.4', des:'10.20.3.5', severity:'info', timestamp:'1495093983'

As shown above, each type of log data can include multiple fields, and each field can include a corresponding field name and field value. Correspondingly, each type of log data may include a corresponding field delimiter and a key-value (Key-Value, KV) delimiter. Among them, the field separator is used to separate two adjacent fields in the data, and the KV separator is used to separate the field name and field value of each field in the data. It should be understood that the field name is used as the key of the field, and the field value is used as the value corresponding to the key of the field.

As shown above, the field separators and KV separators in various types of log data such as event logs, traffic logs, and domain name logs are different. For example, the event log uses a space as the field separator and an equal sign as the KV separator. For another example, the traffic log uses a space as the field separator and a colon as the KV separator. For another example, the domain name log uses a comma as the field separator and a colon as the KV separator.

Further, as shown above, field names corresponding to the same information in various types of log data such as event logs, traffic logs, and domain name logs are also different. Exemplarily, taking the source ip as an example, the name of the field describing the source ip in the event log is src_ip, the name of the field describing the source ip in the traffic log is srcIp, and the name of the field describing the source ip in the domain name log is src. Exemplarily, taking the target ip as an example, the field name describing the target ip in the event log is dst_ip, while the field name describing the target ip in the traffic log is dstIp, and the field name describing the target ip in the domain name log is dst. Not the same, there is no uniform standard.

In an illustrated embodiment, it should be understood that the data format of the same type of data sent by different data sources will also be different, for example, it is also an event log, the format of the event log in the data source 100a and the event log in the data source 100b It will also be different, and so on, so I won’t go into details here.

In this way, after the cloud service platform receives the data sent by each data source, it cannot parse each data according to a unified standard, and thus cannot accurately know the data structure of each data, for example, it cannot accurately segment the fields in each data, and cannot Know exactly what field names and field values are contained in each field, and more. In short, the data sent by each data source is equivalent to unstructured and non-standardized data for the cloud service platform, which makes the cloud service platform unable to perform accurate and effective data management.

Based on this, in an illustrated embodiment, the cloud service platform can first obtain the sample data sets in the data source 100a, the data source 100b, and the data source 100c, and respectively analyze the The data structure of the sample data in the sample data set, and then generate data parsing rules corresponding to the data in the data source 100a, the data source 100b, and the data source 100c based on the analysis results. Correspondingly, the follow-up cloud service platform can quickly and accurately analyze the data structure of the corresponding data in the data source 100a, data source 100b, and data source 100c based on the generated data analysis rules, so as to effectively manage the data based on the analysis results.

Exemplarily, taking the data source 100a as an example, the cloud service platform may acquire the sample data set in the data source 100a, and then analyze the data structure of the sample data in the sample data set. Exemplarily, field delimiters and KV delimiters in the sample data may be analyzed. Then, the cloud service platform can generate corresponding data parsing rules based on the analysis results. Wherein, the data parsing rule may be used to indicate the data structure of the corresponding data in the data source 100a, for example, indicate field delimiters and KV delimiters in the data. Then, when the data source 100a sends corresponding data to the cloud service platform to enable the cloud service platform to perform data management, the cloud service platform can analyze the data structure of the data sent by the data source 100a based on the pre-generated data analysis rules, and based on the analysis Results data management.

It should be understood that since data in the same data source may also have different data formats, the above data parsing rules generated based on sample data may generally be used to indicate the data structure of data having the same format as the sample data.

Exemplarily, the sample data set in the data source 100a may include data related to the target product held by manufacturer A, and correspondingly, the data parsing rules generated based on the sample data set may be used to indicate each The data structure of the data.

Exemplarily, the sample data set in the data source 100a may include event logs related to the target product, and correspondingly, the data parsing rules generated based on the sample data set may be used to indicate each event log related to the target product data structure.

Exemplarily, the sample data set in the data source 100a may include traffic logs related to the target product, and correspondingly, the data parsing rules generated based on the sample data set may be used to indicate the traffic logs related to the target product data structure.

Exemplarily, the sample data set in the data source 100a may include domain name logs related to the target product, and correspondingly, the data parsing rules generated based on the sample data set may be used to indicate the domain name logs related to the target product Data structures, etc., will not be listed here.

In an illustrated embodiment, the sample data set may be part of the data sent by the data source 100a to the cloud service platform in real time, that is, after the cloud service platform receives the data sent by the data source 100a, it can start from the Part of the data is selected as sample data, and corresponding data analysis rules are generated in real time based on the sample data, so as to analyze the data structure of all the data received this time, etc., which are not specifically limited in this specification.

Exemplarily, the data source 100a sends 1000 event logs related to the target product to the cloud service platform so that the cloud service platform can perform data management. After receiving the 1000 event logs sent by the data source 100a, the cloud service platform can retrieve Select 50 event logs from the 1,000 event logs as sample data, and generate corresponding data analysis rules based on these 50 event logs in real time, and then the cloud service platform can perform data analysis on all 1,000 event logs received based on the analysis rules. Structural analysis, etc., are not specifically limited in this specification.

In this way, the cloud service platform in this application can automatically generate corresponding data analysis rules based on the sample data in the data source to be accessed, and then perform efficient and accurate data structure analysis on the data sent by the data source based on the data analysis rules , so as to effectively manage the data in the data source based on the analysis results, which greatly improves the data analysis efficiency of the cloud service platform, and further improves the data access efficiency and data management of the cloud service platform for each data source efficiency.

In an illustrated embodiment, the above-mentioned data source 100a, data source 100b, and data source 100c may be a server with the above-mentioned functions or a server cluster composed of multiple servers, etc. Specific limits.

In an illustrated embodiment, the above-mentioned cloud service platform may include a Security Operations Center (SecurityOperations Center, SOC), specifically, it may include a server with the above-mentioned functions or a server cluster composed of multiple servers, etc., this specification This is not specifically limited.

Please refer to FIG. 2 . FIG. 2 is a schematic flowchart of a data parsing method provided by an exemplary embodiment. The method can be applied to the system architecture shown in FIG. 1 or a similar system architecture. Specifically, the method can be applied to the cloud service platform shown in FIG. 1 . As shown in FIG. 2 , the method may specifically include the following steps S101-S103.

Step S101, acquiring a sample data set corresponding to any target data source among multiple data sources.

In an illustrated embodiment, the cloud service platform can interface with multiple data sources for unified data management of the data in each data source. Exemplarily, the multiple data sources may be data sources of different manufacturers, or data sources of the same manufacturer for different products, etc., which are not specifically limited in this specification.

In an illustrated embodiment, after connecting with multiple data sources, the cloud service platform can obtain a sample data set corresponding to any target data source in the multiple data sources, wherein the sample data set can include multiple bar sample data.

In an illustrated embodiment, the sample data set may specifically include data related to a target product (such as a firewall product) in the target data source. In an illustrated embodiment, the sample data set may also include various types of data related to the target product in the target data source. In an illustrated embodiment, the sample data set may also include data related to multiple products in the target data source, etc., which is not specifically limited in this specification.

In an illustrated embodiment, the version of any product in the target data source may be continuously updated, and the data format of various data related to the product may also be continuously changed. Therefore, when any product in the target data source is updated, the cloud service platform can obtain the sample data set related to the updated product.

In an illustrated embodiment, the target data source may send the sample data set to the cloud service platform through the syslog protocol, and correspondingly, the cloud service platform receives the sample data set sent by the target data source. Correspondingly, the sample data in the sample data set may be log data, such as various types of log data such as event logs, traffic logs, or domain name logs, which is not specifically limited in this specification. In some possible implementation manners, the sample data in the sample data set may also be any other possible type of data, which is not specifically limited in this specification.

Exemplarily, the sample data set may include three pieces of sample data as shown below. Exemplarily, the three pieces of sample data may be event logs related to firewall products in the target data source.

Sample data 1: module='syslog#info'severity='debug-1'type='url-http'session_id='1'time='1495093983'src_ip='10.20.3.4'dst_ip='10.20.3.5'proto ='TCP'

Sample data 2: module='syslog#info'severity='debug-1'type='web'session_id='2'time='1495093984'src_ip='10.20.3.4'dst_ip='10.20.3.5'proto=' TCP'

Sample data 3: module='syslog#info'severity='debug-1'type='tomcat'session_id='3'time='1495093985'src_ip='10.20.3.50'dst_ip='10.20.3.10'proto=' TCP'

Step S102, analyzing the data structure of the sample data in the sample data set, and generating a data parsing rule corresponding to the target data source based on the analysis result.

In an illustrated embodiment, after acquiring the sample data set sent by the target data source, the cloud service platform can analyze the data structure of the sample data in the sample data set, and generate corresponding data parsing rules based on the analysis result.

In an illustrated embodiment, the generated data parsing rules can be used to indicate the data structure of the corresponding data in the target data source. Further, the data parsing rule may be used to indicate the data structure of the data in the target data source having the same format as the sample data.

Exemplarily, taking the sample data set including event logs related to the firewall product in the target data source as an example, the generated data parsing rules can be used to indicate the data structure of the event logs related to the firewall product in the target data source , etc., this specification does not specifically limit it.

Exemplarily, taking the sample data set including traffic logs related to the firewall product in the target data source as an example, the generated data parsing rules can be used to indicate the data structure of the traffic logs related to the firewall product in the target data source , etc., this specification does not specifically limit it.

Exemplarily, taking the sample data set including event logs related to firewall products and network traffic detection products in the target data source as an example, two corresponding data parsing rules can be generated based on the sample data set, the two data The parsing rules can be respectively used to indicate the data structure of the event log related to the firewall product and the network traffic detection product, etc., which are not specifically limited in this specification.

Exemplarily, taking the sample data set including event logs and domain name logs related to the firewall product in the target data source as an example, two corresponding data parsing rules can be generated based on the sample data set, and the two data parsing rules It can be respectively used to indicate the data structure of the event log related to the firewall product and the data structure of the domain name log, etc., which are not specifically limited in this specification.

In an illustrated embodiment, the above data parsing rules may include a first parsing rule for indicating field delimiters and KV delimiters in the data.

In an illustrated embodiment, after the cloud service platform acquires the sample data set sent by the target data source, it may count the number of various delimiters contained in the sample data in the sample data set in each sample data. Among them, the various separators contained in the sample data can be composed of characters other than uppercase and lowercase letters and single quotation marks and double quotation marks in the ASCII code, such as spaces, commas, colons, equal signs, exclamation marks, minus signs and wells No., etc., which are not specifically limited in this specification.

Exemplarily, take the example that the sample data set contains the above-mentioned sample data 1, sample data 2 and sample data 3, sample data 1, sample data 2 and sample data 3 all contain pound sign, minus sign, equal sign and space. kind of delimiter. Count the numbers of pound signs, minus signs, equal signs, and spaces in sample data 1, sample data 2, and sample data 3 respectively, and you can get the following four arrays.

Hash sign: [1, 1, 1], indicating that the number of the hash sign in sample data 1 is 1, the number in sample data 2 is 1, and the number in sample data 3 is 1.

Minus sign: [2, 1, 1], indicating that the number of minus signs in sample data 1 is 2, the number in sample data 2 is 1, and the number in sample data 3 is 1.

Equal signs: [8, 8, 8], indicating that the number of equal signs in sample data 1 is 8, the number in sample data 2 is 8, and the number in sample data 3 is 8.

Space: [7, 7, 7], indicating that the number of spaces as a delimiter in sample data 1 is 7, the number in sample data 2 is 7, and the number in sample data 3 is 7.

Further, in an illustrated embodiment, the cloud service platform can determine the field delimiter and KV delimiter.

In an illustrated embodiment, the cloud service platform can determine the separator with the largest number in each sample data as the KV separator of the sample data set, and set the number of separators in each sample data to be less than the KV separator A delimiter is determined as the field delimiter for the sample data set.

Exemplarily, still taking the above sample data 1, sample data 2 and sample data 3 as examples, the number of equal signs in each sample data is the largest, so the equal sign can be determined as the KV separator of the sample data set, and the space is in The number in each sample data is one less than the equal sign, so spaces can be identified as field separators for sample data sets.

In an illustrated embodiment, the cloud service platform can further determine the KV delimiter and the field delimiter of the sample data set more accurately based on the difference in the number of each delimiter among different sample data.

In an illustrated embodiment, the cloud service platform may determine the separator with the largest number in each sample data and the smallest difference in number between different sample data as the KV separator of the sample data set. Correspondingly, the cloud service platform may determine the separator whose number in each sample data is one less than the KV separator and whose number has the smallest difference between different sample data as the field separator of the sample data set.

In an illustrated embodiment, the cloud service platform can also calculate the number variance of various separators among different sample data through variance calculation based on the counted number of various separators in each sample data (i.e. fluctuations in quantity). Further, the cloud service platform can also determine the field separator and the KV separator of the sample data set in combination with the variance of the number of various separators among different sample data.

Exemplarily, still taking the above sample data 1, sample data 2 and sample data 3 as examples, the number of equal signs in each sample data is the largest, and the number of equal signs in different sample data is 8, and the variance is the smallest, so it can be Determine the equal sign as the KV delimiter for the sample dataset. Correspondingly, the number of spaces in different sample data is 7, the variance is the smallest, and the number of spaces in each sample data is one less than the equal sign, so the space can be determined as the field separator of the sample data set.

In an illustrated embodiment, the cloud service platform may generate the first type of parsing rules based on the analyzed field delimiters and KV delimiters of the sample data set above. Correspondingly, the first type of parsing rule can be used to indicate the field delimiter and KV delimiter in the data to be parsed.

In an illustrated embodiment, the above data parsing rule may further include a second parsing rule for indicating the field type of each field in the data.

In an illustrated embodiment, after the cloud service platform obtains the sample data set sent by the target data source, and analyzes the field delimiter and KV delimiter of the sample data, it can separate character to identify the fields in each sample data, and the field names and field values that each field contains.

Further, in an illustrated embodiment, the cloud service platform can match the field values of each field in each sample data with multiple field types respectively. Exemplarily, the multiple field types include, for example, time, value, IP, and character string, which are not specifically limited in this specification.

In an illustrated embodiment, regular expressions of multiple field types can be defined first, and then the field values of each field in each sample data are compared with the regular expressions of multiple field types according to the matching priority. Match to determine one or more field types that match the field value for each field in each sample data.

In an illustrated embodiment, for any target field in each sample data, if the field types that match the field value of the target field are all target types among multiple field types, then the target type Determined as the field type corresponding to the target field.

In an illustrated embodiment, for any target field in each sample data, if the field type matching the field value of the target field includes multiple field types, the corresponding A target type with a wider range of data is determined as the field type corresponding to the target field. For example, if among multiple sample data, the field type matched by the target field in one sample data is a numeric type, and the field type matched by the same target field in another sample data is a string type, consider If the data range corresponding to the string type is larger than the data range corresponding to the numeric type, the string type can be determined as the field type of the target field.

Exemplarily, still taking the above sample data 1, sample data 2 and sample data 3 as examples, the field types matching the src_ip field in each sample data are all ip types, then it can be determined that the field type of the src_ip field is ip type. If the field types matching the module field in each sample data are all string types, it can be determined that the field type of the module field is a string type. The field types matching the session_id field in each sample data are all numeric types, so it can be determined that the field type of the session_id field is a numeric type. Exemplarily, the field types that match the time field in each sample data include numerical type and time type, further considering that the data range of time can cover the data range of numerical values, and the field name of this field contains keywords such as time , so it can be determined that the field type of the time field is the time type.

In an illustrated embodiment, the cloud service platform may generate a second type of parsing rule based on the determined field type corresponding to each field in the sample data. Correspondingly, the second type of parsing rule may be used to indicate the field type of each field in the data to be parsed.

In an illustrated embodiment, the above-mentioned data parsing rules may further include a third type of parsing rules for indicating standard fields corresponding to each field in the data.

In an illustrated embodiment, after the cloud service platform obtains the sample data set sent by the target data source, and analyzes the field delimiter and KV delimiter of the sample data, it can separate character to identify the fields in each sample data, and the field names and field values that each field contains.

In an illustrated embodiment, the cloud service platform can calculate the similarity between each field in each sample data and various standard fields. Among them, the standard fields, as a set of standards, can be built into the cloud service platform. Exemplarily, the time field timestamp, the source ip field src.ip, the destination ip field dst.ip, etc. may be preset standard fields, which are not specifically limited in this specification.

In an illustrated embodiment, the cloud service platform can use a preset similarity calculation method to calculate the similarity between each field in each sample data and various standard fields.

It should be noted that this specification does not specifically limit the preset similarity calculation method.

In an illustrated embodiment, the aforementioned preset similarity calculation method may include a cosine similarity calculation method and/or an edit distance calculation method, etc., which are not specifically limited in this specification. In some possible implementation manners, the cloud service platform may also use any other possible calculation method, which is not specifically limited in this specification.

In an illustrated embodiment, the cloud service platform can actually compare the field names of each field in each sample data with a variety of standard fields, and then calculate the difference between each field in each sample data and a variety of standard fields. similarity between fields. Exemplarily, still taking the above sample data 1, sample data 2 and sample data 3 as examples, the cloud service platform can calculate the similarity between the src_ip field in each sample data and the standard source ip field src.ip, for example 95%, and calculate the similarity between the src_ip field and the standard target ip field dst.ip, for example, 30%, and calculate the similarity between the src_ip field and the standard time field timestamp, for example, 1%, etc. Wait, no more examples here.

Further, in an illustrated embodiment, the cloud service platform can determine from various standard fields the similarity between each field in each sample data and the standard fields in the sample data. The standard fields corresponding to each field, and map each field with the corresponding standard field.

In an illustrated embodiment, the cloud service platform may determine a standard field whose similarity with each field in the sample data is greater than a preset threshold among the various standard fields as a standard field corresponding to each field, and Each field is mapped with the corresponding standard field. Exemplarily, the preset threshold is, for example, 80%, 90% or 95%, etc., which is not specifically limited in this specification.

In an illustrated embodiment, if the similarities between the target field in the sample data and the above-mentioned various standard fields are all less than or equal to the preset threshold, the standard field with the largest similarity can be determined as being similar to the target field. The corresponding standard field can be labeled with a recommended mapping label to prompt the user to manually check and modify the mapping result, so as to ensure the accuracy of the standard field mapping.

In an illustrated embodiment, the cloud service platform can further combine the field types of each field in the sample data to determine the standard field corresponding to each field, etc., which is not specifically limited in this specification. Exemplarily, if the field type of the time field in the above sample data 1, sample data 2, and sample data 3 is a time type, and the similarity with the standard time field timestamp is greater than the preset threshold, the standard time field can be The field timestamp is determined as a standard field corresponding to the time field, and the standard time field timestamp is mapped to the time field.

Exemplarily, still taking the above sample data 1, sample data 2 and sample data 3 as examples, src_ip in the sample data can be mapped to the standard source ip field src.ip, and dst_ip in the sample data can be mapped to the standard target ip The field dst.ip, the time in the sample data can be mapped to the standard time field timestamp, etc., and will not be listed here.

In an illustrated embodiment, the cloud service platform may generate a third type of parsing rule based on the mapping relationship between each field in the sample data and the corresponding standard field. Correspondingly, the third type of parsing rules can be used to indicate standard fields corresponding to each field in the data to be parsed.

Step S103, based on the data parsing rules, perform data structure analysis on the target data to be parsed in the target data source.

In an illustrated embodiment, the cloud service platform may analyze the data structure of the target data to be analyzed sent by the target data source based on the obtained data analysis rules, and perform data management on the target data based on the analysis result. It is understandable that after the cloud service platform analyzes the data structure of the target data based on the data analysis rules, it can obtain structured and standardized target data, so that the cloud service platform can perform accurate and effective data management on it.

In an illustrated embodiment, the target data to be parsed may be data having the same data format as the above sample data, for example, an event log related to the same product.

In an illustrated embodiment, the cloud service platform can determine the field delimiter and KV delimiter of the target data to be parsed based on the first type of parsing rule included in the data parsing rule, so as to determine each fields and the field names and field values within each field. In this way, the cloud service platform can specify specific information such as source ip, target ip and time in the target data.

Further, in an illustrated embodiment, the cloud service platform can determine the field type corresponding to each field in the target data based on the second type of parsing rule included in the data parsing rule, such as time, value, string Or IP, etc.

Furthermore, in an illustrated embodiment, the cloud service platform can map each field in the target data to its corresponding standard field based on the third type of parsing rule contained in the data parsing rule, so as to finally obtain the structured and standardized target data.

In an illustrated embodiment, the target data to be parsed may also include the above sample data, and the cloud service platform may select part of the target data from the target data after receiving the target data to be parsed from the target data source As sample data, corresponding data parsing rules are generated in real time based on the sample data, so as to analyze the data structure of all target data received this time, etc., and this description does not specifically limit this.

To sum up, the cloud service platform in this application can first obtain the sample data in each data source, then analyze the data structure of the sample data in each data source, and automatically generate the data corresponding to the data in each data source based on the analysis results. Corresponding data parsing rules. Furthermore, the cloud service platform can quickly and accurately analyze the data structure of the data to be analyzed in each data source based on the automatically generated data analysis rules, so that the subsequent cloud service platform can analyze the data in each data source based on the analysis results. Effective and reliable data management. In this way, this application can automatically generate corresponding data analysis rules through the analysis of sample data, thereby greatly improving the data analysis and data management efficiency of the cloud service platform, further improving the service quality of the cloud service platform, and meeting the actual needs of customers .

Corresponding to the realization of the above-mentioned method flow, the embodiment of this specification also provides a data parsing device. Please refer to FIG. 3 . FIG. 3 is a schematic structural diagram of a data parsing device provided by an exemplary embodiment. The device 30 can be applied to the cloud service platform in the system architecture shown in FIG. 1 , and can be connected to multiple data sources respectively, so as to perform unified data management on the data in the multiple data sources. As shown in Figure 3, the device 30 includes:

An acquisition unit 301, configured to acquire a sample data set corresponding to any target data source among the plurality of data sources;

The first parsing rule generation unit 302 is configured to analyze the data structure of the sample data in the sample data set, and generate a data parsing rule corresponding to the target data source based on the analysis result; the data parsing rule is used to indicate the Describe the data structure of each data in the target data source;

The parsing unit 303 is configured to analyze the data structure of the target data to be parsed in the target data source based on the data parsing rules.

In an illustrated embodiment, the data parsing rules include a first type of parsing rule for indicating field separators and Key-Value key-value separators in data; wherein, the field separators are used to separate data The two adjacent fields in the data, the key-value delimiter is used to separate the field name and field value of each field in the data, the field name is used as the key of the field, and the field value is used as the key corresponding to the field value;

The first parsing rule generating unit 302 is specifically used for:

Counting the number of various delimiters contained in the sample data in the sample data set in each sample data;

Based on the counted quantity, determine the field delimiter and the key value delimiter of the sample data set among the various delimiters;

The first type of parsing rules are generated based on the field delimiter and the key value delimiter of the sample data set.

In an illustrated implementation manner, the first parsing rule generating unit 302 is specifically configured to:

Determining the separator with the largest number in each sample data as the key-value separator of the sample data set;

A delimiter whose number is one less than the key-value delimiter in each sample data is determined as a field delimiter of the sample data set.

In an illustrated implementation manner, the first parsing rule generating unit 302 is specifically configured to:

Determining the separator with the largest number in each sample data and the smallest difference in number between different sample data as the key-value separator of the sample data set;

The number of separators in each sample data is one less than the key-value separator, and the number of separators between different sample data is the smallest is determined as the field separator of the sample data set.

In an illustrated implementation manner, the first parsing rule generating unit 302 is specifically configured to:

Calculate the number variance of each delimiter between different sample data;

Determining the delimiter with the largest quantity in each sample data and the minimum quantity variance between different sample data as the key-value delimiter of the sample data set;

The number of separators in each sample data is one less than the key-value separator and the separator with the smallest number variance between different sample data is determined as the field separator of the sample data set.

In an illustrated embodiment, the data parsing rule further includes a second type of parsing rule for indicating the field type corresponding to each field in the data;

The device 30 also includes a second parsing rule generation unit 304, configured to:

Match the field values of each field in each sample data with multiple field types;

If the field type matched with the field value of any target field in each sample data is the target type in the plurality of field types, then determine the target type as the field type corresponding to the target field ;

If the field type matched with the field value of the target field in each sample data includes multiple field types, determine the target type corresponding to a wider range of data among the multiple field types as the target type that matches the target The field type corresponding to the field;

The second type of parsing rules are generated based on the determined field types corresponding to each field in the sample data.

In an illustrated embodiment, the data parsing rule further includes a third type of parsing rule for indicating the standard field corresponding to each field in the data;

The device 30 also includes a third parsing rule generation unit 305, configured to:

Use the preset similarity calculation method to calculate the similarity between each field in each sample data and various standard fields;

determining a standard field whose similarity with each field is greater than a preset threshold from the plurality of standard fields, and mapping each field with a corresponding standard field;

The third type of parsing rules are generated based on the mapping relationship between each field in the sample data and the corresponding standard field.

In an illustrated embodiment, the preset similarity calculation method includes a cosine similarity calculation method and/or an edit distance calculation method.

In an illustrated embodiment, the sample data in the sample data set and the target data to be parsed include log data in the target data source.

For the implementation process of the functions and effects of each unit in the above-mentioned device 30, refer to the descriptions of the corresponding embodiments in the above-mentioned Figs. 1-2 for details, and details are not repeated here. It should be understood that the above-mentioned device 30 may be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, the processor (CPU) of the device reads the corresponding computer program instructions into the memory and runs them. From the perspective of hardware, in addition to CPU and memory, the equipment where the above-mentioned device is located usually includes other hardware such as chips for wireless signal transmission and reception, and/or boards and other hardware for realizing network communication functions.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical modules, that is, they may be located in One place, or it can be distributed to multiple network modules. Part or all of the units or modules can be selected according to actual needs to achieve the purpose of the solution in this specification. It can be understood and implemented by those skilled in the art without creative effort.

The devices, units, and modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.

Corresponding to the foregoing method embodiments, the embodiments of this specification also provide a server. Please refer to FIG. 4 . FIG. 4 is a schematic structural diagram of a server provided by an exemplary embodiment. Exemplarily, the server 1000 may include one or more servers in the cloud service platform shown in FIG. 1 above, which is not specifically limited in this specification. As shown in FIG. 4, the server 1000 may include a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected through a bus or in other ways. As shown in FIG. 4 , the memory 1002 includes a computer-readable storage medium 1003 , and the computer-readable storage medium 1003 stores a computer program executable by the processor 1001 . The processor 1001 may be a general processor, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiments. When the processor 1001 is running the stored computer program, it can execute each step of the data analysis method in the embodiment of this specification, including: obtaining a sample data set corresponding to any target data source among multiple data sources; analyzing the sample data the data structure of the sample data in the collection, and generate data analysis rules corresponding to the target data source based on the analysis results; the data analysis rules are used to indicate the data structure of each data in the target data source; based on the The data parsing rule is to analyze the data structure of the target data to be parsed in the target data source, and perform data management on the target data based on the parsing result, and so on.

For a detailed description of each step of the above data parsing method, please refer to the previous content, and details will not be repeated here.

Corresponding to the above method embodiments, the embodiments of this specification also provide a computer-readable storage medium, on which computer programs are stored. When these computer programs are run by a processor, the data in the embodiments of this specification are executed. The steps of the analysis method. For details, please refer to the descriptions of the embodiments corresponding to FIGS. 1-2 above, and details are not repeated here.

The above descriptions are only preferred embodiments of this specification, and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included in this specification. within the scope of protection.

In a typical configuration, an end device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data.

Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Those skilled in the art should understand that the embodiments of this specification may be provided as methods, systems or computer program products. Accordingly, the embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present specification may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. .

Claims (11)

1. The data analysis method is applied to a cloud service platform, and the cloud service platform is respectively in butt joint with a plurality of data sources; the method comprises the following steps:

acquiring a sample data set corresponding to any target data source in the plurality of data sources;

analyzing a data structure of sample data in the sample data set, and generating a data analysis rule corresponding to the target data source based on an analysis result; the data parsing rule is used for indicating a data structure of each data in the target data source;

and carrying out data structure analysis on the target data to be analyzed in the target data source based on the data analysis rule.

2. The method of claim 1, the data parsing rules comprising a first type of parsing rule for indicating field separators and Key-Value separators in data; the key value separator is used for separating a field name and a field value of each field in the data, wherein the field name is used as a key of the field, and the field value is used as a value corresponding to the key of the field;

the analyzing the data structure of the sample data in the sample data set and generating a data analysis rule corresponding to the target data source based on the analysis result comprises the following steps:

counting the number of a plurality of separators contained in sample data in the sample data set in each sample data;

determining field separators and key separators of the sample data set from the plurality of separators based on the counted number;

the first type parsing rule is generated based on field separators and key-value separators of the sample data set.

3. The method of claim 2, the determining field separators and key separators of the sample data set among the plurality of separators based on the counted number, comprising:

Determining the most number of separators in each sample data as key separators of the sample data set;

a number of separators in each sample data that is one less than the key-value separators is determined as field separators for the sample data set.

4. A method according to claim 3, the determining the most number of separators in each sample data as key separators of the sample data set, comprising:

determining as a key-value separator of the sample data set a separator having a largest number in each sample data and a smallest number of differences between different sample data;

the determining a number of separators in each sample data that is one less than the key separators as field separators of the sample data set, comprising:

the field separator of the sample data set is determined as a separator having one less number in each sample data than the key separator and the smallest difference in number between different sample data.

5. The method of claim 4, the determining, as a key-value separator for the set of sample data, a separator that is the most numerous in each sample data and that has the smallest number of differences between different sample data, comprising:

Calculating a variance in the number of each separator between the different sample data;

determining as a key-value separator of the sample data set a separator having the largest number in each sample data and the smallest number variance between different sample data;

the determining, as a field separator of the sample data set, a separator having one fewer number in each sample data than the key separator and a smallest difference in number between different sample data, comprising:

a field separator of the sample data set is determined as a separator having one fewer number in each sample data than the key-value separator and a minimum number variance between different sample data.

6. The method of claim 2, the data parsing rules further comprising a second type parsing rule for indicating a field type corresponding to each field in the data;

the method further comprises the steps of:

respectively matching field values of various fields in each sample data with various field types;

if the field type matched with the field value of any target field in each sample data is the target type in the multiple field types, determining the target type as the field type corresponding to the target field;

If the field type matched with the field value of the target field in each sample data comprises a plurality of field types, determining the target type with wider data range corresponding to the plurality of field types as the field type corresponding to the target field;

and generating the second type analysis rule based on the determined field types corresponding to the fields in the sample data.

7. The method of claim 2, the data parsing rules further comprising a third type of parsing rule for indicating standard fields corresponding to respective fields in data;

the method further comprises the steps of:

calculating the similarity between each field in each sample data and various standard fields by using a preset similarity calculation method;

determining standard fields with similarity to each field larger than a preset threshold value from the plurality of standard fields, and mapping each field with the corresponding standard field;

and generating the third type of analysis rule based on the mapping relation between each field in the sample data and the corresponding standard field.

8. The method of any of claims 1-7, sample data in the sample data set and the target data to be parsed comprising log data in the target data source.

9. The data analysis device is applied to a cloud service platform, and the cloud service platform is respectively in butt joint with a plurality of data sources; the device comprises:

the acquisition unit is used for acquiring a sample data set corresponding to any target data source in the plurality of data sources;

a first analysis rule generating unit, configured to analyze a data structure of sample data in the sample data set, and generate a data analysis rule corresponding to the target data source based on an analysis result; the data parsing rule is used for indicating a data structure of each data in the target data source;

and the analysis unit is used for carrying out data structure analysis on the target data to be analyzed in the target data source based on the data analysis rule.

10. A server applied to a cloud service platform, the server comprising a memory and a processor; the memory has stored thereon a computer program executable by the processor; the processor, when running the computer program, performs the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of claims 1 to 8.