CN116414816A - A massive data processing method and system based on heterogeneous data sources - Google Patents

Abstract

The invention relates to a massive data processing method and system based on heterogeneous data sources. The method includes steps such as writing service call data into storage, querying service call data, regularly cleaning data files, and performing intelligent early warning through shell scripts. This method can support the rapid expansion of the service system, greatly increase the concurrency of the interface, support the call of the billion-level interface, and can be used for the storage and call of massive data; it also supports system expansion without the perception of external users, and the interface can be called normally during the expansion. This method adopts multi-data source engine collaborative query, supports cross-data source query, is applicable to various query scenarios, improves real-time data query capabilities and multi-dimensional analysis capabilities, and this method can intelligently clean up useless stored data and files in a timely and rapid manner. Manual inspection and cleaning are not required for operation and maintenance, which reduces labor and machine costs.

Description

A massive data processing method and system based on heterogeneous data sources

technical field

The invention belongs to the technical field of data processing methods of mixed data sources, and in particular relates to a massive data processing method and system based on heterogeneous data sources.

Background technique

Many application systems need to obtain the number of published services, interface failure rate, interface call time-consuming, and interface call volume (day, week, month, year) Information such as the number of supported applications, etc., all need to be statistically analyzed based on the interface call data. At present, most existing systems adopt an architectural solution based on mongodb+mysql+operation and maintenance manual data clearing. mongodb is used to store call data, and timing tasks process mongodb data to form statistical data, and then store statistical data in a mysql single table. Operation and maintenance regularly (every year) manually clean up useless data files in mongodb.

However, for the above-mentioned solution based on mongodb+mysql+operation and maintenance manual data clearing, if the number of call interface users increases sharply due to emergencies, the number of service calls reaches tens of millions or billions per day, and the concurrent number of single interfaces per second reaches thousands (such as Personal information query and other service interfaces provided by cities with a population of tens of millions), there will be the following defects:

1. In the above solution, the daily call volume of the interface can only support up to 10,000 levels, and it is impossible to support the situation where the number of calls reaches the daily level of 100 million due to emergencies.

2. MongoDB stores data files, which is difficult to expand. The expansion of mongo cluster is complicated, and the service will be stopped during the expansion process. The machine cost and labor cost are huge. When the concurrency suddenly increases, it is difficult to support expansion in a short period of time in a way that users do not perceive .

3. Interface call statistics are stored in a single table mysql. If 1w interfaces are opened to the outside world, there will be more than 130,000 pieces of data generated every day, 4 million pieces per month, and 50 million pieces per year. When the amount of mysql data in a single table exceeds 5000w, it will touch the disk IO bottleneck, which will lead to slow query, low efficiency, interface call timeout, or even unavailability.

4. Based on mysql and mongodb storage middleware, it does not support cross-data source query, is not friendly to complex query and statistics, and the query efficiency is difficult to meet complex OLAP requirements.

5. Manually clear data files. For a single-day data file size of less than 1G, a general file server with a disk size of 1T, it is enough to manually clear the data plan once in the operation and maintenance year. However, for a daily data volume of more than 50G, the data disk will be full in just a dozen days, requiring O&M to check the disk frequently, manually clean up files, and consume labor costs. If the cleanup is not timely, the service will be unavailable.

Contents of the invention

Terminology Explanation

Presto: Presto is Facebook's open source, fully memory-based parallel computing, distributed SQL query engine, suitable for interactive analysis queries.

Catalog: Catalog is the data source. Each data source connection has a name. A Catalog can contain multiple Schemas, and the data source is referenced through the connector. You can see all the data sources connected to Presto through the show catalogs command.

Schema: Equivalent to a database, a Schema contains multiple data tables. Use the show schemas from 'catalog_name' command to list all Schemas under the Catalog.

Kafka: Kafka is a high-throughput distributed publish-subscribe message system that can handle all action stream data of consumers in the website.

In order to overcome the defects of the existing mongodb+mysql+operation and maintenance manual data clearing architecture scheme, the present invention proposes a new massive data processing method based on heterogeneous data sources. The inventive method aims to solve the following problems:

1. In view of the situation that the daily service calls of a single interface reach tens of millions or even billions due to emergencies, a data storage solution that is easy to expand is proposed to solve the problem of difficult expansion.

2. Optimize the storage solution to solve the problem that the service platform is unavailable due to database io blocking and database freezing when the data volume of a single mysql table exceeds 5000w.

3. Improve single-table query speed and query efficiency.

4. For multiple data sources, realize cross-data source query to meet complex OLAP requirements.

5. Propose automatic cleaning rules and cleaning algorithms, realize intelligent cleaning of data files, replace manual cleaning of data, and save labor costs.

Overall, the present invention achieves the above-mentioned purpose through the following technical concepts and strategies:

(1) Efficient expansion solution: replace the main data storage engine, use mongodb only as a temporary storage middleware, and store the final data in ElasticaSearch. Since ElasticaSearch expansion is to increase machine nodes in the cluster, it is easier to achieve expansion.

(2) mysql intelligently divides tables according to specific rules and algorithms.

(3) In order to realize multiple queries such as single-table query and complex OLAP, multiple data source engines (mysql, mongodb, ElasticaSearch, redis) are used to store data, and multiple data sources cooperate to complete queries to improve query efficiency and share queries pressure.

(4) Introduce the Presto engine to support cross-data source query and analysis of mysql and ElasticaSearch.

(5) A variety of data cleaning solutions such as scheduled scheduling tasks, shell scripts, and database scheduled tasks, regularly clean up mongodb, mysql statistical tables, and ElasticaSearch.

(6) Introduce intelligent early warning. After passing the above scheme, if the system still has server pressure or database pressure that cannot be handled, it will be inspected on a daily basis and intelligently send early warning messages to the administrator.

Specifically, the present invention provides a method for processing massive data based on heterogeneous data sources. The method includes:

S1. Service call data is written into storage, and the storage includes mongodb and Elasticsearch;

S2. Query service call data;

S3. Regularly clean up data files;

S4. Carry out intelligent early warning through shell script.

Further, according to some embodiments of the present invention, the service call data in step S1 of the heterogeneous data source-based mass data processing method of the present invention includes API service call time, apiId, application name, parameters, time-consuming, error information, Error code information record json string.

Further, the method for processing massive data based on heterogeneous data sources in the present invention writes and stores service call data in step S1, including:

S11. Asynchronously write the specific information of the service call data into mongodb for data statistics query, create a new collection for mongodb every day, start a scheduled task at 23:59 every day, create a new mongodb table, write the data of the day into the data of the day surface;

S12. Asynchronously write the specific information of the service call data into Elasticsearch for user query, and create a new index for Elasticsearch rolling every month (create a template, generate a new index based on the template and creation strategy rollover every month);

S13. Make statistics on mongodb data every 10 seconds, according to the dimensions of year, month, day, and hour, the amount of interface calls, average time-consuming, number of failures, and number of calls, and save the statistical data to the mysql statistics table;

S14. Create a new data statistics table for mysql every year, and create a new table regularly at the end of each year. The new table will synchronize the table data of the previous month, and the table data of the next month will be written into the new table. Create a new data table at 23:00:00 on the last day of each year. The name of the table is dytj_current date year. At the same time, the data in December is synchronized to the new data table. On February 1st every year, the data in December of the previous year in the new table is regularly cleaned up.

Further, according to some embodiments of the present invention, when the massive data processing method based on heterogeneous data sources of the present invention suddenly increases the pressure on the database and/or server during the operation of the system, and the resource space is insufficient, the capacity expansion is performed in the following manner :

(1) For mysql, through the mysql sub-table strategy, when there is insufficient resource space, add machines to mysql, and store historical data and current data in sub-databases;

(2) For mongodb, mongodb only stores temporary data, as a temporary storage middleware, when there is insufficient resource space, only keep 1 day of data, and delete the intermediate table data;

(3) For Elasticsearch, evaluate the server resources that need to be increased, calculate the number of machines that need to be expanded according to the call volume, and perform horizontal expansion;

(4) For the application server, evaluate the server resources that need to be increased according to the total concurrency and the concurrency supported by a single server configuration, and perform horizontal expansion.

Further, the query service invocation data described in step S2 of the mass data processing method based on heterogeneous data sources in the present invention includes:

(1) Single table query

①Query the statistical information of API service calls, and obtain statistical data information in the dimensions of year, month, day, and hour by querying the mysql statistical table;

②Query the API service call data information, and obtain the data information of the API service name, call time, application name, data content, parameters, time-consuming, and error codes by querying Elasticsearch;

(2) Cross data source query

① Configure the catalog of Presto, find the catalog directory in the Presto installation directory, then add the connector file in the catalog directory, create mysql.properties, mongodb.properties, Elasticsearch.properties, and configure the connector information (address, user name, password, etc. );

② Write mixed query sql. The method of writing mixed query sql is the same as that of ordinary sql. The table name of the query is written as catalog.schame.tableName (catalog is the name of properties) to complete cross-data source query.

Further, in the above-mentioned mass data processing method based on heterogeneous data sources, the query API service call data information in the single table query also includes the data information obtained based on the query for aggregate analysis;

The aggregation analysis includes error data distribution analysis, error type distribution analysis, and most error reporting time period analysis.

Further, the regular cleaning of data files in step S3 of the mass data processing method based on heterogeneous data sources in the present invention includes:

(1) mongodb data cleaning

Set a scheduled deletion task for data cleaned up on a daily basis. The data is valid for 2 days, and the mongodb data of the previous 2 days are deleted every day;

(2) Elasticsearch data cleaning

①Elasticsearch synchronously sets the data timing deletion task when creating the index, the data is valid for 1 year, and the data is automatically deleted when it expires;

②Manually write a shell script for detection and put it into the server to check the remaining disk space regularly every day. When the remaining disk space is less than 10%, delete files according to the importance and index creation time strategy until the disk is left Space up to 80%;

(3) mysql data cleaning

① Set up a scheduled deletion task for data cleaned up on a yearly basis. The data is valid for 3 years, and the data statistics table 3 years ago will be automatically cleaned up when it expires;

②Manually write a shell script for detection and put it into the server to check the remaining disk space regularly every day. When the remaining disk space is less than 10%, delete files according to the importance and data storage time strategy until the disk The remaining space exceeds 10%;

③ For the new event of the mysql database, check the data volume of a single table every day. If the data volume of a single table exceeds 100 million, write data into the alarm table and notify the platform maintenance personnel to check and deal with it.

Further, the intelligent early warning through the shell script described in step S4 of the mass data processing method based on the heterogeneous data source of the present invention includes:

S41. Write an intelligent early warning shell script, and then create the xtyj.sh file at /home/opt;

S42. query the disk space, if the disk space occupancy rate reaches 85%, then send the message of XT_YJ_TOPIC to Kafka, the message content includes server ip, cpu and disk usage;

S43. Set a scheduled task for the intelligent early warning shell script, and execute the script at 1:00 every day;

S44. The back-end code monitors the message of Kafka corresponding to the topic XT_YJ_TOPIC, and after receiving the message, sends an early warning message to the system administrator;

S45. After receiving the warning message, the system administrator checks the abnormality of the server and performs corresponding processing.

After passing the above solution, if the system still has server pressure or database pressure that cannot be handled, it will be inspected on a daily basis and intelligently send an early warning message to the system administrator.

In the second aspect, the present invention also provides a massive data processing system based on heterogeneous data sources, and the processing system includes:

The storage module is used to write the service call data into storage, and the storage includes mongodb and Elasticsearch;

A query module, used to query service call data;

The cleaning module is used to regularly clean up data files;

The early warning module is used for intelligent early warning through shell scripts.

In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above-mentioned massive data processing method based on heterogeneous data sources are realized.

In summary, the massive data processing method based on heterogeneous data sources in the present invention has the following advantages:

(1) The method of the present invention can support the rapid expansion of the service system, greatly increase the concurrency of the interface, support the call of the billion-level interface, and can be used for the storage and call of massive data; and support the expansion of the system without the perception of external users. During the expansion, the interface can be call normally.

(2) The method of the present invention adopts multi-data source engine collaborative query, supports cross-data source query, is applicable to various query scenarios, and improves real-time data query capability and multi-dimensional analysis capability.

(3) The method of the present invention can intelligently clean up useless stored data and files in a timely and rapid manner, without manual inspection and cleaning for operation and maintenance, and reduces labor and machine costs.

(4) The method of the present invention provides an intelligent early warning function, and when the pressure on the database and server suddenly increases sharply, an early warning can be sent to the platform maintenance personnel in the form of email or short message.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings that need to be used in the embodiments of the present invention are briefly introduced below. Obviously, the following drawings are only some embodiments recorded in the present invention. As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

Fig. 1 is the overall implementation flowchart of the method of the present invention.

Fig. 2 is a block diagram of a service call data storage solution in the method of the present invention.

Fig. 3 is a flow chart of the service call data storage process in the method of the present invention.

Fig. 4 is a block diagram of a service call data query scheme in the method of the present invention.

Fig. 5 is a block diagram of a data timing cleaning scheme in the method of the present invention.

Fig. 6 is a flow chart of the intelligent early warning process in the method of the present invention.

Fig. 7 is a schematic diagram of the composition and structure of the data processing system of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention can also be implemented or applied through other different specific implementation modes, and the details in this specification can also be based on different viewpoints Various modifications or changes may be made without departing from the spirit of the invention.

Simultaneously, it should be understood that the protection scope of the present invention is not limited to the following specific embodiments; protected range.

Embodiment: A method for processing massive data based on heterogeneous data sources

As shown in Figure 1, this method comprises the following steps:

S1. Service call data is written to storage

Hypothetical background:

The number of interfaces provided by the service platform is as follows, divided into high-concurrency call interfaces and low-concurrency call interfaces:

mysql data volume:

The amount of statistical data generated (items/day) = total number of services * 24 * 1.5 ≈ 11w pieces;

The amount of statistical data generated (articles/year) = the amount of statistical data generated (articles/day)*365≈4000w articles.

mongodb data volume:

Detailed data volume (per day) = number of high-concurrency interfaces * average daily call volume * data volume + number of low-concurrency interfaces * average daily call volume * data volume

＝5000w*10*1kb+1w*3000*1kb

≈500G.

ElasticaSearch data volume:

Monthly data volume = data volume (per day) * 30 ≈ 15T;

Quarterly data volume = data volume (per day) * 90 ≈ 45T;

Annual data volume = data volume (per day) * 365 ≈ 180T.

As shown in Figure 2 and Figure 3, the service call data storage process includes the following steps:

S11. Service call data is written into mongodb asynchronously

Asynchronously write service call data, such as API service call time, apiId, application name, parameters, time-consuming, error information, error code information record json string, etc., into mongodb for data statistics query.

mongodb table by day

Create a new collection for mongodb every day, start a scheduled task at 23:59 every day, create a new mongodb table, write the data of the day into the data table of the day, and only keep the data of 2 days.

S12. Service call data is asynchronously written to Elasticsearch

Asynchronously write the specific information of the above service call data into Elasticsearch for user query.

Elasticsearch generates new indexes on a monthly basis

Create templates to roll out new indexes for Elasticsearch based on templates and policies every month.

S13. Save statistics to mysql

Every 10 seconds, the mongodb data is counted according to the dimensions of year, month, day, and hour, the number of interface calls, the average time spent, the number of failures, and the number of calls, and the statistical data is saved to the mysql statistics table;

(1) Synchronize statistical data insertAndUpdateHourData by hour;

(2) Synchronize statistical data insertAndUpdateDayData by day;

(3) Synchronize statistical data insertAndUpdateMonthData on a monthly basis;

(4) Synchronize statistical data insertAndUpdateYearData by year.

S14.mysql statistics table by year

A new data table is created for mysql every year, and a new table is created by a scheduled task at the end of each year. The new table synchronizes the table data of the previous month, and the table data of the next month is written into the new table. Create a new data table at 23:00:00 on the last day of each year. The table name is dytj_current date year. At the same time, the December data is synchronized to the new data table. On February 1st every year, the scheduled task cleans up the new table in December of the previous year The data.

expansion

When the system is running, after the data file cleaning solution is applied, if the pressure on the database and server suddenly increases sharply, and the resource space is insufficient, the intelligent early warning function will send an early warning to the platform maintenance personnel by email, SMS, etc., prompting which server and which application have resources insufficient. For mysql, mongodb, Elasticsearch, and application servers, different solutions are adopted for capacity expansion or disposal.

(1)mysql

For mysql, through the mysql sub-table strategy, when there is insufficient resource space, add machines to mysql, and store historical data and current data in sub-databases;

(2) mongodb

For mongodb, mongodb only stores temporary data, as a temporary storage middleware, when there is insufficient resource space, only one day of data is kept, and other data in the intermediate table are deleted;

(3) Elastic search

For Elasticsearch, evaluate the server resources that need to be increased, perform horizontal expansion, and calculate the number of machines that need to be expanded according to the call volume:

Daily data volume = ∑ daily call volume of statistical form service * 1K;

If you need to keep data for 1 year, the data storage capacity is:

1-year data volume = data volume per day * 365.

(4) Application server

For the application server, according to the total concurrency and the concurrency supported by a single server configuration, evaluate the server resources that need to be increased, and perform horizontal expansion.

Total concurrency (per second) = call volume per hour stored in the statistics table/3600.

S2. Query service call data

As shown in Figure 4, service call data query includes single-table query and cross-data source query. The specific process is as follows:

(1) Single table query (simple query)

(1) Query the statistical information of API service calls. By querying the mysql statistical table, you can directly obtain the statistical data information in the dimensions of year, month, day, and hour;

(2) Query the API service call data information, and obtain the specific data information of the API service name, call time, application name, data content, parameters, time-consuming, and error code by querying Elasticsearch;

It also supports aggregated analysis based on the data information obtained from the query, such as the analysis of statistical information such as error data distribution, error type distribution, and the most error-reported time period.

(2) Cross-data source query (complex query)

Querying complex business scenarios requires a joint query of mysql, Elasticsearch, and mongodb. By using Presto, cross-data source queries are realized.

(1) Configure the catalog of Presto, find the catalog directory in the Presto installation directory, then add the connector file in the catalog directory, create mysql.properties, mongodb.properties, Elasticsearch.properties, and configure the connector information, including address, user name , password, etc.;

(2) Write mixed query sql to complete cross-data source query. The method of writing mixed query sql is the same as that of ordinary sql. The table name of the query is written as catalog.schame.tableName, and catalog is the name of properties, as shown in the following table:

data source type catalog Table Name mysql mysql mysql.library name.table name mongodb mongodb mongodb.library name.table name Elasticsearch Elasticsearch Elasticsearch. library name. table name

S3. Regularly clean up data files

As shown in Figure 5, data file cleaning includes the following processes:

(1) mongodb data cleaning

Set a scheduled data deletion task that is cleaned up on a daily basis. The data is valid for 2 days, and the mongodb data of the previous 2 days are deleted every day (collection name is dyrz_collection of the date of the previous 2 days);

(2) Elasticsearch data cleaning

①Elasticsearch synchronously sets the data timing deletion task when creating the index, the data is valid for 1 year, and the data is automatically deleted when it expires;

②Manually write a shell script for detection and put it into the server to check the remaining disk space regularly every day. When the remaining disk space is less than 10%, delete files according to the importance and index creation time strategy until the disk is left Space up to 80%;

(3) mysql data cleaning

① Set up a scheduled deletion task for data cleaned up on a yearly basis. The data is valid for 3 years, and the data statistics table 3 years ago will be automatically cleaned up when it expires;

②Manually write a shell script for detection and put it into the server to check the remaining disk space regularly every day. When the remaining disk space is less than 10%, delete files according to the importance and data storage time strategy until the disk The remaining space exceeds 10%;

③ For the new event of mysql database, check the data volume of a single table every day. If the data volume of a single table exceeds 100 million, write data into the alarm table, and use emails and text messages to notify the platform maintenance personnel to check. If there is any abnormality, take corresponding measures to deal with.

S4. Intelligent early warning through shell script

As shown in Figure 6, intelligent early warning includes the following processes:

S41. Write an intelligent early warning shell script, and then create the xtyj.sh file at /home/opt;

S42. query the disk space, if the disk space occupancy rate reaches 85%, then send the message of XT_YJ_TOPIC to Kafka, the message content includes server ip, cpu and disk usage;

S43. Set a scheduled task for the intelligent early warning shell script, execute the script at 1:00 every day,

0 1 *** /home/opt/xtyj.sh;

S44. The back-end code listens to the message of Kafka corresponding to the topic XT_YJ_TOPIC, and after receiving the message, sends an email, a short message and other early warning messages to the system administrator;

S45. After receiving the warning message, the system administrator checks the abnormality of the server and performs corresponding processing.

A massive data processing system based on heterogeneous data sources, as shown in Figure 7, includes:

The storage module is used to write the service call data into storage, and the storage includes mongodb and Elasticsearch;

A query module, used to query service call data;

The cleaning module is used to regularly clean up data files;

The early warning module is used for intelligent early warning through shell scripts.

The above descriptions are only examples of the present invention, and are not intended to limit the present invention. Various modifications and variations of the present invention will occur to those skilled in the art. Any modification, substitution, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims (10)

1. A massive data processing method based on heterogeneous data sources, characterized in that the method comprises: S1. Service call data is written into storage, and the storage includes mongodb and Elasticsearch; S2. Query service call data; S3. Regularly clean up data files; S4. Carry out intelligent early warning through shell script. 2. The method for processing massive data based on heterogeneous data sources according to claim 1, wherein the service call data in step S1 includes API service call time, apiId, application name, parameters, time-consuming, and error information , Error code information record json string. 3. The method for processing massive data based on heterogeneous data sources according to claim 2, characterized in that, the service calling data in step S1 is written into storage, comprising: S11. Asynchronously write the specific information of the service call data into mongodb for data statistics query, and create a new collection for mongodb every day; S12. Asynchronously write the specific information of the service call data into Elasticsearch for user query, and create a new index for Elasticsearch every month; S13. Make statistics on mongodb data every 10 seconds, according to the dimensions of year, month, day, and hour, the amount of interface calls, average time-consuming, number of failures, and number of calls, and save the statistical data to the mysql statistics table; S14. Create a new data table for mysql every year. 4. The method for processing massive data based on heterogeneous data sources according to claim 3, characterized in that, when the pressure on the database and/or server suddenly increases sharply during the operation of the system, and the resource space is insufficient, the following method is adopted: Expansion: (1) For mysql, through the mysql sub-table strategy, when there is insufficient resource space, add machines to mysql, and store historical data and current data in sub-databases; (2) For mongodb, mongodb only stores temporary data, as a temporary storage middleware, when there is insufficient resource space, only keep 1 day of data, and delete the intermediate table data; (3) For Elasticsearch, evaluate the server resources that need to be increased, calculate the number of machines that need to be expanded according to the call volume, and perform horizontal expansion; (4) For the application server, evaluate the server resources that need to be increased according to the total concurrency and the concurrency supported by a single server configuration, and perform horizontal expansion. 5. The method for processing massive data based on heterogeneous data sources according to claim 3, wherein the query service call data in step S2 includes: (1) Single table query ①Query the statistical information of API service calls, and obtain statistical data information in the dimensions of year, month, day, and hour by querying the mysql statistical table; ②Query the API service call data information, and obtain the data information of the API service name, call time, application name, data content, parameters, time-consuming, and error codes by querying Elasticsearch; (2) Cross data source query ① Configure the catalog of Presto, find the catalog directory in the Presto installation directory, then add the connector file in the catalog directory, create mysql.properties, mongodb.properties, Elasticsearch.properties, and configure the connector information; ② Write the sql of the mixed query, write the table name of the query as catalog.schame.tableName, and complete the cross-data source query. 6. The method for processing massive amounts of data based on heterogeneous data sources according to claim 5, wherein the query API service call data information in the single table query also includes data information obtained based on the query for aggregate analysis; The aggregation analysis includes error data distribution analysis, error type distribution analysis, and most error reporting time period analysis. 7. The method for processing massive data based on heterogeneous data sources according to claim 3, wherein the regular cleaning of data files in step S3 includes: (1) mongodb data cleaning Set a scheduled deletion task for data cleaned up on a daily basis. The data is valid for 2 days, and the mongodb data of the previous 2 days are deleted every day; (2) Elasticsearch data cleaning ①Elasticsearch synchronously sets the data timing deletion task when creating the index, the data is valid for 1 year, and the data is automatically deleted when it expires; ②Manually write a shell script for detection and put it into the server to check the remaining disk space regularly every day. When the remaining disk space is less than 10%, delete files according to the importance and index creation time strategy until the disk is left Space up to 80%; (3) mysql data cleaning ① Set up a scheduled deletion task for data cleaned up on a yearly basis. The data is valid for 3 years, and the data statistics table 3 years ago will be automatically cleaned up when it expires; ②Manually write a shell script for detection and put it into the server to check the remaining disk space regularly every day. When the remaining disk space is less than 10%, delete files according to the importance and data storage time strategy until the disk The remaining space exceeds 10%; ③ For the new event of the mysql database, check the data volume of a single table every day. If the data volume of a single table exceeds 100 million, write data into the alarm table and notify the platform maintenance personnel to check and deal with it. 8. the massive data processing method based on heterogeneous data source according to claim 3, is characterized in that, described in step S4 carries out intelligent early warning by shell script, comprises: S41. Write an intelligent early warning shell script, and then create the xtyj.sh file at /home/opt; S42. query the disk space, if the disk space occupancy rate reaches 85%, then send the message of XT_YJ_TOPIC to Kafka, the message content includes server ip, cpu and disk usage; S43. Set a scheduled task for the intelligent early warning shell script, and execute the script at 1:00 every day; S44. The back-end code monitors the message of Kafka corresponding to the topic XT_YJ_TOPIC, and after receiving the message, sends an early warning message to the system administrator; S45. After receiving the warning message, the system administrator checks the abnormality of the server and performs corresponding processing. 9. A massive data processing system based on heterogeneous data sources, characterized in that the processing system comprises: The storage module is used to write the service call data into storage, and the storage includes mongodb and Elasticsearch; A query module, used to query service call data; The cleaning module is used to regularly clean up data files; The early warning module is used for intelligent early warning through shell scripts. 10. A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for processing massive data based on heterogeneous data sources according to any one of claims 1-8 is realized A step of.

Images