CN111865725B - Flow consumption analysis method and system based on log - Google Patents
Flow consumption analysis method and system based on log Download PDFInfo
- Publication number
- CN111865725B CN111865725B CN202010744240.8A CN202010744240A CN111865725B CN 111865725 B CN111865725 B CN 111865725B CN 202010744240 A CN202010744240 A CN 202010744240A CN 111865725 B CN111865725 B CN 111865725B
- Authority
- CN
- China
- Prior art keywords
- data
- analysis
- traffic
- network traffic
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 150
- 238000005206 flow analysis Methods 0.000 claims abstract description 52
- 238000012546 transfer Methods 0.000 claims abstract description 24
- 238000004590 computer program Methods 0.000 claims description 9
- 230000002688 persistence Effects 0.000 claims description 7
- 230000002159 abnormal effect Effects 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 30
- 238000007405 data analysis Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000003068 static effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003012 network analysis Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a flow consumption analysis method, a flow consumption analysis system, computer equipment and a computer readable storage medium based on logs, wherein a configuration file is maintained in advance, and safe shell protocol channels and hypertext transfer protocol connections of all hosts contained in an application cluster are established according to the configuration file; based on the secure shell protocol channel, sending a shell layer command for acquiring network traffic analysis subdata to all hosts contained in the application cluster; receiving network flow analysis subdata returned by the hypertext transfer protocol connection based on the shell layer command; according to the network traffic analysis subdata, counting network traffic analysis data of the application cluster, wherein the network traffic analysis data comprise traffic byte data; and analyzing the flow byte data by using a preset rule to obtain a flow analysis result, positioning the abnormal flow problem very quickly, and providing huge assistance for system cost control and malicious access.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a flow consumption analysis method and system based on logs.
Background
The most typical method is a content distribution network technology, that is, all static resources and other dynamic resources with higher consumption flow are cached to each node nearest to a user to reduce the transmission distance of the public network flow, so as to reduce the cost of the public network flow, but how to accurately take the resource information with higher consumption? It is necessary to discard all static resources to the content distribution node, but as resources are continuously increased, if a certain artificial error causes individual resources to be discarded to the content distribution node, a huge loss may be caused. The traditional traffic consumption analysis mode generally captures the times and traffic of accessing a source IP on network equipment, but this cannot help to locate which application cluster consumes the traffic, and the traffic cannot be analyzed in a slave manner even when a plurality of systems share the same public network IP; or a data packet analysis tool is deployed by high technology and manual components to analyze specific consumption conditions, so that a low-cost and more accurate-analysis method for analyzing the consumption of the stream data is needed.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, a computer device, and a computer-readable storage medium for analyzing traffic consumption based on a log, so as to solve the problems of inaccurate traffic data consumption analysis and high cost.
The embodiment of the invention solves the technical problems through the following technical scheme:
a log-based traffic consumption analysis method, comprising:
maintaining a configuration file in advance, and establishing a secure shell protocol channel and a hypertext transfer protocol connection with all hosts contained in an application cluster according to the configuration file, wherein the configuration file comprises address information of all hosts and login information of all hosts;
based on the secure shell protocol channel, sending a shell layer command for acquiring network traffic analysis subdata to all hosts contained in the application cluster;
receiving network flow analysis subdata returned by the hypertext transfer protocol connection based on the shell layer command;
according to the network traffic analysis subdata, network traffic analysis data of the application cluster are counted, wherein the network traffic analysis data comprise traffic byte data;
analyzing the traffic byte data by using a preset rule to obtain a traffic analysis result;
and outputting the flow analysis result.
Further, the counting network traffic analysis data of the application cluster according to the network traffic analysis subdata, where the network traffic analysis data includes traffic byte data includes:
writing the network flow quantum data into a blank file in a preset format to obtain a network flow data file;
and counting the total data volume of flow byte data in the network flow data file based on the network flow data file, wherein the total data volume is used as the network flow analysis data of the application cluster.
Further, the pre-maintaining the configuration file establishes a secure shell protocol channel and a hypertext transfer protocol connection with all hosts included in the application cluster according to the configuration file, where the configuration file includes address information of all hosts and login information of all hosts, and includes:
acquiring address information of all hosts in the application cluster from the configuration file;
and establishing security shell protocol channels of all hosts corresponding to the address information and hypertext transfer protocol connection corresponding to the address information.
Further, the calculating the traffic byte data by using a preset rule to obtain a traffic analysis result includes:
and acquiring flow byte data of the application cluster in unit time as the flow analysis result.
Further, based on the secure shell protocol channel, sending a shell command for obtaining the network traffic analysis sub-data to all hosts included in the application cluster includes:
sending the shell layer command for acquiring the network traffic analysis subdata to all hosts contained in the application cluster through the secure shell protocol channel at preset time;
the receiving, based on the shell layer command, network traffic analysis subdata returned by the hypertext transfer protocol connection includes:
receiving new network flow analysis subdata returned according to the shell layer command;
the statistics of the network traffic analysis data of the application cluster according to the network traffic analysis subdata includes:
and updating the network traffic analysis data according to the new network traffic analysis subdata.
Further, the method comprises:
and storing the flow analysis result into a data persistence system of the block chain.
Further, the method further comprises:
and responding to a preset condition that the flow analysis result accords with, and outputting alarm information.
The invention also provides a flow consumption analysis system based on the log, which comprises:
the system comprises a channel establishing module, a configuration module and a control module, wherein the channel establishing module is used for maintaining configuration files in advance, establishing the connection with the security shell protocol channels and the hypertext transfer protocol of all hosts contained in an application cluster according to the configuration files, and the configuration files comprise the address information of all hosts and the login information of all hosts;
the shell layer command sending module is used for sending a shell layer command for acquiring network traffic analysis subdata to all hosts contained in the application cluster based on the secure shell protocol channel;
the network flow analysis subdata acquisition module is used for receiving network flow analysis subdata returned by the hypertext transfer protocol connection based on the shell layer command;
the network traffic analysis data statistics module is used for performing statistics on the network traffic analysis data of the application cluster according to the network traffic analysis subdata, wherein the network traffic analysis data comprises traffic byte data;
the flow analysis result generation module is used for analyzing the flow byte data by using a preset rule to obtain a flow analysis result;
and the flow analysis result output module is used for outputting the flow analysis result.
In order to achieve the above object, the present invention also provides a computer device, which includes a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the log-based traffic consumption analysis method as described above when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored therein a computer program, which is executable by at least one processor to cause the at least one processor to perform the steps of log-based traffic consumption analysis as described above.
According to the flow consumption analysis method, the flow consumption analysis system, the computer equipment and the computer readable storage medium based on the logs, network flow analysis word data of all hosts contained in an application cluster are directly obtained; according to the network flow analysis word data, network flow analysis data are counted, and the network flow analysis data comprise time data and flow byte data; calculating the network traffic analysis data by using a preset rule to obtain a traffic analysis result; the flow analysis result is output, and the number of related flow bytes is directly returned from the network flow log without filtering by other log processing tools, so that a large amount of log synchronization processes are eliminated, and the data accuracy is high compared with the method of reading data from a log cloud or reading data from other intermediate log systems; the method can visually see the arrangement of the traffic consumption application clusters in any time period from the page, thereby quickly positioning which application cluster, consuming bandwidth resources when, and being also used for detecting traffic abnormality, quickly positioning the problem of traffic abnormality, and providing huge assistance for system cost control and malicious access.
The invention is described in detail below with reference to the drawings and specific examples, but the invention is not limited thereto.
Drawings
FIG. 1 is a flowchart illustrating steps of a log-based traffic consumption analysis method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a specific process of pre-maintaining the configuration file, establishing a secure shell protocol channel and a hypertext transfer protocol connection with all hosts included in the application cluster according to the configuration file, where the configuration file includes address information of all hosts and login information of all hosts;
fig. 3 is a schematic diagram of a specific flow of the step of counting network traffic analysis data of the application cluster according to the network traffic analysis sub-data, where the network traffic analysis data includes traffic byte data;
fig. 4 is a schematic diagram illustrating a specific process of counting network traffic analysis data of the application cluster according to the network traffic analysis subdata, where the network traffic analysis data includes traffic byte data according to another embodiment of the present invention;
FIG. 5 is a diagram illustrating program modules according to a second embodiment of the present invention;
FIG. 6 is a diagram of a hardware structure of a third embodiment of the computer apparatus according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Technical solutions between the embodiments may be combined with each other, but must be based on the realization of the technical solutions by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
Example one
Referring to fig. 1, a flowchart illustrating steps of a method for analyzing log traffic consumption according to an embodiment of the present invention is shown. It is to be understood that the flow charts in the embodiments of the present method are not intended to limit the order in which the steps are performed. The following description is given by taking a computer device as an execution subject, specifically as follows:
step S100: the method comprises the steps of maintaining configuration files in advance, establishing security shell protocol channels and hypertext transfer protocol connections with all hosts contained in an application cluster according to the configuration files, wherein the configuration files comprise address information of all the hosts and login information of all the hosts.
Application cluster refers to a cluster in which many hosts are centralized to perform the same service, but the client appears as if there is only one host, and multiple hosts are included in one application cluster. The application cluster can utilize a plurality of computers to perform parallel computation so as to obtain high computation speed, and also can use a plurality of computers to perform backup so that any one computer breaks down the whole system and can still normally run.
In an exemplary embodiment, as shown in fig. 2, step S100 further includes:
step S101: acquiring address information of all hosts in the application cluster from the configuration file;
step S102: and establishing security shell protocol channels of all hosts corresponding to the address information and hypertext transfer protocol connection corresponding to the address information.
Specifically, in an exemplary embodiment, the method for establishing the secure shell protocol channels with all hosts included in the application cluster according to the configuration file includes: acquiring address information of all hosts in the application cluster and login information of all hosts from the configuration file; establishing security shell protocol channels of all hosts corresponding to the address information, and logging in all hosts in the application cluster by using the login information; establishing a hypertext transfer protocol connection with all hosts included in the application cluster comprises: and acquiring the address information of all the hosts from the configuration file, and establishing the hypertext transfer protocol connection corresponding to the address information.
Step S200: and sending a shell layer command for acquiring the network traffic analysis subdata to all the hosts contained in the application cluster based on the secure shell protocol channel.
Specifically, each host generates a network traffic log during operation. In one embodiment, the network traffic log refers to an access log (acc. log) of all hosts in an application cluster, wherein access. log refers to a file for a server to record each request, and is generally used for analyzing the access condition of a website. For example: the flow of the website, the visit volume of the website, the network address of the sender, the network address of the receiver, the request type, the request return time, the flow byte returned according to the request, and the like.
Because the network flow log includes the flow of the website and the access amount of the website, and also includes data such as the network address of the sender, the network address of the receiver, the type of the request, the request return time, and the flow byte returned according to the request, we do not need such much data during analysis, and therefore, the network flow analysis subdata needs to be identified and extracted. In this embodiment, the network analysis subdata refers to byte data of the request return time and the flow returned according to the request of each host.
In one embodiment, the Shell command is used for extracting time data and flow byte data of network flow quantum logs of all hosts, and the Shell language is a program written in C language and is a bridge for users to use Linux. The Shell language is both a command language and a programming language. Specifically, by adding a shell script in advance, the shell script is executed so as to extract time data and traffic byte data of network traffic logs of all hosts.
Step S300: and receiving network flow analysis subdata returned by the hypertext transfer protocol connection based on the shell layer command.
Specifically, in some embodiments of this embodiment, a specific process of extracting time data and traffic byte data of a network stream quantum log by using a shell script is as follows:
firstly, adding a catalog of the network flow quantum log in a shell script, namely a position where the generated network flow self-log is stored, setting extraction keywords such as time, accbyte and the like in the shell script, positioning the shell script to the position of the network flow quantum log through the catalog T when executing the shell script, extracting time data and flow byte data from the network flow quantum log according to the keywords, and finally obtaining data required by network flow data analysis.
Step S400: and according to the network traffic analysis subdata, counting network traffic analysis data of the application cluster, wherein the network traffic analysis data comprises traffic byte data.
In an exemplary embodiment, referring to fig. 3, step S400 may further include:
step S401: writing the network flow quantum data into a blank file in a preset format to obtain a network flow data file;
specifically, after the network traffic subdata is identified and extracted, the extracted network traffic subdata is written into a blank file according to a preset format, wherein the preset format can comprise two columns of data, one column of data is request return time, and the other column of data is traffic byte data returned according to the request. The blank files may be text files, WORD files, and EXCEL files, and the present scheme is not limited herein. Compared with the method for acquiring flow analysis data from other intermediate log systems, the method has the advantages that by directly acquiring the network flow sub-logs of all the hosts without filtering by other log processing tools, a large amount of log synchronization processes are eliminated, and on one hand, the efficiency and the data accuracy are improved; and on the other hand, the traceability of subsequent data is ensured.
Step S402: and counting the total data volume of flow byte data in the network flow data file based on the network flow data file, wherein the total data volume is used as the network flow analysis data of the application cluster.
Specifically, the counting of the total data volume of the traffic byte data in the network traffic data file refers to calculating the sum of the traffic byte data returned by all the hosts in the application cluster according to the request within the same request return time, and in an exemplary embodiment, the counted total data volume of the traffic byte data is also written into another blank file to obtain the network traffic data comprehensive data file.
An example network traffic data integrated data file is as follows:
the first column of data is request return time, the second column of data is flow byte data returned according to the request, wherein the flow byte data returned according to the request can be counted once every minute by taking minutes as a unit according to the request return time; the byte data of the traffic returned by the request may also be counted every second, which is expressed by taking the second as a unit, and the scheme is not limited herein. In this embodiment, the above network traffic integrated data file represents, in units of one minute of request return time: when the time is 10:05:00, the sum of the traffic byte data returned by all the hosts in the application cluster is 3568921, when the time is 10:06:00, the sum of the traffic byte data returned by all the hosts in the application cluster is 3523181, and when the time is 10:07:00, the sum of the traffic byte data returned by all the hosts in the application cluster is 2968719.
In another embodiment, as shown in fig. 4, the method for analyzing the log flow consumption further includes:
step S110: and sending the shell layer command for acquiring the network traffic analysis subdata to all hosts contained in the application cluster through the secure shell protocol channel at a preset period.
The shell layer command for acquiring the network traffic analysis subdata is generated and sent by a specified server according to a preset period, the preset period can be set according to the requirements of developers, wherein the set unit of the period can be month, week, day, hour, minute and the like, the preset period for generating and sending the network traffic quantum log acquisition command can be one week, one day, every 6 hours, every 30 minutes and the like, and the preset period can be specifically determined according to the requirements of the developers and the performance of the server, and the scheme is not limited herein. In this embodiment, the preset period is 3 minutes, and data acquisition is delayed for 3 minutes after the network traffic log is generated, so that on one hand, the accuracy of the data is ensured, and on the other hand, the traceability of subsequent data is ensured.
Step S111: and receiving new network flow analysis subdata returned according to the shell layer command.
Step S112: and updating the network traffic analysis data according to the new network traffic analysis subdata.
Step S500: and analyzing the network traffic analysis data by using a preset rule to obtain a traffic analysis result.
In an exemplary embodiment, step S500 may further include:
and acquiring flow byte data of the application cluster in unit time as the flow analysis result.
Specifically, in the exemplary embodiment, the average flow data is bytes of flow data per minute. The specific calculation formula is as follows: and the flow data per minute is the sum of byte data of the flow 8/1024/1024/60, and the formula shows that the bandwidth per minute is calculated according to the network flow analysis data, and finally the average flow data of the application cluster is obtained.
Step S600: and outputting the flow analysis result.
Specifically, after the traffic data analysis result is obtained, the traffic data analysis result is sent to the front end, and is displayed in an exemplary graph or table manner on a display interface (for example, a WEB interface for monitoring the traffic data in real time), so that the user can more intuitively know the traffic data analysis result.
In an embodiment, the obtained traffic analysis result is displayed on a front-end interface of a user side in a form of a histogram, when the traffic analysis result is displayed in the form of the histogram, an abscissa of the histogram is time data, each time data is provided with a plurality of application clusters, and an ordinate is a traffic analysis result, that is, in the time, traffic byte data corresponding to each application cluster is provided, so that a technician can visually find the traffic analysis result corresponding to each application cluster in each time period, and further prompt the technician to excessively occupy the application clusters of network bandwidth in advance, so that the technician can process resources corresponding to the application clusters in advance, and avoid too large network traffic consumption.
In another embodiment, the method may further include:
and storing the flow analysis result into a data persistence system of the block chain.
Specifically, in order to facilitate a technician to check a traffic analysis result at any time, in some embodiments of this embodiment, after obtaining a data traffic analysis result, the traffic analysis result is output, and the traffic analysis result may be stored in a data persistence system, where the data persistence system may be a database system, and in addition, a network traffic sub-log, a network data file, and a network data integrated file may all be stored in the data persistence system, so as to facilitate subsequent statistics and verification.
And uploading the flow analysis result to the block chain can ensure the safety of the block chain. The user equipment may download the traffic analysis result from the blockchain to verify whether the traffic analysis result is tampered. The blockchain referred to in this example is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
In another embodiment, the method may further include:
and responding to a preset condition that the flow analysis result accords with, and outputting alarm information.
Specifically, in addition to displaying the flow data analysis result on the front-end interface, the embodiment of the present invention may also monitor the flow data analysis result, and by presetting the trigger condition, when the flow data analysis result is abnormal, the preset alarm condition may be triggered to send an alarm. And when the average flow data calculated in the flow data analysis result is higher than the preset normal average flow data, triggering an alarm. Wherein the preset threshold is set by a person skilled in the art empirically. In another embodiment, the preset threshold is obtained by analyzing historical flow data. The present solution is not limited herein.
In this embodiment, network traffic logs of all hosts included in an application cluster are obtained; according to the network flow logs, network flow analysis data are counted, and the network flow analysis data comprise time data and flow byte data; calculating the network traffic analysis data by using a preset rule to obtain a traffic analysis result; and outputting the flow analysis result. Compared with the method for acquiring flow analysis data from other intermediate log systems, the method has the advantages that by directly acquiring the network flow sub-logs of all the hosts without filtering by other log processing tools, a large number of log synchronization processes are eliminated, and on one hand, the efficiency and the data accuracy are improved; on the other hand, the traceability of subsequent data is ensured; the traffic analysis result is displayed on a WEB interface, so that the traffic consumption application cluster arrangement in any time period can be visually seen from the page, the application cluster can be very quickly positioned, the bandwidth resource is consumed when, meanwhile, the traffic anomaly detection method can also be used for detecting traffic anomaly, the problem of positioning traffic anomaly is very quick, and huge assistance is provided for system cost control and malicious access.
Example two
Continuing to refer to FIG. 5, a block diagram of a log-based traffic consumption analysis system of the present invention is shown. In this embodiment, the log-based traffic consumption analysis system 20 may include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to implement the present invention and implement the above-described log-based traffic consumption analysis method. The program module referred to in the embodiments of the present invention refers to a series of computer program instruction segments capable of performing specific functions, and is more suitable for describing the execution process of the log-based traffic consumption analysis system 20 in a storage medium than the program itself. The following description will specifically describe the functions of the program modules of the present embodiment:
a channel establishing module 200, configured to maintain a configuration file in advance, and establish a secure shell protocol channel and a hypertext transfer protocol connection with all hosts included in an application cluster according to the configuration file, where the configuration file includes address information of all hosts and login information of all hosts;
in an exemplary embodiment, the channel setup module 200 is further configured to:
acquiring address information of all hosts in the application cluster from the configuration file;
and establishing security shell protocol channels of all hosts corresponding to the address information and hypertext transfer protocol connection corresponding to the address information.
A shell layer command sending module 202, configured to send a shell layer command for obtaining network traffic analysis sub-data to all hosts included in the application cluster based on the secure shell protocol channel;
in an exemplary embodiment, shell command sending module 202 is further configured to:
and sending the shell layer command for acquiring the network traffic analysis subdata to all hosts contained in the application cluster through the secure shell protocol channel at preset time.
A network traffic analysis subdata obtaining module 204, configured to receive network traffic analysis subdata returned by the hypertext transport protocol connection based on the shell layer command;
in an exemplary embodiment, the network traffic analysis sub-data obtaining module 204 is further configured to:
and receiving new network flow analysis subdata returned according to the shell layer command.
A network traffic analysis data statistics module 206, configured to, according to the network traffic analysis subdata, count network traffic analysis data of the application cluster, where the network traffic analysis data includes traffic byte data;
in an exemplary embodiment, the network traffic analysis data statistics module 206 is further configured to:
and updating the network traffic analysis data according to the new network traffic analysis subdata.
In an exemplary embodiment, the network traffic analysis data statistics module 206 is further configured to:
writing the network flow quantum data into a blank file in a preset format to obtain a network flow data file;
and counting the total data volume of flow byte data in the network flow data file based on the network flow data file, wherein the total data volume is used as the network flow analysis data of the application cluster.
A traffic analysis result generation module 208, configured to analyze the network traffic analysis data using a preset rule to obtain a traffic analysis result;
in the exemplary embodiment, traffic analysis result generation module 208 is further configured to:
and acquiring flow byte data of the application cluster in unit time as the flow analysis result.
And a flow analysis result output module 210, configured to output the flow analysis result.
In an exemplary embodiment, the traffic analysis result output module 210 is further configured to:
and storing the flow analysis result into a data persistence system of the block chain.
In an exemplary embodiment, the traffic analysis result output module 210 is further configured to:
and responding to a preset condition that the flow analysis result accords with, and outputting alarm information.
EXAMPLE III
Fig. 6 is a schematic diagram of a hardware architecture of a computer device according to a third embodiment of the present invention. In the present embodiment, the computer device 2 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a preset or stored instruction. The computer device 2 may be a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers), and the like. As shown in FIG. 6, the computer device 2 includes, but is not limited to, at least a memory 21, a processor 22, a network interface 23, and a log-based traffic consumption analysis system 20, which may be communicatively coupled to each other via a system bus. Wherein:
in this embodiment, the memory 21 includes at least one type of computer-readable storage medium including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 21 may be an internal storage unit of the computer device 2, such as a hard disk or a memory of the computer device 2. In other embodiments, the memory 21 may also be an external storage device of the computer device 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the computer device 2. Of course, the memory 21 may also comprise both internal and external memory units of the computer device 2. In this embodiment, the memory 21 is generally used for storing an operating system installed in the computer device 2 and various types of application software, such as the program codes of the log-based traffic consumption analysis system 20 described in the above embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
The network interface 23 may comprise a wireless network interface or a wired network interface, and the network interface 23 is generally used for establishing communication connection between the computer device 2 and other electronic apparatuses. For example, the network interface 23 is used to connect the computer device 2 to an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 2 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, and the like.
It is noted that fig. 6 only shows the computer device 2 with components 20-23, but it is understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
In this embodiment, the log-based traffic consumption analysis system 20 stored in the memory 21 may be further divided into one or more program modules, and the one or more program modules are stored in the memory 21 and executed by one or more processors (in this embodiment, the processor 22) to complete the present invention.
For example, fig. 5 is a schematic diagram illustrating program modules of a second embodiment of implementing the log-based traffic consumption analysis system 20, in this embodiment, the log-based traffic consumption analysis system 20 may be divided into a channel establishing module 200, a shell command sending module 202, a network traffic analysis sub-data obtaining module 204, a network traffic analysis data statistics module 206, a traffic analysis result generating module 208, and a traffic analysis result outputting module 210. The program modules referred to herein are a series of computer program instruction segments capable of performing specific functions, and are more suitable than programs for describing the execution process of the log-based traffic consumption analysis system 20 in the computer device 2. The specific functions of the channel establishing module 200 and the traffic analysis result output module 210 have been described in detail in the above embodiments, and are not described herein again.
Example four
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of the present embodiment is used for storing a log-based traffic consumption analysis system 20, and when being executed by a processor, the log-based traffic consumption analysis method described in the above embodiments is implemented.
The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages and disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes performed by the present invention or directly or indirectly applied to other related technical fields are also included in the scope of the present invention.
Claims (10)
1. A log-based traffic consumption analysis method is characterized by comprising the following steps:
maintaining a configuration file in advance, and establishing a secure shell protocol channel and a hypertext transfer protocol connection with all hosts contained in an application cluster according to the configuration file, wherein the configuration file comprises address information of all hosts and login information of all hosts;
sending a shell layer command for acquiring network traffic analysis subdata to all hosts contained in the application cluster based on the secure shell protocol channel; the network traffic analysis subdata comprises time data of a network traffic log and traffic byte data of the network traffic log, and the shell layer commands extract the network traffic analysis subdata from the network traffic logs of all the hosts;
receiving network flow analysis subdata returned by the hypertext transfer protocol connection based on the shell layer command;
according to the network traffic analysis subdata, counting network traffic analysis data of the application cluster, wherein the network traffic analysis data comprise traffic byte data;
analyzing the traffic byte data by using a preset rule to obtain a traffic analysis result;
and outputting the flow analysis result.
2. The log-based traffic consumption analysis method according to claim 1, wherein the counting network traffic analysis data of the application cluster according to the network traffic analysis subdata, the network traffic analysis data including traffic byte data includes:
writing the network flow quantum data into a blank file in a preset format to obtain a network flow data file;
and counting the total data volume of flow byte data in the network flow data file based on the network flow data file, wherein the total data volume is used as the network flow analysis data of the application cluster.
3. The log-based traffic consumption analysis method according to claim 2, wherein the calculating the traffic byte data by using the preset rule to obtain the traffic analysis result comprises:
and acquiring the traffic byte data of the application cluster in unit time as the traffic analysis result.
4. The log-based traffic consumption analysis method according to claim 3, wherein the pre-maintaining a configuration file, establishing a secure shell protocol channel and a hypertext transfer protocol connection with all hosts included in an application cluster according to the configuration file, and the configuration file includes address information of all hosts and login information of all hosts, including:
acquiring address information of all hosts in the application cluster from the configuration file;
and establishing security shell protocol channels of all hosts corresponding to the address information and hypertext transfer protocol connection corresponding to the address information.
5. The log-based traffic consumption analysis method of claim 4, wherein sending, based on the secure shell protocol channel, a shell command for obtaining network traffic analysis sub-data to all hosts included in the application cluster comprises:
sending the shell layer command for acquiring the network traffic analysis subdata to all hosts contained in the application cluster through the secure shell protocol channel at preset time;
the receiving, based on the shell layer command, network traffic analysis subdata returned by the hypertext transfer protocol connection includes:
receiving new network flow analysis subdata returned according to the shell layer command;
the counting network traffic analysis data of the application cluster according to the network traffic analysis subdata, wherein the network traffic analysis data including traffic byte data includes:
and updating the network traffic analysis data according to the new network traffic analysis subdata.
6. The log-based traffic consumption analysis method of claim 5, further comprising:
and storing the flow analysis result into a data persistence system of the block chain.
7. The log-based traffic consumption analysis method of claim 6, further comprising:
and responding to a preset condition that the flow analysis result accords with, and outputting alarm information.
8. A log-based traffic consumption analysis system, comprising:
the system comprises a channel establishing module, a configuration module and a control module, wherein the channel establishing module is used for maintaining configuration files in advance, establishing the connection with the security shell protocol channels and the hypertext transfer protocol of all hosts contained in an application cluster according to the configuration files, and the configuration files comprise the address information of all hosts and the login information of all hosts;
a shell layer command sending module, configured to send a shell layer command for obtaining network traffic analysis sub-data to all hosts included in the application cluster based on the secure shell protocol channel; the network traffic analysis subdata comprises time data of a network traffic log and traffic byte data of the network traffic log, and the shell layer commands extract the network traffic analysis subdata from the network traffic logs of all the hosts;
the network flow analysis subdata acquisition module is used for receiving network flow analysis subdata returned by the hypertext transfer protocol connection based on the shell layer command;
the network traffic analysis data statistics module is used for performing statistics on the network traffic analysis data of the application cluster according to the network traffic analysis subdata, wherein the network traffic analysis data comprises traffic byte data;
the traffic analysis result generation module is used for analyzing the traffic byte data by using a preset rule to obtain a traffic analysis result;
and the flow analysis result output module is used for outputting the flow analysis result.
9. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the computer program, carries out the steps of the log-based traffic consumption analysis method according to any of claims 1 to 7.
10. A computer-readable storage medium, having stored therein a computer program executable by at least one processor to cause the at least one processor to perform the steps of the log-based traffic consumption analysis method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010744240.8A CN111865725B (en) | 2020-07-29 | 2020-07-29 | Flow consumption analysis method and system based on log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010744240.8A CN111865725B (en) | 2020-07-29 | 2020-07-29 | Flow consumption analysis method and system based on log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111865725A CN111865725A (en) | 2020-10-30 |
| CN111865725B true CN111865725B (en) | 2022-09-23 |
Family
ID=72946041
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010744240.8A Active CN111865725B (en) | 2020-07-29 | 2020-07-29 | Flow consumption analysis method and system based on log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111865725B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104954189A (en) * | 2015-07-07 | 2015-09-30 | 上海斐讯数据通信技术有限公司 | Automatic server cluster detecting method and system |
| CN105049232A (en) * | 2015-06-19 | 2015-11-11 | 成都艾尔普科技有限责任公司 | Network information log audit system |
| CN109525645A (en) * | 2018-10-22 | 2019-03-26 | 郑州云海信息技术有限公司 | A kind of method and system for collecting the log of distributed storage cluster |
| CN110249321A (en) * | 2017-09-29 | 2019-09-17 | 甲骨文国际公司 | For the system and method that capture change data use from distributed data source for heterogeneous target |
| CN110781143A (en) * | 2019-11-05 | 2020-02-11 | 北纬通信科技南京有限责任公司 | Method and device for querying and extracting server logs |
| CN111092852A (en) * | 2019-10-16 | 2020-05-01 | 平安科技（深圳）有限公司 | Network security monitoring method, device, equipment and storage medium based on big data |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9178935B2 (en) * | 2009-03-05 | 2015-11-03 | Paypal, Inc. | Distributed steam processing |
| US9747592B2 (en) * | 2011-08-16 | 2017-08-29 | Verizon Digital Media Services Inc. | End-to-end content delivery network incorporating independently operated transparent caches and proxy caches |
-
2020
- 2020-07-29 CN CN202010744240.8A patent/CN111865725B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049232A (en) * | 2015-06-19 | 2015-11-11 | 成都艾尔普科技有限责任公司 | Network information log audit system |
| CN104954189A (en) * | 2015-07-07 | 2015-09-30 | 上海斐讯数据通信技术有限公司 | Automatic server cluster detecting method and system |
| CN110249321A (en) * | 2017-09-29 | 2019-09-17 | 甲骨文国际公司 | For the system and method that capture change data use from distributed data source for heterogeneous target |
| CN109525645A (en) * | 2018-10-22 | 2019-03-26 | 郑州云海信息技术有限公司 | A kind of method and system for collecting the log of distributed storage cluster |
| CN111092852A (en) * | 2019-10-16 | 2020-05-01 | 平安科技（深圳）有限公司 | Network security monitoring method, device, equipment and storage medium based on big data |
| CN110781143A (en) * | 2019-11-05 | 2020-02-11 | 北纬通信科技南京有限责任公司 | Method and device for querying and extracting server logs |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111865725A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111563016B (en) | Log collection and analysis method and device, computer system and readable storage medium | |
| CN112416728A (en) | Buried point data acquisition method and device, client device and readable storage medium | |
| CN112231271B (en) | Data migration integrity verification method, device, equipment and computer readable medium | |
| CN114500690B (en) | Interface data processing method and device, electronic equipment and storage medium | |
| CN111131221B (en) | Interface checking device, method and storage medium | |
| CN110046073B (en) | Log collection method and device, equipment and storage medium | |
| CN110610196A (en) | Desensitization method, system, computer device and computer-readable storage medium | |
| CN112039900A (en) | Network security risk detection method, system, computer device and storage medium | |
| CN113449339A (en) | Log collection method, system, computer device and computer readable storage medium | |
| CN110932918B (en) | Log data acquisition method and device and storage medium | |
| CN113157524B (en) | Big data based exception problem solving method, system, equipment and storage medium | |
| CN111680104B (en) | Data synchronization method, device, computer equipment and readable storage medium | |
| CN110958292A (en) | File uploading method, electronic device, computer equipment and storage medium | |
| CN111858605A (en) | Database automatic auditing method, system, equipment and storage medium | |
| CN112702228A (en) | Service current limiting response method and device, electronic equipment and readable storage medium | |
| CN111770022A (en) | Link monitoring-based capacity expansion method, system, equipment and computer storage medium | |
| CN118474166A (en) | Data sharing and cooperative processing platform based on cloud computing | |
| CN113051224A (en) | File transmission method and device, electronic equipment and computer readable storage medium | |
| CN111865725B (en) | Flow consumption analysis method and system based on log | |
| CN119512855A (en) | Abnormal data monitoring method, device, equipment and medium | |
| CN111767161B (en) | Remote call depth recognition method, device, computer equipment and readable storage medium | |
| CN112685253A (en) | Front-end error log collection method, device, equipment and storage medium | |
| CN112231194A (en) | Index abnormity root analysis method and device and computer readable storage medium | |
| CN112256532A (en) | Test interface generation method and device, computer equipment and readable storage medium | |
| CN117874674A (en) | Network about vehicle order monitoring and alarming method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |