CN109284257B

CN109284257B - Log writing method and device, electronic equipment and storage medium

Info

Publication number: CN109284257B
Application number: CN201810912188.5A
Authority: CN
Inventors: 陈小龙; 李辉
Original assignee: Ruijie Networks Co Ltd
Current assignee: Ruijie Networks Co Ltd
Priority date: 2018-08-10
Filing date: 2018-08-10
Publication date: 2021-01-05
Anticipated expiration: 2038-08-10
Also published as: CN109284257A

Abstract

The embodiment of the invention provides a log writing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a remote direct data access RDMA system running log; judging whether the RDMA system operation log is an emergency log; if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase; if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS); and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS. The log writing method provided by the embodiment of the invention improves the real-time large-scale throughput concurrent processing capacity of the burst log of the system and solves the problem of orderly storage of a large amount of irregular burst logs.

Description

Log writing method and device, electronic equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of networks, in particular to a log writing method and device, electronic equipment and a storage medium.

Background

In the field of data centers, data transmission has the characteristics of high bandwidth and low delay, and performance tests need to pay attention to the probability of occurrence of bandwidth, packet loss, delay and other abnormal conditions, so that data transmission performance needs to be tested to prevent the occurrence of packet loss, delay and other abnormal conditions in transmission. Existing RDMA technology has become the primary network connection technology for computing and data networks in 25Gbps/100Gbps data center networks. RDMA technology, all known as remote direct data access, is created to account for server-side data processing delays in network transmissions. RDMA provides device capability to directly access remote host memory, and does not require computer operating system and CPU intervention to perform this task, thus resulting in low overhead. RDMA technology provides high throughput, low latency networks. The RDMA service is widely applied, meanwhile, the transmission of the RDMA Data depends on a switch with a Data Center Bridging (DCB) characteristic, and the RDMA Data transmission can be ensured by adopting the switch with the DCB characteristic to transmit the Data. In a big data platform under an RDMA scene, log monitoring needs to be carried out on a server and network equipment, and when a client subscribes an event request or equipment has an abnormal fault, an access log is generated. Due to the characteristics of high bandwidth and low delay of RDMA, data transmission and processing are more frequent, the number of generated logs is large, the working frequency of an RDMA network card is generally 25Gbps, and the frequency can also reach 50Gbps/100 Gbps in a binding mode.

In addition, most of the access log files generated when the event analysis request is submitted are disordered, if the disordered log files are directly written into the RDMA system, the subsequent processing is very troublesome, and meanwhile, some important log information is not monitored in time, so that the event is not processed in time, and the network interruption is caused.

Disclosure of Invention

Aiming at the defects in the prior art, the embodiment of the invention provides a log writing method and device, electronic equipment and a storage medium.

In a first aspect, an embodiment of the present invention provides a log writing method, including:

acquiring a remote direct data access RDMA system running log;

judging whether the RDMA system operation log is an emergency log;

if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase;

if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS);

and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

As in the above method, optionally, the determining whether the RDMA system operation log is an urgent log includes:

determining the urgency of the RDMA system operation log according to a preset grade division principle;

if the urgency degree is greater than a preset urgency degree threshold value, determining that the RDMA system operation log is an urgency log;

otherwise, determining the RDMA system operation log as a non-urgent log.

As in the foregoing method, optionally, after determining whether the RDMA system operation log is an urgent log, the method further includes:

determining log data corresponding to the RDMA system operation log according to a preset semi-structured data format, wherein the preset semi-structured data format comprises: the RDMA system runs the urgency of the log, the log name, the log generation time and the log value.

As in the foregoing method, optionally, if the RDMA system operation log is an urgent log, writing the urgent log into a memory database system Hbase includes:

and if the operation log of the RDMA system is an emergency log, writing the log data corresponding to the emergency log into a memory database system Hbase according to the emergency degree.

As for the above method, optionally, the writing the log data corresponding to the emergency log into the memory database system Hbase according to the degree of emergency includes:

caching the log data into a first memory queue according to the corresponding urgency;

and sequentially writing the log data in the first memory queue into a memory database system Hbase.

As in the foregoing method, optionally, the writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS includes:

acquiring an emergency log queue in the memory database system Hbase;

caching the emergency log queue into a second memory queue;

and sequentially writing the emergency logs corresponding to the log data in the second memory queue into a disk distributed file system (HDFS).

As above, optionally, the writing the non-urgent log into the HDFS includes:

caching the log data corresponding to the non-urgent log into a third memory queue according to the corresponding urgency degree of the log data;

and sequentially writing the non-urgent logs corresponding to the log data in the third memory queue into a disk distributed file system (HDFS).

In a second aspect, an embodiment of the present invention provides a log writing apparatus, including:

the acquisition module is used for acquiring the remote direct data access RDMA system operation log;

the judging module is used for judging whether the RDMA system operation log is an emergency log;

the emergency log caching module is used for writing the emergency log into a memory database system Hbase if the RDMA system running log is the emergency log;

the non-urgent log storage module is used for writing the non-urgent log into a disk distributed file system (HDFS) if the operation log of the RDMA system is a non-urgent log;

and the emergency log storage module is used for writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS if the RDMA system is judged to be idle.

Optionally, the determining module is specifically configured to:

otherwise, determining the RDMA system operation log as a non-urgent log.

The above apparatus, optionally, further comprises:

the preprocessing module is configured to determine log data corresponding to the RDMA system operation log according to a preset semi-structured data format, where the preset semi-structured data format includes: the RDMA system runs the urgency of the log, the log name, the log generation time and the log value.

Optionally, the urgent log caching module is specifically configured to:

As the above apparatus, optionally, the emergency log caching module includes:

the first cache unit is used for caching the log data into a first memory queue according to the corresponding urgency;

and the first writing unit is used for sequentially writing the log data in the first memory queue into a memory database system Hbase.

As the above device, optionally, the emergency log storage module includes:

the acquiring unit is used for acquiring an emergency log queue in the memory database system Hbase;

the second buffer unit is used for buffering the emergency log queue into a second memory queue;

and the second writing unit is used for sequentially writing the emergency logs corresponding to the log data in the second memory queue into the disk distributed file system HDFS.

As the above apparatus, optionally, the non-urgent log storage module includes:

a third cache unit, configured to cache the log data corresponding to the non-urgent log into a third memory queue according to the corresponding urgency level of the log data;

and the third writing unit is used for sequentially writing the non-urgent logs corresponding to the log data in the third memory queue into the disk distributed file system HDFS.

In a third aspect, an embodiment of the present invention provides an electronic device, including:

the processor and the memory are communicated with each other through a bus; the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform a method comprising: acquiring a remote direct data access RDMA system running log; judging whether the RDMA system operation log is an emergency log; if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase; if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS); and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

In a fourth aspect, an embodiment of the present invention provides a storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the following method: acquiring a remote direct data access RDMA system running log; judging whether the RDMA system operation log is an emergency log; if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase; if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS); and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

According to the log writing method provided by the embodiment of the invention, the operation log of the RDMA system is divided into the urgent log and the non-urgent log, the urgent log is stored in the memory database system Hbase, and the urgent log stored in the Hbase is stored in the disk distributed file system HDFS in an idle state, so that the real-time large-scale throughput concurrent processing capacity of the burst log of the system is improved, high-frequency log sampling is adopted for urgent log information, low-frequency log sampling is adopted for non-urgent log information, and the problem of orderly storage of a large amount of irregular burst logs is solved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a log writing method according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a log writing apparatus according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a log writing system according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a second flash system in the log writing system according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is a schematic flow chart of a log writing method according to an embodiment of the present invention, as shown in fig. 1, the method includes:

step S11, acquiring a remote direct data access RDMA system operation log;

specifically, in a big data platform in an RDMA scenario, a data center employs a state log of an RDMA protocol of an RoCE network, where RoCE is an RDMA over converted Ethernet abbreviation, and is a network protocol that allows RDMA to be used over Ethernet. The log writing device obtains multiple remote direct data access RDMA system running logs on a server or a network device, such as a two-layer switch or a three-layer switch, for example, the RDMA system running log is obtained through a Linux Shell script, and the running log may be: register values when the RDMA network card runs, attribute values output by an ethtool command, or operation result values of RDMA test programs such as OFED Perftest.

Step S12, judging whether the RDMA system operation log is an urgent log;

specifically, after the RDMA system operation log is obtained, whether the log is an urgent log is determined, for example, the RDMA system operation logs of different types are divided in advance, and the RDMA system operation log is divided into two categories, that is, an urgent log and a non-urgent log, for example, an important burst high-frequency log is used as the urgent log, or the RDMA system operation log corresponding to the control packet is divided into the urgent log, and the RDMA system operation log corresponding to the data packet is divided into the non-urgent log, and the like. After the RDMA system operation log is obtained, whether the RDMA system operation log is an urgent log or a non-urgent log is judged according to the type of the RDMA system operation log.

Step S13, if the RDMA system operation log is an emergency log, writing the emergency log into a memory database system Hbase;

specifically, the working frequency of the RDMA network card is generally 25/10Gbps, the working frequency of the network card in the binding working mode is as high as 100/40Gpbs, and high-frequency log data needs to be written into a high-throughput low-delay distributed memory database system Hbase. If the RDMA system running log is judged to be an emergency log, the emergency log is written into the memory database system Hbase, high-frequency log sampling is adopted for emergency log information, and the problem of orderly storage of a large number of irregular emergency logs is solved. The data in which Hbase exists are finally written into the disk distributed file system HDFS.

Step S14, if the RDMA system operation log is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS);

specifically, if the operation log of the RDMA system is judged to be a non-urgent log, the non-urgent log can be directly written into a disk distributed file system (HDFS) for persistence, the HDFS is a Hadoop distributed file system, the HDFS can provide high-throughput data access, the method is very suitable for application on a large-scale data set, the non-urgent log is written into the HDFS, low-frequency log sampling is adopted for non-urgent log information, and due to the fact that the sampling frequency of the HDFS is low, if all the operation logs are directly written into the HDFS, the problem that real-time burst throughput is insufficient is caused.

And step S15, if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

Specifically, when the RDAM system is idle, the urgent log stored in the Hbase of the memory database system needs to be written into the HDFS of the disk distributed file system for persistence, so that all RDMA system logs are finally stored in the HDFS of the disk distributed file system. For example, by monitoring the system memory usage rate, when the system is judged to be idle, the emergency log stored in the Hbase is written into the HDFS, and the emergency log in the Hbase is emptied, so that the Hbase has enough space to store the emergency log when the RDMA system operation log is collected subsequently.

On the basis of the foregoing embodiment, further, the determining whether the RDMA system operation log is an urgent log includes:

otherwise, determining the RDMA system operation log as a non-urgent log.

Specifically, in order to sort out a disordered log file, a grade division principle can be preset, and the log is divided into N grades, wherein the urgency is from 1 to N, and the higher the urgency value is, the more important the log is, and the higher the log sampling frequency is. For example, the RDMA system log urgency level corresponding to the control packet is 8, the RDAM system log urgency level corresponding to the data packet is 5, the RDMA system log urgency level corresponding to the out-of-order packet is 1, and the like. After the RDMA system operation log is obtained, according to a preset grade division principle, the urgency degree of the RDMA system operation log is determined, and then whether the log urgency degree is greater than a preset urgency degree threshold is judged, for example, if the urgency degree threshold is set to 5, the RDMA system operation log with the urgency degree greater than 5 is an urgency log, and the RDMA system operation log with the urgency degree less than or equal to 5 is a non-urgency log.

According to the log writing method provided by the embodiment of the invention, the operation log of the RDMA system is divided into the urgent log and the non-urgent log according to the urgency degree, the urgent log is stored in the memory database system Hbase, the urgent log stored in the Hbase is stored in the disk distributed file system HDFS in an idle state, the real-time large-scale throughput concurrent processing capacity of the burst log of the system is improved, high-frequency log sampling is adopted for the urgent log information, low-frequency log sampling is adopted for the non-urgent log information, and the problem of orderly storage of a large amount of irregular burst logs is solved.

On the basis of the foregoing embodiments, further after determining whether the RDMA system operation log is an urgent log, the method further includes:

Specifically, after an RDMA system operation log is acquired, according to a preset grade division principle, the urgency degree corresponding to the RDMA system operation log is determined, then whether the RDMA system operation log is an urgency log is judged, and the RDMA system operation log is converted into a log data Event according to a preset semi-structured data format, wherein the Event structure is as follows:

the method comprises the steps of obtaining a running log of the RDMA system, obtaining a Level, a Name, a Value, a Time and an Info, wherein the Level is the emergency degree of the running log of the RMDA system, the Name is the Name of the running log of the RDMA system, the Value is a specific numerical Value of the running log of the RDMA system, the Time is the Time generated by the running log of the RDMA system, and. The method has the advantages that the RDMA system operation log is converted into the log data Event in a semi-structured data format, and the emergency Event is processed according to the priority. And then writing the log data Event into the corresponding Hbase or HDFS. When the log file is read from the Hbase or the HDFS, the urgency of the log file can be checked, and the log file with high urgency is preferentially processed.

According to the log writing method provided by the embodiment of the invention, the operation log of the RDMA system is divided into the emergency log and the non-emergency log according to the emergency degree, the log data in the semi-structured data format corresponding to the emergency log is stored in the memory database system Hbase, the log stored in the Hbase is stored in the disk distributed file system HDFS when the system is idle, the real-time large-scale throughput concurrent processing capacity of the burst log of the system is improved, the emergency log information is subjected to high-frequency log sampling, the non-emergency log information is subjected to low-frequency log sampling, the problem of orderly storage of a large number of irregular burst logs is solved, and the emergency event is processed according to the priority level by converting the operation log into the log data in the semi-structured data format.

On the basis of the foregoing embodiments, further, if the RDMA system operation log is an urgent log, writing the urgent log into a memory database system Hbase includes:

Specifically, after the obtained RDMA system running log is converted into a log data Event in a semi-structured data format, the same urgency log data Event is stored in the same Hbase file according to a time sequence, and meanwhile, the same urgency log data Event in a non-urgency log can also be stored in the same HDFS file according to a time sequence, so that the RDMA system running log is in the same level in the same Hbase file or the same HDFS file, corresponding processing can be performed according to different Hbase files or HDFS files, and the problem of orderly storage of a large number of irregular burst logs is solved.

According to the log writing method provided by the embodiment of the invention, the log data in the semi-structured data format corresponding to the operation log of the RDMA system is stored in the Hbase or the HDFS of the disk distributed file system according to the urgency and the time sequence, so that the real-time large-scale throughput concurrent processing capacity of the burst log of the system is improved, the high-frequency log sampling is adopted for the emergency log information, the low-frequency log sampling is adopted for the non-emergency log information, the problem of orderly storage of a large amount of irregular burst logs is solved, and the emergency event is processed according to the priority by converting the operation log into the log data in the semi-structured data format.

On the basis of the foregoing embodiments, further, the writing the log data corresponding to the emergency log into the memory database system Hbase according to the degree of emergency includes:

Specifically, because the log obtaining speed is greater than the log writing speed, in practical application, the log data can be cached into the memory queue according to the corresponding urgency degree, the log data is recorded as a first memory queue, the log data with the same urgency degree is cached into the same first memory queue, then the log data in the first memory queue is sequentially written into the memory database system Hbase, and the urgent log queue is stored in the Hbase, so that the problem of orderly and real-time writing of a large amount of urgent and irregular data is solved.

According to the log writing method provided by the embodiment of the invention, the log data in the semi-structured data format corresponding to the operation log of the RDMA system is stored in the Hbase or the HDFS of the disk distributed file system according to the urgency and the time sequence, so that the real-time large-scale throughput concurrent processing capacity of the burst log of the system is improved, high-frequency log sampling is adopted for the urgent log information, low-frequency log sampling is adopted for the non-urgent log information, and the problem of orderly and real-time writing of a large amount of urgent irregular data is solved.

On the basis of the foregoing embodiments, further, the writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS includes:

acquiring an emergency log queue in the memory database system Hbase;

caching the emergency log queue into a second memory queue;

Specifically, when the disk distributed file system is idle, the emergency logs in the memory database system Hbase need to be written into the HDFS of the disk distributed file system for persistence, an emergency log queue in the memory database system Hbase is obtained first, then the emergency log queue is cached into the memory queue and recorded as a second memory queue, then the emergency logs corresponding to the log data in the second memory queue are written into the HDFS of the disk distributed file system in sequence, the logs of the same level are stored into the same HDFS file according to a time sequence, and the structure of Event data stored in the memory database is a data table in the database and is stored into the HDFS and then is stored into the data file. The data volume of the Event message in the Hbase is small, generally dozens of bytes to hundreds of bytes, the HDFS file is generally dozens of megabytes to gigabytes, when the Event data in the Hbase is written into the HDFS, the Event data need to be classified into different second memory queues according to the log urgency degree rule, the urgent logs corresponding to the Event data in the different second memory queues are aggregated to the HDFS file according to the time sequence, and the problem of orderly storage of a large number of irregular burst logs is solved.

According to the log writing method provided by the embodiment of the invention, high-frequency log sampling is adopted for log information with high urgency, low-frequency log sampling is adopted for non-urgent log information, an emergent important high-frequency log is stored in an Hbase memory database system, the important high-frequency log of the Hbase memory database system is stored in an HDFS disk distributed file system in idle, and the problem of real-time large-scale throughput processing capability of the emergent log of the system is solved.

On the basis of the foregoing embodiments, further, the writing the non-urgent log into a disk distributed file system HDFS includes:

Specifically, the non-urgent logs can be stored in the HDFS of the disk distributed file system according to the urgency degree, the operation logs of the RDMA system are firstly obtained, whether the operation logs of the RDMA system are the non-urgent logs and the urgency degree corresponding to the non-urgent logs are determined, the non-urgent logs are converted into log data in a preset semi-structured data format, the log data are cached in a memory queue according to the urgency degree and are recorded as a third memory queue, then the non-urgent logs corresponding to the log data in the third memory queue are sequentially written into the HDFS of the disk distributed file system, the logs of the same level are stored in the same HDFS file according to the time sequence, and the problem of orderly storage of a large number of irregular burst logs is solved.

Fig. 2 is a schematic structural diagram of a log writing apparatus according to an embodiment of the present invention, and as shown in fig. 2, the apparatus includes: the system comprises an acquisition module 21, a judgment module 22, an emergency log cache module 23, a non-emergency log storage module 24 and an emergency log storage module 25, wherein:

the obtaining module 21 is configured to obtain a remote direct data access RDMA system operation log; the judging module 22 is configured to judge whether the RDMA system operation log is an urgent log; the urgent log caching module 23 is configured to write the urgent log into a memory database system Hbase if the RDMA system operation log is an urgent log; the non-urgent log storage module 24 is configured to write the non-urgent log into a disk distributed file system HDFS if the RDMA system operation log is a non-urgent log; and the urgent log storage module 25 is configured to write the urgent log in the memory database system Hbase into the disk distributed file system HDFS if it is determined that the RDMA system is idle.

Specifically, the obtaining module 21 obtains an RDMA system running log through a Linux Shell script, where the running log may be: register values when the RDMA network card runs, attribute values output by an ethtool command, or operation result values of RDMA test programs such as OFED Perftest. The judging module 22 judges whether the log is an emergency log, if the log is the emergency log, the emergency log caching module 23 writes the emergency log into a memory database system Hbase, so that high-frequency log sampling is adopted for emergency log information, and the problem of orderly storage of a large number of irregular emergency logs is solved; if the log is a non-urgent log, the non-urgent log storage module 24 directly writes the non-urgent log into the disk distributed file system HDFS for persistence, then judges whether the system is idle, and when the system is idle, the urgent log storage module 25 writes the urgent log already stored in the Hbase into the HDFS and empties the urgent log in the Hbase, so that the Hbase has enough space to store the urgent log when the RDMA system runs the log subsequently. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

The log writing device provided by the embodiment of the invention divides the operation log of the RDMA system into the urgent log and the non-urgent log, stores the urgent log into the memory database system Hbase, and stores the urgent log stored in the Hbase into the disk distributed file system HDFS in an idle state, thereby improving the real-time large-scale throughput concurrent processing capability of the burst log of the system, and solving the problem of orderly storage of a large amount of irregular burst logs by adopting high-frequency log sampling for the urgent log information and low-frequency log sampling for the non-urgent log information.

On the basis of the foregoing embodiment, further, the determining module is specifically configured to:

otherwise, determining the RDMA system operation log as a non-urgent log.

Specifically, in order to sort out a disordered log file, a grade division principle can be preset, and the log is divided into N grades, wherein the urgency is from 1 to N, and the higher the urgency value is, the more important the log is, and the higher the log sampling frequency is. After the RDMA system operation log is obtained, the determining module determines the urgency of the RDMA system operation log according to a preset level dividing principle, and then determines whether the log urgency is greater than a preset urgency threshold, for example, if the urgency threshold is set to 5, the RDMA system operation log with the urgency greater than 5 is an urgency log, and the RDMA system operation log with the urgency less than or equal to 5 is a non-urgency log. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

According to the log writing device provided by the embodiment of the invention, the operation log of the RDMA system is divided into the urgent log and the non-urgent log according to the urgency degree, the urgent log is stored in the memory database system Hbase, the urgent log stored in the Hbase is stored in the disk distributed file system HDFS in an idle state, the real-time large-scale throughput concurrent processing capacity of the burst log of the system is improved, high-frequency log sampling is adopted for the urgent log information, low-frequency log sampling is adopted for the non-urgent log information, and the problem of orderly storage of a large amount of irregular burst logs is solved.

On the basis of the above embodiments, further, the apparatus further includes:

Specifically, after the RDMA system running log is obtained, the preprocessing module determines the urgency degree corresponding to the RDMA system running log according to a preset level division principle, then judges whether the RDMA system running log is an urgency log, and converts the RDMA system running log into a log data Event according to a preset semi-structured data format, wherein the Event structure is as follows:

the method comprises the steps of obtaining a running log of the RDMA system, obtaining a Level, a Name, a Value, a Time and an Info, wherein the Level is the emergency degree of the running log of the RMDA system, the Name is the Name of the running log of the RDMA system, the Value is a specific numerical Value of the running log of the RDMA system, the Time is the Time generated by the running log of the RDMA system, and. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

The log writing device provided by the embodiment of the invention divides the operation log of the RDMA system into the urgent log and the non-urgent log according to the urgency degree, stores the log data of a semi-structured data format corresponding to the urgent log into the memory database system Hbase, and stores the log stored in the Hbase into the disk distributed file system HDFS when the system is idle, thereby improving the real-time large-scale throughput concurrent processing capability of the burst log of the system, and adopts high-frequency log sampling for the urgent log information and low-frequency log sampling for the non-urgent log information, thereby solving the problem of orderly storing a large amount of irregular burst logs, and realizing the processing of the urgent event according to the priority by converting the operation log into the log data of the semi-structured data format.

On the basis of the foregoing embodiments, further, the urgent log caching module is specifically configured to:

Specifically, after the obtained RDMA system running log is converted into a log data Event in a semi-structured data format, the emergency log cache module stores the same emergency log data Event into the same Hbase file according to a time sequence, and simultaneously stores the same emergency log data Event in a non-emergency log into the same HDFS file according to a time sequence, so that the RDMA system running log is of the same level in the same Hbase file or the same HDFS file, corresponding processing can be performed according to different Hbase files or HDFS files, and the problem of orderly storage of a large number of irregular burst logs is solved. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

The log writing device provided by the embodiment of the invention stores the log data in the semi-structured data format corresponding to the operation log of the RDMA system into the Hbase or the HDFS of the disk distributed file system according to the urgency and the time sequence, improves the real-time large-scale throughput concurrent processing capability of the burst log of the system, adopts high-frequency log sampling for the emergency log information and low-frequency log sampling for the non-emergency log information, solves the problem of orderly storage of a large amount of irregular burst logs, and realizes the priority processing of emergency events by converting the operation log into the log data in the semi-structured data format.

On the basis of the foregoing embodiments, further, the urgent log caching module includes:

Specifically, because the log obtaining speed is greater than the log writing speed, in practical application, the first cache unit caches the log data into the memory queue according to the corresponding urgency degree, the log data is recorded as a first memory queue, the log data with the same urgency degree is cached into the same first memory queue, then the first writing unit sequentially writes the log data in the first memory queue into the memory database system Hbase, and the urgent log queue is stored in the Hbase, so that the problem of orderly and real-time writing of a large amount of urgent and irregular data is solved. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

The log writing device provided by the embodiment of the invention stores the log data in a semi-structured data format corresponding to the operation log of the RDMA system into the Hbase or the HDFS of the disk distributed file system according to the urgency and the time sequence, improves the real-time large-scale throughput concurrent processing capacity of the burst log of the system, and solves the problem of orderly real-time writing of a large amount of urgent irregular data by adopting high-frequency log sampling for urgent log information and low-frequency log sampling for non-urgent log information.

On the basis of the foregoing embodiments, further, the emergency log storage module includes:

Specifically, the obtaining unit obtains an emergency log queue in a memory database system Hbase, the second caching unit caches the emergency log queue in the memory queue, the emergency log queue is recorded as a second memory queue, the second writing unit sequentially writes emergency logs corresponding to log data in the second memory queue into a disk distributed file system HDFS, logs of the same level are stored into the same HDFS file according to a time sequence, and the structure of Event data stored in the Hbase is a data table in the database and is stored as a data file after being stored in the HDFS. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

According to the log writing device provided by the embodiment of the invention, high-frequency log sampling is adopted for log information with high urgency, low-frequency log sampling is adopted for non-urgent log information, an emergent important high-frequency log is stored in the Hbase memory database system, the important high-frequency log of the Hbase memory database system is stored in the HDFS disk distributed file system in idle, and the problem of real-time large-scale throughput processing capability of the emergent log of the system is solved.

On the basis of the foregoing embodiments, further, the non-urgent log storage module includes:

Specifically, the non-urgent logs can be stored in the HDFS of the disk distributed file system according to the urgency degree, the operation logs of the RDMA system are firstly obtained, whether the operation logs of the RDMA system are the non-urgent logs and the urgency degree corresponding to the non-urgent logs are determined, the non-urgent logs are converted into log data in a preset semi-structured data format, the log data are cached in the memory queue according to the urgency degree by the third caching unit and are recorded as a third memory queue, then the non-urgent logs corresponding to the log data in the third memory queue are sequentially written into the HDFS of the disk distributed file system by the third writing unit, the logs of the same level are stored in the same HDFS file according to the time sequence, and the problem of orderly storage of a large number of irregular burst logs is solved. The apparatus provided in the embodiment of the present invention is configured to implement the method, and its functions specifically refer to the method embodiment, which is not described herein again.

Fig. 3 is a schematic structural diagram of a log writing system according to an embodiment of the present invention, and as shown in fig. 3, in an actual application, an Apache flash log management system may be adopted, and includes a first flash system and a second flash system. Each Flume system comprises three components, namely a Source component, a channel component and a Sink component. A first flash system is deployed on each server and used for grading logs, writing the logs into Hbase or HDFS. And selecting one server from the plurality of servers to deploy a second flash system for writing the data in the Hbase into the HDFS.

The first Flume system comprises a Source component, an emergency channel component, a non-emergency channel component, an Hbase Sink component and an HDFS Sink component, and the second Flume system comprises: hbase Source module, Mem channel module and HDFS sink module. Wherein: the Source component is used for acquiring the RDMA system running log, and the emergency Event channel component is used for caching the log data Event corresponding to the emergency RDMA system running log in a memory queue form; non-emergency channel component: the device comprises a component for caching log data Event corresponding to a non-urgent RDMA system running log in a memory queue form, and a component for Hbase sink: the system comprises a distributed memory database, a log data Event and a log data processing module, wherein the log data Event is used for storing a log corresponding to an RDMA system operation log in a distributed memory database mode; HDFS sink component: for storing RDMA system run logs in the form of disk distributed file systems. Fig. 4 is a schematic structural diagram of a second flash system in the log writing system according to the embodiment of the present invention, and as shown in fig. 4, the second flash system includes: hbase Source component, multiple Mem channel components, and multiple HDFS sink components. The multi-queue second flash system is used for writing the Event data with different urgency degrees into different HDFS file queues, and the problem of orderly and real-time writing of a large amount of urgent and irregular data is solved. Wherein: the Hbase source component is used for receiving Event data from a first Flume system Hbase Sink and forwarding the Event data to a downstream channel cache component, the Mem channel components are used for caching the Event data from a second Flume system Hbase source component, each Mem channel component is used for caching the Event data with the same priority, the data of events 1, 2, 3 and the like with the same urgency are cached in a priority queue form, the Event data are forwarded to a Sink corresponding to the downstream and written into the components, and the Mem channel components are mainly used for caching a large amount of urgent Event data. For example, the first Mem channel component buffers priority queue 1, the second Mem channel component buffers priority queue 2, the third Mem channel component buffers priority queue 3, etc. The HDFS sink component is used for receiving Event data from the second flash system and writing logs corresponding to the Event data into the HDFS disk distributed file system. Event data in different Mem channel queues are aggregated to the HDFS file according to chronological order. The structure of Event data stored in Hbase is a data table in a database, and the Event data is stored into an HDFS and then is stored as a data file. The emergent important high-frequency logs are stored in an Hbase memory database system, and the important high-frequency logs of the Hbase memory database system are stored in an HDFS disk distributed file system in an idle state, so that the problem of real-time large-scale throughput processing capacity of the emergent logs of the system is solved.

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 5, the electronic device includes: a processor (processor)51, a memory (memory)52, and a bus 53;

wherein, the processor 51 and the memory 52 complete the communication with each other through the bus 53;

the processor 51 is configured to call program instructions in the memory 52 to perform the methods provided by the above-mentioned method embodiments, including, for example: acquiring a remote direct data access RDMA system running log; judging whether the RDMA system operation log is an emergency log; if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase; if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS); and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

An embodiment of the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer can execute the methods provided by the above method embodiments, for example, the method includes: acquiring a remote direct data access RDMA system running log; judging whether the RDMA system operation log is an emergency log; if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase; if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS); and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

Embodiments of the present invention provide a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause the computer to perform the methods provided by the above method embodiments, for example, the methods include: acquiring a remote direct data access RDMA system running log; judging whether the RDMA system operation log is an emergency log; if the RDMA system running log is an emergency log, writing the emergency log into a memory database system Hbase; if the operation log of the RDMA system is a non-urgent log, writing the non-urgent log into a disk distributed file system (HDFS); and if the RDMA system is judged to be idle, writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS.

Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

The above-described embodiments of the apparatuses and the like are merely illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the embodiments of the present invention, and are not limited thereto; although embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log writing method, comprising:

acquiring a remote direct data access RDMA system running log;

judging whether the RDMA system operation log is an emergency log;

2. The method of claim 1, wherein the determining whether the RDMA system run log is an urgent log comprises:

otherwise, determining the RDMA system operation log as a non-urgent log.

3. The method of claim 2, wherein after determining whether the RDMA system run log is an urgent log, further comprising:

4. The method of claim 3, wherein writing the urgent log to a memory database system (Hbase) if the RDMA system oplog is an urgent log comprises:

5. The method according to claim 4, wherein writing the log data corresponding to the emergency log into a memory database system Hbase according to the degree of emergency comprises:

6. The method according to claim 5, wherein writing the emergency log in the memory database system Hbase into the disk distributed file system HDFS comprises:

acquiring an emergency log queue in the memory database system Hbase;

caching the emergency log queue into a second memory queue;

7. The method according to any one of claims 3 to 6, wherein said writing said non-urgent log into a disk distributed file system (HDFS) comprises:

8. A log writing apparatus, comprising:

9. The apparatus of claim 8, wherein the determining module is specifically configured to:

otherwise, determining the RDMA system operation log as a non-urgent log.

10. The apparatus of claim 9, further comprising:

11. The apparatus of claim 10, wherein the emergency log caching module is specifically configured to:

12. The apparatus of claim 11, wherein the emergency log cache module comprises:

13. The apparatus of claim 12, wherein the emergency log storage module comprises:

14. The apparatus according to any of claims 10-13, wherein the non-emergency log storage module comprises:

15. An electronic device, comprising:

the processor and the memory are communicated with each other through a bus; the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of claims 1 to 7.

16. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.