CN111786828B - Log processing method executed in network equipment and network equipment - Google Patents
Log processing method executed in network equipment and network equipment Download PDFInfo
- Publication number
- CN111786828B CN111786828B CN202010606536.3A CN202010606536A CN111786828B CN 111786828 B CN111786828 B CN 111786828B CN 202010606536 A CN202010606536 A CN 202010606536A CN 111786828 B CN111786828 B CN 111786828B
- Authority
- CN
- China
- Prior art keywords
- log
- storage space
- request information
- user
- network device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 36
- 230000004044 response Effects 0.000 claims abstract description 12
- 230000036961 partial effect Effects 0.000 claims description 11
- 230000009471 action Effects 0.000 abstract description 9
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000006399 behavior Effects 0.000 description 36
- 238000012545 processing Methods 0.000 description 16
- 238000004590 computer program Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 10
- 230000002829 reductive effect Effects 0.000 description 10
- 230000015654 memory Effects 0.000 description 9
- 230000002159 abnormal effect Effects 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 239000000203 mixture Substances 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000009472 formulation Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000003542 behavioural effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present disclosure provides a log processing method executed in a network device, and relates to the field of computer technology, the method includes: in response to receiving the request information, determining an importance level of a user providing the request information, generating a first log having a first data amount based on an operation action performed with respect to the request information in a case where the importance level of the user is a first level, and storing the first log in a first storage space, generating a second log having a second data amount based on an operation action performed with respect to the request information in a case where the importance level of the user is a second level lower than the first level, and storing the second log in the second storage space, wherein the second data amount is lower than the first data amount. The method can be applied to video scenes and can be executed by network equipment related to video type application programs.
Description
Technical Field
The present disclosure relates to the field of computer technology, and more particularly, to a log processing method performed in a network device and a network device.
Background
With the rapid development of mobile terminals and internet technology, a variety of applications have been developed. When a business is abnormal, each application generally hopes to quickly locate the business abnormality so as to process in time to ensure user experience. A complete service request goes from client to server to upstream of the server, and anomalies in the service may occur in any link. In locating traffic anomalies, it is often necessary to track a complete data link site.
However, in implementing the concepts of the present disclosure, the inventors found that there are at least the following problems in the related art: the larger the log quantity is, the more complete the data site can be restored, so that the exception positioning is facilitated, but the required processing resources and data reading and writing resources are larger, the processing time is longer, and the real-time performance of the exception positioning is difficult to meet.
Disclosure of Invention
In view of this, the present disclosure provides a log processing method performed in a network device and a network device.
One aspect of the present disclosure provides a log processing method performed in a network device, including: in response to receiving request information, determining an importance level of a user providing the request information, generating a first log having a first data amount based on an operation action performed with respect to the request information in a case where the importance level of the user is a first level, and storing the first log in a first storage space, and generating a second log having a second data amount based on an operation action performed with respect to the request information in a case where the importance level of the user is a second level lower than the first level, and storing the second log in a second storage space, wherein the second data amount is lower than the first data amount.
According to an embodiment of the disclosure, the generating a first log having a first data volume based on the operational behavior performed for the request information includes: the first log is generated based on all behavior data generated by the operation behavior performed with respect to the request information.
According to an embodiment of the present disclosure, the generating a second log having a second data amount based on the operation behavior performed for the request information includes: the second log is generated based on partial behavior data generated by an operation behavior performed with respect to the request information.
According to an embodiment of the present disclosure, the read-write speed of the first storage space is greater than the read-write speed of the second storage space.
According to an embodiment of the disclosure, the first storage space stores the first log by way of a queue.
According to an embodiment of the present disclosure, a user having an importance level of a first level satisfies at least one of the following conditions: the access frequency of the user is larger than a first threshold, the access time of the user is larger than a second threshold, the historical feedback times of the user are larger than a third threshold, and the user is a preset user.
According to an embodiment of the present disclosure, the method further comprises: the method includes obtaining the first log from the first storage space, obtaining a third log provided by another network device associated with the first log from a third storage space, the third log being a third log having a third amount of data generated by operational actions performed by the other network device with respect to the request information, stitching the first log and the third log into a first full link log, and storing the first full link log to a log data repository for log querying.
According to an embodiment of the present disclosure, for the same request information, the first log and the third log have the same log identifier, and the obtaining, from a third storage space, a third log provided by another network device associated with the first log includes: based on the log identification of the first log, a third log having the same log identification as the log identification of the first log is obtained from the third storage space.
According to an embodiment of the present disclosure, the method further comprises: in response to the splice instruction, performing the following operations: the method includes obtaining the second log from the second storage space, obtaining a fourth log provided by another network device associated with the second log from a fourth storage space, the fourth log being a fourth log having a fourth data volume generated by an operation behavior performed by the other network device with respect to the request information, wherein the fourth data volume is lower than the third data volume, and stitching the second log and the fourth log into a second full link log.
According to an embodiment of the present disclosure, for the same request information, the second log and the fourth log have the same log identifier, and the obtaining, from the fourth storage space, the fourth log provided by another network device associated with the second log includes: and acquiring a fourth log with the same log identification as the second log from the fourth storage space based on the log identification of the second log.
Another aspect of the present disclosure provides a network device comprising: one or more processors, storage means for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method as described above.
According to an embodiment of the present disclosure, the network device comprises a network device associated with a video class application.
According to an embodiment of the disclosure, the network device is a client.
According to an embodiment of the disclosure, the network device is a server.
Another aspect of the present disclosure provides a computer-readable medium storing computer-executable instructions that, when executed, are configured to implement a method as described above.
Another aspect of the present disclosure provides a computer program comprising computer executable instructions which when executed are for implementing a method as described above.
Another aspect of the present disclosure provides a computer program product, comprising a computer program. The computer program is for implementing the method as described above when being executed by a processor.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:
FIG. 1 schematically illustrates an exemplary system architecture of a log processing method according to an embodiment of the present disclosure;
fig. 2 schematically illustrates a flowchart of a log processing method performed in a network device according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a log processing method performed in a network device according to another embodiment of the present disclosure;
fig. 4 schematically illustrates a schematic diagram of a network device according to an embodiment of the present disclosure;
fig. 5 schematically illustrates a flowchart of a log processing method performed in a network device according to still another embodiment of the present disclosure;
fig. 6 schematically illustrates a schematic diagram of a network device according to another embodiment of the present disclosure; and
Fig. 7 schematically illustrates a block diagram of a network device according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
Embodiments of the present disclosure provide a log processing method performed in a network device, the method including: in response to receiving the request information, a level of importance is determined for the user providing the request information. In the case where the importance level of the user is the first level, a first log having a first data amount is generated based on the operation behavior performed for the request information, and the first log is stored to the first storage space. In the case where the importance level of the user is a second level lower than the first level, a second log having a second data amount is generated based on the operation behavior performed for the request information, and the second log is stored to a second storage space, wherein the second data amount is lower than the first data amount.
Fig. 1 schematically illustrates an exemplary system architecture 100 of a log processing method according to an embodiment of the present disclosure.
It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios.
As shown in fig. 1, a system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as a short video platform class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.
It should be noted that, the log processing method provided in the embodiment of the present disclosure may be generally executed by the server 105. Alternatively, the log processing method provided by the embodiment of the present disclosure may also be executed by the terminal apparatuses 101, 102, 103. Alternatively, the log processing method provided by the embodiment of the present disclosure may also be performed by a server or a server cluster that is different from the terminal devices 101, 102, 103 and the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.
For example, in the disclosed embodiments, a user may initiate request information through any one of the terminal devices 101, 102, 103 (e.g., terminal device 101, but not limited thereto). The terminal device 101 may perform a corresponding operation action in response to receiving the request information, generate a log of the client, and transmit the request information to the server 105. The server 105 may perform a corresponding operation for the request information, generate a log of the server, and feed back the processing result to the terminal device 101.
In an embodiment of the present disclosure, the terminal device 101 may generate a full log on request information when processing the request information of a user having an importance level of a first level, and may generate a partial log on request information when processing the request information of a user having an importance level of a second level lower than the first level.
In another embodiment of the present disclosure, the server 105 may generate a full log of request information about a user having a first level of importance when processing the request information, and may generate a partial log of request information about a user having a second level of importance lower than the first level when processing the request information.
The log processing method provided by the embodiment of the disclosure can be applied to video scenes, for example. For example, the terminal device 101 may be a client of a video class application program, and the server 105 may be a server of the video class application program. The request information initiated by the user through the terminal device 101 may be, for example, request information for searching for video.
According to the embodiment of the disclosure, the full log is generated by the request information of the user with higher importance level, and the partial log is generated by the request information of the user with lower importance level, so that the consumption and occupation of resources can be reduced while the completeness of the log is satisfied, and the log processing efficiency is improved.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically illustrates a flowchart of a log processing method performed in a network device according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S201 to S203.
According to the embodiment of the disclosure, the network device may be a client or a server, for example. For example, the following operations S201 to S203 may be performed at the client side, or the following operations S201 to S203 may be performed at the server side.
In response to receiving the request information, an importance level with the user who provided the request information is determined in operation S201.
According to the embodiment of the present disclosure, the user may be classified into a plurality of importance levels by a preset condition. For example, the importance level of the user satisfying the preset condition may be a first level, and the importance levels of the remaining users may be a second level.
In the embodiment of the present disclosure, the preset condition may be, for example, at least one of the following conditions: the access frequency of the user is larger than a first threshold, the access time of the user is larger than a second threshold, the historical feedback times of the user are larger than a third threshold, and the user is a preset user.
It may be appreciated that, a user who frequently pays attention to an application program, a user who pays attention to an application program for a long time, a user who performs opinion feedback on a certain function of an application program, or an internal developer of an application program, etc. may find problems and anomalies in an application program more easily, and the embodiments of the present disclosure may set importance levels of the users to a first level, and importance levels of the remaining users to a second level, so that different log generation standards may be executed for users of different levels.
According to the embodiment of the disclosure, the importance level of each user can be updated periodically. For example, the composition of the user whose importance level is at the first level may be appropriately adjusted according to the coverage situation and the repeatability of the abnormality found by the history, so that the coverage of the service of interest to the user at the first level is more comprehensive.
In the embodiment of the present disclosure, a threshold of the number of users at the first level may also be set. For example, the number of users in the first level must not exceed 10000 (only by way of example), so that the data volume of the log can be controlled, the processing time is reduced, and the real-time performance of the generation of the full-link log is ensured. As another example, the number of users at the first level must not be less than 5000 (by way of example only), so that coverage for each business scenario can be ensured.
In operation S202, in the case where the importance level of the user is the first level, a first log having a first data amount is generated based on the operation behavior performed for the request information, and the first log is stored to the first storage space.
According to the embodiment of the disclosure, in the case where the importance level of the user is the first level, the first log may be generated based on all the behavior data generated by the operation behavior performed with respect to the request information of the user. That is, a full log is generated for request information of a core user that is important.
For example, the client may record various behavioral data of its own request information for the core user to generate a full log of the client. The server may also record various behavior data of the request information of the core user to generate a full log of the server. For example, the server may record complete request data, response data, parameters of the interactive request with the upstream nodes and returned results, and information about which node the current request is to be processed at, the node, etc.
The embodiment of the disclosure does not limit the log content of the full log, and a person skilled in the art can set the log content according to actual situations. The present disclosure only contemplates that the first log may be more comprehensive in composition and may be used to build a more complete data link site to quickly and accurately locate business anomalies.
According to the embodiment of the disclosure, the generated first log may be stored in the first storage space. The first storage space may be, for example, a Redis database, and for example, the server may store the full log generated by the first storage space into the Redis database through network I/O. The first storage space may store the first log, for example, by way of a queue.
In an embodiment of the present disclosure, the first log may include a plurality of logs related to the request information. For example, regarding a request for information requiring execution of a plurality of operational acts, an operational act may generate at least one log, or at least one operational act may be packaged to generate a set of logs. Each log or set of logs may be directly stored in the queue of the first storage space each time the log or set of logs is generated. The queue may be, for example, a first-in-first-out queue, so that a subsequent time-sequential retrieval of the first log from the first storage space may be facilitated.
In operation S203, in case that the importance level of the user is a second level lower than the first level, a second log having a second data amount is generated based on the operation behavior performed with respect to the request information, and the second log is stored to the second storage space, wherein the second data amount is lower than the first data amount.
According to the embodiment of the present disclosure, in the case where the importance level of the user is the second level lower than the first level, the second log may be generated based on the partial behavior data generated by the operation behavior performed with respect to the request information of the user. That is, a non-full log is generated for the request information of the non-core user.
For example, the client may record partial behavior data itself for the non-core user's request information to generate a non-full log of the client. The server may also record partial behavior data of its own request information for the non-core user, so as to generate a non-full log of the server.
According to the embodiment of the disclosure, the data volume of the non-full-volume log may be 20% of the data volume of the full-volume log (only by way of example), so that the data volume of the log may be reduced, and occupation and consumption of resources may be reduced.
In the embodiment of the present disclosure, the non-full log may be all behavior data about part of the behavior of the request information (e.g., all behavior data about core behavior), or may be part of the behavior data about all of the behavior of the request information (e.g., only time information about each behavior is recorded), or may also be part of the behavior data about part of the behavior of the request information.
The embodiment of the disclosure does not limit log content of the non-full log, and a person skilled in the art can set the log content according to actual situations. The present disclosure only contemplates that the second log has a smaller amount of data that can be used to build a somewhat complete data link site to enable locating business anomalies.
According to the embodiment of the disclosure, the generated second log may be stored in the second storage space. The second storage space may be, for example, a disk, and the server may write the generated non-full log into the disk through disk I/O.
In the embodiment of the present disclosure, the read-write speed of the first storage space may be greater than the read-write speed of the second storage space. According to the embodiment of the disclosure, the full log can be stored through the first storage space with higher read-write speed, so that the server side or the client side can quickly store the generated full log into the first storage space and also can quickly read the first log from the first storage space, and the instantaneity of splicing to form the full link log can be improved. The embodiment of the disclosure can store the non-full log through the second storage space with larger storage capacity, so that a large amount of log data produced by a large amount of non-core users can be accommodated.
According to the embodiment of the disclosure, the total log is generated for a small number of core users, so that the data volume of the log can be reduced, the data processing speed is improved, the instantaneity of forming the full link log is improved, the integrity of the full link log can be ensured, and the exception positioning can be conveniently and rapidly carried out.
According to the embodiment of the disclosure, the data volume of the logs can be reduced by generating the non-full logs for a large number of non-core users, all business scenes can be covered comprehensively, and under the condition that the full logs cannot locate the anomalies, the data analysis can be performed through the non-full logs so as to locate the anomalies.
According to the embodiment of the disclosure, the full logs and the non-full logs are stored through different storage spaces, so that a large amount of I/O resources of the non-full logs written into the disk are not occupied, the full logs can be rapidly obtained, and the instantaneity of forming the full link logs is improved.
Fig. 3 schematically illustrates a flowchart of a log processing method performed in a network device according to another embodiment of the present disclosure. Fig. 4 schematically shows a schematic diagram of a network device according to an embodiment of the present disclosure. The log processing method of the embodiment of the present disclosure is described below with reference to fig. 3 and 4.
As shown in fig. 3, the method includes operations S301 to S304.
The method may be performed, for example, by the first network device in fig. 4, according to an embodiment of the present disclosure. The first network device may be, for example, a client, a server, or a server cluster different from the client and the server.
In operation S301, a first log is acquired from a first storage space.
According to the embodiment of the present disclosure, the server may generate the first log based on all behavior data generated by the operation behaviors performed for the request information of the user of the first level, and store the first log in the first storage space. The first network device may read the first log from the first storage space.
For example, the server may store the full log generated based on the request information of the core user in the first storage space, so that the first network device may obtain the full log generated by the server from the first storage space.
In operation S302, a third log provided by another network device associated with the first log is obtained from a third storage space, the third log being a third log having a third data amount generated by an operation behavior performed by the other network device with respect to the request information.
According to the embodiment of the present disclosure, the client may generate the third log based on all behavior data generated by the operation behavior performed with respect to the request information of the user of the first level, and store the third log in the third storage space. The first network device may read the third log from the third storage space.
For example, the client may store the full log generated based on the request information of the core user in a third storage space, so that the first network device may obtain the full log generated by the client from the third storage space.
In an embodiment of the present disclosure, the first storage space and the third storage space may be different storage spaces. For example, the first storage space and the third storage space may be provided in different network nodes. Alternatively, the first storage space and the third storage space may be provided in different storage areas of the same network node. For example, the first storage space may be disposed in a network device other than the server and the client, and the third storage space may be disposed in the client.
In another embodiment of the present disclosure, the first storage space and the third storage space may also be the same storage space. For example, two queues may be disposed in the storage space, where one is used to store a third log generated by the client and the other is used to store a first log generated by the server. Or a queue can be arranged in the storage space, and the third log generated by the client and the first log generated by the server are stored in the queue. The storage space can be arranged in the client, the server and other network equipment different from the client and the server.
According to embodiments of the present disclosure, each log may have a log identification, and all logs generated for the same request information may have the same log identification.
In an embodiment of the present disclosure, a third log having the same log identification as the first log may be obtained from the third storage space based on the log identification of the first log.
For example, in response to the log data having the log identifier 1 being acquired from the third storage space or from the first storage space, all the log data having the log identifier 1 may be acquired from the first storage space and the third storage space, so that all the log data about a certain request information may be acquired to generate a full link log based on the log data.
In operation S303, the first log and the third log are spliced into a first full link log.
According to the embodiment of the disclosure, all logs with the same log identification can be spliced to generate a first full-link log based on time sequence or data flow.
It can be understood that the first log and the third log are full logs generated by the server and the client respectively, and the first full link log generated based on full log splicing can have more perfect data information, so that the data site can be recovered more completely, and the speed and accuracy of abnormal positioning are improved.
In the embodiment of the disclosure, the formats of the obtained first log and the obtained third log may be normalized, so that the processed first log and the processed third log may have the same format, which is convenient for splicing.
In operation S304, a first full link log is stored to a log data repository for log queries.
According to the embodiment of the disclosure, the generated first full-link log can be stored in the search engine ES, so that when business abnormality occurs, the full-link log about an abnormal scene is conveniently retrieved based on keywords, and the abnormality can be positioned.
According to the embodiment of the disclosure, the full-link log is generated based on the full-quantity log splice generated by the client and the server, so that the data site can be recovered more completely, and the accuracy of abnormal positioning is improved.
The client and the server in the embodiment of the disclosure only generate the full log aiming at the core user, have smaller data volume, can timely finish the splicing of the full link log, and improve the timeliness of the full link log generation so as to quickly provide complete log data for an abnormal scene.
Fig. 5 schematically illustrates a flowchart of a log processing method performed in a network device according to still another embodiment of the present disclosure. Fig. 6 schematically illustrates a schematic diagram of a network device according to another embodiment of the present disclosure. The log processing method of the embodiment of the present disclosure is described below with reference to fig. 5 and 6.
As shown in fig. 5, the method includes operations S501 to S503.
The method may be performed, for example, by the second network device in fig. 6, according to an embodiment of the present disclosure. The network device may be, for example, a client, a server, a first network device similar to that of fig. 4, or a server cluster different from the client, the server, and the first network device.
In operation S501, a second log is acquired from a second storage space.
According to the embodiment of the present disclosure, the server may generate the second log based on partial behavior data generated by the operation behavior performed with respect to the request information of the user of the second level, and store the second log in the second storage space. The second network device may read the second log from the second storage space.
For example, the server may store the non-full log generated based on the request information of the non-core user in the second storage space, so that the second network device may obtain the non-full log generated by the server from the second storage space.
In operation S502, a fourth log provided by another network device associated with the second log, which is a fourth log having a fourth data amount generated by an operation action performed by the other network device with respect to the request information, is acquired from the fourth storage space, the fourth data amount being lower than the third data amount.
According to the embodiment of the disclosure, the client may generate the fourth log based on the partial behavior data generated by the operation behavior performed with respect to the request information of the user of the second level, and store the fourth log in the fourth storage space. The second network device may read the fourth log from the fourth storage space.
For example, the client may store a non-full log generated based on the request information of the non-core user in a fourth storage space, so that the second network device may obtain the non-full log generated by the client from the fourth storage space.
In an embodiment of the present disclosure, the second storage space and the fourth storage space may be different storage spaces. For example, the second storage space and the fourth storage space may be provided in different network devices. Alternatively, the second storage space and the fourth storage space may be disposed in different storage areas of the same network device.
In another embodiment of the present disclosure, the second storage space and the fourth storage space may also be the same storage space.
According to embodiments of the present disclosure, each log may have a log identification, and all logs generated for the same request information may have the same log identification.
In an embodiment of the present disclosure, a fourth log having the same log identification as the second log may be acquired from the fourth storage space based on the log identification of the second log.
For example, in response to acquiring log data having log identity 2 from the second storage space or from the fourth storage space, all log data having the log identity 2 may be acquired from the second storage space and the fourth storage space, so that all logs regarding a certain request information may be acquired to generate a full link log based on the log concatenation.
In operation S503, the second log and the fourth log are spliced into a second full link log.
According to the embodiment of the disclosure, all logs with the same log identification can be spliced to generate a second full link log based on time sequence or data flow.
It can be understood that the second log and the fourth log are non-full logs generated by the server and the client respectively, and the data processing amount can be reduced and the processing speed can be improved by generating the second full link log based on the splicing of the non-full logs, so that the resources, occupation and consumption can be reduced.
In the embodiment of the disclosure, the formats of the obtained second log and the obtained fourth log may be normalized, so that the processed second log and the processed fourth log may have the same format, which is convenient for splicing.
In an embodiment of the present disclosure, the splice instruction may be issued periodically, and the second network device may perform operations S501 to S503 in response to the splice instruction. For example, the non-full logs generated in the week may be spliced to generate a second full-link log at intervals of one week and stored in a log data repository, which may be the same as or different from the log data repository in fig. 4, so that when a traffic abnormality occurs, the second full-link log about the abnormal scene may be retrieved based on the keywords, and the abnormality may be located.
In another embodiment of the present disclosure, the tester may issue a splice instruction when a non-full log is required, and the second network device may perform operations S501 to S503 in response to the splice instruction. For example, when the full log cannot cover all abnormal scenes, the tester can extract the log about the business from the non-full log based on the business needing to locate the abnormality, and then splice to generate a second full link log.
The second full-link log generated based on the non-full log splice generated by the client and the server can cover almost all service scenes, so that support can be provided when the data site needs to be restored.
The client and the server in the embodiment of the disclosure only generate partial logs for non-core users, so that the data volume can be reduced, and the disk I/O pressure and the occupation of a processor can be reduced.
Fig. 7 schematically illustrates a block diagram of a network device according to an embodiment of the disclosure. The network device shown in fig. 7 is merely an example, and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.
According to the embodiment of the present disclosure, the network device shown in fig. 7 may be, for example, a client, a server, or another server or a server cluster different from the client and the server.
As shown in fig. 7, a network device 700 according to an embodiment of the present disclosure includes a processor 701 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. The processor 701 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 701 may also include on-board memory for caching purposes. The processor 701 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.
In the RAM 703, various programs and data required for the operation of the network device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The processor 701 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 702 and/or the RAM 703. Note that the program may be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, the network device 700 may also include an input/output (I/O) interface 705, the input/output (I/O) interface 705 also being connected to the bus 704. The network device 700 may also include one or more of the following components connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 702 and/or RAM 703 and/or one or more memories other than ROM 702 and RAM 703 described above.
The present disclosure also provides a computer program product comprising a computer program. The above-described computer program, when executed by a processor, implements a method according to embodiments of the present disclosure.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.
Claims (12)
1. A log processing method performed in a network device, comprising:
in response to receiving the request information, determining a level of importance with a user providing the request information;
Generating a first log based on all behavior data generated by the operation behavior performed for the request information and storing the first log in a first storage space, wherein the data amount of the first log is a first data amount, in the case that the importance level of the user is a first level;
generating a second log based on partial behavior data generated by the operation behavior performed with respect to the request information and storing the second log in a second storage space, wherein a data amount of the second log is a second data amount lower than the first data amount, in a case where the importance level of the user is a second level lower than the first level;
acquiring the first log from the first storage space; obtaining a third log provided by a client associated with the first log from a third storage space, the third log being a third log having a third amount of data generated by an operational behavior performed by the client for the request information; and splicing the first log and the third log into a first full-link log; and
responding to a splicing instruction, and acquiring the second log from the second storage space; obtaining a fourth log provided by a client associated with the second log from a fourth storage space, the fourth log being a fourth log having a fourth amount of data generated by an operational behavior performed by the client with respect to the request information, wherein the fourth amount of data is lower than the third amount of data; splicing the second log and the fourth log into a second full-link log; the splice instruction is issued regularly or by a tester when a non-full log is required to be used;
The client is a client of the video application program, and the request information is request information for searching the video.
2. The method of claim 1, wherein the read-write speed of the first storage space is greater than the read-write speed of the second storage space.
3. The method of claim 1, wherein the first storage space stores the first log by way of a queue.
4. The method of claim 1, wherein a user having a first level of importance meets at least one of the following conditions:
the access frequency of the user is greater than a first threshold;
the access time of the user is longer than a second threshold;
the historical feedback times of the user are larger than a third threshold value; and
the user is a preset user.
5. The method of claim 1, further comprising:
the first full link log is stored to a log data repository for log querying.
6. The method of claim 5, wherein the first log and the third log have the same log identity for the same request information;
the retrieving, from a third storage space, a third log provided by a client associated with the first log, comprising: based on the log identification of the first log, a third log having the same log identification as the log identification of the first log is obtained from the third storage space.
7. The method of claim 1, wherein the second log and the fourth log have the same log identity for the same request information;
the obtaining, from a fourth storage space, a fourth log provided by a client associated with the second log, comprising: and acquiring a fourth log with the same log identification as the second log from the fourth storage space based on the log identification of the second log.
8. A network device, comprising:
one or more processors;
storage means for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.
9. The network device of claim 8, wherein the network device comprises a network device associated with a video-type application.
10. The network device of claim 8 or 9, wherein the network device is a client.
11. The network device according to claim 8 or 9, wherein the network device is a server.
12. A computer readable medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010606536.3A CN111786828B (en) | 2020-06-29 | 2020-06-29 | Log processing method executed in network equipment and network equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010606536.3A CN111786828B (en) | 2020-06-29 | 2020-06-29 | Log processing method executed in network equipment and network equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111786828A CN111786828A (en) | 2020-10-16 |
| CN111786828B true CN111786828B (en) | 2023-07-11 |
Family
ID=72761301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010606536.3A Active CN111786828B (en) | 2020-06-29 | 2020-06-29 | Log processing method executed in network equipment and network equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111786828B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112306871A (en) * | 2020-10-29 | 2021-02-02 | 广州博冠信息科技有限公司 | Data processing method, device, equipment and storage medium |
| CN116932549A (en) * | 2023-07-21 | 2023-10-24 | 企知道科技有限公司 | Intelligent model-based platform data storage method, system, medium and equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101192227A (en) * | 2006-11-30 | 2008-06-04 | 阿里巴巴公司 | Log file analytical method and system based on distributed type computing network |
| CN107678908A (en) * | 2017-06-23 | 2018-02-09 | 平安科技（深圳）有限公司 | Log recording method, device, computer equipment and storage medium |
| CN109727137A (en) * | 2018-12-18 | 2019-05-07 | 杭州茂财网络技术有限公司 | A log reporting method and system based on user risk assessment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4276672B2 (en) * | 2006-10-03 | 2009-06-10 | シャープ株式会社 | Image processing device |
| CN110851471B (en) * | 2018-07-27 | 2025-02-21 | 北京京东尚科信息技术有限公司 | Distributed log data processing method, device and system |
| CN111046010A (en) * | 2019-11-13 | 2020-04-21 | 泰康保险集团股份有限公司 | Log storage method, device, system, electronic equipment and computer readable medium |
-
2020
- 2020-06-29 CN CN202010606536.3A patent/CN111786828B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101192227A (en) * | 2006-11-30 | 2008-06-04 | 阿里巴巴公司 | Log file analytical method and system based on distributed type computing network |
| CN107678908A (en) * | 2017-06-23 | 2018-02-09 | 平安科技（深圳）有限公司 | Log recording method, device, computer equipment and storage medium |
| CN109727137A (en) * | 2018-12-18 | 2019-05-07 | 杭州茂财网络技术有限公司 | A log reporting method and system based on user risk assessment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111786828A (en) | 2020-10-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108549569B (en) | A method and device for searching information in an application program | |
| CN113590447B (en) | Buried point processing method and device | |
| CN113590372A (en) | Log-based link tracking method and device, computer equipment and storage medium | |
| CN111786828B (en) | Log processing method executed in network equipment and network equipment | |
| CN110866031B (en) | Database access path optimization method and device, computing equipment and medium | |
| CN107291835B (en) | Search term recommendation method and device | |
| CN113495825A (en) | Line alarm processing method and device, electronic equipment and readable storage medium | |
| CN114861054B (en) | Information collection method, device, electronic device and storage medium | |
| CN117076280A (en) | Policy generation method and device, electronic equipment and computer readable storage medium | |
| CN111694639A (en) | Method and device for updating address of process container and electronic equipment | |
| CN112084114B (en) | Method and apparatus for testing interfaces | |
| CN119106750A (en) | Task processing method based on large model, device, equipment and medium | |
| CN119336731A (en) | Data storage method and device | |
| CN107273362B (en) | Data processing method and apparatus thereof | |
| CN113590425B (en) | Data processing method, apparatus, device, medium, and program product | |
| CN117806977A (en) | Test method, apparatus, device, medium and program product | |
| CN116521544A (en) | Method and device for processing test information | |
| CN114693358A (en) | Data processing method, device, electronic device and storage medium | |
| CN119788741B (en) | Business processing methods, apparatus, equipment and computer-readable media | |
| CN113535562A (en) | Test data generation method, system, electronic device, medium and program product | |
| CN114564408B (en) | Cause analysis method, device, electronic device, medium and computer program product | |
| CN114817314B (en) | Data processing method, device, electronic device and storage medium | |
| CN111178014A (en) | Method and device for processing business process | |
| CN114201214B (en) | File generation method, device, electronic device, medium and program product | |
| CN120297399B (en) | Knowledge base construction method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |