CN117033332A - Log processing method, device and equipment - Google Patents

Abstract

The application discloses a log processing method, device and equipment, and belongs to the technical field of log processing. The log processing method comprises the following steps: acquiring first log data uploaded by a service system, wherein the first log data comprises a first service tag; determining a log data flow limiting index value of a first service label according to a hash table, wherein the hash table comprises at least one hash node, the hash node is established based on different service labels and hash functions, and the hash node is used for storing the corresponding relation between the service labels and the flow limiting index value; and under the condition that the current limiting index value meets the current limiting condition, prohibiting the first log data from being stored in a cache space of the log system.

Description

Log processing methods, devices and equipment

Technical field

This application belongs to the technical field of log processing, and specifically relates to a log processing method, device and equipment.

Background technique

With the development of technology, electronic devices include more and more applications, and each application generates more and more log data. In order to facilitate subsequent search and analysis, log data needs to be cached.

However, the cache space is limited, so in subsequent uses, the cache will be exhausted and the log data can no longer be stored, resulting in the loss of this part of the log data.

Contents of the invention

The purpose of the embodiments of this application is to provide a log processing method, device and equipment that can solve the problem of log loss due to the exhaustion of the cache of the log system.

In the first aspect, embodiments of the present application provide a log processing method, including:

Obtain the first log data uploaded by the business system, where the first log data includes the first business tag;

Determine the log data current limiting indicator value of the first business tag according to the hash table. The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node is used to store the business tag and the hash function. Correspondence between current limiting indicator values;

When the current limiting indicator value meets the current limiting condition, the first log data is prohibited from being stored in the cache space of the log system.

In the second aspect, embodiments of the present application provide a log processing device, including:

The acquisition module is used to acquire the first log data uploaded by the business system, where the first log data includes the first business tag;

The determination module is used to determine the log data current limit indicator value of the first business tag according to the hash table. The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node is To store the corresponding relationship between business labels and current limiting indicator values;

The processing module is configured to prohibit storing the first log data into the cache space of the log system when the current limiting indicator value meets the current limiting condition.

In a third aspect, embodiments of the present application provide an electronic device. The electronic device includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the program or instructions are executed by the processor, the implementation is as in the first aspect. steps of the method.

In a fourth aspect, embodiments of the present application provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the programs or instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

In a fifth aspect, embodiments of the present application provide a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method described in the first aspect.

The embodiment of the present application obtains the first log data uploaded by the business system. The first log data includes the first business tag; the log data current limit indicator value of the first business tag is determined according to the hash table, and the hash table includes at least one hash node. , the hash node is established based on different business labels and hash functions. The hash node is used to store the correspondence between the business label and the current limiting indicator value; when the current limiting indicator value meets the current limiting condition, the first log is prohibited from being The data is stored in the cache space of the log system. That is, after obtaining the first log data uploaded by the business system, the embodiment of this application can determine the current limiting flag value of the first business tag from the hash table, and when the current limiting flag value meets the current limiting condition, it is prohibited to use the first log data uploaded by the business system. One log data is stored in the cache space of the log system. This can avoid the situation where the output volume of log data of a certain business type is large and exhaust the cache of the log system, which in turn causes the log data of other business types to be unable to be cached and thus lost, effectively improving the This ensures the balance of log data caching for each business type.

Description of the drawings

Figure 1 is a flow chart of a log processing method provided by an embodiment of the present application;

Figure 2 is a flow chart of a log processing method provided by an embodiment of the present application;

Figure 3 is a schematic diagram of a log processing process provided by an embodiment of the present application;

Figure 4 is a schematic structural diagram of a log processing device provided by an embodiment of the present application;

Figure 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the scope of protection of this application.

The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It is to be understood that the figures so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in orders other than those illustrated or described herein, and that "first," "second," etc. are distinguished Objects are usually of one type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the related objects are in an "or" relationship.

When developers need to analyze bugs that occur in electronic devices, they need to rely on the logs cached by the log system on the electronic devices to troubleshoot and analyze the problem.

With the development of technology, electronic devices support more and more businesses, and each business generates more and more logs. In this case, how to save the logs of each business is a serious challenge for the log system.

Currently, there are two main methods: first, by increasing the priority of the log thread to alleviate the problem of log loss; second, by increasing the cache space of the log system.

For the first method, due to the increased priority of the log thread, it occupies a lot of central processing unit (CPU) time, causing the CPU of the user interface (UI) thread of the electronic device and the packet sending and receiving thread to be exhausted. The occupancy rate will decrease, which will lead to screen operation freezes, slow network response, etc., affecting the overall performance of the electronic device and reducing the user experience.

Regarding the second method, increasing the cache space of the log system can only delay the time when the problem is exposed, but cannot fundamentally solve the problem of large log output, exhaustion of cache space, and log loss.

In summary, none of the currently provided log processing solutions can effectively solve the problem of log loss caused by the log system's cache being exhausted.

To this end, embodiments of the present application provide a log processing method, device and equipment, which can effectively solve the problem of log loss due to exhaustion of the cache of the log system. The log processing method, device and equipment provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios.

It should be noted that the log processing solution provided by the embodiments of this application can also be applied to web requests, etc. That is, not only can logs be flow-limited, but web requests or other content can also be flow-limited. The processing process is similar.

Figure 1 is a flow chart of a log processing method provided by an embodiment of the present application. This method can be applied to a log system of an electronic device. The electronic device can be a ulog-based smart device such as a mobile phone, a computer, a wearable smart device, and others. Embedded devices such as routers, etc. Wearable smart devices can be, for example, smart watches, smart bracelets, etc.

As shown in Figure 1, the log processing method may include the following steps:

S110. Obtain the first log data uploaded by the business system.

Wherein, the first log data includes a first service tag.

S120. Determine the log data current limiting indicator value of the first business tag according to the hash table.

The hash table includes at least one hash node. The hash node is established based on different business labels and hash functions. The hash node is used to store the corresponding relationship between the business label and the current limiting indicator value.

S130. When the current limiting indicator value meets the current limiting condition, it is prohibited to store the first log data in the cache space of the log system.

The embodiment of the present application obtains the first log data uploaded by the business system. The first log data includes the first business tag; the log data current limit indicator value of the first business tag is determined according to the hash table, and the hash table includes at least one hash node. , the hash node is established based on different business labels and hash functions. The hash node is used to store the correspondence between the business label and the current limiting indicator value; when the current limiting indicator value meets the current limiting condition, the first log is prohibited from being The data is stored in the cache space of the log system. That is, after obtaining the first log data uploaded by the business system, the embodiment of this application can determine the current limiting flag value of the first business tag from the hash table, and when the current limiting flag value meets the current limiting condition, it is prohibited to use the first log data uploaded by the business system. One log data is stored in the cache space of the log system. This can avoid the situation where the output volume of log data of a certain business type is large and exhaust the cache of the log system, which in turn causes the log data of other business types to be unable to be cached and thus lost, effectively improving the This ensures the balance of log data caching for each business type.

The above steps are explained in detail below, as follows:

In S110, the business system is used to process different businesses. When the business system processes different businesses, it can generate corresponding log data for each business and upload the log data to the log system, which will be cached by the log system for convenience. Subsequent user search and analysis.

The first service label may be a character string, used to characterize the service type of the first log data, and different service types may correspond to different service labels. Setting different business tags for different business types can avoid the centralized storage of log data for one or a few types of business, resulting in the loss of log data for other types of business when the corresponding log data is subsequently cached.

In addition to the first service tag, the first log data may also include specific log content.

For example, the log system can obtain the first log data uploaded by the business system through the log management interface.

In S120, the hash table includes at least one hash node, and one hash node corresponds to one first service label. For example, if the log system receives log data of three service labels, then there will be three corresponding hash nodes. In actual application, The number of hash nodes can be updated in real time based on log data received by the logging system.

For example, the hash node can be established based on the business tag and the hash function. For example, the business tag can be hashed based on the hash function to obtain the hash result, and then the corresponding hash node can be established based on the hash result. And store the hash result in association with the hash node.

The embodiment of the present application does not limit the hash function. For example, string hash functions such as BKDRHash, APHash, DJBHash, JSHash, RSHash, SDBMHash, PJWHash, and ELFHash may be used.

Through the above string hash function, the first business tag can be converted from a string into an identity document (ID) value, so that when the hash table is subsequently searched based on the first business tag, the first business tag only needs to be Just compare the ID value corresponding to the business tag with the ID value stored in each hash node in the hash table, which can greatly reduce the time taken by string comparison.

The current limiting indicator value may be an indicator value capable of limiting the current flow of the first log data, and the current limiting indicator value may be updated in real time based on the log data incoming from the business system.

For example, the current limiting indicator value may include a current limiting flag bit. For example, when the current limiting flag bit is 0, it indicates that the first log data can be cached; when the current limiting flag bit is 1, it indicates that caching of the first log data is prohibited. .

For example, the current limiting index value may also include the current limiting start time. For example, if the difference between the current time and the current limiting start time is less than a preset difference, caching of the first log data may be prohibited. That is, if the first log data is received in a short period of time, For a large amount of first log data, it is necessary to implement a current limiting policy on the first log data to prevent it from exhausting the cache space of the log system. The current time may be the time at which the current limiting indicator value is determined.

The current limiting indicator values of different business labels can be stored in association with the hash nodes corresponding to the business labels.

For example, after the log system obtains the first log data, it can search the hash table to determine the current limiting indicator value of the first business tag, thereby providing a basis for subsequent execution of the current limiting policy.

In S130, the current limiting condition is used to limit the current flow of the first log data, that is, it is prohibited to store the first log data in the cache space of the log system.

For example, when the current limiting indicator value meets the current limiting condition, the first log data is prohibited from being stored in the cache space of the log system. When the current limiting indicator value does not meet the current limiting condition, the first log data is allowed to be stored in the log system's cache space. cache space.

The embodiments of this application can effectively filter out spam, invalid, redundant and unrestricted junk log data by limiting the flow of log data of different business types, thereby preventing these junk log data from occupying more cache space and causing normal Log data cannot be cached and is lost.

In some embodiments, the above S120 may include the following steps:

Compare the first business tag and the hash table;

When the first service label is not included in the hash table, a hash node of the first service label is established and a current limiting indicator value corresponding to the first service label is initialized, and the initialized current limiting indicator value is determined as the first The log data current limit indicator value of the business tag;

When the hash table includes the first service label, the log data current limiting indicator value of the first service label is determined according to the hash node corresponding to the first service label.

In some embodiments, in order to improve the efficiency of table lookup, a hash function can be used to perform a hash operation on the first business tag to obtain the first hash value, and the first hash value can be compared with each hash in the hash table. If the first hash value is the same as a hash value in the hash table, it can be determined that the first business tag exists in the hash table. If the first hash value is the same as each hash value in the hash table If the values are all different, it can be determined that the first service tag does not exist in the hash table.

In some embodiments, when the first service label does not exist in the hash table, a hash node of the first service label may be established, and the current limiting indicator value of the first service label may be initialized. For example, a new hash node can be created, a hash function is used to perform a hash operation on the first business tag, a hash value is obtained, and the hash value is stored in the newly created hash node.

For example, assuming that the hash table initially contains 5 hash nodes, a new hash node can be added at the end of the hash table.

When a new hash node is added, the current limiting indicator value of the first service tag can be initialized. For example, the statistical time can be initialized, the starting value of the statistical value is 1, and the current limiting flag is 0. There is no current limiting at this time. Starting time. The statistical time is the recording time corresponding to the starting value of the statistical value.

At this time, the initialized current limiting indicator value can be used as the current limiting indicator value of the first business tag. When the first log data is subsequently received, the current limiting indicator value stored on the hash node can be updated.

In some embodiments, when the first service label exists in the hash table, the log data current limiting indicator value of the first service label may be determined according to the hash node corresponding to the first service label. For example, the current limiting indicator value stored on the corresponding hash node may be determined as the log data current limiting indicator value of the first service tag.

Embodiments of the present application can compare the first service label and the hash table to determine the current limiting index value of the first service label, providing a basis for subsequent execution of the current limiting strategy. At the same time, when comparing the first business tag and the hash table, you can first perform a hash operation on the first business tag based on the hash function to obtain the first hash value, and then compare the first hash value with each of the hash tables. The hash value of the hash node is compared, which can greatly reduce the time taken by string comparison and improve the efficiency of table lookup.

In some embodiments, the hash functions mentioned in the above embodiments can be selected as needed. For example, several common hash algorithms can be selected, and then several hash algorithms can be tested based on several test data to select one. The hash algorithm with the best effect is used in the embodiment of this application.

For example, three hashing algorithms can be selected, namely:

Algorithm 1: Add the ASC codes of each character of each string string, and then take the remainder of %1024;

Algorithm 2: Based on Algorithm 1, add the asc code of the last character of the string to the remainder;

Algorithm 3: Based on the BKDRHash algorithm, and then take the remainder of %1024.

For example, test data can include the following three types:

Data 1: 100 randomly generated strings, each string length 8;

Data 2: 500 randomly generated strings, each string length 8;

Data 3: 700 randomly generated strings, each string length 8;

For each algorithm, the following two methods can be used to create tables and look up tables:

Method 1: Calculate a value of data X through the HASH function, and the value is recorded as tagid. When establishing a hash table, if there is no data corresponding to the tagid (hash node), write it directly; if there is data corresponding to the tagid and the keyword does not match, add a new hash node backward; after searching for the hash In the table, if the tagid is inconsistent, the next hash node will be traversed backwards and the search will continue. Data X here can be the above-mentioned data 1, data 2 and data 3, and the HASH function can be the above-mentioned algorithm 1, algorithm 2 and algorithm 3.

Method 2: Calculate a value from the data X through the HASH function, and the value is recorded as tagi d. When establishing the hash table, if there is no data corresponding to tagi d (hash node), write it directly; if there is data corresponding to tagi d and the keywords do not match, the hash table is traversed from the beginning and the empty bits are put in; When searching the hash table, if tagi d is inconsistent, the hash table will be traversed from the beginning and the search will continue.

For each algorithm, you can test data 1, data 2, and data 3 according to the above methods one and two, respectively, to get the total time to build the table, the total time to look up the table, and the average time of each string. and other parameters, see Table 1 and Table 2 for details.

Table 1 Parameters corresponding to mode 1

Table 2 Parameters corresponding to method 2

By analyzing the above parameters, it can be seen that the number of conflicts between Method 1 and Algorithm 3 is smaller, and the average time consumption is smaller. Therefore, Algorithm 3 can be used as the hash function of the embodiment of the present application to perform a hash operation on the first business tag. .

Among them, when two hash values are the same but the keywords of the corresponding strings are different, the two hash values are said to conflict in the hash table. For example, when two hash values are the same but the keys are different, a new hash node can be added to the hash table.

The above-mentioned Algorithm 1, Algorithm 2 and Algorithm 3 are just one example. In actual application, other algorithms can also be tested, and the process is similar.

The first hash value in the above embodiment is also the tagid in the above example, that is, the first service tag can be operated based on the BKDRHash algorithm, and then the operation result %1024 is modulated to obtain the first hash value. That is to say, the embodiment of the present application can convert the first business tag from a string into a numerical value. In this way, when searching the hash table for the first business tag, the string comparison can be converted into a numerical comparison, thereby greatly improving the efficiency of table lookup. .

For example, when comparing the first business tag and the hash table, you can start from the first hash node of the hash table, that is, compare the first hash value with the hash value of the first hash node. , if the two are different, continue to traverse a hash node, that is, perform table lookup according to the above method 1, which can reduce the table lookup time.

In this embodiment of the present application, the current limiting condition is related to the current limiting indicator value. For example, the current limiting indicator value may include a current limiting flag and a current limiting start time;

Correspondingly, the current limiting index value satisfying the current limiting condition includes: the current limiting flag bit is the first flag bit, and the first difference between the current limiting start time and the first time is less than the first threshold, and the first time is according to the hash table The time to determine the log data current limiting indicator value of the first business tag.

The current limiting start time is the time to limit the first log data cache. For example, the current limiting start time can be determined based on the current limiting flag bit. For example, if the current limiting flag bit is 0, it means no current limiting, and if the current limiting flag bit is 1, it means current limiting, then the current limiting start time can be the time when the current limiting flag bit switches from 0 to 1. For example, if the current limit flag switches from 0 to 1 at 9:05, then the current limit start time is 9:05.

The first flag bit is a flag bit that restricts storage of the first log data. For example, the first flag bit may be set to 1.

The first time is the time when the log data current limiting indicator value of the first business tag is determined based on the hash table, that is, the time when the current limiting indicator value is determined.

For example, when the current limiting flag bit of the first service tag is 1, and the first difference between the first time and the current limiting start time is less than the first threshold, the first log data may be prohibited from being stored in the cache of the log system. space. The size of the first threshold can be set according to actual needs, for example, it can be set to 1 minute or 2 minutes.

The embodiments of this application can limit the flow of a large amount of log data incoming from the business system in a short period of time. This can avoid the problem of running out of cache space, causing other normal log data to be lost due to the exhaustion of cache space, and improves the efficiency of various aspects. The balance of caching of business type log data.

In some embodiments, the current limiting indicator value may also include a statistical value and a statistical time. The statistical value represents the number of first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value;

Accordingly, as shown in Figure 2, the business processing method may include the following steps:

S210. Obtain the first log data uploaded by the business system.

S220. Determine the log data current limiting indicator value of the first service tag according to the hash table.

S230. When the current limiting indicator value meets the current limiting condition, it is prohibited to store the first log data in the cache space of the log system.

S240. When at least one of the current limiting flag bit and the first difference value does not meet the current limiting condition, add 1 to the statistical value to obtain an updated statistical value.

S250. When the updated statistical value is greater than the current limiting threshold, obtain the second difference between the first time and the statistical time.

S260. When the second difference value is less than the second threshold, it is prohibited to store the first log data in the cache space of the log system.

S270, reset statistical values, current limiting flag bits and current limiting start time.

Among them, the process of S210-S230 is the same as the above-mentioned process of S110-S130. For details, please refer to the above-mentioned S110-S130. For the sake of concise description, the details will not be repeated here.

The other steps mentioned above are explained in detail below, as follows:

In S240, for example, at least one of the current limiting flag bit and the first difference value does not meet the current limiting condition. The current limiting flag bit may be the second flag bit, and the first difference value is less than the first threshold; it may also be The current limiting flag is a first flag, and the first difference is not less than the first threshold; the current limiting flag can also be a second flag, and the first difference is not less than the first threshold.

The second flag bit is used to indicate that the current flow of the first log data is not limited. For example, the second flag bit can be set to 0.

The statistical value is used to count the number of first log data obtained by the log system. For example, when at least one of the current limiting flag bit and the current limiting start time does not meet the current limiting condition, the statistical value can be increased by 1, thereby updating the number of first log data obtained by the log system to determine whether the subsequent Provides the basis for limiting the flow of first log data.

For example, when the current limiting flag bit of the first service tag is 0, 1 may be added to the statistical value. For example, if the first difference between the first time and the current limiting start time is not less than the first threshold, 1 may be added to the statistical value. For example, when the current limiting flag is 0 and the first difference between the first time and the current limiting start time is not less than the first threshold, 1 may be added to the statistical value.

In S250, if the updated statistical value is greater than the current limiting threshold, the second difference between the first time and the statistical time can be obtained. The statistical time is the recording time corresponding to the starting value of the statistical value. The starting value of the statistical value can be 0 or 1. For details, see the description below.

In S260, if the second difference between the first time and the statistical time is less than the second threshold, the first log data may be prohibited from being stored in the cache space of the log system. If the second difference between the first time and the statistical time is not If the value is less than the second threshold, the first log data may be allowed to be stored in the cache space of the log system. The size of the second threshold can be set according to actual needs. For example, the second threshold may be set to 1 minute.

In S270, when the second difference between the first time and the statistical time is less than the second threshold, in addition to prohibiting the storage of the first log data into the cache space of the log system, the statistical value, the current limiting flag bit and the Current limiting start time.

For example, the statistical value can be reset to 0, the current limiting flag bit can be reset from 0 to 1, and the current limiting start time can be set. There is no need to update the statistical time at this time.

In actual application, S260 and S270 can be executed at the same time, or S260 can be executed first and then S270, or S270 can be executed first and then S260. You can choose one of S230 and S240 to execute.

For example, when the second difference between the first time and the statistical time is not less than the second threshold, in addition to allowing the first log data to be stored in the cache space of the log system, the statistical value, statistical time and limit also need to be updated. stream flag.

For example, the statistical value can be reset to 0, and the statistical time can be reset, that is, the statistical time starts counting when the statistical value is reset to 0. For hash nodes that have been current-limited, the current-limiting flag can be updated from 1 to 0, that is, the current limitation is lifted. That is to say, the embodiment of this application only limits the flow of log data within a short period of time.

In the embodiment of the present application, when at least one of the current limiting flag bit and the current limiting start time does not meet the current limiting condition, the first log data can be further limited based on the statistical value and statistical time of the first log data, that is, this method The application embodiment can limit the flow of a large amount of log data output by the business system in a short period of time to prevent it from excessively occupying the cache space of other normal log data.

In some embodiments, after S110, the log processing method may also include the following steps:

Record the second time when the first log data is obtained, and start timing from the second time;

If the first log data is not received within the preset timing period, the current limiting indicator value corresponding to the first log data in the hash table is deleted from the first hash node corresponding to the first service label.

For example, for each first log data obtained, a second time at which the first log data is obtained can be recorded, and the timing starts from the second time. If the first log data is not received again within the preset timing period, , at this time, the hash value of the first business tag and the corresponding current limit flag value can be removed from the corresponding hash node, and the hash node can be retained. For example, the preset timing duration can be set to 3 minutes.

Taking the correspondence between the first business label and the first hash node as an example, if the log system has not received the first log data within the preset timing period, the hash of the first business label stored on the first hash node can be value and the corresponding current limiting indicator value are removed.

This can reduce the memory footprint of the hash table and reduce the probability of collision of the corresponding hash value in the hash table.

Since the log data of a certain business tag has not been transmitted to the log system for a period of time, some hash nodes may be empty, that is, they do not contain the hash value and the corresponding current limit flag value. Therefore, in the hash of the first business tag When the value is different from the hash value of each hash node, you can also directly insert the hash value of the first business tag and the corresponding current limit flag value into the empty hash node, that is, build it according to the above method. surface.

The embodiment of this application uses business tags to distinguish different business types, and establishes a hash table based on the hash function to count the log output frequency and output quantity of different business types, and limit the storage of log data that exceeds the threshold. This is used to achieve the purpose of real-time monitoring and flow limiting of log data.

The following uses an example to illustrate the log processing process provided by the embodiment of this application, as follows:

As shown in Figure 3, the log system receives log data uploaded by the business system. The log data is distinguished by business tags (strings). The business tags are hashed through a hash algorithm to obtain the hash value, and then the hash is determined. Whether the value exists in the hash table. If the hash value exists in the hash table, further determine the current limit flag and the difference between the first time and the current limit start time. If the current limit flag is 1 and the difference is less than When the first threshold is reached, the log data corresponding to the hash value is prohibited from being stored in the cache space of the log system, and the next piece of log data is continued to be judged.

If the current limiting flag is 0, and/or, the difference is not less than the first threshold, add 1 to the statistical value of the hash node corresponding to the hash value, and then further determine whether the updated statistical value is greater than the current limiting threshold. , if the updated statistical value is not greater than the current limiting threshold, the log data corresponding to the hash value is allowed to be stored in the cache space of the log system. If the updated statistical value is greater than the current limiting threshold, it can be further determined that the first time and The relationship between the statistical time difference and the second threshold.

If the difference between the first time and the statistical time is less than the second threshold, the statistical value, current limiting flag bit, and current limiting start time of the corresponding hash node are reset, and the corresponding log data is prohibited from being stored in the cache space of the log system. . If the difference between the first time and the statistical time is not less than the second threshold, the corresponding log data is allowed to be stored in the cache space of the log system, and the statistical value of the corresponding hash node (set to 0) and the statistical time are reset. For hash nodes that have been current-limited, update their current-limit flag bits to 0 to lift the current limit.

If the hash value does not exist in the hash table, the corresponding hash node is established, and the statistical time, statistical value (set to 1), and current limit flag (set to 0) are set.

Through the above solution, the balance of the log output of each business type can be significantly improved, that is, the normal cache space will not be enough and the logs will not be lost because of a certain business log crazily refreshing the screen. That is to say, part of the logs of each business type can be stored in the cache space. In addition, the business tags are hashed based on the hash function to convert the string comparison into a hash value comparison, which greatly improves the processing performance of the log system.

It should be noted that when the current limiting flag value meets the current limiting condition, in the embodiment of the present application, in addition to prohibiting the storage of log data in the cache space, an alarm can also be issued, or the log data with current limiting can also be added to a blacklist. Add unlimited log data to the whitelist, etc.

It should be noted that, for the log processing method provided by the embodiment of the present application, the execution subject may be a log processing device, or a processing module in the log processing device for executing the log processing method. In the embodiment of the present application, the log processing device executing the log processing method is taken as an example to illustrate the log processing device provided by the embodiment of the present application.

Figure 4 is a schematic structural diagram of a log processing device provided by an embodiment of the present application.

As shown in Figure 4, the log processing device 400 may include:

The acquisition module 401 is used to acquire the first log data uploaded by the business system, where the first log data includes the first business tag;

Determination module 402, configured to determine the log data current limiting indicator value of the first service tag according to a hash table. The hash table includes at least one hash node, and the hash node is based on different service tags and hashes. The function is established, and the hash node is used to store the corresponding relationship between the business tag and the current limiting indicator value;

The processing module 403 is configured to prohibit storing the first log data in the cache space of the log system when the current limiting indicator value meets the current limiting condition.

The embodiment of the present application obtains the first log data uploaded by the business system. The first log data includes the first business tag; the log data current limit indicator value of the first business tag is determined according to the hash table, and the hash table includes at least one hash node. , the hash node is established based on different business labels and hash functions. The hash node is used to store the correspondence between the business label and the current limiting indicator value; when the current limiting indicator value meets the current limiting condition, the first log is prohibited from being The data is stored in the cache space of the log system. That is, after obtaining the first log data uploaded by the business system, the embodiment of this application can determine the current limiting flag value of the first business tag from the hash table, and when the current limiting flag value meets the current limiting condition, it is prohibited to use the first log data uploaded by the business system. One log data is stored in the cache space of the log system. This can avoid the situation where the output volume of log data of a certain business type is large and exhaust the cache of the log system, which in turn causes the log data of other business types to be unable to be cached and thus lost, effectively improving the This ensures the balance of log data caching for each business type.

In some possible implementations of the embodiments of this application, the determination module 401 is specifically used to:

Compare the first business tag and the hash table;

When the first service label is not included in the hash table, a hash node of the first service label is established and a current limiting indicator value corresponding to the first service label is initialized, and the initialized current limiting indicator value is determined as the first The log data current limit indicator value of the business tag;

When the hash table includes the first service label, the log data current limiting indicator value of the first service label is determined according to the hash node corresponding to the first service label.

In some possible implementations of the embodiments of this application, the current limiting indicator value includes a current limiting flag and a current limiting start time;

The current limiting indicator value satisfies the current limiting conditions including: the current limiting flag bit is the first flag bit, and the first difference between the current limiting start time and the first time is less than the first threshold, and the first time is determined based on the hash table. The time of the log data current limiting indicator value of the business tag.

In some possible implementations of the embodiments of this application, the current limiting indicator value also includes a statistical value and a statistical time. The statistical value represents the number of first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value;

The processing module 403 is also configured to, after the determination module 402 determines the log data current limiting index value of the first service tag according to the hash table, if at least one of the current limiting flag bit and the first difference value does not meet the current limiting condition. , add 1 to the statistical value to obtain the updated statistical value;

The acquisition module 401 is also configured to acquire the second difference between the first time and the statistical time when the updated statistical value is greater than the current limiting threshold;

The processing module 403 is also configured to prohibit storing the first log data into the cache space of the log system when the second difference is less than the second threshold; and to reset the statistical value, the current limiting flag bit, and the current limiting start time. .

In some possible implementations of the embodiments of this application, the processing module 403 is also configured to record the second time when the first log data is obtained after the acquisition module obtains the first log data uploaded by the business system, and start timing from the second time. ;as well as,

If the first log data is not received within the preset timing period, the current limiting indicator value corresponding to the first log data in the hash table is deleted from the first hash node corresponding to the first service label.

The embodiment of this application uses business tags to distinguish different business types, and establishes a hash table based on the hash function to count the log output frequency and output quantity of different business types, and limit the storage of log data that exceeds the threshold. This is used to achieve the purpose of real-time monitoring and flow limiting of log data.

The log processing device applied to electronic equipment in the embodiment of the present application may be a device or a component in the electronic equipment, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than the terminal. For example, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device (MID), or augmented reality (AR)/virtual reality (VR). ) equipment, robots, wearable devices, ultra-mobile personal computers (UMPC), netbooks or personal digital assistants (PDA), etc., and can also be servers, network attached storage (Network Attached Storage, NAS) , personal computer (PC), television (TV), teller machine or self-service machine, etc., the embodiments of this application are not specifically limited.

The electronic device in the embodiment of the present application may be an electronic device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of this application.

The log processing device applied to electronic equipment provided by the embodiment of the present application can implement each process in the embodiment of the log processing method applied to electronic equipment in Figures 1 to 3. To avoid duplication, the details will not be described here.

As shown in Figure 5, this embodiment of the present application also provides an electronic device, including a processor 501 and a memory 502. The memory 502 stores programs or instructions that can be run on the processor 501. When the program or instructions are executed by the processor 501 Each step of the above embodiment of the screen projection method can be implemented and the same technical effect can be achieved. To avoid repetition, the details will not be described again here.

It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

FIG. 6 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.

The electronic device 600 includes but is not limited to: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, processor 610, etc. part.

Those skilled in the art can understand that the electronic device 600 may also include a power supply (such as a battery) that supplies power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby managing charging, discharging, and function through the power management system. Consumption management and other functions. The structure of the electronic device shown in Figure 6 does not constitute a limitation on the electronic device. The electronic device may include more or less components than shown in the figure, or combine certain components, or arrange different components, which will not be described again here. .

Among them, the input unit 604 is used to obtain the first log data uploaded by the business system, where the first log data includes the first business tag;

Processor 610, configured to determine the log data current limiting indicator value of the first business tag according to the hash table. The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node Used to store the correspondence between business tags and current limiting indicator values; and,

When the current limiting indicator value meets the current limiting condition, the first log data is prohibited from being stored in the cache space of the log system.

The embodiment of the present application obtains the first log data uploaded by the business system. The first log data includes the first business tag; the log data current limit indicator value of the first business tag is determined according to the hash table, and the hash table includes at least one hash node. , the hash node is established based on different business labels and hash functions. The hash node is used to store the correspondence between the business label and the current limiting indicator value; when the current limiting indicator value meets the current limiting condition, the first log is prohibited from being The data is stored in the cache space of the log system. That is, after obtaining the first log data uploaded by the business system, the embodiment of this application can determine the current limiting flag value of the first business tag from the hash table, and when the current limiting flag value meets the current limiting condition, it is prohibited to use the first log data uploaded by the business system. One log data is stored in the cache space of the log system. This can avoid the situation where the output volume of log data of a certain business type is large and exhaust the cache of the log system, which in turn causes the log data of other business types to be unable to be cached and thus lost, effectively improving the This ensures the balance of log data caching for each business type.

In some possible implementations of the embodiments of this application, the processor 610 is specifically used to:

Compare the first business tag and the hash table;

When the first service label is not included in the hash table, a hash node of the first service label is established and a current limiting indicator value corresponding to the first service label is initialized, and the initialized current limiting indicator value is determined as the first The log data current limit indicator value of the business tag;

When the hash table includes the first service label, the log data current limiting indicator value of the first service label is determined according to the hash node corresponding to the first service label.

In some possible implementations of the embodiments of this application, the current limiting indicator value includes a current limiting flag and a current limiting start time;

The current limiting indicator value satisfies the current limiting conditions including: the current limiting flag bit is the first flag bit, and the first difference between the current limiting start time and the first time is less than the first threshold, and the first time is determined based on the hash table. The time of the log data current limiting indicator value of the business tag.

In some possible implementations of the embodiments of this application, the current limiting indicator value also includes a statistical value and a statistical time. The statistical value represents the number of first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value;

The processor 610 is also configured to: after the determination module determines the log data current limiting indicator value of the first service tag according to the hash table, if at least one of the current limiting flag bit and the first difference value does not meet the current limiting condition, Add 1 to the statistical value to obtain the updated statistical value;

The input unit 604 is also used to obtain the second difference between the first time and the statistical time when the updated statistical value is greater than the current limiting threshold;

The processor 610 is also configured to prohibit storing the first log data into the cache space of the log system when the second difference is less than the second threshold; and to reset the statistical value, the current limiting flag bit, and the current limiting start time. .

In some possible implementations of the embodiments of this application, the processor 610 is also configured to record the second time when the first log data is obtained after the acquisition module obtains the first log data uploaded by the business system, and start timing from the second time. ;as well as,

If the first log data is not received within the preset timing period, the current limiting indicator value corresponding to the first log data in the hash table is deleted from the first hash node corresponding to the first service label.

The embodiment of this application uses business tags to distinguish different business types, and establishes a hash table based on the hash function to count the log output frequency and output quantity of different business types, and limit the storage of log data that exceeds the threshold. This is used to achieve the purpose of real-time monitoring and flow limiting of log data.

It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (Graphics Processing Unit, GPU) 6041 and a microphone 6042. The graphics processor 6041 is responsible for the processing of images by an image capture device (such as Process the image data of still pictures or videos obtained by the camera). The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072 . Touch panel 6071, also called touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be described again here.

Memory 609 may be used to store software programs as well as various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, Image playback function, etc.) etc. Additionally, memory 609 may include volatile memory or non-volatile memory, or memory 609 may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (Random Access Memory, RAM), static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synch link DRAM) , SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). Memory 609 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, where the application processor mainly handles operations related to the operating system, user interface, application programs, etc., Modem processors mainly process wireless communication signals, such as baseband processors. It can be understood that the above modem processor may not be integrated into the processor 610.

Embodiments of the present application also provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the program or instructions are executed by the processor, each process of the above embodiment of the log processing method applied to electronic devices is implemented, and can achieve the same technical effect, so to avoid repetition, we will not repeat them here.

Wherein, the processor is the processor in the electronic device described in the above embodiment. Readable storage media include computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks or optical disks.

An embodiment of the present application also provides a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above embodiment of the log processing method applied to electronic devices. And can achieve the same technical effect. To avoid repetition, they will not be described again here.

It should be understood that the chips mentioned in the embodiments of this application may also be called system-on-chip, system-on-a-chip, system-on-a-chip or system-on-chip, etc.

Embodiments of the present application provide a computer program product. The program product is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above log processing method embodiments applied to electronic devices, and can To achieve the same technical effect, to avoid repetition, we will not repeat them here.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions may be performed, for example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a computer software product that is essentially or contributes to related technologies. The computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including several instructions to cause a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of this application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Inspired by this application, many forms can be made without departing from the purpose of this application and the scope protected by the claims, all of which fall within the protection of this application.

Claims (11)

1. A log processing method, comprising:

acquiring first log data uploaded by a service system, wherein the first log data comprises a first service tag;

determining a log data flow limiting index value of the first service label according to a hash table, wherein the hash table comprises at least one hash node, the hash node is established based on different service labels and hash functions, and the hash node is used for storing the corresponding relation between the service labels and the flow limiting index value;

and under the condition that the current limiting index value meets the current limiting condition, prohibiting the first log data from being stored in a cache space of a log system.

2. The method of claim 1, wherein determining the log data restriction index value for the first traffic label from the hash table comprises:

comparing the first service tag with the hash table;

under the condition that the hash table does not comprise the first business label, a hash node of the first business label is established, a flow limiting index value corresponding to the first business label is initialized, and the initialized flow limiting index value is determined to be the log data flow limiting index value of the first business label;

And under the condition that the hash table comprises the first service label, determining a log data flow limiting index value of the first service label according to the hash node corresponding to the first service label.

3. The method of claim 1, wherein the current limit indicator value comprises a current limit flag bit and a current limit start time;

the current limiting index value meeting a current limiting condition includes: the current limiting flag bit is a first flag bit, a first difference value between the current limiting start time and a first time is smaller than a first threshold value, and the first time is the time for determining the log data current limiting index value of the first service tag according to a hash table.

4. The method according to claim 3, wherein the current limit index value further includes a statistic value representing the number of the first log data acquired and a statistic time corresponding to a start value of the statistic value;

after the determining the log data flow restriction index value of the first service tag according to the hash table, the method further includes:

adding 1 to the statistical value under the condition that at least one of the current limiting zone bit and the first difference value does not meet the current limiting condition to obtain an updated statistical value;

Acquiring a second difference value between the first time and the statistical time under the condition that the updated statistical value is larger than a current limiting threshold value;

if the second difference value is smaller than a second threshold value, the first log data is forbidden to be stored in a cache space of the log system;

and resetting the statistic value, the current limit flag bit and the current limit starting time.

5. The method of claim 1, wherein after the obtaining the first log data uploaded by the service system, the method further comprises:

recording a second time for acquiring the first log data, and starting timing from the second time;

and deleting the current limiting index value corresponding to the first log data in the hash table from the first hash node corresponding to the first service label under the condition that the first log data is not received within a preset time duration.

6. A log processing apparatus, comprising:

the system comprises an acquisition module, a service system and a service system, wherein the acquisition module is used for acquiring first log data uploaded by the service system, and the first log data comprises a first service tag;

the determining module is used for determining the log data flow limiting index value of the first service label according to a hash table, the hash table comprises at least one hash node, the hash node is established based on different service labels and hash functions, and the hash node is used for storing the corresponding relation between the service labels and the flow limiting index value;

And the processing module is used for prohibiting the first log data from being stored in the cache space of the log system under the condition that the current limiting index value meets the current limiting condition.

7. The apparatus according to claim 6, wherein the determining module is specifically configured to:

comparing the first service tag with the hash table;

under the condition that the hash table does not comprise the first business label, a hash node of the first business label is established, a flow limiting index value corresponding to the first business label is initialized, and the initialized flow limiting index value is determined to be the log data flow limiting index value of the first business label;

and under the condition that the hash table comprises the first service label, determining a log data flow limiting index value of the first service label according to the hash node corresponding to the first service label.

8. The apparatus of claim 6, wherein the current limit indicator value comprises a current limit flag bit and a current limit start time;

the current limiting index value meeting a current limiting condition includes: the current limiting flag bit is a first flag bit, a first difference value between the current limiting start time and a first time is smaller than a first threshold value, and the first time is the time for determining the log data current limiting index value of the first service tag according to a hash table.

9. The apparatus of claim 8, wherein the current limit index value further comprises a statistics value and a statistics time, the statistics value representing the number of the first log data acquired, the statistics time being a recording time corresponding to a start value of the statistics value;

the processing module is further configured to add 1 to the statistics value under a condition that at least one of the current limit flag bit and the first difference value does not satisfy the current limit condition after the determining module determines the log data current limit index value of the first service label according to the hash table, so as to obtain an updated statistics value;

the obtaining module is further configured to obtain a second difference value between the first time and the statistical time when the updated statistical value is greater than a current limiting threshold;

the processing module is further configured to prohibit storing the first log data to a cache space of the log system if the second difference value is smaller than a second threshold value; and resetting the statistics, the current limit flag bit and the current limit start time.

10. The apparatus of claim 6, wherein the processing module is further configured to record a second time for acquiring the first log data after the acquiring module acquires the first log data uploaded by the service system, and start timing from the second time; the method comprises the steps of,

And deleting the current limiting index value corresponding to the first log data in the hash table from the first hash node corresponding to the first service label under the condition that the first log data is not received within a preset time duration.

11. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the method of any one of claims 1 to 5.