CN109992469B - Method and device for merging logs - Google Patents

Abstract

The invention provides a method and a device for merging logs, wherein the method comprises the steps of writing received logs into a set non-persistent cache and a persistent cache in a replicative manner, wherein the cache time of the non-persistent cache to the logs is smaller than that of the persistent cache to the logs. Searching the logs with the same field in the non-persistent cache, acquiring and merging various logs meeting preset backtracking conditions from the searched logs with the same field, and caching the merged logs into the persistent cache. According to the embodiment of the invention, the received log is replicatively written into the set non-persistent cache and the persistent cache, so that a large number of acquisition requests can be prevented from being directly distributed into the persistent cache when the log is acquired later, and the log processing pressure of the persistent cache is reduced. And moreover, the logs with the same fields are combined, so that the storage space of the logs can be effectively saved, and the subsequent centralized management of the logs is facilitated.

Description

A method and device for merging logs

technical field

The invention relates to the field of computer technology, in particular to a method and device for merging logs.

Background technique

Logs are a very broad concept in computer systems, and any program (such as operating system kernel, various application servers, etc.) may output logs. Although the content, scale and purpose of the log are different, its overall function is to record the running status of the software and store the event information generated by the system.

With the continuous development of computer technology, the current log volume is increasing, especially when there are many log backtracking rules (for example, the backtracking rules of the same log include log backtracking every 5 minutes or 1 hour or 5 hours) , It is necessary to continuously obtain logs from the cache database storing logs. Due to the poor performance of the persistent cache, this will put a lot of pressure on the persistent cache database, which will easily affect the log processing performance of the log processing system. Therefore, how to provide sufficient flexibility and good scalability for log caching and processing is a major challenge at present.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a method and device for merging logs that overcome the above problems or at least partially solve the above problems.

According to one aspect of the present invention, a method for merging logs is provided, including:

Write the receive log duplicatively into the set non-persistent cache and persistent cache, wherein the cache duration of the non-persistent cache to the log is shorter than the cache duration of the persistent cache to the log;

Find a log with the same field in the non-persistent cache;

Various logs satisfying preset backtracking conditions are obtained from the found logs of the same field and merged, and the merged logs are cached in the persistent cache.

Optionally, if no log with the same field is found in the non-persistent cache, continue to search for a log with the same field from the persistent cache.

Optionally, before the non-persistent cache and the persistent cache are written to receive log replication, it also includes: setting the first-level cache and the second-level cache as the non-persistent cache, and setting the third-level cache as the non-persistent cache. persistent cache.

Optionally, look for logs with the same fields in the non-persistent cache, including:

A log with the same field is searched from the first-level cache and the second-level cache in sequence.

Optionally, the first-level cache includes redis cluster, the second-level cache includes aerospike, and the third-level cache includes hbase.

Optionally, the method is applied to the kafka system, and the non-persistent cache and the persistent cache of the set receive log replication are written, including:

Logs with the same field in the receiving log are assigned to the same worker trip based on the kafka system, wherein the field includes a user ID or other unique identification keys of the user;

Each worker process writes the allocated log replicas into the non-persistent cache and the persistent cache.

Optionally, the distributing the logs with the same fields in the received logs to the same worker trip based on the kafka system includes:

Use the flink keyby operation to assign logs with the same field to the same worker trip based on the kafka system.

Optionally, look for logs with the same fields in the non-persistent cache, including:

A backtracking operation is started when the preset backtracking time is reached, and a log with the same field is searched in the non-persistent cache.

Optionally, the method also includes:

When the preset backtracking time is reached and the backtracking operation is started, it is judged whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache;

If so, directly search for a log with the same field from the persistent cache.

Optionally, the copying of the received log into the set non-persistent cache and persistent cache includes:

Receive the log and add a timestamp to the log, and write the log with the timestamp into the set non-persistent cache and persistent cache.

Optionally, when the preset backtracking time is reached and the backtracking operation is started, judging whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache includes:

When the preset backtracking time is reached and the backtracking operation is started, it is judged according to the timestamp of the log whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache.

Optionally, the various logs satisfying preset backtracking conditions include:

Logs corresponding to multiple constituent objects in the complete session action, wherein the multiple constituent objects in the complete session action are determined according to business requirements.

Optionally, the method further includes: if various logs satisfying the preset backtracking conditions cannot be obtained from the found logs of the same field, creating a topic according to the found logs of the same field, and setting the maximum look back in time;

When the maximum backtracking time is reached, it is searched from the non-persistent cache and the persistent cache whether there are various logs that are the same as the log fields in the topic and satisfy the preset backtracking conditions;

If so, merge multiple logs with the same fields and satisfy preset backtracking conditions, and cache the merged logs into the persistent cache.

According to another aspect of the present invention, a device for merging logs is also provided, including:

The writing module is adapted to write the received log duplicatively into the set non-persistent cache and persistent cache, wherein the cache duration of the non-persistent cache to the log is shorter than the cache duration of the persistent cache to the log;

A search module, adapted to search for logs with the same field in the non-persistent cache;

The merging module is adapted to acquire and merge various logs satisfying preset backtracking conditions from the found logs of the same field, and cache the merged logs into the persistent cache.

Optionally, the search module is also suitable for:

If no log with the same field is found in the non-persistent cache, continue to search for a log with the same field from the persistent cache.

Optionally, the device also includes:

The setting module is adapted to set the first-level cache and the second-level cache as non-persistent caches before the writing module writes the received log replication into the set non-persistent cache and persistent cache, and sets the third-level The cache acts as a persistent cache.

Optionally, the search module is also suitable for:

A log with the same field is searched from the first-level cache and the second-level cache in sequence.

Optionally, the first-level cache includes redis cluster, the second-level cache includes aerospike, and the third-level cache includes hbase.

Optionally, the device is applied to the kafka system, and the writing module is also suitable for:

Logs with the same field in the receiving log are assigned to the same worker trip based on the kafka system, wherein the field includes a user ID or other unique identification keys of the user;

Each worker process writes the allocated log replicas into the non-persistent cache and the persistent cache.

Optionally, the writing module is also suitable for:

Use the flink keyby operation to assign logs with the same field to the same worker trip based on the kafka system.

Optionally, the search module is also suitable for:

A backtracking operation is started when the preset backtracking time is reached, and a log with the same field is searched in the non-persistent cache.

Optionally, the device also includes:

The judging module is adapted to determine whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache when the preset backtracking time is reached and the backtracking operation is started;

If yes, the search module directly searches the log with the same field from the persistent cache.

Optionally, the writing module is also suitable for:

Receive the log and add a timestamp to the log, and write the log with the timestamp into the set non-persistent cache and persistent cache.

Optionally, the judging module is also suitable for:

When the preset backtracking time is reached and the backtracking operation is started, it is judged according to the timestamp of the log whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache.

Optionally, the various logs satisfying preset backtracking conditions include:

Logs corresponding to multiple constituent objects in the complete session action, wherein the multiple constituent objects in the complete session action are determined according to business requirements.

Optionally, the device also includes:

Establishing a module adapted to create a topic based on the found logs of the same field if the merging module cannot obtain multiple logs that meet the preset backtracking conditions from the found logs of the same field, and setting the maximum backtracking for the established topic time;

When the search module reaches the maximum backtracking time, it searches from the non-persistent cache and the persistent cache whether there are various logs that are the same as the log fields in the topic and satisfy the preset backtracking conditions;

If so, the merging module merges various logs having the same fields and meeting preset backtracking conditions, and caches the merged logs into the persistent cache.

According to another aspect of the present invention, an electronic device is also provided, including:

processor; and

A memory arranged to store computer-executable instructions which, when executed, cause the processor to perform the method of merging logs according to any of the above embodiments.

According to yet another aspect of the present invention, a computer storage medium is also provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs are used when an electronic device including multiple application programs When executing, the electronic device is made to execute the method for merging logs according to any of the above embodiments.

In the embodiment of the present invention, when a log is received, the received log is duplicatively written into the set non-persistent cache and the persistent cache, wherein the non-persistent cache has a shorter cache duration for the log than the persistent cache pair Log cache duration. After writing the log, you can also search for logs with the same fields in the non-persistent cache, and obtain and merge various logs that meet the preset backtracking conditions from the found logs with the same fields, and cache the merged logs to the persistent cached. Therefore, in the embodiment of the present invention, by setting the non-persistent cache and the persistent cache, and writing the received logs to the set non-persistent cache and the persistent cache, it is possible to avoid a large number of subsequent log acquisitions. Get requests are directly assigned to the persistent cache to reduce the log processing pressure of the persistent cache. Further, merging the logs with the same field can also effectively save the storage space of the logs, and facilitate subsequent centralized management of the logs.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Those skilled in the art will be more aware of the above and other objects, advantages and features of the present invention according to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:

FIG. 1 shows a schematic flowchart of a method for merging logs according to an embodiment of the present invention;

FIG. 2 shows a schematic flowchart of a method for merging logs according to another embodiment of the present invention;

FIG. 3 shows a schematic structural diagram of an apparatus for merging logs according to an embodiment of the present invention;

FIG. 4 shows a schematic structural diagram of an apparatus for merging logs according to another embodiment of the present invention;

Figure 5 shows a block diagram of a computing device for performing the method for merging logs according to the present invention; and

FIG. 6 shows a storage unit for holding or carrying program codes for realizing the method for merging logs according to the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the above technical problems, the embodiment of the present invention provides a method for merging logs, which can be applied in the Kafka system. Kafka is a high-throughput distributed publish-subscribe message system, which can handle consumer-scale For all action flow data in the website, the main goal of kafka development is to build a data processing framework for processing massive logs, user behavior and website operation statistics. Fig. 1 shows a schematic flowchart of a method for merging logs according to an embodiment of the present invention. Referring to Fig. 1, the method at least includes step S102 to step S106.

In step S102, copying the received log is written into the set non-persistent cache and persistent cache, wherein the cache duration of the log in the non-persistent cache is shorter than the cache duration of the log in the persistent cache.

In this step, copying the log into the set non-persistent cache and the persistent cache refers to not only writing the log to the set non-persistent cache after receiving a log, but also Write the log to the set persistent cache. The way to write the log can be to write two types of caches (ie, non-persistent cache and persistent cache) at the same time, or write to one type of cache first, and then write to another type of cache, which is not done in the embodiment of the present invention. Specific limits.

Step S104, searching for a log with the same field in the non-persistent cache.

In this step, the logs with the same fields mean that although the contents of the logs are different (or the same), they have the same fields. Wherein, the field may include a user ID, and may also include other unique identification keys of the user, and the like.

Step S106, obtaining and merging various logs satisfying preset backtracking conditions from the found logs of the same field, and caching the merged logs into a persistent cache.

In the embodiment of the present invention, when a log is received, the received log is duplicatively written into the set non-persistent cache and the persistent cache, wherein the non-persistent cache has a shorter cache duration for the log than the persistent cache for the log cache duration. After writing the log, you can also search for logs with the same fields in the non-persistent cache, and obtain and merge various logs that meet the preset backtracking conditions from the found logs with the same fields, and cache the merged logs to the persistent cached. Therefore, in the embodiment of the present invention, by setting the non-persistent cache and the persistent cache, and writing the received logs to the set non-persistent cache and the persistent cache, it is possible to avoid a large number of subsequent log acquisitions. Get requests are directly assigned to the persistent cache to reduce the log processing pressure of the persistent cache. Further, merging the logs with the same field can also effectively save the storage space of the logs, and facilitate subsequent centralized management of the logs.

Referring to step S102 above, in an embodiment of the present invention, two types of caches, the non-persistent cache and the persistent cache, need to be set before the received log is replicated into the set non-persistent cache and the persistent cache. For example, the first-level cache and the second-level cache can be set as non-persistent caches, and the third-level cache can be set as persistent caches. Among them, the first-level cache can use redis cluster (ie Redis cluster), Redis is an open source Key-Value database, and provides API (Application Program Interface, application program interface) in multiple languages. The second-level cache can use the aerospike database, which is a distributed and scalable NoSql (Not Only SQL, non-relational database). The third-level cache can use the hbase database, which is a distributed, column-oriented open source database. The databases used by the caching at all levels here are only illustrative, and may also include other types of databases, which are not specifically limited in this embodiment of the present invention.

In the embodiment of the present invention, the non-persistent cache can cache logs for a long time much shorter than the persistent cache for logs. For example, the cache duration of the non-persistent cache is 1 hour, 2 hours, etc., and the cache duration of the persistent cache is 10 days, 20 days, etc., or the cache duration of the persistent cache can not be set for a specific time, that is As long as the logs cached in the persistent cache are not manually deleted, the logs can be kept forever. If the non-persistent cache includes the first-level cache and the second-level cache, you can set corresponding cache durations for the first-level cache and the second-level cache respectively, and the cache durations of the two caches can be the same or different. For example, if the cache duration of the first-level cache and the second-level cache are both 1 hour, then the logs in the two-level cache will be updated every 1 hour, that is, the original cached logs will be deleted, and the newly received logs will continue to be cached . If the persistent cache includes a third-level cache, its cache duration is 10 days. Similarly, the logs in the third-level cache are updated every 10 days. However, if the third-level cache does not configure the cache duration, the logs in it It can be stored permanently, of course, users can manually delete the logs in the persistent cache. The embodiment of the present invention does not specifically limit the cache time of the non-persistent cache and the persistent cache.

In the embodiment of the present invention, the received logs are cached in databases at all levels rather than system memory. The main reason is that if the logs are stored in the system memory, the logs in the memory may be lost when the system is restarted. , that is, before the system restarts, the logs cached in the memory will be missed, so that only the logs in the persistent cache (such as the hbase database) remain. When there are subsequent business use logs, the logs can only be obtained from the persistent cache, which may make the processing performance of the persistent cache unable to keep up with the needs of the business. By setting non-persistent cache and persistent cache, such as first-level cache, second-level cache, and third-level cache, the logs in each level of cache will not be lost after the system is restarted, and the logs in each level of cache can satisfy Various business needs avoid the log processing pressure of increasing the persistent cache.

Referring to step S104 above, in an embodiment of the present invention, if the non-persistent cache includes a first-level cache and a second-level cache. Then, when looking for logs with the same field in the non-persistent cache, you can search for logs with the same field from the first-level cache and the second-level cache in turn, that is, first look for the logs with the same field from the first-level cache log, if yes, you can obtain multiple logs that meet the preset backtracking conditions from logs with the same field, and if not, look for logs with the same field from the second-level cache. Certainly, if the log with the same field is still not found in the second-level cache, the log with the same field may continue to be searched from the persistent cache (such as the third-level cache).

In an embodiment of the present invention, the process of searching for a log with the same field in the non-persistent cache is the backtracking process of the cached log. When the preset backtracking time is reached and the backtracking operation (that is, the join operation) is started, a log with the same field is searched in the non-persistent cache. Wherein, the preset look-back time may be 5 minutes, 1 hour, 5 hours, etc., which is not specifically limited in this embodiment of the present invention.

In this embodiment, when the preset backtracking time is reached and the backtracking operation is started, it is also possible to first determine whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache, and if so, it can be directly searched from the persistent cache Logs with the same fields. Since the cache time of the non-persistent cache is short, that is, the storage time of the log in it is short, the log that needs to be backtracked may not be in the non-persistent cache when the preset backtracking time is reached. Therefore, when judging the backtracking time and the log When the write time difference exceeds the cache duration of the non-persistent cache, you can directly look up logs with the same field from the persistent cache to reduce unnecessary operation processes and improve backtracking efficiency.

Among them, when judging whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache, it can be judged based on the timestamp of the log in the non-persistent cache whether the difference between the lookback time and the log writing time exceeds the cache duration of the non-persistent cache. The timestamp of the log can be a timestamp added to the log after the log is received, and then the log with the timestamp is replicated and written into the set non-persistent cache and persistent cache.

Referring to step S106 above, in the embodiment of the present invention, the various logs satisfying the preset backtracking conditions may be the logs corresponding to multiple constituent objects in the complete session action. Among them, multiple constituent objects in the complete session action are determined according to business requirements. For example, in business 1, a complete session action may include user traffic bidding, display, and click operations, where bidding, display, and click operations are the three components of the complete session action of business 1. In business 2, a complete session action may include bidding and display of user traffic, excluding click operations, where bidding and display are two components of the complete session action of business 2. Therefore, the complete session actions corresponding to different business requirements are also different.

In this embodiment, if various logs satisfying the preset backtracking condition cannot be obtained from the found logs of the same field. At this point, various logs that do not meet the preset backtracking conditions can be treated as special log types, that is, a topic is created based on the found logs of the same field, and the maximum backtracking time is set for the created topic. And when the maximum backtracking time is reached, check from the non-persistent cache and the persistent cache whether there are various logs that have the same log field as the topic and meet the preset backtracking conditions. If they exist, merge them with the same field and meet the preset Various logs with backtracking conditions set, and the merged logs are cached in the persistent cache. Among them, the Kafka system can use topics to classify and manage logs. One business can apply for multiple topics, and the logs of one topic can also be shared by multiple businesses.

For example, in business 1 mentioned above, only the logs corresponding to user A’s traffic bidding and display can be found during the backtracking process, but no logs corresponding to user A’s click operations can be found, that is, the backtracking is not successful. At this point, you can create a topic based on the found user A's log, and set the maximum backtracking time for the established topic. For example, if the maximum backtracking time is set to 5 hours, then after 5 hours, the non-persistent cache and persistent cache Check again whether there are logs corresponding to the bidding, display, and click operations of user A's traffic. If yes, merge the found logs and cache the merged logs in the persistent cache.

What needs to be explained here is that most of the backtracking operations can be done in seconds. The maximum backtracking time set refers to the maximum backtracking time that the business side can tolerate, that is, the business side can tolerate the maximum number of backtracking success logs waiting time. Of course, for user logs that have not been backtracked successfully, before reaching the maximum backtracking time (such as 5 hours, etc.), it is possible that the user logs that have not been backtracked successfully before meet the preset backtracking conditions. For example, in the previous backtracking process, only the logs corresponding to user A’s traffic bidding and display can be found, but no logs corresponding to user A’s click operations can be found, and the maximum backtracking time is set to 5 hours. If in the next 1 If the click log is received within minutes, the backtracking operation can no longer be performed when the set maximum backtracking time is reached.

Therefore, the maximum backtracking time can actually be considered as a bottom-line guarantee, that is, to try whether the log can be backtracked successfully within the maximum tolerance time on the business side. Because some business rules are that even if the backtracking is not successful within the maximum tolerance time (ie, the maximum backtracking time), it is necessary to merge some of the backtracking successful logs.

Continuing to refer to step S106, in a real-time method of the present invention, after the log is merged successfully, it can be directly cached in the persistent cache for offline job processing. Of course, after the log is successfully merged, it can be directly distributed outside the center of the Kafka system for real-time job processing.

The embodiment of the present invention also provides another method for merging logs. Fig. 2 shows a schematic flowchart of a method for merging logs according to another embodiment of the present invention. Referring to Fig. 2, the method at least includes step S202 to step S212.

Step S202, assigning logs with the same fields in the received logs to the same worker journey in the Kafka-based system. Wherein, the field may include a user ID or other unique identification keys of the user.

In this step, the flink keyby operation (such as hash) can be used to assign logs with the same field to the same worker process in the kafka-based system. Flink is an "open source technology stack" similar to spark, which can provide batch processing, streaming computing, graph computing, interactive query, machine learning and other functions. By assigning logs with the same field to the same worker itinerary, it is possible to ensure that the logs with the same field can be merged more conveniently in the future, and then effectively ensure the consistency of the logs when the system is expanded or upgraded. For example, through the flink keyby operation, the log with the user ID "11" is allocated to worker1, the log with the user ID "12" is allocated to worker2, and the log with the user ID "13" is allocated to worker3.

In step S204, each worker process writes the assigned log into the non-persistent cache and the persistent cache.

In this step, since the log of the same field has been assigned to the same worker trip above, each worker trip can copy the log of the same field assigned to itself to the non-persistent cache and persistence cached. For example, in combination with the above embodiment, worker1 is assigned to the log with user ID "11", and worker2 is assigned to the log with user ID "12", then worker1 will copy the log with user ID "11" into the non-persistent cache In the persistent cache, worker2 writes the log replication of the user ID "12" into the non-persistent cache and the persistent cache, and worker3 writes the log replication of the user ID "13" into the non-persistent cache and persistent cache.

Step S206, check whether there is a log with the same field in the non-persistent cache, if yes, execute step S208; if not, execute step S210.

In step S208, various logs satisfying preset backtracking conditions are acquired from the found logs of the same field and merged, and the merged logs are cached in a persistent cache, and step S212 is executed.

Step S210, check whether there is a log with the same field in the persistent cache, if yes, execute step S208; if not, execute step S212, and end.

The embodiment of the present invention can perform flow processing on the received logs based on the kafka system and flink, and then realize the expansion and contraction of the system through the configuration of parallelism.

Based on the same inventive concept, an embodiment of the present invention also provides an apparatus for merging logs. FIG. 3 shows a schematic structural diagram of an apparatus for merging logs according to an embodiment of the present invention. Referring to FIG. 3 , an apparatus 300 for merging logs includes writing Input module 310, search module 320 and merge module 330.

The functions of the various components or devices of the log merging apparatus 300 according to the embodiment of the present invention and the connection relationship between the various parts are now introduced:

The writing module 310 is adapted to write the received log duplicatively into the set non-persistent cache and persistent cache, wherein the cache duration of the non-persistent cache to the log is shorter than the cache duration of the persistent cache to the log;

The search module 320, coupled with the write module 310, is adapted to search for logs with the same field in the non-persistent cache;

The merging module 330, coupled with the search module 320, is adapted to obtain and merge various logs satisfying preset backtracking conditions from the found logs of the same field, and cache the merged logs into a persistent cache. Among them, the various logs satisfying the preset backtracking conditions may include logs corresponding to multiple constituent objects in the complete session action, wherein the multiple constituent objects in the complete session action are determined according to business requirements.

In an embodiment of the present invention, the search module 320 is further adapted to continue searching for a log with the same field from the persistent cache if no log with the same field is found in the non-persistent cache.

In an embodiment of the present invention, the device for merging logs can also be applied to the kafka system, and the writing module 310 is also suitable for distributing logs with the same field in the received log to the same worker trip based on the kafka system, wherein the field Including the user ID or other unique identification keys of the user. Each worker trip writes the assigned log replicas into the non-persistent cache and the persistent cache.

In an embodiment of the present invention, the write module 310 is also adapted to assign logs with the same field to the same worker process in the kafka-based system by using the flink keyby operation.

In an embodiment of the present invention, the search module 320 is further adapted to start a backtracking operation when the preset backtracking time is reached, and search for a log with the same field in the non-persistent cache.

The embodiment of the present invention also provides another device for merging logs. Referring to FIG. 4 , the device 300 for merging logs includes a setting module 340 , a judging module 350 and a building module 360 in addition to the above modules.

The setting module 340, coupled with the writing module 310, is suitable for setting the first-level cache and the second-level cache as non-persistent cache, and set the third-level cache as a persistent cache.

The judging module 350 is coupled with the writing module 310 and the searching module 320 respectively, and is suitable for starting after the writing module 310 writes the duplication of the received log into the set non-persistent cache and the persistent cache, when the preset backtracking time is reached and starts During the backtracking operation, it is judged whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache, and if so, the search module 320 directly searches the log with the same field from the persistent cache.

The establishment module 360 is separately coupled with the merge module 330 and the search module 320, and is suitable for if the merge module 330 cannot obtain various logs satisfying preset backtracking conditions from the logs of the same field found, then according to the logs of the same field found The log creates a topic, and sets the maximum backtracking time for the created topic. When the maximum backtracking time is reached, the search module 320 searches from the non-persistent cache and the persistent cache whether there are multiple logs that are the same as the log fields in the topic and satisfy the preset backtracking conditions. If so, the merge module 330 merges them with the same Fields and various logs that meet the preset backtracking conditions, and the merged logs are cached in the persistent cache.

In an embodiment of the present invention, the search module 320 is further adapted to search logs with the same field from the first-level cache and the second-level cache in sequence. Among them, the first-level cache can include redis cluster, the second-level cache can include aerospike, and the third-level cache can include hbase.

In an embodiment of the present invention, the writing module 310 is further adapted to receive the log, add a time stamp to the log, and write the log carrying the time stamp into the configured non-persistent cache and persistent cache.

In an embodiment of the present invention, the judging module 350 is further adapted to, when the preset backtracking time is reached and the backtracking operation is started, judge whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache according to the timestamp of the log.

According to any one of the above preferred embodiments or a combination of multiple preferred embodiments, the embodiments of the present invention can achieve the following beneficial effects:

In the embodiment of the present invention, when a log is received, the received log is duplicatively written into the set non-persistent cache and the persistent cache, wherein the non-persistent cache has a shorter cache duration for the log than the persistent cache pair Log cache duration. After writing the log, you can also search for logs with the same fields in the non-persistent cache, and obtain and merge various logs that meet the preset backtracking conditions from the found logs with the same fields, and cache the merged logs to the persistent cached. Therefore, in the embodiment of the present invention, by setting the non-persistent cache and the persistent cache, and writing the received logs to the set non-persistent cache and the persistent cache, it is possible to avoid a large number of subsequent log acquisitions. Get requests are directly assigned to the persistent cache to reduce the log processing pressure of the persistent cache. Further, merging the logs with the same field can also effectively save the storage space of the logs, and facilitate subsequent centralized management of the logs.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that although some embodiments herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. And form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the apparatus for merging logs according to the embodiment of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

According to yet another aspect of the present invention, there is also provided an electronic device, comprising a processor; and a memory arranged to store computer-executable instructions, which when executed cause the processor to perform the The method of merging logs.

According to another aspect of the present invention, a computer storage medium is also provided, wherein the computer storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device Execute the method for merging logs according to any of the above embodiments.

For example, Figure 5 illustrates a computing device that may implement a method of merging logs. The computing device conventionally includes a processor 510 and a computer program product in the form of a memory 520 or a computer-readable medium. Memory 520 may be electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. The memory 520 has a storage space 530 storing a program code 531 for performing any method step in the method described above. For example, the storage space 530 storing program codes may include respective program codes 531 for respectively implementing various steps in the above methods. These program codes can be read from or written into one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. Such a computer program product is typically, for example, a portable or fixed storage unit as shown in FIG. 6 . The storage unit may have storage segments, storage spaces, etc. arranged similarly to the memory 520 in the computing device of FIG. 5 . The program code can eg be compressed in a suitable form. Typically, the memory unit includes computer readable code 531' for performing the method steps of the present invention, i.e. code readable by a processor such as 510, which when executed by a computing device causes the computing device to execute steps in the method described above.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Claims (22)

1. A method of merging logs, the method being applied to a kafka system, comprising:

writing received logs into a set non-persistent cache and a persistent cache in a replicative manner, wherein the cache time of the non-persistent cache to the logs is smaller than that of the persistent cache to the logs, and the non-persistent cache comprises a first-level cache and a second-level cache;

When reaching the preset backtracking time and starting the backtracking operation, judging whether the difference between the backtracking time and the log writing time exceeds the cache time of the non-persistent cache;

if yes, directly searching the logs with the same fields from the persistent cache;

acquiring and combining various logs meeting preset backtracking conditions from the searched logs in the same field, and caching the combined logs into the persistent cache;

if multiple logs meeting preset backtracking conditions cannot be obtained from the searched logs in the same field, building a topic according to the searched logs in the same field, and setting the maximum backtracking time for the built topic;

searching whether various logs which are the same as the log fields in the topic and meet preset backtracking conditions exist in the non-persistent cache and the persistent cache when the maximum backtracking time is reached;

if yes, merging various logs which have the same field and meet preset backtracking conditions, and caching the merged logs into the persistent cache.

2. The method of claim 1, further comprising:

if the logs with the same fields are not found in the non-persistent cache, continuing to find the logs with the same fields from the persistent cache.

3. The method of claim 1 or 2, wherein prior to the replicative writing of the received log into the set non-persistent cache and persistent cache, further comprising: setting a first-level cache and a second-level cache as non-persistent caches, and setting a third-level cache as a persistent cache.

4. The method of claim 3, wherein looking up the log with the same field in the non-persistent cache comprises:

and searching logs with the same field from the first-level cache and the second-level cache in sequence.

5. The method of claim 3, wherein the first level cache comprises a redis cluster, the second level cache comprises an aerosepike, and the third level cache comprises hbase.

6. The method of claim 1 or 2, wherein the method is applied to a kafka system, the replicative writing of a received log into a set non-persistent cache and persistent cache, comprising:

distributing the logs with the same fields in the received logs to the same worker travel in the kafka system, wherein the fields comprise user IDs or other unique identification keys of users;

and each worker stroke duplicatively writes the allocated log into the non-persistent cache and the persistent cache.

7. The method of claim 6, wherein the assigning logs with the same fields in the received log to the same worker run in the kafka-based system comprises:

the flink key operation is used to distribute the logs with the same fields to the same worker run in the kafka system.

8. The method of claim 1, wherein the replicatively writing the received log into the set non-persistent cache and persistent cache comprises:

and receiving the log, adding a time stamp to the log, and copying the log carrying the time stamp into the set non-persistent cache and the persistent cache.

9. The method of claim 8, wherein when the preset trace back time is reached and the trace back operation is started, determining whether a difference between the trace back time and the log writing time exceeds a cache duration of the non-persistent cache comprises:

when the preset backtracking time is reached and the backtracking operation is started, judging whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache according to the time stamp of the log.

10. The method according to claim 1 or 2, wherein the plurality of logs satisfying a preset backtracking condition include:

And the logs corresponding to the plurality of component objects in the complete session action are obtained by determining the plurality of component objects in the complete session action according to the service requirement.

11. An apparatus for merging logs, wherein the apparatus is applied to a kafka system, comprising:

the writing module is suitable for writing the received log into a set non-persistent cache and a persistent cache, wherein the cache time of the non-persistent cache to the log is smaller than that of the persistent cache to the log, and the non-persistent cache comprises a first-level cache and a second-level cache;

the judging module is suitable for judging whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache when the preset backtracking time is reached and the backtracking operation is started;

if yes, the searching module is suitable for directly searching the logs with the same fields from the persistent cache;

the merging module is suitable for acquiring and merging various logs meeting preset backtracking conditions from the searched logs in the same field, and caching the merged logs into the persistent cache;

the establishing module is suitable for establishing a topic according to the searched logs of the same field if the merging module cannot acquire various logs meeting preset backtracking conditions from the searched logs of the same field, and setting the maximum backtracking time for the established topic;

The searching module searches whether various logs which are the same as the log fields in the topic and meet preset backtracking conditions exist in the non-persistent cache and the persistent cache when the maximum backtracking time is reached;

if yes, the merging module merges various logs which have the same field and meet preset backtracking conditions, and caches the merged logs into the persistent cache.

12. The apparatus of claim 11, wherein the lookup module is further adapted to:

if the logs with the same fields are not found in the non-persistent cache, continuing to find the logs with the same fields from the persistent cache.

13. The apparatus of claim 11 or 12, further comprising:

the setting module is suitable for setting the first-level cache and the second-level cache as non-persistent caches and setting the third-level cache as persistent caches before the writing module is used for reproducibly writing the received log into the set non-persistent caches and persistent caches.

14. The apparatus of claim 13, wherein the lookup module is further adapted to:

and searching logs with the same field from the first-level cache and the second-level cache in sequence.

15. The apparatus of claim 13, wherein the first level cache comprises a redis cluster, the second level cache comprises an aeropike, and the third level cache comprises a hbase.

16. The apparatus of claim 11 or 12, the writing module further adapted to:

distributing the logs with the same fields in the received logs to the same worker travel in the kafka system, wherein the fields comprise user IDs or other unique identification keys of users;

and each worker stroke duplicatively writes the allocated log into the non-persistent cache and the persistent cache.

17. The apparatus of claim 16, wherein the writing module is further adapted to:

the flink key operation is used to distribute the logs with the same fields to the same worker run in the kafka system.

18. The apparatus of claim 17, wherein the writing module is further adapted to:

and receiving the log, adding a time stamp to the log, and copying the log carrying the time stamp into the set non-persistent cache and the persistent cache.

19. The apparatus of claim 18, wherein the determination module is further adapted to:

When the preset backtracking time is reached and the backtracking operation is started, judging whether the difference between the backtracking time and the log writing time exceeds the cache duration of the non-persistent cache according to the time stamp of the log.

20. The apparatus according to claim 11 or 12, wherein the plurality of logs satisfying a preset backtracking condition include:

and the logs corresponding to the plurality of component objects in the complete session action are obtained by determining the plurality of component objects in the complete session action according to the service requirement.

21. An electronic device, comprising:

a processor; and

a memory arranged to store computer executable instructions which when executed cause the processor to perform the method of merging logs according to any of claims 1-10.

22. A computer storage medium storing one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of merging logs according to any of claims 1-10.