CN111913922B - Binary structured log generation method, device, equipment and storage medium - Google Patents

Abstract

Embodiments of the present invention disclose methods, devices, equipment and storage media for generating binary structured logs. The method includes: obtaining an application request that matches user behavior, where the application request includes: behavior data in the form of key-value pairs; The standard structure model matching the application request processes the behavioral data in the form of key-value pairs in the application request to obtain a structured log that matches the application request; processes the structured log as Binary structured log and written to file. The technical solution of the embodiment of the present invention uses a standard structure model to process behavioral data in application requests and then generates structured logs without generating any intermediate files, saving storage resources and avoiding complex log parsing processes. At the same time, through The serialization operation obtains binary structured logs, which greatly reduces the log volume, facilitates log transmission, and reduces resource consumption during data transmission.

Description

Binary structured log generation method, device, equipment and storage medium

Technical field

Embodiments of the present invention relate to the field of the Internet, and in particular to methods, devices, equipment and storage media for generating binary structured logs.

Background technique

With the continuous development of Internet technology, the number of Internet application groups and business types are constantly growing. The operation of various businesses causes the Internet to generate a large number of logs every day, and these logs are an important part of Internet big data analysis. Therefore, they are extremely important. High mining and analysis value.

To process these huge amounts of logs, you first need to perform log parsing, that is, extract meaningful fields and transform unstructured data into structured data. Usually all logs are printed into text files, and then the text files are parsed by writing a log analysis program.

However, different types of log parsing programs need to be written for different formats of logs. Changes in the log format require downstream parsing programs to also change accordingly. Therefore, the parsing cost is high, and the parsing process is complex and error-prone. In addition, due to the excessive amount of log data, Large, a large amount of resources are consumed during storage and transmission, so storage and transmission costs are extremely high.

Contents of the invention

Embodiments of the present invention provide methods, devices, computer equipment and storage media for generating binary structured logs. In order to directly generate structured logs according to user application requests, it avoids the complex log parsing process and saves resource consumption in the storage and transmission process of log data.

In the first aspect, embodiments of the present invention provide a method for generating binary structured logs, including:

Obtain an application request that matches the user's behavior, where the application request includes: behavior data in the form of key-value pairs;

According to the standard structure model that matches the application request, the behavioral data in the form of key-value pairs in the application request is processed to obtain a structured log that matches the application request; wherein, the standard structure The volume model includes: at least one standard field name, and an area to be filled that matches the standard field name;

The structured log is processed into a binary structured log and written to a file to generate a binary structured file.

In the second aspect, embodiments of the present invention provide a device for generating binary structured logs, including:

The application request acquisition module is used to obtain application requests that match user behavior, where the application requests include: behavior data in the form of key-value pairs;

A structured log acquisition module, configured to process the behavioral data in the form of key-value pairs in the application request according to a standard structure model that matches the application request, and obtain a structured log that matches the application request. Log; wherein, the standard structure model includes: at least one standard field name, and an area to be filled that matches the standard field name;

A binary structured log acquisition module is used to process the structured log into a binary structured log and write it to a file to generate a binary structured file.

In a third aspect, embodiments of the present invention provide a computer device, which includes: one or more processors;

A storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the binary structured log generation method described in any embodiment of the present invention.

In a fourth aspect, embodiments of the present invention also provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating a binary structured log as described in any embodiment of the present invention is implemented.

The technical solution of the embodiment of the present invention uses a standard structure model to process behavioral data in application requests, and then directly generates structured logs without generating any intermediate files, saving storage resources, avoiding complex log parsing processes, and saving time. Analyze costs. At the same time, binary structured logs are obtained through serialization operations, which are easy for computers to read, reduce log volume, facilitate log transmission, improve data transmission efficiency, and reduce resource consumption during data transmission.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and therefore do not It should be regarded as a limitation of the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a binary structured log generation method provided by Embodiment 1 of the present invention;

Figure 2 is a flow chart of a binary structured log generation method provided by Embodiment 2 of the present invention;

Figure 3 is a structural block diagram of a binary structured log generation device provided in Embodiment 3 of the present invention;

Figure 4 is a structural block diagram of a computer device provided by Embodiment 4 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It can be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for convenience of description, only some but not all structures related to the present invention are shown in the drawings.

In addition, it should be noted that, for convenience of description, only part but not all of the content related to the present invention is shown in the drawings. Before discussing example embodiments in more detail, it should be mentioned that some example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe various operations (or steps) as a sequential process, many of the operations may be performed in parallel, concurrently, or simultaneously. Additionally, the order of operations can be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The processing may correspond to a method, function, procedure, subroutine, subroutine, or the like.

Embodiment 1

Figure 1 is a flow chart of a method for generating binary structured logs provided in Embodiment 1 of the present invention. This embodiment can be applied to situations where structured logs are directly generated according to user application requests without generating intermediate files. This method can be implemented by the present invention. The binary structured log generation device in the example is executed. This device can be implemented in software and/or hardware, and can generally be integrated on the server. As shown in Figure 1, this method specifically includes the following operations:

S110. Obtain an application request that matches the user's behavior, where the application request includes: behavior data in the form of key-value pairs.

Generally speaking, when a client user uses an application (APP), he or she will send various application requests to the server, such as route search requests, navigation requests, or purchase requests. The above application request is generally sent to the server in the form of an HTTP request. The HTTP request includes behavioral data in the form of various key-value pairs. Therefore, the above HTTP request is generally printed directly to a file by the server and the above file is used as log data. for storage.

User behavior includes at least one behavioral data, involving one or more of many elements such as time, place, person, interaction, and interactive content; for example, if the user searches for a certain item, then the time of the search behavior includes the time when the search occurs and/or the time spent on the page; the location includes the user's geographical location, network address and/or click location on the page; the person includes the user's personal information, such as the account information logged in or registered when searching, and the terminal used Device, platform, channel and/or network information, for example, the terminal device name and/or IP address used in the search; interaction, that is, the type of operation performed this time is search; interaction content, that is, the search content and related information searched out .

Application requests match user behavior. For example, if a user issues a search request, it represents the user's search behavior; if a user issues a purchase request, it represents the user's purchasing behavior. Application requests can be triggered by clicking function buttons, for example, clicking the "Search" button or "Purchase" button to trigger a search request or purchase request; they can also be triggered by clicking text or pictures, for example, clicking text or pictures to view detailed content triggers the user's Browsing request; it can also be triggered by the biometric recognition function of the terminal device, for example, the payment request is triggered by fingerprint recognition or facial recognition of the mobile terminal; in the embodiment of the present invention, optionally, there is no specific limit on the triggering method of the application request. . The application request can be issued by an application client installed on the smart terminal, or by a page accessed by a third-party browser. In this embodiment of the present invention, optionally, the source of the application request is not specifically limited.

The application request includes: behavioral data in the form of key-value pairs. Key-value pair is a data storage method, including key name and key value. The key name represents the name of the behavioral data, and the key value represents the value assigned to the behavioral data, that is, the specific content assigned; for example, a key-value pair is "Amount = 50 yuan", then "amount" is the key name of the behavior data, and "50 yuan" represents the value of the behavior data, that is, the specific content assigned.

S120. Process the behavioral data in the form of key-value pairs in the application request according to the standard structure model that matches the application request, and obtain a structured log that matches the application request; wherein, The standard structure model includes: at least one standard field name, and an area to be filled that matches the standard field name.

In the existing technology, the server uses application requests as log data, uses text log generation tools, such as log4j and glog, etc., to print and generate text logs, parses the text logs by writing a log analysis program, and then performs structured processing on the text logs. , but different text logs require writing different log analysis programs. Therefore, once the log format of the log source changes, the downstream parsing program must also be modified, and the parsing cost is extremely high. Compared with the existing technology, this embodiment uses the standard structure model to directly generate a structured log from the user's application request. It does not need to go through the step of printing the application request to generate a file. No intermediate file is generated, and the application request is Direct storage as structured logs saves storage resources and avoids complex log parsing processes.

The standard structure model is established based on various information included in the application request, and different application requests correspond to different standard structure models. The standard structure model can be built manually by staff. For example, for a user's purchase request, it includes multiple information such as the user's name, contact information, product name, order time and address, so it can be pre-established including the above information. The standard structure model in the application is used to process purchase requests; the standard structure model can also be automatically created according to the application request. For example, when the obtained application request does not find the corresponding standard structure model, a standard structure model is created according to the application request. structure model, and use the key name of the key-value pair included in the application request as the standard field name of the standard structure model, and set an area to be filled for each standard field name. In the embodiment of the present invention, optionally, there is no specific limitation on the way to establish the standard structure model.

The standard structure model includes at least one standard field name and an area to be filled that matches the standard field name. Optionally, in this embodiment of the present invention, a standard field name is sequentially obtained in the standard structure model as the target field name; if there is a target key name matching the target field name in the behavioral data, then Fill the key value corresponding to the target key name in the area to be filled corresponding to the target field name in the standard structure model; return to the execution to obtain a standard field name in the standard structure model as The target field name is operated until all standard field names in the standard structure model are processed. If there is no target field value matching the target field name in the behavioral data, the filling process of the area to be filled corresponding to the target field name in the standard structure model is abandoned.

For example, the standard field names of the standard structure model include: "Amount", "Time", "Product Name" and "Account Type", as well as the areas to be filled that match each standard field name; the application request in the form of key-value pairs Behavioral data includes: "amount = 10 yuan", "time = 5 o'clock", "order number = 208A", "product name = clothes" and "account type = member"; the standard structure model is obtained sequentially according to the above filling rules "Amount", "Time", "Product Name" and "Account Type" are used as target field names, and based on the behavioral data in the form of key-value pairs in the application request, the corresponding key values are filled in the area to be filled, and the final structured The log includes: "amount = 10 yuan", "time = 5 o'clock", "product name = clothes" and "account type = member".

Optionally, in this embodiment of the present invention, a set function can be used to fill the key value corresponding to the target key name in the area to be filled corresponding to the target field name in the standard structure model. The set function is an associative container that is used to store data of the same data type and can retrieve data from the same data collection. In particular, the value of each member in the set function is unique and cannot be modified.

S130. Process the structured log into a binary structured log and write it into a file to generate a binary structured file.

In this embodiment of the present invention, optionally, a serialization processing tool is used to process the structured log into a binary structured log. The serialization processing tool includes Protobuf. Serialization is a process of converting the state information of a target object into a storable and/or transmissible form to facilitate data storage and/or transmission. In this embodiment of the present invention, this form is a binary format. Protobuf is a protocol used to serialize structured data. It is flexible and efficient. The code flow after Protobuf serialization is smaller and faster, making it suitable for performing large-volume serialization operations.

Since the binary number system is widely used in computers, processing structured logs into binary structured logs is not only easier for computers to read, but also reduces the size of the log, reduces resource consumption during the transmission process, and improves subsequent Efficiency in big data processing.

Further, after the binary structured file is generated, the binary structured file can be stored locally or sent to a specialized log storage platform for log storage.

In the embodiment of the present invention, if the binary structured file is stored locally, viewing permissions can also be opened to users who have log viewing needs. Before the user views a certain log, the binary structured log needs to be converted into a format that is convenient for the user. Reading data format.

Optionally, after processing the structured log into a binary structured log and writing it to a file, if a viewing request for the binary structured log is detected, use a deserialization processing tool to convert the The binary structured log is reprocessed into the structured log and displayed. Deserialization is a process of converting a storable or transmissible form such as a binary format into an object. In the embodiment of the present invention, it is a process of converting the object into a structured log to display the actual form of the object.

The technical solution of the embodiment of the present invention uses a standard structure model to process behavioral data in application requests, and then directly generates structured logs without generating any intermediate files, saving storage resources, avoiding complex log parsing processes, and saving time. Analyze costs. At the same time, binary structured logs are obtained through serialization operations, which are easy for computers to read, reduce log volume, facilitate log transmission, improve data transmission efficiency, and reduce resource consumption during data transmission.

Embodiment 2

Figure 2 is a flow chart of a method for generating binary structured logs provided in Embodiment 2 of the present invention. This embodiment is embodied based on the above embodiment. In this embodiment, the application request also includes: a request type. Correspondingly, the method of this embodiment specifically includes the following operations:

S210. Obtain an application request that matches the user's behavior, where the application request includes: behavior data and request type in the form of key-value pairs.

Application requests are divided into different request types according to different functions. Multiple application requests can correspond to one request type. For example, if you click on different texts or pictures to view detailed content, the application requests are different, but the request types are the same, and they are all browsing requests.

S220: Obtain a log type that matches the request type, and obtain a standard structure model that matches the application request among multiple alternative standard structure models according to the log type.

One request type corresponds to one log type. For example, the above browsing request corresponds to the browsing log. One log type corresponds to multiple alternative standard structure models, and the standard structure model matching the application request is obtained from the multiple alternative standard structure models. For example, the standard structure model corresponding to the above browsing log includes a standard structure model for processing image browsing requests and a standard structure model for processing text browsing requests. The corresponding standards are obtained according to specific application requests. Structural model.

S230. Process the behavioral data in the form of key-value pairs in the application request according to the standard structure model that matches the application request, and obtain a structured log that matches the application request; wherein, The standard structure model includes: at least one standard field name, and an area to be filled that matches the standard field name.

S240. Process the structured log into a binary structured log and write it into a file to generate a binary structured file.

Optionally, the method for generating binary structured logs in the embodiment of the present invention also includes: receiving at least one type of standard structure model uploaded by at least one log data generation platform, and generating a model that matches the standard structure model. View; the standard structure model is associated with the log type; publishing the model view to at least one log data usage platform that matches the log data generation platform, so that the log data usage platform uses the model view to view the data related to the log data generation platform. The binary structured file corresponding to the above-mentioned standard structure model.

Obtain the standard structure model from the log data generation platform. Since the standard structure model is associated with the log type, the model view generated by the standard structure model can reflect the structure of the corresponding log type; the model view can be a table ( table), column (column) or view (view) and other formats. The platform that generates log data and the platform that uses these log data are often different platforms. When the log data using platform receives the matching model views published by the log data generating platform, it can use these model views to view the log data generated by the log data generating platform. Binary structured file.

In particular, when staff optimized functions at the code level, the log format changed, the behavioral data in the form of key-value pairs in application requests changed, the matching standard structure model and the generated structured logs and binary structures Corresponding changes will also occur in the log. For example, after the navigation request is issued, the original function only provides routes generated based on the starting point and arrival point. However, after the function is optimized, the display of congestion conditions is added. At this time, the application request is added Behavioral data in the form of key-value pairs related to congestion conditions is included. Statistics on congestion conditions are also added to the standard structure model corresponding to the application request, and the generated structured logs and binary structured logs include Records related to congestion conditions.

When the log data usage platform uses the model view after function optimization to view the binary structured file before function optimization, it can still be obtained normally. Although the log format has changed, backward compatibility is guaranteed, that is, the model view of the function optimization version can be opened. The binary structured file of the version before function optimization; however, because the binary structured file was generated before function optimization, it does not include related records after function optimization, so the model view after function optimization will not open the binary structured file. Display this record, for example, the above-mentioned record related to congestion conditions. This record is not included in the binary structured file generated before function optimization. Therefore, use the model view after function optimization to open the binary structured file before function optimization. Records related to congestion conditions will not be displayed.

When the log data usage platform uses the model view before function optimization to view the binary structured file after function optimization, it can also be obtained normally. Although the log format has changed, forward compatibility is guaranteed, that is, the model view of the version before function optimization can be opened. The binary structured file of the function-optimized version; however, since the model view was generated before function optimization, it does not include related records after function optimization, so opening the binary structured file in the model view before function optimization will not display the related records. Records, for example, the above-mentioned records related to congestion conditions, are not included in the model view generated before function optimization. Therefore, using the model view before function optimization to open the binary structured file after function optimization will not display the same Records related to congestion conditions.

The technical solution of the embodiment of the present invention obtains a matching log type according to the request type of the application request, and then obtains the target standard structure model from multiple standard structure models matching the log type, so that each application request only needs to Just search for the required standard structure model in a log type that matches its request type, instead of searching for the target standard structure model in all standard structure models, which improves the acquisition efficiency and thus improves the conversion of each application request into Structured log conversion speed. At the same time, the model view is used to manage the log format, achieving forward and backward compatibility of the log format, ensuring that the logs before and after function optimization can be used normally, and avoiding data loss.

Embodiment 3

Figure 3 is a structural block diagram of a binary structured log generation device provided in Embodiment 3 of the present invention. The device includes: an application request acquisition module 310, a structured log acquisition module 320 and a binary structured log acquisition module 330.

The application request acquisition module 310 is used to obtain application requests that match user behavior, where the application requests include: behavior data in the form of key-value pairs;

The structured log acquisition module 320 is configured to process the behavioral data in the form of key-value pairs in the application request according to the standard structure model that matches the application request, and obtain a structure that matches the application request. Log; wherein, the standard structure model includes: at least one standard field name, and an area to be filled that matches the standard field name;

The binary structured log acquisition module 330 is used to process the structured log into a binary structured log, write it into a file, and generate a binary structured file.

The technical solution of the embodiment of the present invention uses a standard structure model to process behavioral data in application requests, and then directly generates structured logs without generating any intermediate files, saving storage resources, avoiding complex log parsing processes, and saving time. Analyze costs. At the same time, binary structured logs are obtained through serialization operations, which are easy for computers to read, reduce log volume, facilitate log transmission, improve data transmission efficiency, and reduce resource consumption during data transmission.

Optionally, based on the above technical solution, the structured log acquisition module 320 may specifically include:

The target field name acquisition unit is used to sequentially acquire a standard field name as the target field name in the standard structure model;

A key value filling unit, configured to fill the key value corresponding to the target key name in the standard structure model and the Within the area to be filled corresponding to the target field name;

The return unit is used to return and execute the operation of sequentially obtaining a standard field name as a target field name in the standard structure model until the processing of all standard field names in the standard structure model is completed.

Optionally, based on the above technical solution, the structured log acquisition module 320 may also include:

A filling abandonment unit is configured to, if there is no target field value matching the target field name in the behavior data, abandon the filling process of the area to be filled corresponding to the target field name in the standard structure model.

Optional, based on the above technical solution, the key value filling unit is specifically used for:

Using the set function, the key value corresponding to the target key name is filled in the area to be filled corresponding to the target field name in the standard structure model.

Optionally, based on the above technical solution, the binary structured log acquisition module 330 is specifically used for:

Use a serialization processing tool to process the structured log into a binary structured log.

Optionally, based on the above technical solution, the device for generating binary structured logs may also include:

A deserialization processing module, configured to use a deserialization processing tool to reprocess the binary structured log into the structured log and display it when a viewing request for the binary structured log is detected.

Optionally, based on the above technical solution, the application request may also include: request type.

Optionally, based on the above technical solution, the device for generating binary structured logs may also include:

A standard structure model acquisition module is used to obtain a log type that matches the request type, and according to the log type, obtain a standard structure that matches the application request among multiple alternative standard structure models. Model.

Optionally, based on the above technical solution, the device for generating binary structured logs may also include:

A model view generation module, configured to receive at least one type of standard structure model uploaded by at least one log data generation platform, and generate a model view that matches the standard structure model; the standard structure model is associated with the log type;

A log structured type acquisition module, configured to publish the model view to at least one log data usage platform that matches the log data generation platform, so that the log data usage platform uses the model view to view the standard structure The binary structured file corresponding to the model.

The above-mentioned device can execute the binary structured log generation method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method. For technical details that are not described in detail in this embodiment, please refer to the method provided by any embodiment of the present invention.

Embodiment 4

Figure 4 is a schematic structural diagram of a computer device provided in Embodiment 4 of the present invention. 4 illustrates a block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention. The computer device 12 shown in FIG. 4 is only an example and should not bring any limitations to the functions and scope of use of the embodiments of the present invention.

As shown in Figure 4, computer device 12 is embodied in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components, including system memory 28 and processing unit 16.

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerated port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect ( PCI) bus.

Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and nonvolatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . Computer device 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in Figure 4, commonly referred to as a "hard drive"). Although not shown in FIG. 4, a disk drive may be provided for reading and writing removable non-volatile disks (e.g., "floppy disks"), and for removable non-volatile optical disks (e.g., CD-ROM, DVD-ROM). or other optical media) that can read and write optical disc drives. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of embodiments of the invention.

A program/utility 40 having a set of (at least one) program modules 42, which may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other programs Modules, as well as program data, each of these examples or some combination may include an implementation of a network environment. Program modules 42 generally perform functions and/or methods in the described embodiments of the invention.

Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with Any device (eg, network card, modem, etc.) that enables the computer device 12 to communicate with one or more other computing devices. This communication may occur through input/output (I/O) interface 22. Furthermore, computer device 12 may also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18 . It should be understood that, although not shown in the figures, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.

The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing the binary structured log generation method provided by the embodiment of the present invention: obtaining application requests that match user behavior, The application request includes: behavioral data in the form of key-value pairs; according to a standard structure model matching the application request, the behavioral data in the form of key-value pairs in the application request is processed to obtain the The application requests a matching structured log; wherein the standard structure model includes: at least one standard field name, and an area to be filled that matches the standard field name; the structured log is processed into a binary structured log, And write it to the file to generate a binary structured file.

Embodiment 5

Embodiment 5 of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating a binary structured log as provided by all the inventive embodiments of the present application is implemented:

Obtain an application request that matches the user's behavior, where the application request includes: behavior data in the form of key-value pairs;

According to the standard structure model that matches the application request, the behavioral data in the form of key-value pairs in the application request is processed to obtain a structured log that matches the application request; wherein, the standard structure The volume model includes: at least one standard field name, and an area to be filled that matches the standard field name;

The structured log is processed into a binary structured log and written to a file to generate a binary structured file.

Any combination of one or more computer-readable media may be employed. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples (non-exhaustive list) of computer readable storage media include: electrical connections having one or more conductors, portable computer disks, hard drives, random access memory (RAM), read only memory (ROM), Erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. As used herein, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including - but not limited to - electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .

Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present invention may be written in one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional Procedural programming language—such as "C" or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).

Note that the above are only the preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, it can also include more other equivalent embodiments, and the present invention The scope is determined by the scope of the appended claims.

Claims (9)

1. A method for generating a binary structured log, comprising:

acquiring an application request matched with user behaviors; wherein the application request includes: behavior data in the form of key-value pairs;

according to a standard structure model matched with the application request, processing the behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;

processing the structured log into a binary structured log, writing the binary structured log into a file, and generating a binary structured file;

receiving at least one standard structure model uploaded by at least one log data generation platform, and generating a model view matched with the standard structure model; wherein the standard structure model is associated with a log type;

issuing the model view to at least one log data use platform matched with the log data generation platform so that the log data use platform can view a binary structured file corresponding to the standard structure body model by utilizing the model view;

the processing the behavior data in the form of the key value pair in the application request according to the standard structure model matched with the application request to obtain a structured log matched with the application request comprises the following steps:

sequentially acquiring a standard field name from the standard structure body model as a target field name;

if the behavior data has the target key name matched with the target field name, filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model;

and returning to execute the operation of sequentially acquiring one standard field name from the standard structure body model as a target field name until the processing of all the standard field names in the standard structure body model is completed.

2. The method according to claim 1, further comprising, after sequentially acquiring one standard field name as a target field name in the standard structure model:

and if the behavior data does not have the target field value matched with the target field name, discarding filling processing of the region to be filled corresponding to the target field name in the standard structural body model.

3. The method according to claim 1, wherein the filling the key value corresponding to the target key name into the region to be filled corresponding to the target field name in the standard structure model includes:

and filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model by adopting a set function.

4. The method of claim 1, wherein the processing the structured log into a binary structured log comprises:

and processing the structured log into the binary structured log by using a serialization processing tool.

5. The method of claim 1, wherein after said processing the structured log into a binary structured log and writing to a file, generating a binary structured file, comprising:

and when detecting a viewing request for the binary structured log, reprocessing the binary structured log into the structured log by adopting a deserializing processing tool and displaying the structured log.

6. The method of claim 1, wherein the application request further comprises: a request type;

before the behavior data in the form of the key value pair in the application request is processed according to the standard structure model matched with the application request to obtain the structured log matched with the application request, the method further comprises the following steps:

and acquiring a log type matched with the request type, and acquiring a standard structural body model matched with the application request from a plurality of alternative standard structural body models according to the log type.

7. A binary structured log generation apparatus, comprising:

the application request acquisition module is used for acquiring application requests matched with user behaviors; wherein the application request includes: behavior data in the form of key-value pairs;

the structured log acquisition module is used for processing the behavior data in the form of the key value pair in the application request according to a standard structure model matched with the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;

the binary structured log acquisition module is used for processing the structured log into a binary structured log, writing the binary structured log into a file and generating a binary structured file;

the model view generation module is used for receiving at least one type of standard structure body model uploaded by the at least one log data generation platform and generating a model view matched with the standard structure body model; wherein the standard structure model is associated with a log type;

the log structure type acquisition module is used for issuing the model view to at least one log data use platform matched with the log data generation platform so that the log data use platform can view binary structured files corresponding to the standard structure body model by utilizing the model view;

wherein, the structured log obtaining module includes: the system comprises a target field name acquisition unit, a key value filling unit and a return unit;

the target field name acquisition unit is used for sequentially acquiring a standard field name in the standard structure body model as a target field name;

the key value filling unit is used for filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure body model if the target key name matched with the target field name exists in the behavior data;

and the return unit is used for returning and executing the operation of sequentially acquiring one standard field name from the standard structure body model as a target field name until the processing of all the standard field names in the standard structure body model is completed.

8. A computer device, the computer device comprising:

one or more processors;

a storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of generating a binary structured log as recited in any of claims 1-6.

9. A computer storage medium having stored thereon a computer program, which when executed by a processor implements a method of generating a binary structured log according to any of claims 1-6.