CN116016683B - Website data acquisition method, device, equipment and storage medium - Google Patents
Website data acquisition method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116016683B CN116016683B CN202211549893.6A CN202211549893A CN116016683B CN 116016683 B CN116016683 B CN 116016683B CN 202211549893 A CN202211549893 A CN 202211549893A CN 116016683 B CN116016683 B CN 116016683B
- Authority
- CN
- China
- Prior art keywords
- information
- service request
- data
- website
- request information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 91
- 238000000034 method Methods 0.000 title claims abstract description 74
- 230000006870 function Effects 0.000 claims abstract description 34
- 238000012544 monitoring process Methods 0.000 claims abstract description 18
- 238000013480 data collection Methods 0.000 claims description 27
- 230000006399 behavior Effects 0.000 claims description 24
- 238000013507 mapping Methods 0.000 claims description 20
- 238000012545 processing Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000008878 coupling Effects 0.000 claims description 5
- 238000010168 coupling process Methods 0.000 claims description 5
- 238000005859 coupling reaction Methods 0.000 claims description 5
- 238000012795 verification Methods 0.000 claims description 5
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000012508 change request Methods 0.000 claims description 2
- 230000003993 interaction Effects 0.000 claims description 2
- 238000013500 data storage Methods 0.000 claims 1
- 238000012827 research and development Methods 0.000 abstract description 7
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000009933 burial Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 1
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a website data acquisition method, a device, equipment and a storage medium, which relate to the technical field of webpage flow data acquisition, and the log and embedded point function not only provides asynchronous output of the log, ensures that the log output cannot influence the performance of an application program, but also realizes the functions of rear-end hidden and automatic embedded point statistics; the customized custom log class JtyLog packaged by log4j provides the same use mode as log4j, and ensures decoupling of functions and rapid hand-up use of research and development. On the premise of ensuring performance and research and development efficiency, the flow statistics and monitoring functions of the page and the service interface are well realized.
Description
Technical Field
The invention relates to the technical field of webpage flow data acquisition, in particular to a website data acquisition method, device and equipment and a storage medium.
Background
At present, with the development of internet technology, the period from function research and development to production is shorter and shorter, and the function burial point can take data as a support, reflect the response of a user to a website, improve the website flow and the website performance, know the behavior of the user for accessing the website, and provide support for better meeting the user demands. With the popularization of internet products in the life of people, more and more internet products have functional burial points, become big data grippers, such as page browsing number (PV), independent visitor number (UV), IP, page residence time, page operation time, page access times, button clicking times, file downloading times, and the like, and can obtain some important information according to the page burial points, such as the response of users to websites, browsing content according to access habits, and the like.
In the financial industry, especially banks, the management system has more strict control on functional modules, approval processes and the like, but no proper data grip is available for supporting the optimization direction and strategy of decision-making functions of the management layer. And because of the specificity of the system architecture in the industry, the front end and the rear end are separated, a plurality of micro-service conditions exist at the rear end, a unified embedded point strategy is lacked to advance the flow statistics and monitoring functions of the page and the rear end service interface, and a large data base cannot be provided.
Disclosure of Invention
The invention aims to provide a website data acquisition method, a website data acquisition device, website data acquisition equipment and a storage medium so as to solve the problems. In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
in a first aspect, the present application provides a website data collection method, including:
and acquiring first service request information and first time information corresponding to the first service request information, wherein the first service request information is information describing that a website front-end request calls a back-end micro server interface.
And obtaining a first key parameter according to the first service request information, wherein the first key parameter is key field information which is extracted from the first service request information in a parsing way.
And obtaining user data information according to the first time information and the first key parameters, wherein the user data information is obtained by combining the first service request information of the same micro server interface in the first key parameters according to the sequence of the first time information.
A first log file is generated based on the user data information.
And obtaining website data information according to the first time information, the first log file and a preset storage unit, wherein the website data information is data which stores the first log file into the preset storage unit according to the mapping relation between the first log file and the preset storage unit and the sequence of the first time information, and the preset storage unit is used for dividing a database into a plurality of storage units according to custom fields.
In a second aspect, the application further provides a website data acquisition device, which comprises an acquisition module, an extraction module, a collection module, a log module and a storage module, wherein:
the acquisition module is used for: the method comprises the steps of acquiring first service request information and first time information corresponding to the first service request information, wherein the first service request information is information describing that a website front-end request calls a back-end micro server interface.
And an extraction module: and the first key parameter is key field information which is extracted from the first service request information in a resolving way.
And an aggregation module: and the user data information is information obtained by combining the first service request information of the same micro server interface in the first key parameter according to the sequence of the first time information.
And a log module: for generating a first log file based on the user data information.
And a storage module: the website data information is data which is used for storing the first log file into a preset storage unit according to the sequence of the first time information and according to the mapping relation between the first log file and the preset storage unit, and the preset storage unit is used for dividing a database into a plurality of storage units according to custom fields.
In a third aspect, the present application further provides a website data collection device, including:
a memory for storing a computer program; and the processor is used for realizing the steps of the website data acquisition method when executing the computer program.
In a fourth aspect, the present application further provides a storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements the steps of the website data collection method described above.
The beneficial effects of the invention are as follows:
The log and embedded point function not only provides asynchronous output of the log and ensures that the log output cannot influence the performance of an application program, but also realizes the hidden and automatic embedded point statistics function of the rear end; the customized custom log class JtyLog packaged by log4j provides the same use mode as log4j, and ensures decoupling of functions and rapid hand-up use of research and development. On the premise of ensuring performance and research and development efficiency, the flow statistics and monitoring functions of the page and the service interface are well realized.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a website data acquisition method in an embodiment of the invention;
FIG. 2 is a block diagram of a website data collection device in an embodiment of the invention;
fig. 3 is a block diagram of a website data collecting apparatus in an embodiment of the present invention.
The marks in the figure: 700-website data acquisition device; 710 an acquisition module; 720-an extraction module; 730-a collection module; 740-a log module; 750-a storage module; 760-a verification module; 761-a first configuration unit; 762-a first judging unit; 763-a second judgment unit; 7631-a third processing unit; 770-a section interception module; 771-a second configuration unit; 772—a first acquisition unit; 773-a first extraction unit; 774-a third judgment unit; 775-a first storage unit; 780-a first front end buried point module; 781-a third configuration unit; 782-a second acquisition unit; 783-a second processing unit; 784-a second storage unit; 790-a second front end buried point module; 791-a third acquisition unit; 792-a fourth acquisition unit; 793-a fourth judgment unit; 794-a third storage unit; 800-website data acquisition equipment; 801-a processor; an 802-memory; 803-multimedia component; 804-I/O interface; 805-a communication component.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
Example 1:
Referring to fig. 1, fig. 1 is a flow chart of a website data collection method in this embodiment. The embodiment provides a website data acquisition method, which comprises the steps of S1, S2, S3, S4 and S5.
Step S1, first service request information and first time information corresponding to the first service request information are obtained, wherein the first service request information is information describing that a website front-end request calls a back-end micro server interface.
It can be understood that in this step, when the terminal starts browsing the website information, the terminal sends information for calling the micro server to the back end, generates the first service request information, and records the request time as the first time information. The first service request information includes, but is not limited to: the IP address of the website front end, the IP address of the back end micro server interface, the request URL address, the request data type, the return data type, the user information (user name, user identification, etc.), etc.
Step S101, step S102, and step S103 are further included in step S1, where:
Step S101, configuring interception rules of a gateway, wherein the interception rules are element sets for actively intercepting the service request by the gateway, and each element in the element sets corresponds to interception information.
It can be appreciated that, in this embodiment, the configuration manner of the interception rule includes, but is not limited to: A. configuring a control rule of 'intercepting a data packet with a source address as a website IP address and a target address as a specified micro-server address' at a gateway; B. control rules for 'accessing the interface of the designated micro server for interception' are configured in the gateway.
Step S102, judging whether the first service request information is legal, if the first service request information is legal, judging whether the URL address in the first service request information is in first preset information, wherein the first preset information is URL address information of interception operation of a gateway.
It will be appreciated that in the step of interfacing with the API, the address and parameter structure of the interface is easily grasped by a hacker and sent in a simulated manner, and in order to prevent imposter calls by others, the validity of the interface request is verified to improve its security. In this embodiment, the gateway filter-based management technology is combined with redis and mysql databases to intercept and judge the call request sent by the website. When receiving the first service request information, whether the login of the user is effective and whether the call request is legal are intercepted. If the URL address in the first service request information is legal, judging whether the URL address in the first service request information belongs to an address request to be intercepted or not according to the first preset information.
Step S103, if the URL address in the first service request information exists in the first preset information, judging whether first token information in the first service request information is empty or not; and if the first token information in the first service request information is not empty, obtaining the key parameters according to the first service request information.
Step S103 further includes step S1031, judging whether the first token information is consistent with the second token information of the back-end micro server interface; if the first service request information is inconsistent, returning the first service request information to the front end of the website; if the information is consistent, the information of the front end of the website is put into an attribute attributes in the first service request information.
It can be understood that, in step S103, if the URL address in the first service request information does not exist in the first preset information, the operation is released; otherwise, continuing to judge, acquiring first token information in the first service request information, and acquiring user information from the redis database through the first token information. If the first token information is inconsistent with the second token information, the first token information in the first service request information is empty, request identification is carried out, the first token information is failed, and the first service request information is returned to the front end of the website. If the first token information is consistent with the second token information, indicating that the first token information is valid, then the user information in the information at the front end of the website is placed in the attribute attributes of the first service request information. Based on the first service request information after the user information is added, the service is requested, and some user related information can be directly obtained from the request head, so that the method is convenient and quick.
The unified gateway interception function is utilized in the gateway layer, unified interception and extraction of gateway service interfaces, ip addresses, users, sources, types and other information are realized, unified assignment of JtyLog member variables is ensured during JtyLog initialization, log output is ensured during log printing, and buried point information falling is realized.
Step S2, according to the first service request information, a first key parameter is obtained, wherein the first key parameter is key field information which is analyzed and extracted from the first service request information.
It will be appreciated that in this step, the first service request information is parsed and first key parameters such as interface information, ip address, user information, access source, etc. are extracted.
And step S3, obtaining user data information according to the first time information and the first key parameters, wherein the user data information is obtained by combining the first service request information of the same micro server interface in the first key parameters according to the sequence of the first time information.
And S4, generating a first log file based on the user data information.
Step S5, obtaining website data information according to the first time information, the first log file and a preset storage unit, wherein the website data information is data which stores the first log file into the preset storage unit according to the mapping relation between the first log file and the preset storage unit and the sequence of the first time information, and the preset storage unit is a database which is divided into a plurality of storage units according to a custom field.
It can be understood that log4j is utilized to introduce AsyncLogger, so that the asynchronous output function of the log is realized; meanwhile, log4j, custom log processing class JtyLog and database connection ConnectionFactory are packaged, configurable mapping (such as information of users, contents, items, sources, types, IP addresses, departments, operation functions and the like) of database connection, table fields and JtyLog member variables is realized in log4j2.Xml, and the function of automatically storing the data to a database table when JtyLog logs are asynchronously output is provided. JtyLog is used in the same way as log4j, except that the name is changed and is not felt for the development; the functions of asynchronous log output and automatic log library falling also solve the problems of log and buried point information output and storage on the premise of ensuring service program performance, and realize hidden and automatic functions for interface service function buried points.
Further, the AOP (tangent plane oriented programming) can isolate each part of the service logic, so that the coupling degree among the parts of the service logic is reduced, the reusability of the program is improved, and the development efficiency is improved. The website data acquisition method further includes step S6, step S6 includes step S601, step S602, step S603, step S604, and step S605, where:
Step S601, a cut-in point intercepted by an AOP section is defined, wherein the cut-in point comprises interception interface information and interception method information, the interception interface information is interface information of service combination, and the service combination is a service with complete functions formed by a plurality of micro servers based on a calling coupling relation; the interception method information is an interception rule of a custom annotation.
It can be understood that the paths that mutually request between the back-end micro servers to complete a certain function are packaged into an integral service combination interface, and the integral paths are used as interception interface information. The interception rules configured by the custom notes can be functions, wildcards, custom notes fields, and the like as interception method information.
Step S602, second service request information and second time information corresponding to the second service request information are acquired, wherein the second service request information is information describing the mutual calling among different micro server interfaces at the back end.
Step S603, obtaining a second key parameter set according to the second service request information, where each second key parameter in the second key parameter set is information of a micro server obtained by parsing the second service request information.
It will be appreciated that in this step, the second service request information is parsed and second key parameters of the interface 5 information, ip address, user information, access source, etc. are extracted.
Step S604, determining whether the second key parameters are consistent with the access point, if yes, generating a second log file according to the sequence of the second time information and each second key parameter.
It can be understood that when the second key parameter has a front-end website consistent with the information 0 of the custom access point, the second key parameter corresponding to the front-end website is generated into a second log file according to the time sequence.
Step S605, according to the mapping relationship between the second log file and the preset storage unit, and according to the sequence of the second time information, storing the second log file into the corresponding data in the preset storage unit.
5, The custom section interception class realizes unified interception of each micro-service interface and interception of a specified interface method; by initializing the annotation interface member variables, jtyLog are initialized and used in the section interception class uniformly; and further, the buried point information can be dropped into a database while log output is ensured.
The website data acquisition method further includes step S7, wherein step S7 includes step S701, step S702, step S703 and step S704, and the step S702 includes:
step S701, obtaining buried point configuration data, where the buried point configuration data includes each buried point element set in a website and attribute parameters of the buried point element.
It can be understood that in this step, the embedded point configuration data may be obtained from the local storage of the terminal device, or may be obtained by expanding the plug-in through the browser. The embedded point 5 element can be a button, a picture, a multimedia file connection, a text and the like, and the attribute parameters comprise some user operation behavior parameters (such as clicking, sliding, touching and the like) related to the embedded point element, page statistics parameters (page exposure, exposure time, visitor volume), page link parameters, embedded point element descriptions (position information, functional information and the like) and the like.
Step S702, responding to the access operation of the embedded point element, and obtaining behavior monitoring data corresponding to the access operation, where the access operation is a detectable operation initiated by a website element, and the behavior monitoring data is data describing user behavior and data of accessed information.
It is understood that in this step, the access operation in this embodiment includes, but is not limited to, a click operation, a drag operation, an input operation, a long press operation, and the like. When the embedded point operation on the website page is detected, a series of data after the access operation behavior is generated can be monitored to obtain behavior monitoring data. Behavior monitoring data includes, but is not limited to: user behavior data, attribute data corresponding to the user behavior data, accessed page attribute data, and the like.
Step S703, obtaining a third log file according to a trigger event corresponding to the access operation, where the third log file is data obtained after the server performs corresponding processing on the behavior monitoring data according to the trigger event, and the trigger event is information describing that the website sends a request for changing data to the server.
It will be appreciated that in this step, the embedded point element may bind one or more events simultaneously, each event corresponding to an event listener and an event handler. The event monitor monitors access operation of the embedded point element to detect whether the access operation triggers the event; the event processor executes corresponding event processing operation when the event is triggered, the event in this embodiment may be statistics data, deleting data, and the like, and the server obtains operation result monitoring data after performing statistics or deleting operations on the data in response to the triggering event, and generates a third log file.
Step S704, obtaining website data according to the embedded point configuration data, the third log file and the preset storage unit, where the website data is data that stores the third log file in the preset storage unit according to a mapping relationship between the attribute parameter of the embedded point element and the preset storage unit.
It can be understood that in this step, according to the mapping relationship between the attribute parameters of the embedded point element and the preset storage unit, the user behavior data (access user information, access operation information, etc.), the attribute data (access source information, interface information, etc.) corresponding to the user behavior data, and the accessed page attribute data (page exposure, exposure duration, visitor volume, access frequency, etc.) in the third log file are respectively stored in the corresponding preset storage unit, so that the acquisition of website data can be completed.
The website data acquisition method further includes step S8, step S8 includes step S801, step S802, step S803 and step S804, wherein:
Step S801, obtaining third access request information, where the third access request information is information describing that the website at the front end requests access to the micro server at the back end.
Step S802, based on the third access request information, obtaining first routing information, wherein the first routing information is a process from the IP address of the website to the IP address of the micro server.
Step 803, judging whether the first routing information is in second routing information, wherein the second routing information at least comprises two customized preset routes, and each preset route is a process from an access IP address to a target access IP address; if yes, responding to the third access request information to obtain a fourth log file, wherein the fourth log file is basic attribute information of the micro server.
It can be understood that in this step, the front end uses vue-router to implement unified interception of web site page address routing, and obtains basic attribute information of the micro server during page switching, where the basic attribute information of the micro server includes, but is not limited to, access user information, access source information, interface information, visitor volume, access frequency, and the like.
Step S804, obtaining website data information according to the fourth log file and the preset storage unit, where the website data information is data that the fourth log file is respectively stored in the preset storage unit according to a mapping relationship between the preset storage unit and basic attribute information of the micro server.
The application realizes a tool platform system flow statistics method based on vue route interception, gateway, aop tangent plane interception and log4j technology. The back end provides aspect section interception, the configuration parameters and the custom notes are utilized to monitor the interception method and the request interface data of the unified gateway, and meanwhile, log4j is utilized to customize asynchronous log output and a function embedded point statistics component, so that the hidden and automatic function embedded point statistics function is realized. The log and the embedded point function provide asynchronous output of the log, so that the log output is ensured not to influence the performance of an application program; the automatic buried point function realizes the hidden and automatic buried point statistics function at the rear end; the log4j package JtyLog provides the same use mode as log4j, is insensitive to research, and ensures decoupling of functions and rapid upper hand use of research and development. On the premise of ensuring performance and research and development efficiency, the flow statistics and monitoring functions of the page and the service interface are well realized. The front end uniformly intercepts through vue routes, and the data is synchronized to the rear end by using a buried point function service interface when the page is switched, so that page buried point statistics is realized; meanwhile, a custom embedded point instruction is provided, an event is bound, and access operation statistics of control elements is realized.
Example 2:
Referring to fig. 2, fig. 2 is a block diagram of a website data collection apparatus 700, including an acquisition module 710, an extraction module 720, a collection module 730, a log module 740, and a storage module 750, according to an exemplary embodiment, wherein: acquisition module 710: the method comprises the steps of acquiring first service request information and first time information corresponding to the first service request information, wherein the first service request information is information describing that a website front-end request calls a back-end micro server interface.
Further, the website data collecting device 700 further includes a verification module 760, wherein the verification module 760 includes a first configuration unit 761, a first judgment unit 762, and a second judgment unit 763, wherein:
First configuration unit 761: the method comprises the steps of configuring interception rules of a gateway, wherein the interception rules are element sets for actively intercepting the service request by the gateway, and each element in the element sets corresponds to interception information.
The first judgment unit 762: and the first preset information is URL address information of interception operation of the gateway.
The second judgment unit 763: if the URL address in the first service request information exists in the first preset information, judging whether first token information in the first service request information is empty or not; and if the first token information in the first service request information is not empty, obtaining the key parameters according to the first service request information.
In detail, the second determination unit 763 includes a third processing unit 7631, the third processing unit 7631: the method comprises the steps of judging whether the first token information is consistent with second token information of the back-end micro server interface; if the first service request information is inconsistent, returning the first service request information to the front end of the website; if the information is consistent, the information of the front end of the website is put into an attribute attributes in the first service request information.
Extraction module 720: and the first key parameter is key field information which is extracted from the first service request information in a resolving way.
The aggregation module 730: and the user data information is information obtained by combining the first service request information of the same micro server interface in the first key parameter according to the sequence of the first time information.
Log module 740: for generating a first log file based on the user data information.
Storage module 750: the website data information is data which is used for storing the first log file into a preset storage unit according to the sequence of the first time information and according to the mapping relation between the first log file and the preset storage unit, and the preset storage unit is used for dividing a database into a plurality of storage units according to custom fields.
Further, the website data collecting apparatus 700 further includes a section interception module 770, the section interception module 770 includes a second configuration unit 771, a first obtaining unit 772, a first extracting unit 773, a third judging unit 774 and a first storage unit 775, wherein:
the second configuration unit 771: the access point is used for customizing the AOP section interception, and comprises interception interface information and interception method information, wherein the interception interface information is interface information of service combination, and the service combination is formed by a plurality of micro-clothes based on calling coupling relation
The server forms a service with complete functions; the interception method information is an interception rule of custom annotation 0.
The first acquisition unit 772: the method comprises the steps of acquiring second service request information and second time information corresponding to the second service request information, wherein the second service request information is information describing the interaction among different micro server interfaces at the back end.
The first extraction unit 773: and the second 5 key parameter set is obtained according to the second service request information, and each second key parameter in the second key parameter set is the information of the micro server which is obtained by analyzing the second service request information.
Third judgment unit 774: and the second log file is used for judging whether the second key parameters are consistent with the access point or not, and if so, a second log file is generated according to the sequence of the second time information and each second key parameter.
0 First memory cell 775: and the second log files are respectively stored into the corresponding data in the preset storage units according to the mapping relation between the second log files and the preset storage units and the sequence of the second time information.
Preferably, the website data collection apparatus 700 further includes a first front-end buried point module 780,
The first front-end buried point module 780 includes a third configuration unit 781, a second acquisition unit 5782, a second processing unit 783, and a second storage unit 784, where:
third configuration unit 781: the embedded point configuration data comprises embedded point elements arranged in a website and attribute parameters of the embedded point elements.
The second acquisition unit 782: and the behavior monitoring data are data describing the behavior of the user and data of the accessed information.
Second processing unit 783: the method is used for obtaining a third log file according to the trigger event corresponding to the access operation, wherein the third log file is data obtained after the server performs corresponding processing on the behavior monitoring data according to the trigger event, and the trigger event is information describing that the website sends a data change request to the server.
Second storage unit 784: and the website data is data for storing the third log file into the preset storage unit according to the mapping relation between the attribute parameters of the embedded point elements and the preset storage unit.
Preferably, the website data collection device 700 further includes a second front-end embedded point module 790, where the second front-end embedded point module 790 includes a third acquisition unit 791, a fourth acquisition unit 792, a fourth judgment unit 793, and a third storage unit 794, where:
third acquisition unit 791: and the third access request information is information describing that the website at the front end requests access to the micro server at the back end.
Fourth acquisition unit 792: and the first routing information is obtained based on the third access request information, and is a process from the IP address of the website to the IP address of the micro server.
Fourth judgment unit 793: the method comprises the steps of judging whether the first routing information is in second routing information or not, wherein the second routing information at least comprises two customized preset routes, and each preset route is a process from an access IP address to a target access IP address; if yes, responding to the third access request information to obtain a fourth log file, wherein the fourth log file is basic attribute information of the micro server.
Third storage unit 794: and the website data information is data which is used for respectively storing the fourth log file into the preset storage unit according to the mapping relation between the preset storage unit and the basic attribute information of the micro server.
It should be noted that, regarding the apparatus in the above embodiments, the specific manner in which the respective modules perform the operations has been described in detail in the embodiments regarding the method, and will not be described in detail herein.
Example 3:
Corresponding to the above method embodiment, a website data collecting device 800 is also provided in this embodiment, and a website data collecting device 800 described below and a website data collecting method described above may be referred to correspondingly.
Fig. 3 is a block diagram of a website data collection apparatus 800, shown in accordance with an exemplary embodiment. As shown in fig. 3, the website data collection apparatus 800 may include: a processor 801, a memory 802. The website data collection device 800 may also include one or more of a multimedia component 803, an i/O interface 804, and a communication component 805.
The processor 801 is configured to control the overall operation of the website data collection apparatus 800 to perform all or part of the steps in the website data collection method described above. The memory 802 is used to store various types of data to support operation at the website data collection device 800, which may include, for example, instructions for any application or method operating on the website data collection device 800, as well as application related data such as contact data, messages, pictures, audio, video, and the like. The Memory 802 may be implemented by any type or combination of volatile or non-volatile Memory devices, such as static random access Memory (Static Random Access Memory, SRAM for short), electrically erasable programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, EEPROM for short), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM for short), programmable Read-Only Memory (Programmable Read-Only Memory, PROM for short), read-Only Memory (ROM for short), magnetic Memory, flash Memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and an audio component. Wherein the screen may be, for example, a touch screen, the audio component being for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 802 or transmitted through the communication component 805. The audio assembly further comprises at least one speaker for outputting audio signals. The I/O interface 804 provides an interface between the processor 801 and other interface modules, which may be a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 805 is configured to perform wired or wireless communication between the website data collection device 800 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, near field Communication (NFC for short), 2G, 3G, or 4G, or a combination of one or more thereof, and accordingly the Communication component 805 may comprise: wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the website data collection apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASIC), digital signal Processor (DIGITAL SIGNAL DSP), digital signal processing apparatus (DIGITAL SIGNAL Processing Device DSPD), programmable logic device (Programmable Logic Device PLD), field programmable gate array (Field Programmable GATE ARRAY FPGA), controller, microcontroller, microprocessor, or other electronic component for performing the website data collection methods described above.
In another exemplary embodiment, a computer storage medium is also provided that includes program instructions that, when executed by a processor, implement the steps of the website data collection method described above. For example, the computer storage medium may be the memory 802 described above including program instructions executable by the processor 801 of the website data collection device 800 to perform the website data collection method described above.
Example 4:
corresponding to the above method embodiment, a storage medium is further provided in this embodiment, and a storage medium described below and a website data collection method described above may be referred to correspondingly.
A storage medium having a computer program stored thereon, which when executed by a processor performs the steps of the website data collection method of the above method embodiment.
The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, which may store various program codes.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (6)
1. The website data acquisition method is characterized by comprising the following steps of:
acquiring first service request information and first time information corresponding to the first service request information, wherein the first service request information is information describing that a website front end requests to call a rear end micro server interface;
obtaining a first key parameter according to the first service request information, wherein the first key parameter is key field information which is analyzed and extracted from the first service request information;
Obtaining user data information according to the first time information and the first key parameters, wherein the user data information is obtained by combining the first service request information of the same micro server interface in the first key parameters according to the sequence of the first time information;
generating a first log file based on the user data information;
Obtaining website data information according to the first time information, the first log file and a preset storage unit, wherein the website data information is data which stores the first log file into the preset storage unit according to the mapping relation between the first log file and the preset storage unit and the sequence of the first time information, and the preset storage unit is used for dividing a database into a plurality of storage units according to custom fields;
According to the first service request information, before obtaining the first key parameter, the method further comprises:
configuring an interception rule of a gateway, wherein the interception rule is an element set for actively intercepting the service request by the gateway, and each element in the element set corresponds to interception information;
Judging whether the first service request information is legal or not, if the first service request information is legal, judging whether the URL address in the first service request information is in first preset information, wherein the first preset information is URL address information of interception operation of a gateway;
If the URL address in the first service request information exists in the first preset information, judging whether first token information in the first service request information is empty or not; if the first token information in the first service request information is not empty, obtaining the key parameters according to the first service request information;
The website data acquisition method further comprises the following steps:
the method comprises the steps that an access point for intercepting an AOP section is customized, the access point comprises interception interface information and interception method information, the interception interface information is interface information of service combinations, and the service combinations are services with complete functions formed by a plurality of micro servers based on calling coupling relations; the interception method information is an interception rule of a custom annotation;
Acquiring second service request information and second time information corresponding to the second service request information, wherein the second service request information is information describing the inter-calling among different micro server interfaces at the back end;
obtaining a second key parameter set according to the second service request information, wherein each second key parameter in the second key parameter set is information of a micro server obtained by analyzing the second service request information;
Judging whether the second key parameters are consistent with the access point or not, if so, generating a second log file according to the sequence of the second time information and each second key parameter;
According to the mapping relation between the second log file and the preset storage unit, storing the second log file into the corresponding data in the preset storage unit according to the sequence of the second time information;
The website data acquisition method further comprises the following steps:
Acquiring buried point configuration data, wherein the buried point configuration data comprises buried point elements arranged in a website and attribute parameters of the buried point elements;
responding to the access operation of the embedded point element, acquiring behavior monitoring data corresponding to the access operation, wherein the access operation is an operation initiated by a website element and can be detected, and the behavior monitoring data is data describing user behaviors and data of accessed information;
According to the triggering event corresponding to the access operation, a third log file is obtained, wherein the third log file is data obtained after the server performs corresponding processing on the behavior monitoring data according to the triggering event, and the triggering event is information describing that the website sends a data changing request to the server;
obtaining website data according to the embedded point configuration data, the third log file and the preset storage unit, wherein the website data is data for storing the third log file to the preset storage unit according to the mapping relation between the attribute parameters of the embedded point elements and the preset storage unit;
The website data acquisition method further comprises the following steps:
acquiring third access request information, wherein the third access request information is information describing that the website at the front end requests access to the micro server at the rear end;
obtaining first routing information based on the third access request information, wherein the first routing information is a process from the IP address of the website to the IP address of the micro server;
Judging whether the first routing information is in second routing information or not, wherein the second routing information at least comprises two customized preset routes, and each preset route is a process from an access IP address to a target access IP address;
If yes, responding to the third access request information to obtain a fourth log file, wherein the fourth log file is basic attribute information of the micro server;
And obtaining website data information according to the fourth log file and the preset storage unit, wherein the website data information is data which respectively stores the fourth log file into the preset storage unit according to the mapping relation between the preset storage unit and the basic attribute information of the micro server.
2. The method of claim 1, wherein determining whether first token information in the first service request information is empty comprises:
Judging whether the first token information is consistent with second token information of the back-end micro server interface;
if the first service request information is inconsistent, returning the first service request information to the front end of the website;
if the information is consistent, the information of the front end of the website is put into an attribute attributes in the first service request information.
3.A website data acquisition device, comprising:
The acquisition module is used for: the method comprises the steps of acquiring first service request information and first time information corresponding to the first service request information, wherein the first service request information is information describing that a website front-end request calls a rear-end micro server interface;
and an extraction module: the first key parameter is key field information which is analyzed and extracted from the first service request information;
and an aggregation module: the method comprises the steps of obtaining user data information according to first time information and first key parameters, wherein the user data information is obtained by combining first service request information of the same micro server interface in the first key parameters according to the sequence of the first time information;
and a log module: for generating a first log file based on the user data information;
And a storage module: the website data information is data which is used for storing the first log file into a preset storage unit according to the sequence of the first time information and according to the mapping relation between the first log file and the preset storage unit, and the preset storage unit is used for dividing a database into a plurality of storage units according to custom fields;
the website data acquisition device further comprises a verification module, wherein the verification module comprises:
A first configuration unit: the method comprises the steps of configuring an interception rule of a gateway, wherein the interception rule is an element set for actively intercepting the service request by the gateway, and each element in the element set corresponds to interception information;
A first judgment unit: the method comprises the steps of judging whether first service request information is legal or not, if the first service request information is legal, judging whether a URL address in the first service request information is in first preset information, wherein the first preset information is URL address information of interception operation of a gateway;
A second judgment unit: if the URL address in the first service request information exists in the first preset information, judging whether first token information in the first service request information is empty or not; if the first token information in the first service request information is not empty, obtaining the key parameters according to the first service request information;
the website data acquisition device also comprises a section interception module, wherein the section interception module comprises:
A second configuration unit: the access point is used for customizing the AOP section interception, and comprises interception interface information and interception method information, wherein the interception interface information is interface information of service combinations, and the service combinations are services with complete functions formed by a plurality of micro servers based on calling coupling relations; the interception method information is an interception rule of a custom annotation;
a first acquisition unit: the method comprises the steps of acquiring second service request information and second time information corresponding to the second service request information, wherein the second service request information is information describing the interaction among different micro server interfaces at the back end;
A first extraction unit: the second key parameters are information of a micro server which is obtained by analyzing the second service request information;
a third judgment unit: the second log file is used for judging whether the second key parameters are consistent with the access point or not, if so, a second log file is generated according to the sequence of the second time information and each second key parameter;
A first storage unit: the data storage unit is used for storing the second log files into the corresponding data in the preset storage units according to the mapping relation between the second log files and the preset storage units and the sequence of the second time information;
the website data acquisition device further comprises a first front-end embedded point module, and the first front-end embedded point module comprises:
third configuration unit: the embedded point configuration data comprises embedded point elements arranged in a website and attribute parameters of the embedded point elements;
A second acquisition unit: the method comprises the steps of responding to access operation of the embedded point element, obtaining behavior monitoring data corresponding to the access operation, wherein the access operation is detected operation initiated by a website element, and the behavior monitoring data is data describing user behaviors and data of accessed information;
a second processing unit: the method comprises the steps that a third log file is obtained according to a trigger event corresponding to the access operation, wherein the third log file is data obtained after a server performs corresponding processing on behavior monitoring data according to the trigger event, and the trigger event is information describing that the website sends a data change request to the server;
A second storage unit: the website data is used for obtaining website data according to the embedded point configuration data, the third log file and the preset storage unit, and the website data is data for storing the third log file to the preset storage unit according to the mapping relation between the attribute parameters of the embedded point elements and the preset storage unit;
The website data acquisition device further comprises a second front-end embedded point module, and the second front-end embedded point module comprises:
A third acquisition unit: the method comprises the steps that third access request information is obtained, wherein the third access request information is information describing that a website at the front end requests access to a micro server at the rear end;
Fourth acquisition unit: the first route information is a process from the IP address of the website to the IP address of the micro server;
Fourth judgment unit: the method comprises the steps of judging whether the first routing information is in second routing information or not, wherein the second routing information at least comprises two customized preset routes, and each preset route is a process from an access IP address to a target access IP address;
If yes, responding to the third access request information to obtain a fourth log file, wherein the fourth log file is basic attribute information of the micro server;
A third storage unit: and the website data information is data which is used for respectively storing the fourth log file into the preset storage unit according to the mapping relation between the preset storage unit and the basic attribute information of the micro server.
4. The website data collection device according to claim 3, wherein the second judgment unit includes:
a third processing unit: the method comprises the steps of judging whether the first token information is consistent with second token information of the back-end micro server interface;
if the first service request information is inconsistent, returning the first service request information to the front end of the website;
if the information is consistent, the information of the front end of the website is put into an attribute attributes in the first service request information.
5. A website data collection apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the website data collection method according to any one of claims 1 to 2 when executing the computer program.
6. A storage medium, characterized by: the storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the website data collection method according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211549893.6A CN116016683B (en) | 2022-12-05 | 2022-12-05 | Website data acquisition method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211549893.6A CN116016683B (en) | 2022-12-05 | 2022-12-05 | Website data acquisition method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116016683A CN116016683A (en) | 2023-04-25 |
| CN116016683B true CN116016683B (en) | 2024-08-02 |
Family
ID=86030825
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211549893.6A Active CN116016683B (en) | 2022-12-05 | 2022-12-05 | Website data acquisition method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116016683B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113032421A (en) * | 2021-04-29 | 2021-06-25 | 电子科技大学 | MongoDB-based distributed transaction processing system and method |
| CN115168166A (en) * | 2022-06-24 | 2022-10-11 | 平安消费金融有限公司 | Recording method, device, equipment and storage medium for business data change |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9053188B2 (en) * | 2008-05-12 | 2015-06-09 | Enpulz, Llc | Web browser accessible search engine with statistics gathering persistence |
| US9269061B2 (en) * | 2009-12-10 | 2016-02-23 | Equinix, Inc. | Performance, analytics and auditing framework for portal applications |
| CN113760658A (en) * | 2021-09-02 | 2021-12-07 | 山东派盟网络科技有限公司 | Monitoring method, device and equipment |
-
2022
- 2022-12-05 CN CN202211549893.6A patent/CN116016683B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113032421A (en) * | 2021-04-29 | 2021-06-25 | 电子科技大学 | MongoDB-based distributed transaction processing system and method |
| CN115168166A (en) * | 2022-06-24 | 2022-10-11 | 平安消费金融有限公司 | Recording method, device, equipment and storage medium for business data change |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116016683A (en) | 2023-04-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021189954A1 (en) | Log data processing method and apparatus, computer device, and storage medium | |
| KR102504075B1 (en) | Matching and attributes of user device events | |
| CN108304498A (en) | Webpage data acquiring method, device, computer equipment and storage medium | |
| CN107645508A (en) | A kind of data handling system, method, client and server | |
| EP3343872A1 (en) | System and method for gathering information to detect phishing activity | |
| JP5764255B2 (en) | User operation detection system and user operation detection method | |
| CN107688568A (en) | Acquisition method and device based on web page access behavior record | |
| US11882154B2 (en) | Template representation of security resources | |
| WO2014059906A1 (en) | Method and system for accessing audio/video community virtual rooms | |
| US10943063B1 (en) | Apparatus and method to automate website user interface navigation | |
| US10015253B2 (en) | System and method for preemptive request processing | |
| CN113282591B (en) | Authority filtering method, authority filtering device, computer equipment and storage medium | |
| CN112948264B (en) | Method and device for executing test task, system, storage medium and electronic device | |
| CN103262049A (en) | Method of gathering data of an event-ike nature from electronic forms | |
| US10140377B2 (en) | Data processing, data collection | |
| WO2021189766A1 (en) | Data visualization method and related device | |
| CN115113898A (en) | Dynamic update method, device, computer equipment and storage medium of micro-application | |
| CN116016683B (en) | Website data acquisition method, device, equipment and storage medium | |
| CN111680247B (en) | Local calling method, device and equipment of webpage character strings and storage medium | |
| CN113779437A (en) | Privacy detection method and device, computer storable medium | |
| CN110851346B (en) | Query statement boundary problem detection method, device, equipment and storage medium | |
| CN107301192A (en) | A kind of terminal identification method and identification server | |
| CN112671615A (en) | Method, system and storage medium for collecting operation behavior data of front-end user | |
| CN114301893B (en) | Log management method, system and readable storage medium | |
| US10949605B2 (en) | Interprogram communication with event handling for online enhancements |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |