Disclosure of Invention
The invention aims to provide a traffic acquisition method, a traffic acquisition device, traffic acquisition equipment and a computer readable storage medium for a distributed system, which can solve the problems of single acquired data and difficulty in secondary processing existing in a mode for acquiring traffic information of the distributed system in the prior art.
In order to achieve the above purpose, the invention provides the following technical scheme:
a traffic acquisition method of a distributed system comprises the following steps:
determining any user accessing the distributed system as a target user, and acquiring access information of the target user, wherein the access information comprises user information of the target user, bucket information of a bucket accessed by the target user, time accessed by the target user and the number of bytes uploaded or downloaded by the target user;
writing the access information into each corresponding log, wherein the log comprises logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system;
and receiving an information request sent by the target user, acquiring corresponding access information from a log corresponding to the information request, and returning the acquired access information serving as flow information to the target user.
Preferably, writing the access information into each corresponding log includes:
and writing the access information into each corresponding log, wherein the log comprises logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system at different time periods.
Preferably, writing the access information into each corresponding log includes:
and writing the access information into each log with a corresponding name, wherein the log is named by the belonged time period and the identification of the belonged classification object, and the classification object comprises a distributed system, a bucket and a user.
Preferably, after obtaining the corresponding access information from the log corresponding to the information request, the method further includes:
and judging whether flow calculation is needed, if so, calculating corresponding flow information based on the acquired access information and returning the flow information to the target user, and if not, executing the step of returning the acquired access information to the target user as the flow information, wherein the flow information comprises the uploading or downloading bandwidth, the access times processed per second and the total time used for processing all accesses.
A traffic acquisition apparatus for a distributed system, comprising:
an acquisition module to: determining any user accessing the distributed system as a target user, and acquiring access information of the target user, wherein the access information comprises user information of the target user, bucket information of a bucket accessed by the target user, time accessed by the target user and the number of bytes uploaded or downloaded by the target user;
a write module to: writing the access information into each corresponding log, wherein the log comprises logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system;
a return module to: and receiving an information request sent by the target user, acquiring corresponding access information from a log corresponding to the information request, and returning the acquired access information serving as flow information to the target user.
Preferably, the writing module includes:
a write unit to: and writing the access information into each corresponding log, wherein the log comprises logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system at different time periods.
Preferably, the writing unit includes:
a write subunit to: and writing the access information into each log with a corresponding name, wherein the log is named by the belonged time period and the identification of the belonged classification object, and the classification object comprises a distributed system, a bucket and a user.
Preferably, the method further comprises the following steps:
a determination module configured to: and the return module is used for judging whether flow calculation is needed or not after acquiring corresponding access information from a log corresponding to the information request, if so, calculating corresponding flow information based on the acquired access information and returning the flow information to the target user, and if not, indicating the return module to execute the step of returning the acquired access information serving as the flow information to the target user, wherein the flow information comprises uploaded or downloaded bandwidth, access times processed per second and total time used for processing all accesses.
A traffic acquisition device of a distributed system, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the traffic acquisition method of the distributed system as described in any one of the above when executing the computer program.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a traffic acquisition method of a distributed system as described in any one of the above.
The invention provides a method, a device, equipment and a computer readable storage medium for acquiring flow of a distributed system, wherein the method comprises the following steps: determining any user accessing the distributed system as a target user, and acquiring access information of the target user, wherein the access information comprises user information of the target user, bucket information of a bucket accessed by the target user, time accessed by the target user and the number of bytes uploaded or downloaded by the target user; writing the access information into each corresponding log, wherein the log comprises logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system; and receiving an information request sent by the target user, acquiring corresponding access information from a log corresponding to the information request, and returning the acquired access information serving as flow information to the target user. According to the technical scheme, when a user accesses a distributed system, access information comprising user information, barrel information, access time and byte number uploaded or downloaded is obtained, the access information is stored in each corresponding log, and then the access information is obtained from the log and returned to the user when the user needs to obtain flow information. Different from the prior art that the total access times, the total uploaded and downloaded data volume can only be obtained through a counter, various information when a user accesses a distributed system can be obtained through the mode of refining, collecting and storing the corresponding information, so that the information diversification is realized, and secondary processing in modes of time-sharing processing or user-sharing processing and the like is facilitated.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flowchart of a traffic obtaining method of a distributed system according to an embodiment of the present invention is shown, where the method includes:
s11: determining any user accessing the distributed system as a target user, and acquiring access information of the target user, wherein the access information comprises user information of the target user, bucket information of a bucket accessed by the target user, time accessed by the target user and the number of bytes uploaded or downloaded by the target user.
The execution subject of the traffic acquisition method for the distributed system provided by the embodiment of the present invention may be a corresponding traffic acquisition device, and the device may be disposed in the distributed system. Any user accessing the distributed system can be used as a target user, that is, each user accessing the distributed system can adopt the technical scheme provided by the invention to realize the acquisition and storage of the corresponding access information. The user information may include a user identifier, a user attribute, and the like, the bucket information may include a bucket identifier, a bucket attribute, a bucket space size, and the like, the access time may include an access start time and an access end time, and may also include a time period used for the access, and the like, and the number of bytes uploaded or downloaded is the number of bytes uploaded to the distributed system or downloaded by the distributed system in the corresponding access; of course, the above information can be set according to actual needs, and all are within the protection scope of the present invention. In addition, it should be noted that a bucket is a storage place for implementing data storage in the distributed system, and is a logical concept of data storage, that is, the distributed system may include a plurality of buckets, and the time referred in this application may all be based on the same time standard of the distributed system.
S12: and writing the access information into each corresponding log, wherein the log comprises logs which are respectively used for storing the access information corresponding to each distributed system, each bucket in the distributed system and each user accessing the distributed system.
It should be noted that the log in the present application may be specifically created in advance in a server of the distributed system; the level of the log may include distributed system, bucket, user. Specifically, the log used for storing the access information corresponding to the distributed system is a log with the level of the distributed system, and the log is used for storing the access information corresponding to the distributed system and comprises the access information corresponding to each user accessing the distributed system each time; the log used for storing the access information corresponding to each bucket is a log with the bucket level, and the log is used for storing the access information corresponding to the corresponding bucket and comprises the access information corresponding to each user when each user accesses the corresponding bucket; the log used for storing the access information corresponding to each user is a log with the user level, and the log is used for storing the access information corresponding to the corresponding user, including the access information of the corresponding user when accessing the distributed system each time. If the access information when the user A accesses the bucket A needs to be respectively stored in the logs corresponding to the distributed system, the bucket A and the user A, the hierarchical storage of the logs is realized. When determining the log for storing the bucket and the access information corresponding to the access information, the corresponding log may be located based on the bucket information and the user information included in the access information.
S13: and receiving an information request sent by a target user, acquiring corresponding access information from a log corresponding to the information request, and returning the acquired access information serving as flow information to the target user.
It should be noted that each user may access the distributed system, and may also obtain the traffic information from the distributed system, so that when receiving an information request sent by a destination user, the traffic information corresponding to the information request needs to be returned to the destination user. The information request can include identification of a classified object corresponding to the access information to be acquired, access time and the like, so that the access information corresponding to the information request is positioned and returned to the target user; the classification object refers to an object corresponding to access information of the log storage, and comprises a distributed system, a bucket and a user, and the identifier of the classification object is the identifier of the object corresponding to the access information of the log storage. In addition, after the identification of the classification object corresponding to the information request is determined, only the required access information needs to be acquired from the log of the classification object corresponding to the identification, if the identification is barrel a, only the required access information needs to be acquired from the barrel a, and compared with the case that all the access information is stored in the same log, the hierarchical storage mode does not need to traverse the same log for acquiring the access information, but is acquired from the log corresponding to the classification object in a targeted manner, so that the data acquisition efficiency is obviously and greatly improved.
It should be noted that the access information that can be obtained by the target user may only include the access information corresponding to the target user, or may include the access information corresponding to all users, and the specific setting may be performed according to actual needs, which are all within the protection scope of the present invention.
According to the technical scheme, when a user accesses a distributed system, access information comprising user information, barrel information, access time and byte number uploaded or downloaded is obtained, the access information is stored in each corresponding log, and then the access information is obtained from the log and returned to the user when the user needs to obtain flow information. Different from the prior art that the total access times, the total uploaded and downloaded data volume can only be obtained through a counter, various information when a user accesses a distributed system can be obtained through the mode of refining, collecting and storing the corresponding information, so that the information diversification is realized, and secondary processing in modes of time-sharing processing or user-sharing processing and the like is facilitated.
In addition, in the technical scheme disclosed by the application, the acquired access information is stored in the logs of different levels, so that when the access information is required to be acquired, the classification object corresponding to the access information required to be acquired can be directly positioned, and then the acquisition of the access information is realized. The method and the device for the flow statistics achieve layering of flow statistics in a distributed system, a user level and a bucket level, are more flexible, and the obtained data can be in the running state of the distributed system.
The method for acquiring traffic of a distributed system according to an embodiment of the present invention writes access information into each corresponding log, and may include:
and writing the access information into each corresponding log, wherein the logs comprise logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system at different time periods.
It should be noted that, in the present application, each log may not only correspond to different classification objects, but also correspond to different time periods, such as year, month, day, hour, minute, and the like. Therefore, the access information is written into the log corresponding to the time period to which the access starting moment belongs, and when the log information is acquired, the corresponding log can be positioned from two points of a classification object and the time period, so that the acquisition of the corresponding access information is realized, the information classification storage and acquisition are realized from two levels of time and level, and the data storage and acquisition efficiency is further improved.
The method for acquiring traffic of a distributed system according to an embodiment of the present invention writes access information into each corresponding log, and may include:
and writing the access information into each log with a corresponding name, wherein the log is named by the time period and the identification of the classified object, and the classified object comprises a distributed system, a bucket and a user.
It should be noted that each log may be named by integrating the time period and the classification object, so that the log name may be obtained in the following ways:
1) bucket level: time period + bucket id
2) User level: time period + user id
3) System level: time period + system flag (customizable);
and then the log needing to be positioned can be directly determined by the log name when the access information is written or acquired, and the method is simple and easy to implement.
The method for acquiring traffic of a distributed system according to the embodiment of the present invention may further include, after acquiring corresponding access information from a log corresponding to an information request:
and judging whether flow calculation is needed, if so, calculating corresponding flow information based on the acquired access information and returning the flow information to the target user, and if not, executing a step of returning the acquired access information to the target user as the flow information, wherein the flow information comprises uploading or downloading bandwidth, the access times per second and total time for processing all accesses.
It should be noted that the access information can be directly returned to the user, the user can perform processing such as calculation on the access information according to the actual needs of the user, the access information can also be directly calculated, and then the result obtained by the calculation is returned to the user, so that the user can conveniently obtain the data, and the user experience is improved. Specifically, the uploaded or downloaded bandwidth may be calculated based on the number of bytes uploaded or downloaded in the acquired traffic information and the corresponding time, the number of access times processed per second is a ratio of the total number of access times included in the acquired traffic information to the total time used for all accesses, and the total time used for all accesses is a sum of the times used for accesses included in the acquired traffic information. Therefore, the information statistics of different levels is realized by relying on the logs of the distributed system, and the method is accurate and effective.
An embodiment of the present invention further provides a device for acquiring traffic of a distributed system, as shown in fig. 2, where the device may include:
an obtaining module 11, configured to: determining any user accessing the distributed system as a target user, and acquiring access information of the target user, wherein the access information comprises user information of the target user, barrel information of a barrel accessed by the target user, time accessed by the target user and the number of bytes uploaded or downloaded by the target user;
a write module 12 for: writing the access information into each corresponding log, wherein the logs comprise logs respectively used for storing the access information corresponding to each distributed system, each bucket in the distributed system and each user accessing the distributed system;
a return module 13 for: and receiving an information request sent by a target user, acquiring corresponding access information from a log corresponding to the information request, and returning the acquired access information serving as flow information to the target user.
In the traffic acquiring apparatus of the distributed system provided in the embodiment of the present invention, the writing module may include:
a write unit to: and writing the access information into each corresponding log, wherein the logs comprise logs respectively used for storing the access information corresponding to the distributed system, each bucket in the distributed system and each user accessing the distributed system at different time periods.
In the traffic acquiring apparatus of the distributed system provided in the embodiment of the present invention, the writing unit may include:
a write subunit to: and writing the access information into each log with a corresponding name, wherein the log is named by the time period and the identification of the classified object, and the classified object comprises a distributed system, a bucket and a user.
The traffic acquiring apparatus for a distributed system provided in the embodiments of the present invention may further include:
a determination module configured to: and the return module judges whether flow calculation is needed or not after acquiring corresponding access information from a log corresponding to the information request, if so, calculates the corresponding flow information based on the acquired access information and returns the flow information to the target user, and if not, instructs the return module to execute the step of returning the acquired access information to the target user as the flow information, wherein the flow information comprises the uploading or downloading bandwidth, the access times per second and the total time for processing all accesses.
An embodiment of the present invention further provides a device for acquiring traffic of a distributed system, where the device may include:
a memory for storing a computer program;
a processor for implementing the steps of the traffic acquisition method of the distributed system as described above when executing the computer program.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the traffic obtaining method of any one of the above distributed systems are implemented.
It should be noted that, for the description of the relevant parts in the traffic acquiring device, the equipment and the computer readable storage medium of the distributed system provided in the embodiment of the present invention, reference is made to the detailed description of the corresponding parts in the traffic acquiring method of the distributed system provided in the embodiment of the present invention, and details are not described herein again. In addition, parts of the technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.