CN111881161A - Calculation method, system, device and storage medium for index measurement - Google Patents
Calculation method, system, device and storage medium for index measurement Download PDFInfo
- Publication number
- CN111881161A CN111881161A CN202010731965.3A CN202010731965A CN111881161A CN 111881161 A CN111881161 A CN 111881161A CN 202010731965 A CN202010731965 A CN 202010731965A CN 111881161 A CN111881161 A CN 111881161A
- Authority
- CN
- China
- Prior art keywords
- usage
- metering
- calculated
- index
- storage module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
本发明提供一种指标计量的计算方法、系统、设备和存储介质，其中计算方法包括：根据待计算目标发送的计量请求确定计量指标；对计量请求中的计量参数进行解析，以得到待计算用量，以及获取得到已保存用量；将获取得到的有效用量写入存储模块中；根据有效用量和已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新；将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。本发明的有益效果：采用实时计算有效用量，和采用离线统计计算总量，从而有效解决高并发下实时性能问题。
The invention provides a calculation method, system, device and storage medium for index measurement, wherein the calculation method includes: determining a measurement index according to a measurement request sent by a target to be calculated; analyzing the measurement parameters in the measurement request to obtain the amount to be calculated , and obtain the saved amount; write the obtained effective amount into the storage module; calculate the current calculation amount according to the effective amount and the saved amount, and update the saved amount in the storage module according to the calculated amount ; Store the metering log associated with the effective amount in the storage module, and store the metering log after incremental subscription in a search engine, so that the effective amount and/or the saved amount are analyzed by the search engine according to the metering log. search. The beneficial effects of the invention are as follows: real-time calculation of the effective amount and offline statistical calculation of the total amount are used to effectively solve the real-time performance problem under high concurrency.
Description
技术领域technical field
本发明涉及数据处理技术领域，尤其涉及一种指标计量的计算方法、系统、设备和存储介质。The present invention relates to the technical field of data processing, and in particular, to a calculation method, system, device and storage medium for index measurement.
背景技术Background technique
目前，计量日志通常采用关系型数据库存储，并且计量日志通常采用mysql数据库进行查询，以及计量用量通常采用redis(Remote Dictionary Server，远程字典服务)进行计算存储，即需要计算某个用户某个计量指标时，可以采用redis分布式锁控制并发进行计算，等计量指标计算完毕后更新redis的值释放分布式锁进行其他计算。计量周期(月、年)到了以后定时任务清理redis中的计量用量值，即将计量用量值设置为0。redis是一个开源的使用ANSI C语言编写、支持网络、可基于内存亦可持久化的日志型、Key-Value数据库，并提供多种语言的API(Application Programming Interface，应用程序编程接口)。At present, the metering log is usually stored in a relational database, and the metering log is usually queried using a mysql database, and the metering amount is usually calculated and stored using redis (Remote Dictionary Server, remote dictionary service), that is, it is necessary to calculate a certain user. , you can use the redis distributed lock to control concurrency for calculation, and after the measurement indicators are calculated, update the value of redis to release the distributed lock for other calculations. After the metering period (month, year) arrives, the scheduled task clears the metering usage value in redis, that is, setting the metering usage value to 0. Redis is an open source log-type, Key-Value database written in ANSI C language, supporting network, memory-based and persistent, and providing API (Application Programming Interface) in multiple languages.
上述现有技术存在如下缺陷：The above-mentioned prior art has the following defects:
(1)计量日志过大，业务系统接入计量较多，计量频繁ADD操作集群压力是，以及计量做日志查询时效和性能较差；(1) The metering log is too large, there are many business system access metering, the cluster pressure of frequent metering ADD operations is high, and the timeliness and performance of metering log query are poor;
(2)当前用户计量指标使用redis互斥锁实现，需要保存redis和rds两份计量数据，并且高并发情况下，同一个用户同一个计量请求操作会因为锁的原因导致等待而降低系统性能，影响系统处理能力；(2) The current user metering indicators are implemented using redis mutual exclusion locks, and two metering data, redis and rds, need to be saved, and in the case of high concurrency, the same user's same metering request operation will cause waiting due to locks and reduce system performance. Affect system processing capacity;
(3)计量用量周期会一道被清空，无法查询计量指标每一天的用量，无法提供计量月度或者年度账单。(3) The metering usage cycle will be emptied at the same time, and it is impossible to query the usage of the metering indicators for each day, and it is impossible to provide the metering monthly or annual bill.
发明内容SUMMARY OF THE INVENTION
针对现有技术中存在的上述问题，现提供一种指标计量的计算方法、系统、设备和存储介质。In view of the above problems existing in the prior art, a calculation method, system, device and storage medium for index measurement are provided.
具体技术方案如下：The specific technical solutions are as follows:
一种指标计量的计算方法，其中，计算方法包括：A calculation method for index measurement, wherein the calculation method includes:
根据待计算目标发送的计量请求确定计量指标，计量指标用于表征待计算目标的待计算功能；The measurement index is determined according to the measurement request sent by the target to be calculated, and the measurement index is used to represent the function to be calculated of the target to be calculated;
对计量请求中的计量参数进行解析，以得到待计算用量，以及实时查询存储模块中关联于计量指标的已保存用量；Analyzing the metering parameters in the metering request to obtain the amount to be calculated, and querying the stored amount associated with the metering index in the storage module in real time;
根据已保存用量和待计算用量获取有效用量，将有效用量进行处理，以将有效用量写入存储模块中；Obtain the effective amount according to the saved amount and the amount to be calculated, and process the effective amount to write the effective amount into the storage module;
根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新；Calculate the current total amount of calculation according to the effective amount and the saved amount that has been stored in the storage module, and update the saved amount in the storage module according to the total amount of calculation;
将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。Store the metering log associated with the effective usage in the storage module, and store the metering log after incremental subscription in a search engine, so that the effective usage and/or the saved usage can be searched through the search engine according to the metering log .
优选的，计算方法，其中，根据已保存用量和待计算用量获取有效用量，将有效用量进行处理，以将有效用量写入存储模块中，具体包括以下步骤：Preferably, the calculation method, wherein the effective amount is obtained according to the saved amount and the amount to be calculated, and the effective amount is processed to write the effective amount into the storage module, which specifically includes the following steps:
判断已保存用量和待计算用量的和是否超出一预设用量阈值；Determine whether the sum of the saved usage and the to-be-calculated usage exceeds a preset usage threshold;
若是，确定待计算用量为无效用量，退出执行；If so, determine that the amount to be calculated is invalid, and exit the execution;
若否，确定待计算用量为有效用量；If not, determine the amount to be calculated as the effective amount;
采用一实时流读取有效用量，并将读取的有效用量写入存储模块中。A real-time stream is used to read the effective amount, and the read effective amount is written into the storage module.
优选的，计算方法，其中，根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，具体包括以下步骤：Preferably, the calculation method, wherein, according to the effective amount and the saved amount that has been stored in the storage module, the current total amount of calculation is calculated and obtained, which specifically includes the following steps:
采用一离线统计模块根据有效用量和已经存储到存储模块中的已保存用量的和计算得到当前的计算总量。An off-line statistics module is used to calculate the current calculated total amount according to the sum of the effective amount and the saved amount that has been stored in the storage module.
优选的，计算方法，其中，将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，具体包括以下步骤：Preferably, the calculation method, wherein the metering log associated with the effective usage is stored in a storage module, and the metering log after incremental subscription is stored in a search engine, which specifically includes the following steps:
在存储模块中建立数据通道，以通过数据通道将有效用量的计量日志存储到存储模块中；A data channel is established in the storage module to store the metering log of the effective amount in the storage module through the data channel;
采用一订阅模块对计量日志进行订阅处理，并在进行订阅处理后的计量日志的数量达到一预设订阅增量后，将达到预设订阅增量的所有订阅处理后的计量日志存储到搜索引擎中；A subscription module is used to subscribe the metering logs, and after the number of metering logs after the subscription processing reaches a preset subscription increment, all the subscribed metering logs that reach the preset subscription increment are stored in the search engine middle;
对搜索引擎中的计量日志进行刷新处理。Refresh the metering log in the search engine.
优选的，计算方法，其中，采用异步线程池将计量日志存储到存储模块中。Preferably, the calculation method, wherein the metering log is stored in the storage module by using an asynchronous thread pool.
优选的，计算方法，其中，计量指标包括：存储用量、账号开通用量和智能能力消耗用量。Preferably, the calculation method, wherein the measurement indicators include: storage usage, account activation usage, and intelligent capability consumption.
优选的，计算方法，其中，于获取得到计量指标之后还包括：Preferably, the calculation method, after obtaining the measurement index, further comprises:
判断计量指标是否为真实指标；Determine whether the measurement index is a real index;
于计量指标不为真实指标时，判断是否能查询到计量请求的绑定指标；When the measurement index is not a real index, judge whether the binding index of the measurement request can be queried;
于不能查询到计量请求的绑定指标时，退出执行；Exit execution when the bound indicator of the metering request cannot be queried;
于能查询到计量请求的绑定指标时，判断是否允许计量请求绑定绑定指标；When the binding index of the metering request can be queried, determine whether the binding index of the metering request is allowed to be bound;
于允许计量请求绑定绑定指标时，将绑定指标确定为计量请求的计量指标；When the measurement request is allowed to be bound to the binding indicator, the binding indicator is determined as the measurement indicator of the metering request;
于不允许计量请求绑定绑定指标时，退出执行。Exit execution when the metering request is not allowed to bind the bound indicator.
还包括一种指标计量的计算系统，其中，计算系统包括：It also includes a calculation system for index measurement, wherein the calculation system includes:
指标获取模块，用于根据待计算目标发送的计量请求确定计量指标，计量指标用于表征待计算目标的待计算功能；The indicator acquiring module is used to determine the measurement indicator according to the measurement request sent by the target to be calculated, and the measurement indicator is used to represent the to-be-calculated function of the to-be-calculated target;
解析模块，用于对计量请求中的计量参数进行解析，以得到待计算用量，以及实时查询存储模块中关联于计量指标的已保存用量；The parsing module is used to parse the metering parameters in the metering request to obtain the amount to be calculated, and to query the stored amount associated with the metering index in the storage module in real time;
实时计算模块，用于根据已保存用量和待计算用量获取有效用量，将有效用量进行处理，以将有效用量写入存储模块中；The real-time calculation module is used to obtain the effective amount according to the saved amount and the amount to be calculated, and process the effective amount to write the effective amount into the storage module;
离线更新模块，用于根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新；The offline update module is used to calculate the current total amount of calculation according to the effective amount and the saved amount that has been stored in the storage module, and update the saved amount in the storage module according to the total amount of calculation;
搜索模块，用于将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。The search module is used to store the metering log associated with the effective usage in the storage module, and store the metering log after incremental subscription in a search engine, so that the effective usage and/or Saved usage to search.
还包括一种指标计量的计算设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现上述任意一项的计算方法。Also included is a computing device for index measurement, including a memory, a processor, and a computer program stored on the memory and running on the processor, where the processor implements any of the above computing methods when executing the computer program.
还包括一种计算机可读存储介质，其中，计算机可读存储介质上保存有可在处理器上运行的计算机程序，处理器执行计算机程序时实现上述任意一项的计算方法。Also included is a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program that can be executed on a processor, and when the processor executes the computer program, any one of the above computing methods is implemented.
上述技术方案具有如下优点或有益效果：The above-mentioned technical scheme has the following advantages or beneficial effects:
去除了现有技术中的采用redis锁机制，改用实时计算有效用量，并且采用离线统计计算总量，从而有效解决高并发下实时性能问题。The redis lock mechanism in the prior art is removed, and real-time calculation of effective consumption is used instead, and offline statistics are used to calculate the total amount, thereby effectively solving real-time performance problems under high concurrency.
附图说明Description of drawings
参考所附附图，以更加充分的描述本发明的实施例。然而，所附附图仅用于说明和阐述，并不构成对本发明范围的限制。Embodiments of the present invention are described more fully with reference to the accompanying drawings. However, the accompanying drawings are for illustration and illustration only, and are not intended to limit the scope of the present invention.
图1为本发明指标计量的计算方法的实施例的流程图。FIG. 1 is a flow chart of an embodiment of a calculation method for index measurement according to the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动的前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.
需要说明的是，在不冲突的情况下，本发明中的实施例及实施例中的特征可以相互组合。It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict.
下面结合附图和具体实施例对本发明作进一步说明，但不作为本发明的限定。The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but it is not intended to limit the present invention.
本发明包括一种指标计量的计算方法，其中，计算方法包括：The present invention includes a calculation method for index measurement, wherein the calculation method includes:
根据待计算目标发送的计量请求确定计量指标，计量指标用于表征待计算目标的待计算功能；The measurement index is determined according to the measurement request sent by the target to be calculated, and the measurement index is used to represent the function to be calculated of the target to be calculated;
对计量请求中的计量参数进行解析，以得到待计算用量，以及实时查询存储模块中关联于计量指标的已保存用量；Analyzing the metering parameters in the metering request to obtain the amount to be calculated, and querying the stored amount associated with the metering index in the storage module in real time;
根据已保存用量和待计算用量获取有效用量，将有效用量进行处理，以将有效用量写入存储模块中；Obtain the effective amount according to the saved amount and the amount to be calculated, and process the effective amount to write the effective amount into the storage module;
根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新；Calculate the current total amount of calculation according to the effective amount and the saved amount that has been stored in the storage module, and update the saved amount in the storage module according to the total amount of calculation;
将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。Store the metering log associated with the effective usage in the storage module, and store the metering log after incremental subscription in a search engine, so that the effective usage and/or the saved usage can be searched through the search engine according to the metering log .
在上述实施例中，可以根据待计算目标发送的计量请求获取对应的用于表征待计算目标的待计算功能的计量指标；In the above embodiment, the corresponding measurement index used to represent the to-be-calculated function of the to-be-calculated target may be obtained according to the measurement request sent by the to-be-calculated target;
并且对计量请求中的计量参数进行解析，以得到待计量用量；And analyze the metering parameters in the metering request to obtain the amount to be metered;
其中，并不确定此时得到的待计量用量是否为有效用量，因此可以根据计量指标在存储模块中查找关联于计量指标的已保存用量，并且根据上述查找到的已保存用量和上述计算得到的待计算用量获取有效用量；Among them, it is not certain whether the amount to be metered obtained at this time is an effective amount, so the stored amount associated with the metering index can be searched in the storage module according to the metering index, and the stored amount obtained by the above search and the above calculation can be obtained. Obtain the effective amount from the amount to be calculated;
随后，将有效用量进行处理，以将有效用量写入存储模块中；即此时完成了对有效用量的实时计算；Subsequently, the effective amount is processed to write the effective amount into the storage module; that is, the real-time calculation of the effective amount is completed at this time;
其中，需要对存储模块中的已保存用量进行更新，因此，可以根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新；Among them, the saved usage in the storage module needs to be updated. Therefore, the current calculated total amount can be calculated according to the effective usage and the saved usage that has been stored in the storage module, and the stored usage in the storage module can be calculated according to the total calculated amount. Save usage for updating;
由此可见，上述实施例已经实现了对有效用量的实时计算，和对已保存用量的离线更新；It can be seen that the above-mentioned embodiment has realized the real-time calculation of the effective amount and the offline update of the saved amount;
但是还缺少实现查找有效用量和/或已保存用量的方法，因此本实施例将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。从而实现了通过搜索引擎查找有效用量和/或已保存用量。However, there is still a lack of a method for finding the effective amount and/or the saved amount. Therefore, in this embodiment, the metering log associated with the effective amount is stored in the storage module, and the metering log after incremental subscription is stored in a search engine. , so that the effective usage and/or the saved usage are searched by the search engine according to the metering log. Thus, it is possible to find the effective amount and/or the saved amount through a search engine.
作为优选的实施方式，用户可以通过搜索引擎查找对应计量指标的有效用量；As a preferred embodiment, the user can search for the effective amount of the corresponding measurement index through a search engine;
作为优选的实施方式，用户可以通过搜索引擎查找对应计量指标的已保存用量；As a preferred embodiment, the user can search for the saved usage of the corresponding metering index through a search engine;
作为优选的实施方式，用户可以通过搜索引擎查找对应计量指标的有效用量和已保存用量。As a preferred embodiment, the user can search for the effective usage and the saved usage of the corresponding metering index through a search engine.
在上述实施例中，去除了现有技术中的采用redis锁机制，改用实时计算有效用量，并且采用离线统计计算总量，从而有效解决高并发下实时性能问题。In the above embodiment, the use of the redis lock mechanism in the prior art is removed, the effective amount is calculated in real time, and the total amount is calculated by offline statistics, thereby effectively solving the real-time performance problem under high concurrency.
在上述实施例中，待计算用量和有效用量均为当前运营阶段的用量；而已保存用量包括当前运营阶段前一个阶段的用量和当前运营阶段前一个阶段之前的用量。例如，运营阶段周期为一天，则待计算用量和有效用量均为今天的用量，而已保存用量包括昨天的用量和昨天之前的用量。In the above embodiment, the to-be-calculated usage and the effective usage are the usage in the current operation stage; the saved usage includes the usage in the previous stage of the current operation stage and the usage in the previous stage of the current operation stage. For example, if the operation period is one day, the to-be-calculated usage and the effective usage are both today's usage, and the saved usage includes yesterday's usage and usage before yesterday.
进一步地，在上述实施例中，根据已保存用量和待计算用量获取有效用量，将有效用量进行处理，以将有效用量写入存储模块中，具体包括以下步骤：Further, in the above embodiment, the effective amount is obtained according to the saved amount and the amount to be calculated, and the effective amount is processed to write the effective amount into the storage module, which specifically includes the following steps:
判断已保存用量和待计算用量的和是否超出一预设用量阈值；Determine whether the sum of the saved usage and the to-be-calculated usage exceeds a preset usage threshold;
若是，确定待计算用量为无效用量，退出执行；If so, determine that the amount to be calculated is invalid, and exit the execution;
若否，确定待计算用量为有效用量；If not, determine the amount to be calculated as the effective amount;
采用一实时流读取有效用量，并将读取的有效用量写入存储模块中。A real-time stream is used to read the effective amount, and the read effective amount is written into the storage module.
在上述实施例中，根据计量请求中的计量参数设置上述预设用量阈值。In the above-mentioned embodiment, the above-mentioned preset usage threshold is set according to the metering parameter in the metering request.
其中，预设用量阈值可以根据具体业务进行自定义设置，例如存储用量限制100G时，预设用量阈值为存储用量限制100G；例如账号开通个数限制100个，预设用量阈值为账号开通个数限制100个。The preset usage threshold can be customized according to specific services. For example, when the storage usage is limited to 100G, the preset usage threshold is the storage usage limit of 100G; for example, the number of accounts opened is limited to 100, and the preset usage threshold is the number of accounts opened. Limit 100.
在上述实施例中，在退出执行后将异常状况通过告警信息发出。In the above-mentioned embodiment, the abnormal situation is sent out through the alarm information after the execution is exited.
在上述实施例中，采用实时流将有效用量作为数据源进行读取，并将读取的有效用量写入存储模块中。In the above embodiment, the real-time stream is used to read the effective amount as a data source, and the read effective amount is written into the storage module.
作为优选的实施方式，实时流将有效用量作为数据源进行读取(此时的读取操作可以采用Source Operator操作)，并将读取的有效用量写入存储模块中(此时的写入操作可以为Sink Operator操作)。As a preferred embodiment, the real-time stream reads the effective amount as a data source (the read operation at this time can use the Source Operator operation), and writes the read effective amount into the storage module (the write operation at this time) Can be operated for Sink Operator).
其中，在实时流中会将接收到的有效用量进行字段解析并转换成一个中间数据交给实时流的task模块(一种任务管理器)进行计算并进行seek操作(一种下沉操作)，以将读取的有效用量保存更新到存储模块中。Among them, in the real-time flow, the received effective amount will be field parsed and converted into an intermediate data, which will be handed over to the task module (a kind of task manager) of the real-time flow for calculation and seek operation (a kind of sinking operation), Save the update to the storage module with the read effective amount.
进一步地，在上述实施例中，根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，具体包括以下步骤：Further, in the above-mentioned embodiment, the current calculated total amount is calculated and obtained according to the effective amount and the saved amount that has been stored in the storage module, which specifically includes the following steps:
采用一离线统计模块根据有效用量和已经存储到存储模块中的已保存用量的和计算得到当前的计算总量。An off-line statistics module is used to calculate the current calculated total amount according to the sum of the effective amount and the saved amount that has been stored in the storage module.
在上述实施例中，离线统计模块可以采用MaxComputer离线统计模块。In the above embodiment, the offline statistics module may adopt the MaxComputer offline statistics module.
其中，MaxComputer离线统计模块可支持海量大数据离线计算，即支持较大量级的计算。Among them, the offline statistics module of MaxComputer can support offline computing of massive big data, that is, it can support large-scale computing.
在上述实施例中，待计算用量和有效用量均为当前运营阶段的用量；而已保存用量包括当前运营阶段前一个阶段的用量和当前运营阶段前一个阶段之前的用量。例如，运营阶段周期为一天，则待计算用量和有效用量均为今天的用量，而已保存用量包括昨天的用量和昨天之前的用量，因此计算得到的当前的计算总量包括今天的用量、昨天的用量和昨天之前的用量，将计算总量对存储模块中的已保存用量进行更新，即更新后的已保存用量包括今天的用量和今天之前的用量。In the above embodiment, the to-be-calculated usage and the effective usage are the usage in the current operation stage; the saved usage includes the usage in the previous stage of the current operation stage and the usage in the previous stage of the current operation stage. For example, if the period of the operation phase is one day, the to-be-calculated usage and the effective usage are both today's usage, and the saved usage includes yesterday's usage and usage before yesterday. Therefore, the calculated current total amount includes today's usage and yesterday's usage. Usage and usage before yesterday, the total amount will be calculated to update the saved usage in the storage module, that is, the updated saved usage includes today's usage and usage before today.
进一步地，在上述实施例中，将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，具体包括以下步骤：Further, in the above-mentioned embodiment, the metering log associated with the effective usage is stored in the storage module, and the metering log after incremental subscription is stored in a search engine, which specifically includes the following steps:
在存储模块中建立数据通道，以通过数据通道将有效用量的计量日志存储到存储模块中；A data channel is established in the storage module to store the metering log of the effective amount in the storage module through the data channel;
采用一订阅模块对计量日志进行订阅处理，并在进行订阅处理后的计量日志的数量达到一预设订阅增量后，将达到预设订阅增量的所有订阅处理后的计量日志存储到搜索引擎中；A subscription module is used to subscribe the metering logs, and after the number of metering logs after the subscription processing reaches a preset subscription increment, all the subscribed metering logs that reach the preset subscription increment are stored in the search engine middle;
对搜索引擎中的计量日志进行刷新处理。Refresh the metering log in the search engine.
在上述实施例中，通过建立数据通道，以简单地实现对有效用量的计量日志存储到存储模块中。In the above-mentioned embodiment, by establishing a data channel, the metering log of the effective amount is simply stored in the storage module.
在上述实施例中，订阅模块可以采用OTS(Open Table Service，一种数据库服务)TunnelService(OTS通道服务)进行计量日志的增量订阅处理，将达到预设订阅增量的所有订阅处理后的计量日志存储到搜索引擎中，并在监听到计量日志存储到搜索引擎后对搜索引擎中的计量日志进行刷新处理，防止逐条插入计量日志导致的线程过多。In the above-mentioned embodiment, the subscription module may use OTS (Open Table Service, a database service) TunnelService (OTS channel service) to perform incremental subscription processing of metering logs, and will perform metering of all subscriptions that reach the preset subscription increment after processing. The log is stored in the search engine, and after monitoring the metering log stored in the search engine, the metering log in the search engine is refreshed to prevent too many threads caused by inserting the metering log one by one.
进一步地，在上述实施例中，采用异步线程池将计量日志存储到存储模块中。Further, in the above embodiment, the asynchronous thread pool is used to store the metering log in the storage module.
进一步地，在上述实施例中，计量指标包括：存储用量、账号开通用量和智能能力消耗用量。Further, in the above embodiment, the metering indicators include: storage usage, account activation usage, and intelligent capability consumption.
进一步地，在上述实施例中，于获取得到计量指标之后还包括：Further, in the above embodiment, after obtaining the measurement index, it further includes:
判断计量指标是否为真实指标；Determine whether the measurement index is a real index;
于计量指标不为真实指标时，判断是否能查询到计量请求的绑定指标；When the measurement index is not a real index, judge whether the binding index of the measurement request can be queried;
于能查询到计量请求的绑定指标时，判断是否允许计量请求绑定绑定指标；When the binding index of the metering request can be queried, determine whether the binding index of the metering request is allowed to be bound;
于允许计量请求绑定绑定指标时，将绑定指标确定为计量请求的计量指标；When the measurement request is allowed to be bound to the binding indicator, the binding indicator is determined as the measurement indicator of the metering request;
于不允许计量请求绑定绑定指标时，退出执行；Exit execution when the metering request is not allowed to bind the bound indicator;
于不能查询到计量请求的绑定指标时，退出执行。Exit execution when the bound indicator of the metering request cannot be queried.
作为优选的实施方式，上述待计算目标可以为业务系统，业务系统发送计量请求，计量系统接收到业务系统的计量请求；As a preferred embodiment, the above target to be calculated may be a business system, the business system sends a metering request, and the metering system receives the metering request from the business system;
随后计量系统判断计量指标是否为真实指标，即判断计量指标是否真实存在；Then the measurement system judges whether the measurement index is a real index, that is, whether the measurement index really exists;
于计量指标为真实指标时(即计量指标存在时)，确定计量指标；When the measurement indicators are real indicators (that is, when the measurement indicators exist), determine the measurement indicators;
于计量指标不为真实指标时(即计量指标不存在时)，判断是否能查询到计量请求的绑定指标；When the measurement indicator is not a real indicator (that is, when the measurement indicator does not exist), determine whether the binding indicator of the measurement request can be queried;
于能查询到计量请求的绑定指标时，判断是否允许计量请求绑定绑定指标；When the binding index of the metering request can be queried, determine whether the binding index of the metering request is allowed to be bound;
于允许计量请求绑定绑定指标时，将绑定指标确定为计量请求的计量指标，即将绑定指标自动绑定到计量请求上；When the measurement request is allowed to be bound to the binding indicator, the binding indicator is determined as the measurement indicator of the metering request, that is, the binding indicator is automatically bound to the metering request;
于不允许计量请求绑定绑定指标时，退出执行，并则将异常状况通过告警信息返回给业务系统；When the metering request is not allowed to be bound to the bound index, the execution is exited, and the abnormal situation is returned to the business system through the alarm information;
于不能查询到计量请求的绑定指标时，退出执行，并则将异常状况通过告警信息返回给业务系统。When the binding index of the metering request cannot be queried, the execution is terminated, and the abnormal situation is returned to the business system through the alarm information.
在上述实施例中，在退出执行后将异常状况通过告警信息发出。In the above-mentioned embodiment, the abnormal situation is sent out through the alarm information after the execution is exited.
作为具体的实施方式，如图1所示，As a specific implementation, as shown in Figure 1,
首先，接收到待计算目标发送的计量请求，上述计量请求包括MQ(RocketMq)协议和/或RPC(Remote Procedure Cal)协议，并通过上述MQ协议和/或RPC协议后开始进行有效用度和已保存用度的计算；First, the metering request sent by the target to be calculated is received. The metering request includes the MQ (RocketMq) protocol and/or the RPC (Remote Procedure Cal) protocol. save the calculation of the cost;
需要说明的是，MQ协议是一款开源的分布式消息系统，基于高可用分布式集群技术，提供低延时的、高可靠的消息发布与订阅服务；It should be noted that the MQ protocol is an open source distributed messaging system, based on high-availability distributed cluster technology, providing low-latency, high-reliability message publishing and subscription services;
RPC协议用于远程过程调用。The RPC protocol is used for remote procedure calls.
其中，可以接收到待计算目标发送的计量请求后执行RPC协议，随后开始进行有效用度和已保存用度的计算，即确定计量指标；Among them, the RPC protocol can be executed after receiving the measurement request sent by the target to be calculated, and then the calculation of the effective cost and the saved cost can be started, that is, the measurement index is determined;
也可以接收到待计算目标发送的计量请求后执行MQ协议，随后开始确定计量指标；It is also possible to execute the MQ protocol after receiving the measurement request sent by the target to be calculated, and then start to determine the measurement indicators;
也可以接收到待计算目标发送的计量请求后异步执行RPC协议和MQ协议，随后开始确定计量指标。It is also possible to asynchronously execute the RPC protocol and the MQ protocol after receiving the measurement request sent by the target to be calculated, and then start to determine the measurement indicators.
接着，判断计量指标是否为真实指标，即判断计量指标是否存在；Next, determine whether the measurement index is a real index, that is, determine whether the measurement index exists;
于计量指标为真实指标时(即计量指标存在时)，确定计量指标；When the measurement indicators are real indicators (that is, when the measurement indicators exist), determine the measurement indicators;
于计量指标不为真实指标时(即计量指标不存在时)，判断是否能查询到计量请求的绑定指标；When the measurement indicator is not a real indicator (that is, when the measurement indicator does not exist), determine whether the binding indicator of the measurement request can be queried;
于能查询到计量请求的绑定指标时，判断是否允许计量请求绑定绑定指标；When the binding index of the metering request can be queried, determine whether the binding index of the metering request is allowed to be bound;
于允许计量请求绑定绑定指标时，将绑定指标确定为计量请求的计量指标，即将绑定指标自动绑定到计量请求上；When the measurement request is allowed to be bound to the binding indicator, the binding indicator is determined as the measurement indicator of the metering request, that is, the binding indicator is automatically bound to the metering request;
于不允许计量请求绑定绑定指标时，退出执行，并则将异常状况通过告警信息返回给业务系统；When the metering request is not allowed to be bound to the bound index, the execution is exited, and the abnormal situation is returned to the business system through the alarm information;
于不能查询到计量请求的绑定指标时，退出执行，并则将异常状况通过告警信息返回给业务系统。When the binding index of the metering request cannot be queried, the execution is terminated, and the abnormal situation is returned to the business system through the alarm information.
随后，通过已保存用量和待计算用量的和是否超过预设用量阈值来判断待计量指标是否为有效指标；Subsequently, it is judged whether the to-be-measured indicator is an effective indicator by whether the sum of the saved usage and the to-be-calculated usage exceeds the preset usage threshold;
于已保存用量和待计算用量的和超过预设用量阈值时，确定待计量指标为无效指标，并输出无效指标对应的计量日志；When the sum of the saved usage and the to-be-calculated usage exceeds the preset usage threshold, determine the indicator to be measured as an invalid indicator, and output the metering log corresponding to the invalid indicator;
于已保存用量和待计算用量的和未超过预设用量阈值时，确定待计量指标为有效指标，并输出有效指标对应的计量日志；When the sum of the saved usage and the to-be-calculated usage does not exceed the preset usage threshold, determine the indicator to be measured as a valid indicator, and output the measurement log corresponding to the valid indicator;
然后，采用实时流将有效用量进行处理，以将有效用量写入存储模块中；Then, use the real-time stream to process the effective amount to write the effective amount into the storage module;
紧接着，根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新。Next, the current calculated total amount is calculated according to the effective amount and the saved amount that has been stored in the storage module, and the saved amount in the storage module is updated according to the calculated amount.
其中，可以将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。Among them, the metering log associated with the effective usage can be stored in the storage module, and the metering log after incremental subscription can be stored in a search engine, so that the effective usage and/or the saved amount can be stored in the search engine according to the metering log. Search by usage.
需要说明的是，在上述实施例中，计量请求可以包括多个协议；例如计量请求中包含的MQ协议计量，针对高并发下的异步场景，如网关接口用量计量等；例如计量请求中包含的RPC协议，提供DUBBO(一种服务框架)接口给业务系统同步调用，针对计量时需要提前校验计量用量是否有效或者用量未超等判断；例如计量请求中包含的HTTP协议，提供HTTP接口给业务系统同步调用。It should be noted that, in the above embodiment, the metering request may include multiple protocols; for example, the MQ protocol metering included in the metering request, for asynchronous scenarios under high concurrency, such as gateway interface usage metering, etc.; RPC protocol, which provides DUBBO (a service framework) interface to the business system to call synchronously. For metering, it is necessary to verify in advance whether the metered amount is valid or the amount is not exceeded; for example, the HTTP protocol included in the metering request provides an HTTP interface to the business. System synchronization calls.
还包括一种指标计量的计算系统，其特征在于，计算系统包括：It also includes a computing system for index measurement, characterized in that the computing system includes:
指标获取模块，用于根据待计算目标发送的计量请求确定计量指标，计量指标用于表征待计算目标的待计算功能；The indicator acquiring module is used to determine the measurement indicator according to the measurement request sent by the target to be calculated, and the measurement indicator is used to represent the to-be-calculated function of the to-be-calculated target;
解析模块，用于对计量请求中的计量参数进行解析，以得到待计算用量，以及实时查询存储模块中关联于计量指标的已保存用量；The parsing module is used to parse the metering parameters in the metering request to obtain the amount to be calculated, and to query the stored amount associated with the metering index in the storage module in real time;
实时计算模块，用于根据已保存用量和待计算用量获取有效用量，将有效用量进行处理，以将有效用量写入存储模块中；The real-time calculation module is used to obtain the effective amount according to the saved amount and the amount to be calculated, and process the effective amount to write the effective amount into the storage module;
离线更新模块，用于根据有效用量和已经存储到存储模块中的已保存用量计算得到当前的计算总量，并根据计算总量对存储模块中的已保存用量进行更新；The offline update module is used to calculate the current total amount of calculation according to the effective amount and the saved amount that has been stored in the storage module, and update the saved amount in the storage module according to the total amount of calculation;
搜索模块，用于将与有效用量相关联的计量日志存储到存储模块中，并将进行增量订阅后的计量日志存储到一搜索引擎中，使得通过搜索引擎根据计量日志对有效用量和/或已保存用量进行搜索。The search module is used to store the metering log associated with the effective usage in the storage module, and store the metering log after incremental subscription in a search engine, so that the effective usage and/or Saved usage to search.
本发明指标计量的计算系统的具体实施方式与上述指标计量的计算方法的各实施例基本相同，在此不再赘述。The specific implementations of the calculation system for index measurement of the present invention are basically the same as the above-mentioned embodiments of the calculation method for index measurement, and are not repeated here.
还包括一种指标计量的计算设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现上述任意一项的计算方法。Also included is a computing device for index measurement, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements any of the above computing methods when executing the computer program.
本发明指标计量的计算设备的具体实施方式与上述指标计量的计算方法的各实施例基本相同，在此不再赘述。The specific implementations of the computing device for index measurement according to the present invention are basically the same as the above-mentioned embodiments of the calculation method for index measurement, and are not repeated here.
还包括一种计算机可读存储介质，其中，计算机可读存储介质上保存有可在处理器上运行的计算机程序，处理器执行计算机程序时实现上述任意一项的计算方法。Also included is a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program that can be executed on a processor, and when the processor executes the computer program, any one of the above computing methods is implemented.
本发明计算机可读存储介质的具体实施方式与上述指标计量的计算方法的各实施例基本相同，在此不再赘述。The specific implementations of the computer-readable storage medium of the present invention are basically the same as the above-mentioned embodiments of the calculation methods for index measurement, and are not repeated here.
以上所述仅为本发明较佳的实施例，并非因此限制本发明的实施方式及保护范围，对于本领域技术人员而言，应当能够意识到凡运用本发明说明书及图示内容所作出的等同替换和显而易见的变化所得到的方案，均应当包含在本发明的保护范围内。The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the embodiments and protection scope of the present invention. For those skilled in the art, they should be able to realize that all equivalents made by using the description and illustrations of the present invention The solutions obtained by substitutions and obvious changes shall all be included in the protection scope of the present invention.
此外，需要说明的是，本说明书中所描述的具体实施例，其零、部件的形状、所取名称等可以不同。凡依本发明专利构思所述的构造、特征及原理所做的等效或简单变化，均包括于本发明专利的保护范围内。本发明所属技术领域的技术人员可以对所描述的具体实施例做各种各样的修改或补充或采用类似的方式替代，只要不偏离本发明的结构或者超越本权利要求书所定义的范围，均应属于本发明的保护范围。In addition, it should be noted that, in the specific embodiments described in this specification, the shapes and names of parts and components thereof may be different. All equivalent or simple changes made according to the structures, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. Those skilled in the art to which the present invention pertains can make various modifications or supplements to the described specific embodiments or substitute in similar manners, as long as they do not deviate from the structure of the present invention or go beyond the scope defined by the claims, All should belong to the protection scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010731965.3A CN111881161A (en) | 2020-07-27 | 2020-07-27 | Calculation method, system, device and storage medium for index measurement |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010731965.3A CN111881161A (en) | 2020-07-27 | 2020-07-27 | Calculation method, system, device and storage medium for index measurement |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111881161A true CN111881161A (en) | 2020-11-03 |
Family
ID=73201697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010731965.3A Pending CN111881161A (en) | 2020-07-27 | 2020-07-27 | Calculation method, system, device and storage medium for index measurement |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111881161A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112966007A (en) * | 2021-04-02 | 2021-06-15 | 新华智云科技有限公司 | Search terminal control method and search terminal |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100114954A1 (en) * | 2008-10-28 | 2010-05-06 | Microsoft Corporation | Realtime popularity prediction for events and queries |
| CN104079662A (en) * | 2014-07-14 | 2014-10-01 | 中国联合网络通信集团有限公司 | Measuring system and method |
| US20180341956A1 (en) * | 2017-05-26 | 2018-11-29 | Digital River, Inc. | Real-Time Web Analytics System and Method |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN109428753A (en) * | 2017-08-29 | 2019-03-05 | 西门子公司 | Measuring index acquisition methods, service call record acquisition methods and device |
| CN109802990A (en) * | 2018-12-10 | 2019-05-24 | 视联动力信息技术股份有限公司 | A kind of resource log reading/writing method and device reducing data redundancy |
| US20190205998A1 (en) * | 2018-01-02 | 2019-07-04 | Patrick Schur | System and method for a prosumer crypto social media and cross bridge service collaboration based on operant tag(s) and d-emojis/d-emoticons |
| CN110633186A (en) * | 2019-08-16 | 2019-12-31 | 南方电网科学研究院有限责任公司 | Log monitoring system and implementation method for power metering microservice architecture |
-
2020
- 2020-07-27 CN CN202010731965.3A patent/CN111881161A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100114954A1 (en) * | 2008-10-28 | 2010-05-06 | Microsoft Corporation | Realtime popularity prediction for events and queries |
| CN104079662A (en) * | 2014-07-14 | 2014-10-01 | 中国联合网络通信集团有限公司 | Measuring system and method |
| US20180341956A1 (en) * | 2017-05-26 | 2018-11-29 | Digital River, Inc. | Real-Time Web Analytics System and Method |
| CN109428753A (en) * | 2017-08-29 | 2019-03-05 | 西门子公司 | Measuring index acquisition methods, service call record acquisition methods and device |
| US20190205998A1 (en) * | 2018-01-02 | 2019-07-04 | Patrick Schur | System and method for a prosumer crypto social media and cross bridge service collaboration based on operant tag(s) and d-emojis/d-emoticons |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN109802990A (en) * | 2018-12-10 | 2019-05-24 | 视联动力信息技术股份有限公司 | A kind of resource log reading/writing method and device reducing data redundancy |
| CN110633186A (en) * | 2019-08-16 | 2019-12-31 | 南方电网科学研究院有限责任公司 | Log monitoring system and implementation method for power metering microservice architecture |
Non-Patent Citations (3)
| Title |
|---|
| 党寿江;刘学;王星凯;刘春梅;: "基于Spark Streaming的实时数据采集分析系统设计", 网络新媒体技术, no. 05 * |
| 汤网祥;王金华;赫凌俊;李敏敬;: "大规模软件系统日志汇集服务平台设计与实现", 计算机应用与软件, no. 11 * |
| 王直荣;徐晓峰;: "大数据环境下计量系统日志的存储架构设计", 计量与测试技术, no. 06 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112966007A (en) * | 2021-04-02 | 2021-06-15 | 新华智云科技有限公司 | Search terminal control method and search terminal |
| CN112966007B (en) * | 2021-04-02 | 2022-06-17 | 新华智云科技有限公司 | Search terminal control method and search terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220179860A1 (en) | Database workload capture and replay | |
| US9361342B2 (en) | Query to streaming data | |
| CN108647357B (en) | Method and device for data query | |
| US20140156683A1 (en) | Integrating event processing with map-reduce | |
| US20050055673A1 (en) | Automatic database diagnostic monitor architecture | |
| CN111522870B (en) | Database access method, middleware and readable storage medium | |
| CN112464165B (en) | A method, storage medium and computing device for improving statistical efficiency of measuring points | |
| CN111723004B (en) | Measurement method for agile software development, measurement data output method and device | |
| CN110569399B (en) | Link construction method based on pinpoint log | |
| CN110990227A (en) | A numerical pool application characteristic performance acquisition and monitoring system and its operation method | |
| CN112506953A (en) | Query method, device and storage medium based on Structured Query Language (SQL) | |
| CN119202100A (en) | Real-time processing method and device for equipment operation data of Internet of Things | |
| CN118964469A (en) | A multi-source heterogeneous data integration method and system for data integration middleware | |
| CN111782671A (en) | An Optimization Method for Realizing CDC Based on Distributed Database | |
| CN116910079B (en) | A Flink method, system, device, and storage medium for implementing delayed association of CDC data dimension tables | |
| CN111881161A (en) | Calculation method, system, device and storage medium for index measurement | |
| CN105303430B (en) | Transaction index acquisition method and device | |
| US11709752B2 (en) | Pause and resume in database system workload capture and replay | |
| CN120066966A (en) | Database compatibility testing method and system | |
| CN116821422A (en) | Data full-link tracking and analysis methods, devices, electronic equipment and storage media | |
| CN115099769B (en) | A risk control approval platform for automatic approval of automobile finance orders | |
| CN111221846A (en) | Automatic translation method and device for SQL (structured query language) statements | |
| Felius | Assessing the performance of distributed PostgreSQL | |
| CN119718819B (en) | Thread pool runtime status monitoring method and device | |
| CN119066084B (en) | A data calculation method, system, intelligent terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201103 |