[go: up one dir, main page]

CN114722059B - Real-time data statistics method and system - Google Patents

Real-time data statistics method and system Download PDF

Info

Publication number
CN114722059B
CN114722059B CN202210340719.4A CN202210340719A CN114722059B CN 114722059 B CN114722059 B CN 114722059B CN 202210340719 A CN202210340719 A CN 202210340719A CN 114722059 B CN114722059 B CN 114722059B
Authority
CN
China
Prior art keywords
data
time
reported data
period
statistics
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210340719.4A
Other languages
Chinese (zh)
Other versions
CN114722059A (en
Inventor
毕鲁君
丁振
王滨
黄林弟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Zhengtai Iot Technology Co ltd
Zhejiang Chint Instrument and Meter Co Ltd
Original Assignee
Zhejiang Zhengtai Iot Technology Co ltd
Zhejiang Chint Instrument and Meter Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Zhengtai Iot Technology Co ltd, Zhejiang Chint Instrument and Meter Co Ltd filed Critical Zhejiang Zhengtai Iot Technology Co ltd
Priority to CN202210340719.4A priority Critical patent/CN114722059B/en
Publication of CN114722059A publication Critical patent/CN114722059A/en
Application granted granted Critical
Publication of CN114722059B publication Critical patent/CN114722059B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2379Updates performed during online database operations; commit processing
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01RMEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R21/00Arrangements for measuring electric power or power factor
    • G01R21/007Adapted for special tariff measuring
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2462Approximate or statistical queries

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Fuzzy Systems (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Power Engineering (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明公开了一种实时数据统计方法及系统,包括:获取预先存储到消息队列中设备上报数据和对应的采集时间,并根据上报数据和对应的采集时间至少形成第一上报数据、第二上报数据;从消息队列中获取第二上报数据及对应采集时间并从缓存区获取第一上报数据及对应采集时间;当第一上报数据对应采集时间与第二上报数据对应采集时间属于同一统计时段,将第二上报数据存储到数据库第一存储统计时段中,并将第二上报数据及对应采集时间写入缓存区;当第一上报数据对应采集时间与第二上报数据对应采集时间不属于同一统计时段,将设备数据存储到数据库第二存储统计时段中,并将第二上报数据及对应采集时间写入缓存区。

The present invention discloses a real-time data statistics method and system, comprising: obtaining device reported data and corresponding collection time pre-stored in a message queue, and forming at least first reported data and second reported data according to the reported data and the corresponding collection time; obtaining the second reported data and the corresponding collection time from the message queue and obtaining the first reported data and the corresponding collection time from a buffer area; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, storing the second reported data in a first storage statistical period of a database, and writing the second reported data and the corresponding collection time into the buffer area; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the device data in a second storage statistical period of the database, and writing the second reported data and the corresponding collection time into the buffer area.

Description

Real-time data statistics method and system
Technical Field
The invention relates to the technical field of instruments and meters of the Internet of things, in particular to a real-time data statistics method and a real-time data statistics system.
Background
The internet of things is provided with a large number of devices (such as electric quantity collection devices, water quantity collection devices or air consumption collection devices) with data collection and metering functions, and the devices have remote communication functions and can upload instantaneous values of device use data to a statistics platform according to time sequences. Taking electricity metering as an example, the electricity metering mode of the existing statistical platform generally uses the accumulated quantity of electricity as a data source, when the statistical platform needs to count the hour, day, month and year electricity consumption of the electricity collection device as the basis of energy consumption statistical analysis, the traditional statistical mode mostly adopts timing statistics, when the hour electricity consumption is used as the basis of energy consumption analysis, the current hour counts the electricity of a plurality of hours, the method has hysteresis for electricity statistics, and the electricity statistics task needs to process a large amount of data in a statistical period, and at the moment, the load of the system is higher.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the defects that the electric quantity statistics method has hysteresis in the prior art, and the electric quantity statistics task needs to process a large amount of data in one statistics period, so that the system load is high, thereby providing the real-time data statistics method and the system.
According to a first aspect, the embodiment of the invention discloses a real-time data statistics method, which comprises the steps of obtaining device report data and corresponding collection time which are stored in a message queue in advance, forming at least first report data and second report data according to the report data and the corresponding collection time, obtaining the second report data and the corresponding collection time from the message queue, obtaining the first report data and the corresponding collection time from a cache area, comparing the corresponding collection time of the first report data with the corresponding collection time of the second report data, storing the second report data into a first storage statistics period of a database when the corresponding collection time of the first report data and the corresponding collection time of the second report data belong to the same statistics period, and writing the second report data into the cache area when the corresponding collection time of the first report data and the corresponding collection time of the second report data do not belong to the same statistics period, and storing the device data into the second storage statistics period and writing the corresponding collection time of the second report data into the cache area.
Optionally, the stored statistical period data comprises a device unique number, a date, a statistical period, a starting value, a final value, a starting value acquisition time and a final value acquisition time.
Optionally, when the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data belong to the same statistical period, storing the second reporting data into the first storage statistical period of the database, wherein the method comprises the steps of replacing the last value of the first reporting data statistics in the current statistical period by the last value of the acquired second reporting data stored into the first storage statistical period, keeping the initial value unchanged, writing the second reporting data and the corresponding acquisition time into a cache region, and updating the data of the cache region.
Optionally, when the corresponding collection time of the first reporting data and the corresponding collection time of the second reporting data do not belong to the same statistical period, storing the second reporting data into a second storage statistical period of the database, wherein the method comprises the steps of using the obtained end value of the first reporting data stored into the first storage statistical period as a starting value of the second storage statistical period, storing the end value of the second reporting data stored into the second storage statistical period as an end value of the second storage statistical period, keeping the starting value and the end value of the first storage statistical period unchanged, writing the second reporting data and the corresponding collection time pair into a cache region, and updating the data of the cache region.
Optionally, before acquiring the device report data and the corresponding acquisition time stored in the message queue in advance and forming at least first report data and second report data according to the report data and the corresponding acquisition time, the method further comprises the steps of acquiring the first report data and the corresponding acquisition time from the message queue for the first time, forming at least one first storage statistic period in a database, and writing the first report data and the acquisition time acquired for the first time into a cache region.
Optionally, the types of the storage statistics time period comprise storage statistics hour time period, storage statistics day time period, storage statistics week time period, storage statistics month time period and storage statistics year time period.
Optionally, before the acquiring the device report data and the corresponding acquisition time stored in the message queue in advance and forming at least first report data and second report data according to the report data and the corresponding acquisition time, the method further comprises the steps of firstly acquiring the first report data and the corresponding acquisition time from the message queue, forming at least two of a storage statistics hour period, a storage statistics day period, a storage statistics week period, a storage statistics month period and a storage statistics year period type in a database, forming corresponding first storage statistics periods in different types of storage statistics periods respectively, and writing the first report data acquired for the first time and the acquisition time into a cache region.
Optionally, the device reported data comprises a device unique number and a table code value.
Optionally, the reported data comprises any one of water consumption metering equipment, water consumption, gas consumption metering equipment, gas consumption metering equipment and electricity consumption.
According to a second aspect, the embodiment of the invention also discloses a real-time data statistics system, which executes the real-time data statistics method according to the first aspect or any optional implementation manner of the first aspect, wherein the system comprises a message queue module, a real-time statistics service program module, a database module and a cache module, wherein the real-time statistics service program module acquires the device report data and the corresponding acquisition time from the message queue module, performs time period comparison judgment on the device report data and the corresponding acquisition time, performs data statistics update on the judgment processing result according to the database statistics rule, and updates the latest report data and the corresponding acquisition time in the cache module.
The technical scheme of the invention has the following advantages:
The real-time data statistics method comprises the steps of obtaining device report data and corresponding collection time which are stored in a message queue in advance, forming at least first report data and second report data according to the report data and the corresponding collection time, obtaining the second report data and the corresponding collection time from the message queue, obtaining the first report data and the corresponding collection time from a cache area, comparing the corresponding collection time of the first report data with the corresponding collection time of the second report data, storing the second report data into a first storage statistics period of a database when the corresponding collection time of the first report data and the corresponding collection time of the second report data belong to the same statistics period, and writing the second report data and the corresponding collection time into the cache area, and storing the device data into the second storage statistics period of the database when the corresponding collection time of the first report data and the corresponding collection time of the second report data do not belong to the same statistics period. The method is a time sequence-based streaming real-time data statistics method, and the data reported by a piece of equipment is obtained to process a piece of data in real time, so that the electricity consumption of the current statistics period can be counted in real time, and meanwhile, the real-time data statistics task does not need to count all the data in one statistics period, so that the load of a statistics system is effectively reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a real-time data statistics method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a real-time data statistics system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram showing a specific example of a real-time data statistics method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram showing a specific example of a real-time data statistics method according to an embodiment of the present invention;
FIGS. 5A-5D are schematic diagrams illustrating a specific example of a real-time data statistics method according to an embodiment of the present invention;
FIG. 6 is a schematic block diagram of a specific example of a real-time data statistics apparatus in an embodiment of the present invention;
fig. 7 is a diagram illustrating an embodiment of an electronic device according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected, mechanically connected, electrically connected, directly connected, indirectly connected via an intermediate medium, and in communication with each other between two elements, and wirelessly connected, or wired. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
The embodiment of the application discloses a real-time data statistics method which can be applied to Internet of things equipment, wherein real-time data comprises any one of water consumption, gas consumption and electricity consumption, and can also be expanded to statistics of other cumulative amounts. In the embodiment of the present application, the real-time data statistics method described in the embodiment will be described by taking the electricity consumption for counting hours as an example. As shown in fig. 1, the method comprises the steps of:
Step 101, acquiring the reporting data and the corresponding acquisition time of the equipment pre-stored in the message queue, and forming at least first reporting data and second reporting data according to the reporting data and the corresponding acquisition time.
The device report data and the corresponding first acquisition time are stored in a message queue, the second report data is the data reported by the current device, and the first report data is the data reported by the device.
Step 102, obtaining the second report data and the corresponding collection time from the message queue, and obtaining the first report data and the corresponding collection time from the buffer.
Illustratively, the buffer area stores the data of the last report (the first report data) and the corresponding acquisition time, and specifically, the last processed data stored in the buffer area may also be referred to as buffer data.
And step 103, comparing the corresponding acquisition time of the first reported data with the corresponding acquisition time of the second reported data.
Step 104, when the corresponding collection time of the first reporting data and the corresponding collection time of the second reporting data belong to the same statistical period, storing the second reporting data and the corresponding collection time into a first storage statistical period of a database, and writing the second reporting data and the corresponding collection time into a cache region.
In an exemplary embodiment, the statistics period is a period that can be preset according to the statistics requirement, the value of the statistics period is not limited in this embodiment, the corresponding collection time of the data (second reporting data) reported by the current device and the corresponding collection time of the data (first reporting data) reported by the previous device belong to the same statistics period, then the second reporting data is stored in the first storage statistics period of the database, and the data (second reporting data) stored by the current device and the corresponding collection time are stored in the cache region.
Step 105, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the device data into a second storage statistical period of a database, and writing the second reported data and the corresponding collection time into a buffer area.
For example, if the collection time corresponding to the data (second report data) reported by the current device and the collection time corresponding to the data (first report data) reported by the previous device do not belong to the same statistical period, the second report data is stored in a second storage statistical period of the database, and the data (second report data) stored by the current device and the corresponding collection time are stored in the cache region.
The real-time data statistics method provided by the invention is a time sequence-based streaming real-time data statistics method, and the data reported by a piece of equipment is obtained to process a piece of data in real time, so that the electricity consumption of the current statistics period can be counted in real time, and meanwhile, the real-time data statistics task does not need to count all the data in one statistics period, thereby effectively reducing the load of a statistics system.
As an alternative embodiment of the invention, the stored statistical period data comprises a device unique number, a date, a statistical period, a starting value, a final value, a starting value acquisition time and a final value acquisition time.
As an optional implementation mode of the invention, the device reported data comprises a device unique number and a table code value.
In this specific real-time example, the statistics period is hours, the data (second reporting data) reported by the current device corresponds to the collection time and the collection time corresponding to the data (first reporting data) reported by the previous device belongs to the same statistics period, then the currently processed statistics data is stored in the first storage statistics period of the database, the data stored in the first storage statistics period comprises a device unique number, a date, a statistics period, a starting value, a final value, a starting value collection time and a final value collection time, the device reporting data comprises a table code value and a device unique number, the table code data is an electric energy reading displayed by the ammeter at the corresponding collection time, the data reported by the previous device is table code data when the value is equal to the sum of the current electricity consumption from the beginning of use of the ammeter, the electricity consumption is calculated according to the table code data, the device unique number is used for distinguishing the device data reported by different devices, the different devices can be stored in the same storage area in advance, and the different devices can be distinguished by using the same storage area. When the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data belong to the same hour, the second reporting data which are processed currently are stored in a first storage statistics period of a database, the second reporting data and the corresponding acquisition time are stored in a buffer area, when the electricity consumption of the current hour needs to be counted, the electricity consumption is equal to the table code data corresponding to the second reporting data minus the initial table code data, the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data do not belong to the same hour, the data (the second storage statistics period) of the statistics period of the corresponding acquisition time of the second reporting data of the device are newly added in the database, the initial table code is the table code data corresponding to the second acquisition time, and the last table code is the table code data corresponding to the first acquisition time.
As an optional implementation manner of the invention, when the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data belong to the same statistical period, storing the second reporting data into a first storage statistical period of the database, wherein the method comprises the steps of replacing the last value of the first reporting data statistics in the current statistical period by the obtained last value of the second reporting data stored into the first storage statistical period, keeping the initial value unchanged, writing the second reporting data and the corresponding acquisition time into a cache region, and updating the data of the cache region.
For example, when the collection time corresponding to the first reporting data and the collection time corresponding to the second reporting data belong to the same statistical period, the table code value corresponding to the first reporting data (last value stored last time) stored in the first storage period of the database is replaced by the table code value corresponding to the current processing data (second reporting data), and the data in the buffer area is updated by using the first reporting data and the corresponding collection time. Specifically, when the current power consumption is required to be counted, the power consumption is equal to the initial table code data subtracted from the table code data corresponding to the second reporting data.
As an optional implementation manner of the invention, the first reporting data corresponding to the collecting time and the second reporting data corresponding to the collecting time do not belong to the same statistical time period, and the storing of the second reporting data into the second storage statistical time period of the database comprises the steps of using the obtained end value of the first reporting data stored into the first storage statistical time period as the initial value of the second storage statistical time period, using the end value of the second reporting data stored into the second storage statistical time period as the end value of the second storage statistical time period, keeping the initial value and the end value of the first storage statistical time period unchanged, writing the second reporting data and the corresponding collecting time pair into a cache area, and updating the data of the cache area.
In an exemplary embodiment, the first reporting data corresponding to the collecting time and the second reporting data corresponding to the collecting time do not belong to the same statistical period, the data (the second stored statistical period) of the statistical period where the second reporting data corresponding to the collecting time of the device is located is newly added in the database, the initial table code takes the table code data corresponding to the first reporting data, the last table code takes the table code data corresponding to the second reporting data, and the second reporting data and the corresponding collecting time are written into the buffer area, and the data of the buffer area is updated.
As an optional implementation manner of the invention, before the acquiring the report data and the corresponding acquisition time of the device stored in the message queue in advance and forming at least the first report data and the second report data according to the report data and the corresponding acquisition time, the method further comprises the steps of acquiring the first report data and the corresponding acquisition time for the first time from the message queue, forming at least one first storage statistic period in a database, and writing the first acquired first report data and the acquisition time into a cache region.
As an alternative embodiment of the invention, the types of the storage statistics time period comprise a storage statistics hour time period, a storage statistics day time period, a storage statistics week time period, a storage statistics month time period and a storage statistics year time period.
The method comprises the steps of obtaining first reporting data and corresponding collecting time from a message queue, forming at least two of storage statistics hour time interval, storage statistics day time interval, storage statistics week time interval, storage statistics month time interval and storage statistics year time interval types in a database, forming corresponding first storage statistics time interval in different types of storage statistics time intervals respectively, and writing the first obtaining first reporting data and the collecting time into a cache region.
For example, if the currently processed data is the device data (first report data) reported by the device for the first time, the cache data at this time is not present, the start table code is the table code data of the current device data, the current device report data is stored in the cache area, and the current power consumption=the start table code data-the table code data of the current device data=0. The storage statistics time period comprises a storage statistics hour time period, a storage statistics day time period, a storage statistics week time period, a storage statistics month time period and a storage statistics year time period, and when the first data is reported by the device, at least two types of the storage statistics hour time period, the storage statistics day time period, the storage statistics week time period, the storage statistics month time period and the storage statistics year time period are formed in the database according to the first reported data and the corresponding acquisition time.
As an optional implementation mode of the invention, the reported data comprises any one of water consumption metering equipment and water consumption, gas consumption metering equipment and gas consumption and electric quantity metering equipment and electric quantity.
The embodiment of the invention also discloses a real-time data statistics system, as shown in fig. 2, which comprises a message queue module, a real-time statistics service program module, a database module and a cache module, wherein the real-time statistics service program module acquires the device report data and the corresponding acquisition time from the message queue module, performs time period comparison judgment with the cache report data and the corresponding acquisition time, performs data statistics update on a judgment processing result according to a database statistics rule, and updates the latest report data and the corresponding acquisition time in the cache module.
In this embodiment, the message queue module may include, but is not limited to, a statistics group of the kafka message queue, the device data collected by the internet of things device may be uniformly reported to the statistics group of the kafka message queue, the time of the data in the message queue is increased for each device, the data structure of the original data of the device stored in the kafka message queue is shown in fig. 3, the data may be in json format or custom format, at least includes three fields of a device unique number, a data collection time (time scale) and a table code value (table code data), and the buffer module may include, but is not limited to, redis for storing the buffer data, where the data structure of the Redis buffer is shown in fig. 4.
Specifically, the real-time data statistics method can count real-time data of hours, days, months and years, wherein an hour statistics table field is a device unique number, a date, an hour, a start table code, an end table code, a start table code time and an end table code time, wherein the device unique number+date+hour is a unique key as shown in fig. 5A, a day statistics table field is a device unique number, a date, a start table code, an end table code, a start table code time and an end table code time, wherein the device unique number+date is a unique key as shown in fig. 5B, a month statistics table field is a device unique number, a date, a start table code, an end table code, a start table code time and an end table code time, wherein the day in the day is fixed to be 1 number, the device unique number+date is a unique key as shown in fig. 5C, a year statistics table field is a device unique number, a start table code, an end table code time and an end table code time, wherein the device unique number+is a unique key as shown in fig. 5D, and the device unique number and any year can be conveniently queried according to the design.
The embodiment of the invention also discloses a real-time data statistics device, as shown in fig. 6, which comprises a first acquisition module 201 for acquiring the device report data and the corresponding acquisition time which are stored in the message queue in advance, and forming at least first report data and second report data according to the report data and the corresponding acquisition time, a second acquisition module 202 for acquiring the second report data and the corresponding acquisition time from the message queue and acquiring the first report data and the corresponding acquisition time from the cache area, a comparison module 203 for comparing the first report data corresponding acquisition time with the second report data corresponding acquisition time, a first storage module 204 for storing the second report data in a first storage statistics period of a database and writing the second report data and the corresponding acquisition time into the cache area when the first report data corresponding acquisition time and the second report data corresponding acquisition time belong to the same statistics period, and a second storage module 205 for storing the second report data and the second report data not belonging to the same statistics period when the first report data corresponding to the second report data corresponding acquisition time and the second report data corresponding to the same statistics period.
The real-time data statistics device comprises a first acquisition module, a second acquisition module, a comparison module and a first storage module, wherein the first acquisition module is used for acquiring report data and corresponding acquisition time of equipment stored in a message queue in advance, forming at least first report data and second report data according to the report data and the corresponding acquisition time, the second acquisition module is used for acquiring the second report data and the corresponding acquisition time from the message queue and acquiring the first report data and the corresponding acquisition time from a cache area, the comparison module is used for comparing the corresponding acquisition time of the first report data with the corresponding acquisition time of the second report data, the first storage module is used for storing the second report data into a first storage statistics period of a database when the corresponding acquisition time of the first report data and the corresponding acquisition time of the second report data belong to the same statistics period, and writing the second report data and the corresponding acquisition time into the cache area, and the second storage module is used for storing the statistics data into the cache area when the corresponding acquisition time of the first report data and the second report data belong to the same statistics period. The device is a time sequence-based streaming real-time data statistics device, processes a piece of data in real time after the table code data reported by a piece of equipment is obtained, can count the electricity consumption of the current statistics period in real time, and meanwhile, the real-time data statistics task does not need to count all the data in one statistics period, so that the load of a statistics system is effectively reduced.
The present invention also provides an electronic device, as shown in fig. 7, which may include a processor 401 and a memory 402, where the processor 401 and the memory 402 may be connected by a bus or other means, and in fig. 7, the connection is exemplified by a bus.
The processor 401 may be a central processing unit (Central Processing Unit, CPU). The Processor 401 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory 402, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the real-time data statistics method in the embodiments of the present invention. The processor 401 executes various functional applications of the processor and data processing, i.e. implements the real-time data statistics method in the above-described method embodiments, by running non-transitory software programs, instructions and modules stored in the memory 402.
The memory 402 may include a storage program area that may store an operating system, application programs required for at least one function, and a storage data area that may store data created by the processor 401, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, such remote memory being connectable to processor 401 through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 402, which when executed by the processor 401, performs the real-time data statistics method in the embodiment shown in fig. 1.
The specific details of the electronic device may be understood correspondingly with respect to the corresponding related descriptions and effects in the embodiment shown in fig. 1, which are not repeated herein.
It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. The storage medium may be a magnetic disk, an optical disc, a Read-Only Memory (ROM), a random access Memory (RandomAccessMemory, RAM), a Flash Memory (Flash Memory), a hard disk (HARD DISK DRIVE, abbreviated as HDD), a Solid state disk (Solid-state disk STATE DRIVE, SSD), or the like, and the storage medium may further include a combination of the above types of memories.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations are within the scope of the invention as defined by the appended claims.

Claims (7)

1.一种实时数据统计方法,其特征在于,包括如下步骤:1. A real-time data statistics method, characterized in that it comprises the following steps: 获取预先存储到消息队列中设备上报数据和对应的采集时间,并根据上报数据和对应的采集时间至少形成第一上报数据、第二上报数据;Acquire the device reported data and the corresponding collection time pre-stored in the message queue, and form at least first reported data and second reported data according to the reported data and the corresponding collection time; 从所述消息队列中获取所述第二上报数据及对应采集时间并从缓存区获取第一上报数据及对应采集时间;Acquire the second reported data and the corresponding collection time from the message queue and acquire the first reported data and the corresponding collection time from the buffer area; 将所述第一上报数据对应采集时间与所述第二上报数据对应采集时间进行比对;Comparing the collection time corresponding to the first reported data with the collection time corresponding to the second reported data; 当所述第一上报数据对应采集时间与所述第二上报数据对应采集时间属于同一统计时段,将所述第二上报数据存储到数据库第一存储统计时段中,并将第二上报数据及对应采集时间写入缓存区;When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, the second reported data is stored in the first statistical storage period of the database, and the second reported data and the corresponding collection time are written into the cache area; 当所述第一上报数据对应采集时间与所述第二上报数据对应采集时间不属于同一统计时段,将所述第二上报数据存储到数据库第二存储统计时段中,并将第二上报数据及对应采集时间写入缓存区;When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the second reported data in the second storage statistical period of the database, and writing the second reported data and the corresponding collection time into the cache area; 所述存储统计时段数据包括:设备唯一号、日期、统计时段、起始值、末值、起始值采集时间、末值采集时间;The stored statistical period data includes: device unique number, date, statistical period, starting value, ending value, starting value collection time, and ending value collection time; 当所述第一上报数据对应采集时间与所述第二上报数据对应采集时间属于同一统计时段,将所述第二上报数据存储到所述数据库第一存储统计时段中,包括:When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, storing the second reported data in the first storage statistical period of the database includes: 利用获取到的第二上报数据存储至第一存储统计时段的末值替换当前统计时段内第一上报数据统计的末值,起始值保留不变,并将第二上报数据和对应采集时间写入缓存区,对缓存区数据更新;The last value of the first reported data in the current statistical period is replaced by the last value of the second reported data stored in the first statistical storage period, the starting value remains unchanged, and the second reported data and the corresponding collection time are written into the cache area to update the data in the cache area; 当所述第一上报数据对应采集时间与所述第二上报数据对应采集时间不属于同一统计时段,将所述第二上报数据存储到所述数据库第二存储统计时段中,包括:When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the second reported data in the second storage statistical period of the database includes: 利用获取到的第一上报数据存储至第一存储统计时段的末值作为第二存储统计时段的起始值,第二上报数据存储至第二存储统计时段的末值作为第二存储统计时段的末值,第一存储统计时段的起始值、末值保持不变,并将第二上报数据和对应采集时间对写入缓存区,对缓存区数据更新。The first reported data obtained and stored in the end value of the first storage statistical period is used as the starting value of the second storage statistical period, and the second reported data obtained and stored in the end value of the second storage statistical period is used as the end value of the second storage statistical period. The starting value and the end value of the first storage statistical period remain unchanged, and the second reported data and the corresponding collection time are written into the cache area to update the cache area data. 2.根据权利要求1所述的方法,其特征在于,所述获取预先存储到消息队列中设备上报数据和对应的采集时间,并根据上报数据和对应的采集时间至少形成第一上报数据、第二上报数据之前,所述方法还包括:2. The method according to claim 1 is characterized in that before obtaining the device reported data and the corresponding collection time pre-stored in the message queue and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method further comprises: 从所述消息队列中第一次获取所述第一上报数据和对应采集时间,在数据库至少形成一个第一存储统计时段,并将第一次获取所述第一上报数据和采集时间写入缓存区。The first reported data and the corresponding collection time are obtained from the message queue for the first time, at least one first storage statistics period is formed in the database, and the first reported data and the collection time obtained for the first time are written into the cache area. 3.根据权利要求1所述的方法,其特征在于,存储统计时段的类型包括:存储统计小时时段、存储统计日时段、存储统计周时段、存储统计月时段、存储统计年时段。3. The method according to claim 1 is characterized in that the types of storage statistics time periods include: storage statistics hourly time period, storage statistics daily time period, storage statistics weekly time period, storage statistics monthly time period, and storage statistics annual time period. 4.据权利要求3所述的方法,其特征在于,所述获取预先存储到消息队列中设备上报数据和对应的采集时间,并根据上报数据和对应的采集时间至少形成第一上报数据、第二上报数据之前,所述方法还包括:4. The method according to claim 3 is characterized in that before obtaining the device reported data and the corresponding collection time pre-stored in the message queue and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method further comprises: 从所述消息队列中第一次获取所述第一上报数据和对应采集时间,在数据库形成存储统计小时时段、存储统计日时段、存储统计周时段、存储统计月时段、存储统计年时段类型中至少两种,并分别在不同类型存储统计时段形成对应第一存储统计时段,并将第一次获取所述第一上报数据和采集时间写入缓存区。The first reported data and the corresponding collection time are obtained from the message queue for the first time, and at least two types of storage statistics hourly period, storage statistics daily period, storage statistics weekly period, storage statistics monthly period, and storage statistics annual period are formed in the database, and corresponding first storage statistics periods are formed in different types of storage statistics period, and the first reported data and the collection time obtained for the first time are written into the cache area. 5.根据权利要求1所述的方法,其特征在于,所述设备上报数据包括:设备唯一号、表码值。5. The method according to claim 1 is characterized in that the data reported by the device includes: a unique device number and a table code value. 6.根据权利要求1所述的方法,其特征在于,所述上报数据包括:水量计量设备及用水量、气量计量设备及用气量和电量计量设备及用电量中的任意一种。6. The method according to claim 1 is characterized in that the reported data includes: any one of: water metering equipment and water consumption, gas metering equipment and gas consumption, and electricity metering equipment and electricity consumption. 7.一种实时数据统计系统,执行如权利要求1-6任一项所述的实时数据统计方法,所述系统包括:消息队列模块、实时统计服务程序模块、数据库模块和缓存模块,其中实时统计服务程序模块从所述消息队列模块获取设备上报数据及对应采集时间,并与缓存中上报数据及对应采集时间进行时段对比判断,将判断处理结果按数据库统计规则进行数据统计更新,并将最新上报数据及对应采集时间在缓存模块中更新。7. A real-time data statistics system, executing the real-time data statistics method according to any one of claims 1 to 6, the system comprising: a message queue module, a real-time statistics service program module, a database module and a cache module, wherein the real-time statistics service program module obtains the device reported data and the corresponding collection time from the message queue module, and compares and judges the time period with the reported data and the corresponding collection time in the cache, updates the data statistics according to the database statistics rules based on the judgment processing results, and updates the latest reported data and the corresponding collection time in the cache module.
CN202210340719.4A 2022-03-31 2022-03-31 Real-time data statistics method and system Active CN114722059B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210340719.4A CN114722059B (en) 2022-03-31 2022-03-31 Real-time data statistics method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210340719.4A CN114722059B (en) 2022-03-31 2022-03-31 Real-time data statistics method and system

Publications (2)

Publication Number Publication Date
CN114722059A CN114722059A (en) 2022-07-08
CN114722059B true CN114722059B (en) 2024-11-29

Family

ID=82242142

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210340719.4A Active CN114722059B (en) 2022-03-31 2022-03-31 Real-time data statistics method and system

Country Status (1)

Country Link
CN (1) CN114722059B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115827721B (en) * 2022-10-14 2025-09-12 神策网络科技(北京)有限公司 Real-time data statistical method, device, electronic device and storage medium

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105868071A (en) * 2016-03-23 2016-08-17 乐视网信息技术(北京)股份有限公司 Monitoring data processing method and device

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8984033B2 (en) * 2005-09-23 2015-03-17 Chicago Mercantile Exchange, Inc. Non-indexed in-memory data storage and retrieval
US7526409B2 (en) * 2005-10-07 2009-04-28 Oracle International Corporation Automatic performance statistical comparison between two periods
CN105306243A (en) * 2014-07-22 2016-02-03 中兴通讯股份有限公司 Stale data collection method, device and system
US11153383B2 (en) * 2016-03-08 2021-10-19 Tanium Inc. Distributed data analysis for streaming data sources
CN110427399A (en) * 2019-06-18 2019-11-08 深圳壹账通智能科技有限公司 Real-time data acquisition method, system, device and storage medium
CN111752916B (en) * 2019-12-30 2024-04-16 北京沃东天骏信息技术有限公司 Data acquisition method and device, computer readable storage medium and electronic equipment
CN112910086B (en) * 2021-01-18 2024-05-31 国网山东省电力公司青岛供电公司 A method and system for verifying data of intelligent substation
CN113127472B (en) * 2021-05-11 2024-03-19 广州宸祺出行科技有限公司 Method and system for real-time deduplication statistics of number of drivers with large reporting amount

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105868071A (en) * 2016-03-23 2016-08-17 乐视网信息技术(北京)股份有限公司 Monitoring data processing method and device

Also Published As

Publication number Publication date
CN114722059A (en) 2022-07-08

Similar Documents

Publication Publication Date Title
US11685281B2 (en) Demand flexibility optimizing scheduler for EV charging and controlling appliances
CN104112345B (en) data monitoring method of remote wireless meter reading system
WO2024031447A1 (en) Carbon data management method, apparatus, electronic device, store medium and computer program product
CN114722059B (en) Real-time data statistics method and system
CN116703135B (en) Power line construction planning analysis and evaluation method
CN107454143A (en) A wide-area measurement mode intelligent inspection method for the measurement performance of digital electric energy meters
CN103440531A (en) Wind power plant short-term wind power prediction system in view of operating state of wind power plant draught fan
CN115526275B (en) Remote multi-source data monitoring and analyzing method and system for electricity utilization safety
CN109360415A (en) A method for identifying abnormal data of road traffic flow
CN111859292B (en) Water supply leakage monitoring method for night water-using active district
CN119477604A (en) A power consumption information collection system and method based on dual-mode communication
CN114283503A (en) The alarm control method for the energy consumption of the cigarette equipment
CN108491432B (en) Electric power system accumulated quantity storage and extraction method based on message abstract, electronic equipment and storage medium
CN107590541B (en) On-site processing method for low success rate defect of load data acquisition
CN103344824B (en) The electric flux integration method measuring based on markers
CN112668916A (en) Equipment index value calculation method and device and electronic equipment
CN107480832A (en) Short-term power predicating method, device and electronic equipment
CN106018931B (en) voltage qualification rate monitoring method and system
CN112419698B (en) Electric energy meter data transmission method, system and device based on distribution line message specification
CN116150132A (en) Enterprise internet of things data processing method, terminal equipment and storage medium
CN111666537B (en) Energy consumption statistical method based on table bottom value
CN115660314A (en) Shadow shielding diagnosis method and device, electronic equipment and storage medium
CN114547018B (en) Method and system for automatically cleaning waste points of SIS real-time database
CN114037113A (en) Intelligent water meter control method, device and system, storage medium and cloud computing center
CN112365703A (en) Method and device for collecting daily freezing data of electric energy meter

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant