Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the defects that the electric quantity statistics method has hysteresis in the prior art, and the electric quantity statistics task needs to process a large amount of data in one statistics period, so that the system load is high, thereby providing the real-time data statistics method and the system.
According to a first aspect, the embodiment of the invention discloses a real-time data statistics method, which comprises the steps of obtaining device report data and corresponding collection time which are stored in a message queue in advance, forming at least first report data and second report data according to the report data and the corresponding collection time, obtaining the second report data and the corresponding collection time from the message queue, obtaining the first report data and the corresponding collection time from a cache area, comparing the corresponding collection time of the first report data with the corresponding collection time of the second report data, storing the second report data into a first storage statistics period of a database when the corresponding collection time of the first report data and the corresponding collection time of the second report data belong to the same statistics period, and writing the second report data into the cache area when the corresponding collection time of the first report data and the corresponding collection time of the second report data do not belong to the same statistics period, and storing the device data into the second storage statistics period and writing the corresponding collection time of the second report data into the cache area.
Optionally, the stored statistical period data comprises a device unique number, a date, a statistical period, a starting value, a final value, a starting value acquisition time and a final value acquisition time.
Optionally, when the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data belong to the same statistical period, storing the second reporting data into the first storage statistical period of the database, wherein the method comprises the steps of replacing the last value of the first reporting data statistics in the current statistical period by the last value of the acquired second reporting data stored into the first storage statistical period, keeping the initial value unchanged, writing the second reporting data and the corresponding acquisition time into a cache region, and updating the data of the cache region.
Optionally, when the corresponding collection time of the first reporting data and the corresponding collection time of the second reporting data do not belong to the same statistical period, storing the second reporting data into a second storage statistical period of the database, wherein the method comprises the steps of using the obtained end value of the first reporting data stored into the first storage statistical period as a starting value of the second storage statistical period, storing the end value of the second reporting data stored into the second storage statistical period as an end value of the second storage statistical period, keeping the starting value and the end value of the first storage statistical period unchanged, writing the second reporting data and the corresponding collection time pair into a cache region, and updating the data of the cache region.
Optionally, before acquiring the device report data and the corresponding acquisition time stored in the message queue in advance and forming at least first report data and second report data according to the report data and the corresponding acquisition time, the method further comprises the steps of acquiring the first report data and the corresponding acquisition time from the message queue for the first time, forming at least one first storage statistic period in a database, and writing the first report data and the acquisition time acquired for the first time into a cache region.
Optionally, the types of the storage statistics time period comprise storage statistics hour time period, storage statistics day time period, storage statistics week time period, storage statistics month time period and storage statistics year time period.
Optionally, before the acquiring the device report data and the corresponding acquisition time stored in the message queue in advance and forming at least first report data and second report data according to the report data and the corresponding acquisition time, the method further comprises the steps of firstly acquiring the first report data and the corresponding acquisition time from the message queue, forming at least two of a storage statistics hour period, a storage statistics day period, a storage statistics week period, a storage statistics month period and a storage statistics year period type in a database, forming corresponding first storage statistics periods in different types of storage statistics periods respectively, and writing the first report data acquired for the first time and the acquisition time into a cache region.
Optionally, the device reported data comprises a device unique number and a table code value.
Optionally, the reported data comprises any one of water consumption metering equipment, water consumption, gas consumption metering equipment, gas consumption metering equipment and electricity consumption.
According to a second aspect, the embodiment of the invention also discloses a real-time data statistics system, which executes the real-time data statistics method according to the first aspect or any optional implementation manner of the first aspect, wherein the system comprises a message queue module, a real-time statistics service program module, a database module and a cache module, wherein the real-time statistics service program module acquires the device report data and the corresponding acquisition time from the message queue module, performs time period comparison judgment on the device report data and the corresponding acquisition time, performs data statistics update on the judgment processing result according to the database statistics rule, and updates the latest report data and the corresponding acquisition time in the cache module.
The technical scheme of the invention has the following advantages:
The real-time data statistics method comprises the steps of obtaining device report data and corresponding collection time which are stored in a message queue in advance, forming at least first report data and second report data according to the report data and the corresponding collection time, obtaining the second report data and the corresponding collection time from the message queue, obtaining the first report data and the corresponding collection time from a cache area, comparing the corresponding collection time of the first report data with the corresponding collection time of the second report data, storing the second report data into a first storage statistics period of a database when the corresponding collection time of the first report data and the corresponding collection time of the second report data belong to the same statistics period, and writing the second report data and the corresponding collection time into the cache area, and storing the device data into the second storage statistics period of the database when the corresponding collection time of the first report data and the corresponding collection time of the second report data do not belong to the same statistics period. The method is a time sequence-based streaming real-time data statistics method, and the data reported by a piece of equipment is obtained to process a piece of data in real time, so that the electricity consumption of the current statistics period can be counted in real time, and meanwhile, the real-time data statistics task does not need to count all the data in one statistics period, so that the load of a statistics system is effectively reduced.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected, mechanically connected, electrically connected, directly connected, indirectly connected via an intermediate medium, and in communication with each other between two elements, and wirelessly connected, or wired. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
The embodiment of the application discloses a real-time data statistics method which can be applied to Internet of things equipment, wherein real-time data comprises any one of water consumption, gas consumption and electricity consumption, and can also be expanded to statistics of other cumulative amounts. In the embodiment of the present application, the real-time data statistics method described in the embodiment will be described by taking the electricity consumption for counting hours as an example. As shown in fig. 1, the method comprises the steps of:
Step 101, acquiring the reporting data and the corresponding acquisition time of the equipment pre-stored in the message queue, and forming at least first reporting data and second reporting data according to the reporting data and the corresponding acquisition time.
The device report data and the corresponding first acquisition time are stored in a message queue, the second report data is the data reported by the current device, and the first report data is the data reported by the device.
Step 102, obtaining the second report data and the corresponding collection time from the message queue, and obtaining the first report data and the corresponding collection time from the buffer.
Illustratively, the buffer area stores the data of the last report (the first report data) and the corresponding acquisition time, and specifically, the last processed data stored in the buffer area may also be referred to as buffer data.
And step 103, comparing the corresponding acquisition time of the first reported data with the corresponding acquisition time of the second reported data.
Step 104, when the corresponding collection time of the first reporting data and the corresponding collection time of the second reporting data belong to the same statistical period, storing the second reporting data and the corresponding collection time into a first storage statistical period of a database, and writing the second reporting data and the corresponding collection time into a cache region.
In an exemplary embodiment, the statistics period is a period that can be preset according to the statistics requirement, the value of the statistics period is not limited in this embodiment, the corresponding collection time of the data (second reporting data) reported by the current device and the corresponding collection time of the data (first reporting data) reported by the previous device belong to the same statistics period, then the second reporting data is stored in the first storage statistics period of the database, and the data (second reporting data) stored by the current device and the corresponding collection time are stored in the cache region.
Step 105, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the device data into a second storage statistical period of a database, and writing the second reported data and the corresponding collection time into a buffer area.
For example, if the collection time corresponding to the data (second report data) reported by the current device and the collection time corresponding to the data (first report data) reported by the previous device do not belong to the same statistical period, the second report data is stored in a second storage statistical period of the database, and the data (second report data) stored by the current device and the corresponding collection time are stored in the cache region.
The real-time data statistics method provided by the invention is a time sequence-based streaming real-time data statistics method, and the data reported by a piece of equipment is obtained to process a piece of data in real time, so that the electricity consumption of the current statistics period can be counted in real time, and meanwhile, the real-time data statistics task does not need to count all the data in one statistics period, thereby effectively reducing the load of a statistics system.
As an alternative embodiment of the invention, the stored statistical period data comprises a device unique number, a date, a statistical period, a starting value, a final value, a starting value acquisition time and a final value acquisition time.
As an optional implementation mode of the invention, the device reported data comprises a device unique number and a table code value.
In this specific real-time example, the statistics period is hours, the data (second reporting data) reported by the current device corresponds to the collection time and the collection time corresponding to the data (first reporting data) reported by the previous device belongs to the same statistics period, then the currently processed statistics data is stored in the first storage statistics period of the database, the data stored in the first storage statistics period comprises a device unique number, a date, a statistics period, a starting value, a final value, a starting value collection time and a final value collection time, the device reporting data comprises a table code value and a device unique number, the table code data is an electric energy reading displayed by the ammeter at the corresponding collection time, the data reported by the previous device is table code data when the value is equal to the sum of the current electricity consumption from the beginning of use of the ammeter, the electricity consumption is calculated according to the table code data, the device unique number is used for distinguishing the device data reported by different devices, the different devices can be stored in the same storage area in advance, and the different devices can be distinguished by using the same storage area. When the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data belong to the same hour, the second reporting data which are processed currently are stored in a first storage statistics period of a database, the second reporting data and the corresponding acquisition time are stored in a buffer area, when the electricity consumption of the current hour needs to be counted, the electricity consumption is equal to the table code data corresponding to the second reporting data minus the initial table code data, the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data do not belong to the same hour, the data (the second storage statistics period) of the statistics period of the corresponding acquisition time of the second reporting data of the device are newly added in the database, the initial table code is the table code data corresponding to the second acquisition time, and the last table code is the table code data corresponding to the first acquisition time.
As an optional implementation manner of the invention, when the corresponding acquisition time of the first reporting data and the corresponding acquisition time of the second reporting data belong to the same statistical period, storing the second reporting data into a first storage statistical period of the database, wherein the method comprises the steps of replacing the last value of the first reporting data statistics in the current statistical period by the obtained last value of the second reporting data stored into the first storage statistical period, keeping the initial value unchanged, writing the second reporting data and the corresponding acquisition time into a cache region, and updating the data of the cache region.
For example, when the collection time corresponding to the first reporting data and the collection time corresponding to the second reporting data belong to the same statistical period, the table code value corresponding to the first reporting data (last value stored last time) stored in the first storage period of the database is replaced by the table code value corresponding to the current processing data (second reporting data), and the data in the buffer area is updated by using the first reporting data and the corresponding collection time. Specifically, when the current power consumption is required to be counted, the power consumption is equal to the initial table code data subtracted from the table code data corresponding to the second reporting data.
As an optional implementation manner of the invention, the first reporting data corresponding to the collecting time and the second reporting data corresponding to the collecting time do not belong to the same statistical time period, and the storing of the second reporting data into the second storage statistical time period of the database comprises the steps of using the obtained end value of the first reporting data stored into the first storage statistical time period as the initial value of the second storage statistical time period, using the end value of the second reporting data stored into the second storage statistical time period as the end value of the second storage statistical time period, keeping the initial value and the end value of the first storage statistical time period unchanged, writing the second reporting data and the corresponding collecting time pair into a cache area, and updating the data of the cache area.
In an exemplary embodiment, the first reporting data corresponding to the collecting time and the second reporting data corresponding to the collecting time do not belong to the same statistical period, the data (the second stored statistical period) of the statistical period where the second reporting data corresponding to the collecting time of the device is located is newly added in the database, the initial table code takes the table code data corresponding to the first reporting data, the last table code takes the table code data corresponding to the second reporting data, and the second reporting data and the corresponding collecting time are written into the buffer area, and the data of the buffer area is updated.
As an optional implementation manner of the invention, before the acquiring the report data and the corresponding acquisition time of the device stored in the message queue in advance and forming at least the first report data and the second report data according to the report data and the corresponding acquisition time, the method further comprises the steps of acquiring the first report data and the corresponding acquisition time for the first time from the message queue, forming at least one first storage statistic period in a database, and writing the first acquired first report data and the acquisition time into a cache region.
As an alternative embodiment of the invention, the types of the storage statistics time period comprise a storage statistics hour time period, a storage statistics day time period, a storage statistics week time period, a storage statistics month time period and a storage statistics year time period.
The method comprises the steps of obtaining first reporting data and corresponding collecting time from a message queue, forming at least two of storage statistics hour time interval, storage statistics day time interval, storage statistics week time interval, storage statistics month time interval and storage statistics year time interval types in a database, forming corresponding first storage statistics time interval in different types of storage statistics time intervals respectively, and writing the first obtaining first reporting data and the collecting time into a cache region.
For example, if the currently processed data is the device data (first report data) reported by the device for the first time, the cache data at this time is not present, the start table code is the table code data of the current device data, the current device report data is stored in the cache area, and the current power consumption=the start table code data-the table code data of the current device data=0. The storage statistics time period comprises a storage statistics hour time period, a storage statistics day time period, a storage statistics week time period, a storage statistics month time period and a storage statistics year time period, and when the first data is reported by the device, at least two types of the storage statistics hour time period, the storage statistics day time period, the storage statistics week time period, the storage statistics month time period and the storage statistics year time period are formed in the database according to the first reported data and the corresponding acquisition time.
As an optional implementation mode of the invention, the reported data comprises any one of water consumption metering equipment and water consumption, gas consumption metering equipment and gas consumption and electric quantity metering equipment and electric quantity.
The embodiment of the invention also discloses a real-time data statistics system, as shown in fig. 2, which comprises a message queue module, a real-time statistics service program module, a database module and a cache module, wherein the real-time statistics service program module acquires the device report data and the corresponding acquisition time from the message queue module, performs time period comparison judgment with the cache report data and the corresponding acquisition time, performs data statistics update on a judgment processing result according to a database statistics rule, and updates the latest report data and the corresponding acquisition time in the cache module.
In this embodiment, the message queue module may include, but is not limited to, a statistics group of the kafka message queue, the device data collected by the internet of things device may be uniformly reported to the statistics group of the kafka message queue, the time of the data in the message queue is increased for each device, the data structure of the original data of the device stored in the kafka message queue is shown in fig. 3, the data may be in json format or custom format, at least includes three fields of a device unique number, a data collection time (time scale) and a table code value (table code data), and the buffer module may include, but is not limited to, redis for storing the buffer data, where the data structure of the Redis buffer is shown in fig. 4.
Specifically, the real-time data statistics method can count real-time data of hours, days, months and years, wherein an hour statistics table field is a device unique number, a date, an hour, a start table code, an end table code, a start table code time and an end table code time, wherein the device unique number+date+hour is a unique key as shown in fig. 5A, a day statistics table field is a device unique number, a date, a start table code, an end table code, a start table code time and an end table code time, wherein the device unique number+date is a unique key as shown in fig. 5B, a month statistics table field is a device unique number, a date, a start table code, an end table code, a start table code time and an end table code time, wherein the day in the day is fixed to be 1 number, the device unique number+date is a unique key as shown in fig. 5C, a year statistics table field is a device unique number, a start table code, an end table code time and an end table code time, wherein the device unique number+is a unique key as shown in fig. 5D, and the device unique number and any year can be conveniently queried according to the design.
The embodiment of the invention also discloses a real-time data statistics device, as shown in fig. 6, which comprises a first acquisition module 201 for acquiring the device report data and the corresponding acquisition time which are stored in the message queue in advance, and forming at least first report data and second report data according to the report data and the corresponding acquisition time, a second acquisition module 202 for acquiring the second report data and the corresponding acquisition time from the message queue and acquiring the first report data and the corresponding acquisition time from the cache area, a comparison module 203 for comparing the first report data corresponding acquisition time with the second report data corresponding acquisition time, a first storage module 204 for storing the second report data in a first storage statistics period of a database and writing the second report data and the corresponding acquisition time into the cache area when the first report data corresponding acquisition time and the second report data corresponding acquisition time belong to the same statistics period, and a second storage module 205 for storing the second report data and the second report data not belonging to the same statistics period when the first report data corresponding to the second report data corresponding acquisition time and the second report data corresponding to the same statistics period.
The real-time data statistics device comprises a first acquisition module, a second acquisition module, a comparison module and a first storage module, wherein the first acquisition module is used for acquiring report data and corresponding acquisition time of equipment stored in a message queue in advance, forming at least first report data and second report data according to the report data and the corresponding acquisition time, the second acquisition module is used for acquiring the second report data and the corresponding acquisition time from the message queue and acquiring the first report data and the corresponding acquisition time from a cache area, the comparison module is used for comparing the corresponding acquisition time of the first report data with the corresponding acquisition time of the second report data, the first storage module is used for storing the second report data into a first storage statistics period of a database when the corresponding acquisition time of the first report data and the corresponding acquisition time of the second report data belong to the same statistics period, and writing the second report data and the corresponding acquisition time into the cache area, and the second storage module is used for storing the statistics data into the cache area when the corresponding acquisition time of the first report data and the second report data belong to the same statistics period. The device is a time sequence-based streaming real-time data statistics device, processes a piece of data in real time after the table code data reported by a piece of equipment is obtained, can count the electricity consumption of the current statistics period in real time, and meanwhile, the real-time data statistics task does not need to count all the data in one statistics period, so that the load of a statistics system is effectively reduced.
The present invention also provides an electronic device, as shown in fig. 7, which may include a processor 401 and a memory 402, where the processor 401 and the memory 402 may be connected by a bus or other means, and in fig. 7, the connection is exemplified by a bus.
The processor 401 may be a central processing unit (Central Processing Unit, CPU). The Processor 401 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory 402, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the real-time data statistics method in the embodiments of the present invention. The processor 401 executes various functional applications of the processor and data processing, i.e. implements the real-time data statistics method in the above-described method embodiments, by running non-transitory software programs, instructions and modules stored in the memory 402.
The memory 402 may include a storage program area that may store an operating system, application programs required for at least one function, and a storage data area that may store data created by the processor 401, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, such remote memory being connectable to processor 401 through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 402, which when executed by the processor 401, performs the real-time data statistics method in the embodiment shown in fig. 1.
The specific details of the electronic device may be understood correspondingly with respect to the corresponding related descriptions and effects in the embodiment shown in fig. 1, which are not repeated herein.
It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. The storage medium may be a magnetic disk, an optical disc, a Read-Only Memory (ROM), a random access Memory (RandomAccessMemory, RAM), a Flash Memory (Flash Memory), a hard disk (HARD DISK DRIVE, abbreviated as HDD), a Solid state disk (Solid-state disk STATE DRIVE, SSD), or the like, and the storage medium may further include a combination of the above types of memories.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations are within the scope of the invention as defined by the appended claims.