CN116582773B - Data collection method, system and computer readable storage medium - Google Patents
Data collection method, system and computer readable storage mediumInfo
- Publication number
- CN116582773B CN116582773B CN202310409554.6A CN202310409554A CN116582773B CN 116582773 B CN116582773 B CN 116582773B CN 202310409554 A CN202310409554 A CN 202310409554A CN 116582773 B CN116582773 B CN 116582773B
- Authority
- CN
- China
- Prior art keywords
- acquisition
- success rate
- period
- data
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/625—Queue scheduling characterised by scheduling criteria for service slots or service orders
- H04L47/6275—Queue scheduling characterised by scheduling criteria for service slots or service orders based on priority
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
本发明涉及通信技术领域，公开一种数据采集方法、系统与计算机可读存储介质。所述数据采集方法包括：针对当前采集时段内的采集任务中的每个采集设备，根据所述每个采集设备的每个数据项的历史采集成功率，确定所述每个数据项的优先级；然后，根据所述每个采集设备的平均历史采集成功率，确定最高优先级的采集设备；最后，对所述最高优先级的采集设备进行关于最高优先级的数据项的采集。由此，本发明根据同一设备的不同数据项的历史采集成功率进行优先级排布，然后根据不同设备的平均历史采集成功率进行优先级排布，并对最高优先级的设备的最高优先级的数据项进行优先采集，从而在面对较多设备或高频采集需求场景时能够极大地提高采集效率。
The present invention relates to the field of communication technology, and discloses a data acquisition method, system, and computer-readable storage medium. The data acquisition method comprises: for each acquisition device in the acquisition task within the current acquisition period, determining the priority of each data item of each acquisition device according to the historical acquisition success rate of each data item; then, determining the highest priority acquisition device according to the average historical acquisition success rate of each acquisition device; finally, collecting the highest priority data item for the highest priority acquisition device. Thus, the present invention arranges priorities according to the historical acquisition success rate of different data items of the same device, and then arranges priorities according to the average historical acquisition success rate of different devices, and preferentially collects the highest priority data items of the highest priority device, thereby greatly improving the acquisition efficiency when facing a large number of devices or high-frequency acquisition demand scenarios.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data acquisition method, system, and computer readable storage medium.
Background
At present, the main communication form between the acquisition type terminal and the acquisition equipment in the low-voltage transformer area is carrier communication. And the application in the terminal is communicated with the local communication module to realize a data acquisition function. And the terminal calculates a specific acquisition plan according to the archives and the acquisition tasks. The standard specifications of most terminals divide the services into different application implementations according to the functions.
Typically, the collection equipment under one distribution or low voltage area is stationary for a period of time. When the collection type business is applied to calculating a specific collection plan, collection data items of collection equipment are arranged according to the sequence of task sequence numbers according to the execution frequency and the task operation time period in the collection task.
The acquisition failure phenomenon can be caused by reasons of communication quality, software version difference and the like of the acquisition equipment under the transformer area, and the phenomenon is more obvious when the number of the acquisition equipment under the transformer area is large or high-frequency acquisition requirements exist. When the collection service application encounters the condition that the collection device is not answered in concurrent reading, the collection device needs to wait for a maximum timeout time, so that the collection efficiency is reduced due to more waiting time, and the collection task cannot be completed in a collection period. The acquisition success rate of different acquisition devices cannot be distinguished by the acquisition method, and the acquisition devices with low acquisition success rate cannot be screened by an automatic means.
Disclosure of Invention
The invention aims to provide a data acquisition method, a system and a computer readable storage medium, which are used for carrying out priority arrangement according to the historical acquisition success rates of different data items of the same equipment, then carrying out priority arrangement according to the average historical acquisition success rates of different equipment, and carrying out priority acquisition on the data item with the highest priority of the equipment with the highest priority, so that the acquisition efficiency can be greatly improved when more equipment or high-frequency acquisition demand scenes are faced.
In order to achieve the above object, a first aspect of the present invention provides a data acquisition method, including, for each of a plurality of acquisition devices in an acquisition task within a current acquisition period, prioritizing each data item of the each acquisition device according to a historical acquisition success rate of the each data item, determining an acquisition device of a highest priority according to an average historical acquisition success rate of the each acquisition device, and performing acquisition on the data item of the highest priority with respect to the acquisition device of the highest priority.
Preferably, the data acquisition method further comprises the steps of obtaining updated data items of the acquisition equipment with the highest priority, wherein the updated data items do not comprise the data items with the highest priority, determining the acquisition equipment with the highest priority again according to the updated average historical acquisition success rate of the acquisition equipment with the highest priority and the average historical acquisition success rate of other acquisition equipment in the acquisition task in the current acquisition period, and acquiring the data items with the highest priority of the newly determined acquisition equipment with the highest priority.
Preferably, the historical acquisition success rate of each data item of each acquisition device is determined by determining the historical acquisition success rate X kj of the jth data item of the kth acquisition device based on the response time C kji of the jth data item of the kth acquisition device in the last acquisition period and the following equation,
Wherein b is the acquisition times corresponding to the jth data item of the kth acquisition device in the last acquisition period, and d is the success rate calculation interval.
Preferably, the average historical acquisition success rate for each acquisition device is determined by determining the average historical acquisition success rate X k for the n data items for the kth device based on the first historical acquisition success rate X kj for the jth data item for the kth device and the following,
Preferably, the data acquisition method further comprises the steps of determining a time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers, wherein the plurality of acquisition time periods are a plurality of continuous acquisition time periods before a current acquisition time period, determining a maximum time period acquisition success rate under the condition that the difference of the time period acquisition success rates in any two of the plurality of acquisition time periods is smaller than or equal to a preset value, and determining the preset maximum concurrency number corresponding to the maximum time period acquisition success rate as a target maximum concurrency number of communication between an acquisition terminal and the plurality of acquisition devices.
Preferably, determining the time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers comprises determining the ratio of the number of data items successfully acquired by all acquisition devices in each of the plurality of preset maximum concurrency numbers to the total number of data items of all acquisition devices as the time period acquisition success rate in each of the plurality of preset maximum concurrency numbers.
Preferably, the data acquisition method further comprises the steps of judging whether the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, determining the shortest completion time according to the response time of each data item of each acquisition device in the acquisition task in the last acquisition period under the condition that the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, and stopping the statistics operation of the historical acquisition success rate under the condition that the total time length of the acquisition tasks in the current acquisition period is larger than the preset multiple of the shortest completion time.
Through the technical scheme, the priority of each data item is creatively determined according to the historical acquisition success rate of each data item of each acquisition device aiming at each acquisition device in a plurality of acquisition devices in an acquisition task in the current acquisition period; then, according to the average historical acquisition success rate of each acquisition device, determining the acquisition device with the highest priority; finally, the acquisition device with the highest priority is subjected to the acquisition of the data item with the highest priority. Therefore, the method and the device for the data acquisition of the invention are used for carrying out priority arrangement according to the historical acquisition success rates of different data items of the same equipment, then carrying out priority arrangement according to the average historical acquisition success rates of different equipment, and carrying out priority acquisition on the data item with the highest priority of the equipment with the highest priority, thereby greatly improving the acquisition efficiency when facing more equipment or high-frequency acquisition demand scenes.
The second aspect of the invention provides a data acquisition system which comprises a first determining device, a second determining device and an acquisition terminal, wherein the first determining device is used for determining the priority of each data item according to the historical acquisition success rate of each data item of each acquisition device aiming at each acquisition device in a plurality of acquisition devices in an acquisition task in a current acquisition period, the second determining device is used for determining the acquisition device with the highest priority according to the average historical acquisition success rate of each acquisition device, and the acquisition terminal is used for acquiring the data item with the highest priority of the acquisition device with the highest priority.
Preferably, the data acquisition system further comprises deleting means for deleting the highest priority data item of the highest priority acquisition device from the acquisition tasks to obtain updated acquisition tasks within the current acquisition period, and third determining means for redetermining the highest priority acquisition device according to an average historical acquisition success rate of each of the updated acquisition tasks, the acquisition terminal further being for performing acquisition of the data item of the highest priority with respect to the redetermined highest priority acquisition device.
Preferably the data acquisition system further comprises first power acquisition means for determining a historical acquisition success rate of each data item of each acquisition device by determining a historical acquisition success rate X kj of a j-th data item of a k-th acquisition device of an i-th acquisition device in a last acquisition period based on a response time C kji of a j-th data item of the k-th acquisition device and the following equation,
Wherein b is the acquisition times corresponding to the jth data item of the kth acquisition device in the last acquisition period, and d is the success rate calculation interval.
Preferably, the data acquisition system further comprises second power acquisition means for determining an average historical acquisition success rate for each acquisition device by determining an average historical acquisition success rate X k for n data items for the kth device based on a first historical acquisition success rate X kj for a j data item for the kth device and the following equation,
Preferably, the data acquisition system further comprises a third power acquisition device for determining a time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers, wherein the plurality of acquisition time periods are a plurality of continuous acquisition time periods before a current acquisition time period, a fourth power acquisition device for determining a maximum time period acquisition success rate if a difference of time period acquisition success rates in any two of the plurality of acquisition time periods is smaller than or equal to a preset value, and a maximum concurrency number acquisition device for determining the preset maximum concurrency number corresponding to the maximum time period acquisition success rate as a target maximum concurrency number of communication between an acquisition terminal and the plurality of acquisition devices.
Preferably, the third power obtaining means is configured to determine a period acquisition success rate in each of a plurality of acquisition periods under a plurality of preset maximum concurrency numbers, including determining a ratio of a number of data items successfully acquired by all acquisition devices in each of the plurality of preset maximum concurrency numbers to a total number of data items of all the acquisition devices as the period acquisition success rate in each of the plurality of preset maximum concurrency numbers.
Preferably, the data acquisition system further comprises a judging device for judging whether the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, a shortest time acquisition device for determining the shortest completion time according to the response time of each data item of each acquisition device in the acquisition task in the last acquisition period when the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, and a control device for stopping the statistics operation of the historical acquisition success rate when the total time of the acquisition tasks in the current acquisition period is greater than a preset multiple of the shortest completion time.
Specific details and benefits of the data acquisition system provided in the embodiments of the present invention can be found in the above description of the data acquisition method, and are not repeated here.
A third aspect of the present invention provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the data acquisition method.
Additional features and advantages of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain, without limitation, the embodiments of the invention. In the drawings:
FIG. 1 is a flow chart of a data acquisition method according to an embodiment of the invention, and
Fig. 2 is a schematic structural diagram of a data acquisition system according to an embodiment of the present invention.
Detailed Description
The following describes specific embodiments of the present invention in detail with reference to the drawings. It should be understood that the detailed description and specific examples, while indicating and illustrating the invention, are not intended to limit the invention.
Fig. 1 is a flowchart of a data acquisition method according to an embodiment of the present invention. As shown in FIG. 1, the data acquisition method may include a step S101 of, for each of a plurality of acquisition devices in an acquisition task within a current acquisition period, prioritizing each data item of the each acquisition device according to a historical acquisition success rate of the each data item, a step S102 of determining a highest priority acquisition device according to an average historical acquisition success rate of the each acquisition device, and a step S103 of acquiring the highest priority data item with respect to the highest priority acquisition device.
The following explanation and explanation are made with respect to the specific procedures of the above steps S101 to S103, respectively.
Step S101, for each of a plurality of acquisition devices in an acquisition task in a current acquisition period, prioritizing each data item according to a historical acquisition success rate of each data item of each acquisition device.
In various embodiments of the present application, each acquisition period may be set appropriately in advance, for example, one acquisition period every 15 minutes.
The data items can be daily freezing, minute freezing, load data, voltage, current, alarm or event record and the like.
Before performing step S101, the data acquisition method further comprises determining a historical acquisition success rate for each data item of each acquisition device.
Determining a historical acquisition success rate X kj of a j-th data item of a k-th acquisition device according to response time C kji of the j-th data item of the k-th acquisition device in a last acquisition period and the following formula,
Wherein b is the acquisition times corresponding to the jth data item of the kth acquisition device in the last acquisition period, and d is the success rate calculation interval.
The embedded acquisition equipment has limited computing resources, so that when the historical acquisition success rate is calculated, the existing mode of verifying the distribution type through acquiring response time distribution characteristics is not adopted, and then a probability curve is fitted. In the above embodiment, because the same device may not support some data items due to differences in software versions, different data items need to be considered when calculating the historical acquisition success rate, and the historical acquisition success rate is calculated by using response time.
The response time in the above equation is described below. If the time when the acquisition terminal sends the data frame corresponding to the j-th data item to the k-th acquisition device is t1 and the time when the acquisition terminal receives the response data frame of the j-th data item is t2, the response time C kj of the j-th data item of the k-th acquisition device is t2-t1 (i.e. the time between the completion of sending the data item to read the frame and the completion of receiving the response frame).
Specifically, in the case where the success rate calculation interval is 5, if the number of acquisitions corresponding to the jth data item of the kth acquisition device in the last acquisition period is 2If the number of acquisitions corresponding to the jth data item of the kth acquisition device in the last acquisition period is 8, thenThe calculation resources of the embedded acquisition equipment are limited, so that a method for verifying the distribution type and fitting a probability curve through acquisition response time distribution characteristics is not adopted when the acquisition success rate is calculated.
Dividing the acquisition task into a plurality of acquisition units according to the maximum support length of the response frame, wherein one acquisition unit contains different data items of the same acquisition equipment. For example, the acquisition tasks may include acquisition unit 1, acquisition unit 2, and acquisition unit 3. The acquisition unit 1 comprises { day-to-freeze, load data, voltage and current } of the acquisition device 1, the acquisition unit 2 comprises { day-to-freeze, load data, voltage and current } of the acquisition device 2, and the acquisition unit 3 comprises { day-to-freeze, load data and voltage } of the acquisition device 3.
According to the response time C 11i of the 1 st data item-day freezing of the acquisition device 1 in the last acquisition period and the above formula (1), the historical acquisition success rate of the 1 st data item-day freezing of the acquisition device 1 can be determined, according to the response time C 12i of the 2 nd data item-load data of the acquisition device 1 in the ith acquisition period and the above formula (1), the historical acquisition success rate of the 2 nd data item-load data of the acquisition device 1 can be determined, according to the response time C 13i of the 3 rd data item-voltage of the acquisition device 1 in the last acquisition period and the above formula (1), the historical acquisition success rate of the 3 rd data item-voltage of the acquisition device 1 can be determined, and according to the response time C 14i of the 4 th data item-current of the acquisition device 1 in the last acquisition period and the above formula (1), the historical acquisition success rate of the 4 th data item-current of the acquisition device 1 can be determined. Similarly, the historical acquisition success rate of the individual data items of the acquisition device 2 and the acquisition device 3 may be determined.
And calculating to obtain a corresponding historical acquisition success rate by taking the acquisition unit as a minimum unit, sequencing the queue elements according to the historical acquisition success rate, and sequentially acquiring each data item of each acquisition device from high to low according to the acquisition success rate.
And according to the historical acquisition success rate of each data item of each acquisition device, which is determined above, the priority of each data item of each acquisition device is ordered through step S101, wherein the higher the historical acquisition success rate is, the higher the priority is. If the historical acquisition success rate of each data item of each acquisition device is ordered in descending order of daily freezing of the acquisition device 1, load data of the acquisition device 1, daily freezing of the acquisition device 2, load data of the acquisition device 2, voltage of the acquisition device 3, current of the acquisition device 1, voltage of the acquisition device 1, current of the acquisition device 2, daily freezing of the acquisition device 3, load data of the acquisition device 3, voltage of the acquisition device 2, the priority of each data item is { daily freezing (11), load data (10), voltage (5), current (6) of the acquisition device 1, the acquisition unit 2 comprises { daily freezing (9), load data (8), voltage (1), current (4) of the acquisition device 2, and the acquisition unit 3 comprises { daily freezing (3), load data (2), voltage (7) of the acquisition device 3.
That is, the data items are ordered from high to low in priority within each acquisition unit according to the historical acquisition success rate without distinguishing front-to-back locations between different acquisition units.
Step S102, determining the acquisition equipment with the highest priority according to the average historical acquisition success rate of each acquisition equipment.
First, the plurality of collection devices (e.g., collection device 1, collection device 2, collection device 3) may be prioritized according to the average historical collection success rate of each collection device, where the higher the average historical collection success rate, the higher the priority. If the average historical acquisition success rate of each acquisition device is sorted into the acquisition device 2, the acquisition device 1 and the acquisition device 3 in descending order, the priority is from high to low, namely the acquisition device 2, the acquisition device 1 and the acquisition device 3, so that the acquisition device with the highest priority can be determined to be the acquisition device 2.
Of course, the acquisition device with the highest average historical acquisition success rate (for example, acquisition device 2) may also be determined according to the average historical acquisition success rate of each acquisition device, and the acquisition device with the highest priority is acquisition device 2.
In one embodiment, the data acquisition method further comprises determining an average historical acquisition success rate for each acquisition device by.
Determining an average historical acquisition success rate X k of n data items of the kth device according to a first historical acquisition success rate X kj of the jth data item of the kth device and the following formula,
Because of the rule of the ammeter communication protocol, a plurality of data items of the same device are read in one frame of data at the same time, and the success rate of collecting n data items of the same device in the same frame can be determined by the above formula.
That is, in the case where the historical acquisition success rate of each data item of each acquisition device is obtained, the historical acquisition success rates of the respective data items in each acquisition device may be averaged, so that the average historical acquisition success rate for each acquisition device may be obtained. If the plurality of acquisition devices are prioritized according to the average historical acquisition success rate, the plurality of acquisition units (corresponding to the plurality of acquisition devices) may be inserted into a list structure according to the priority to form a data request queue.
Step S103, collecting the data item with the highest priority from the collecting device with the highest priority.
In the prior art, if the collection process is executed according to the sequence of the collection tasks, when the collection service application encounters the condition that the collection device does not answer when concurrent reading, the collection device needs to wait for a maximum timeout time, so that more waiting time can reduce the collection efficiency, and the collection tasks cannot be completed in a collection period. In the present embodiment, if the priority from high to low in the data request queue is obtained in step S102, the priority is that of the acquisition device 2, the acquisition device 1, and the acquisition device 3. Thus, the acquisition device of the highest priority can be determined as the acquisition device 2. And, according to the priority of each data item of each acquisition device obtained in step S101, it can be determined that the data item of the highest priority of the acquisition device 2 is daily freeze (9). Therefore, the daily freezing of the collecting device 2 can be collected preferentially. In the acquisition process, even if the condition that the acquisition equipment is not in response is met in the concurrent reading process, the acquisition process does not need to wait for a long time.
In an embodiment wherein the average historical acquisition success rate of the highest priority acquisition device is no longer the highest priority except for the highest priority data item of the highest priority acquisition device, the average historical acquisition success rate of the highest priority acquisition device is lower than the average historical acquisition success rate of other acquisition devices. On this basis, the priorities of the plurality of acquisition devices need to be reordered to update the acquisition device of the highest priority.
The data acquisition method can further comprise the steps of obtaining updated data items of the acquisition equipment with the highest priority, wherein the updated data items do not comprise the data items with the highest priority, determining the acquisition equipment with the highest priority again according to the updated average historical acquisition success rate of the acquisition equipment with the highest priority and the average historical acquisition success rate of other acquisition equipment in the acquisition task in the current acquisition period, and acquiring the data items with the highest priority by the acquisition equipment with the highest priority which is determined again.
The updated data items of the acquisition device with the highest priority are obtained by the acquisition unit 2 including { load data (8), voltage (1), and current (4) } of the acquisition device 2, then updating the average historical acquisition success rate of the acquisition device 2 according to the formula for determining the average historical acquisition success rate, comparing the average historical acquisition success rate of the updated acquisition device 2 with the average historical acquisition success rates of other acquisition devices (for example, acquisition device 1 and acquisition device 3), and if the updated average historical acquisition success rate is still the acquisition device 2, the acquisition device 1 and the acquisition device 3 in descending order, determining that the acquisition device with the highest priority is still the acquisition device 2, the acquisition device 1 and the acquisition device 3 again. And, according to the priority of each data item of each acquisition device obtained in step S101, the data item of the highest priority of the acquisition device 2 can be determined as the load data (8). Therefore, the load data of the acquisition device 2 can be preferentially acquired.
When a new acquisition device is inserted into the data request queue, the devices are re-prioritized.
After each acquisition period is finished, the acquisition success rates of different data items of each device are ordered, the ordering results are inserted into a historical acquisition success rate list, and an automatic deriving means is provided so as to facilitate fault investigation for devices with lower acquisition success rates. In order to facilitate the positioning of fault reasons, the historical acquisition success rate list takes the acquisition success rate of the data item as the minimum unit, and the acquisition success rate of other data items like the equipment is higher, and the acquisition success rate of a certain data item is 0, so that the equipment can judge that the equipment does not support the reading of the data item with high probability.
Considering that the maximum concurrency number (the maximum concurrency number refers to the maximum concurrency data request number allowed when the acquisition terminal performs data reading in the carrier communication process) has a great influence on the acquisition efficiency, in this embodiment, the maximum concurrency number can be subjected to reading efficiency test, so as to obtain the maximum concurrency number with the highest efficiency.
In an embodiment, the data acquisition method can further comprise determining a time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers, wherein the plurality of acquisition time periods are a plurality of continuous acquisition time periods before a current acquisition time period, determining a maximum time period acquisition success rate if a difference between time period acquisition success rates in any two of the plurality of acquisition time periods is smaller than or equal to a preset value, and determining the preset maximum concurrency number corresponding to the maximum time period acquisition success rate as a target maximum concurrency number of communication between an acquisition terminal and the plurality of acquisition devices.
The method comprises the steps of determining the time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers, wherein the time period acquisition success rate in each of the plurality of acquisition time periods under the plurality of preset maximum concurrency numbers comprises the step of determining the ratio of the number of data items successfully acquired by all acquisition devices in each of the plurality of preset maximum concurrency numbers to the total number of the data items of all the acquisition devices as the time period acquisition success rate in each of the plurality of preset maximum concurrency numbers.
For example, the plurality of preset maximum concurrency numbers may be three typical values. And selecting three typical values of the maximum concurrency number for testing, wherein the typical values are obtained through configuration files according to historical application experience. In order to avoid the influence of other factors on reading efficiency as much as possible, the test is required according to the historical data with relatively stable acquisition success rate. And then, when judging that the maximum difference of the time period acquisition success rates between any two of the first three acquisition time periods does not exceed 5, determining the time period acquisition success rate under which typical value is the maximum, determining the typical value as the target maximum concurrency number, otherwise, continuing to wait for a test condition, and determining any one of the three typical values as the target maximum concurrency number.
The embodiment provides a test method to select the maximum concurrency number with higher acquisition success rate in an actual scene. The maximum concurrency number with the highest reading success rate is obtained through testing, and the acquisition efficiency is further improved. Because the maximum concurrency number test is performed in a period when the reading success rate is stable, the influence of other factors on the acquisition success rate can be reduced as much as possible, and the effectiveness of the maximum concurrency number test is ensured.
In an embodiment, the data acquisition method can further comprise judging whether the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, determining the shortest completion time according to the response time of each data item of each acquisition device in the acquisition task in the last acquisition period under the condition that the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, and stopping the statistical operation of the historical acquisition success rate under the condition that the total time length of the acquisition tasks in the current acquisition period is larger than a preset multiple of the shortest completion time.
Specifically, if the acquisition task in the current acquisition period includes an acquisition unit 1, an acquisition unit 2 and an acquisition unit 3, where the acquisition unit 1 includes { day-frozen, load data, voltage, and current } of the acquisition device 1, the acquisition unit 2 includes { day-frozen, load data, voltage, and current } of the acquisition device 2, and the acquisition unit 3 includes { day-frozen, load data, and voltage } of the acquisition device 3, it is determined whether the acquisition task in the previous acquisition period also includes the three acquisition units. And comparing the total time length of the acquisition task in the current acquisition period with the shortest completion time, and if the total time length is 2 times longer than the shortest completion time, indicating that the acquisition success rate in each period under the platform area is higher, and further saving calculation resources by configuring a shutdown success rate statistics function. Since the present application is applicable to more equipment or high frequency acquisition demand scenarios, in most cases the acquisition tasks within multiple consecutive acquisition periods are consistent. If the acquisition task in the last acquisition period does not comprise the three acquisition units, the success rate statistics function is kept on. The first period starting at zero daily automatically turns on the statistics function. Of course, in other embodiments, a reasonable preset multiple may be set according to actual needs.
The existing collection type business application does not consider the historical collection success rate distribution characteristics of collection equipment when collecting data items of different equipment are arranged, and a collection plan cannot be adjusted in a targeted mode, so that collection efficiency is improved. The embodiments can be executed by the acquisition terminal, and aims to improve the acquisition efficiency of the acquisition task by calculating the distribution characteristic of the historical acquisition success rate and arranging the data items according to the rule of preferential acquisition of the equipment with high historical acquisition success rate when the acquisition sequence of the corresponding data items of the specific equipment is arranged in each acquisition period. Because the traditional acquisition method is used for statistics aiming at the equipment with low acquisition success rate and lacks an automatic means, a historical acquisition success rate list of each period needs to be provided for conveniently deriving and performing fault investigation.
In summary, the method creatively aims at each acquisition device in a plurality of acquisition devices in an acquisition task in a current acquisition period, determines the priority of each data item according to the historical acquisition success rate of each data item of each acquisition device, then determines the acquisition device with the highest priority according to the average historical acquisition success rate of each acquisition device, and finally acquires the data item with the highest priority from the acquisition device with the highest priority. Therefore, the method and the device for the data acquisition of the invention are used for carrying out priority arrangement according to the historical acquisition success rates of different data items of the same equipment, then carrying out priority arrangement according to the average historical acquisition success rates of different equipment, and carrying out priority acquisition on the data item with the highest priority of the equipment with the highest priority, thereby greatly improving the acquisition efficiency when facing more equipment or high-frequency acquisition demand scenes.
Fig. 2 is a schematic structural diagram of a data acquisition system according to an embodiment of the present invention. As shown in fig. 2, the data acquisition system may include a first determining means 10 for determining, for each of a plurality of acquisition devices in an acquisition task within a current acquisition period, a priority of each data item of the each acquisition device according to a historical acquisition success rate of the each data item, a second determining means 20 for determining, according to an average historical acquisition success rate of the each acquisition device, an acquisition device of a highest priority, and an acquisition terminal 30 for performing acquisition of the data item of the highest priority with respect to the acquisition device of the highest priority.
Preferably, the data acquisition system further comprises deleting means for deleting the highest priority data item of the highest priority acquisition device from the acquisition tasks to obtain updated acquisition tasks within the current acquisition period, and third determining means for redetermining the highest priority acquisition device according to an average historical acquisition success rate of each of the updated acquisition tasks, the acquisition terminal further being for performing acquisition of the data item of the highest priority with respect to the redetermined highest priority acquisition device.
Preferably the data acquisition system further comprises first power acquisition means for determining a historical acquisition success rate of each data item of each acquisition device by determining a historical acquisition success rate X kj of a j-th data item of a k-th acquisition device of an i-th acquisition device in a last acquisition period based on a response time C kji of a j-th data item of the k-th acquisition device and the following equation,
Wherein b is the acquisition times corresponding to the jth data item of the kth acquisition device in the last acquisition period, and d is the success rate calculation interval.
Preferably, the data acquisition system further comprises second power acquisition means for determining an average historical acquisition success rate for each acquisition device by determining an average historical acquisition success rate X k for n data items for the kth device based on a first historical acquisition success rate X kj for a j data item for the kth device and the following equation,
Preferably, the data acquisition system further comprises a third power acquisition device for determining a time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers, wherein the plurality of acquisition time periods are a plurality of continuous acquisition time periods before a current acquisition time period, a fourth power acquisition device for determining a maximum time period acquisition success rate if a difference of time period acquisition success rates in any two of the plurality of acquisition time periods is smaller than or equal to a preset value, and a maximum concurrency number acquisition device for determining the preset maximum concurrency number corresponding to the maximum time period acquisition success rate as a target maximum concurrency number of communication between an acquisition terminal and the plurality of acquisition devices.
Preferably, the third power obtaining means is configured to determine a period acquisition success rate in each of a plurality of acquisition periods under a plurality of preset maximum concurrency numbers, including determining a ratio of a number of data items successfully acquired by all acquisition devices in each of the plurality of preset maximum concurrency numbers to a total number of data items of all the acquisition devices as the period acquisition success rate in each of the plurality of preset maximum concurrency numbers.
Preferably, the data acquisition system further comprises a judging device for judging whether the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, a shortest time acquisition device for determining the shortest completion time according to the response time of each data item of each acquisition device in the acquisition task in the last acquisition period when the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, and a control device for stopping the statistics operation of the historical acquisition success rate when the total time of the acquisition tasks in the current acquisition period is greater than a preset multiple of the shortest completion time.
Specific details and benefits of the data acquisition system provided in the embodiments of the present invention can be found in the above description of the data acquisition method, and are not repeated here.
An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which when executed by a processor implements the data acquisition method.
The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present invention within the scope of the technical concept of the present invention, and all the simple modifications belong to the protection scope of the present invention.
In addition, the specific features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various possible combinations are not described further.
Those skilled in the art will appreciate that all or part of the steps in implementing the methods of the embodiments described above may be implemented by a program stored in a storage medium, including instructions for causing a single-chip microcomputer, chip or processor (processor) to perform all or part of the steps of the methods of the embodiments described herein. The storage medium includes a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes.
Moreover, any combination of the various embodiments of the invention can be made without departing from the spirit of the invention, which should also be considered as disclosed herein.
Claims (13)
1.A data acquisition method, the data acquisition method comprising:
Determining the priority of each data item according to the historical acquisition success rate of each data item of each acquisition device aiming at each acquisition device in a plurality of acquisition devices in an acquisition task in a current acquisition period;
Determining the acquisition equipment with the highest priority according to the average historical acquisition success rate of each acquisition equipment, wherein the average historical acquisition success rate of each acquisition equipment is calculated according to the historical acquisition success rates of all data items of the acquisition equipment, and
The highest priority acquisition device is subjected to acquisition of data items of the highest priority,
Wherein the historical acquisition success rate of each data item of each acquisition device is determined by:
Response time of jth data item of kth acquisition device according to ith acquisition in last acquisition period And determining a historical acquisition success rate of a j-th data item of the k-th acquisition device,
,
Wherein b is the acquisition times corresponding to the jth data item of the kth acquisition device in the last acquisition period, and d is the success rate calculation interval.
2. The data acquisition method of claim 1, wherein the data acquisition method further comprises:
obtaining updated data items of the highest priority acquisition device, wherein the updated data items do not include the highest priority data items;
Re-determining the acquisition equipment with the highest priority according to the updated average historical acquisition success rate of the acquisition equipment with the highest priority and the average historical acquisition success rate of other acquisition equipment in the acquisition task in the current acquisition period, and
The acquisition of the data item of the highest priority is performed on the newly determined acquisition device of the highest priority.
3. The data acquisition method according to any one of claims 1-2, characterized in that the average historical acquisition success rate of each acquisition device is determined by:
a first historical acquisition success rate according to a j-th data item of the k-th device And determining an average historical acquisition success rate of n data items of the kth device,
。
4. The data acquisition method according to any one of claims 1-2, characterized in that the data acquisition method further comprises:
Determining a time period acquisition success rate in each of a plurality of acquisition time periods under a plurality of preset maximum concurrency numbers, wherein the plurality of acquisition time periods are a plurality of continuous acquisition time periods before a current acquisition time period;
Determining a maximum period acquisition success rate in the event that a difference in period acquisition success rates within any two of the plurality of acquisition periods is less than or equal to a preset value, and
And determining the preset maximum concurrency number corresponding to the maximum period acquisition success rate as the target maximum concurrency number of communication between the acquisition terminal and the plurality of acquisition devices.
5. The method of claim 4, wherein determining a time period acquisition success rate for each of a plurality of acquisition time periods at a plurality of preset maximum concurrency times comprises:
And determining the ratio of the number of data items successfully acquired by all the acquisition devices in each acquisition period under the plurality of preset maximum concurrency numbers to the total number of the data items of all the acquisition devices as the period acquisition success rate in each acquisition period under the plurality of preset maximum concurrency numbers.
6. The data acquisition method according to any one of claims 1-2, characterized in that the data acquisition method further comprises:
judging whether the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period;
Determining a shortest completion time based on a response time of each data item of each acquisition device in the acquisition task in the last acquisition period, in case the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, and
And stopping the statistics operation of the historical acquisition success rate under the condition that the total time length of the acquisition tasks in the current acquisition period is larger than the preset multiple of the shortest completion time.
7. A data acquisition system, the data acquisition system comprising:
the first determining device is used for determining the priority of each data item according to the historical acquisition success rate of each data item of each acquisition device aiming at each acquisition device in a plurality of acquisition devices in an acquisition task in the current acquisition period;
a second determining device for determining the acquisition device with the highest priority according to the average historical acquisition success rate of each acquisition device, wherein the average historical acquisition success rate of each acquisition device is calculated according to the historical acquisition success rates of all data items of the acquisition device, and
An acquisition terminal for performing acquisition of data items of the highest priority with respect to the acquisition device of the highest priority,
The data acquisition system further comprises a first power acquisition device, a second power acquisition device and a third power acquisition device, wherein the first power acquisition device is used for determining the historical acquisition success rate of each data item of each acquisition device by the following method:
Response time of jth data item of kth acquisition device according to ith acquisition in last acquisition period And determining a historical acquisition success rate of a j-th data item of the k-th acquisition device,
,
Wherein b is the acquisition times corresponding to the jth data item of the kth acquisition device in the last acquisition period, and d is the success rate calculation interval.
8. The data acquisition system of claim 7, wherein the data acquisition system further comprises:
deleting means for deleting the highest priority data item of the highest priority acquisition device from the acquisition task to obtain an updated acquisition task within the current acquisition period, and
Third determining means for re-determining the acquisition device of the highest priority based on the average historical acquisition success rate of each acquisition device in the updated acquisition task,
The acquisition terminal is also used for acquiring the data item with the highest priority by the newly determined acquisition equipment with the highest priority.
9. The data acquisition system of any one of claims 7 to 8 further comprising second component power acquisition means for determining an average historical acquisition success rate for each acquisition device by:
a first historical acquisition success rate according to a j-th data item of the k-th device And determining an average historical acquisition success rate of n data items of the kth device,
。
10. The data acquisition system of any one of claims 7 to 8, wherein the data acquisition system further comprises:
A third power obtaining device, configured to determine a period acquisition success rate in each of a plurality of acquisition periods under a plurality of preset maximum concurrency numbers, where the plurality of acquisition periods are a plurality of continuous acquisition periods before a current acquisition period;
Fourth power acquisition means for determining a maximum period acquisition success rate in the case where a difference in period acquisition success rates in any two of the plurality of acquisition periods is less than or equal to a preset value, and
And the maximum concurrency number acquisition device is used for determining the preset maximum concurrency number corresponding to the maximum period acquisition success rate as the target maximum concurrency number of communication between the acquisition terminal and the plurality of acquisition devices.
11. The data acquisition system of claim 10 wherein the third power acquisition means for determining a time period acquisition success rate for each of a plurality of acquisition time periods at a plurality of preset maximum concurrency times comprises:
And determining the ratio of the number of data items successfully acquired by all the acquisition devices in each acquisition period under the plurality of preset maximum concurrency numbers to the total number of the data items of all the acquisition devices as the period acquisition success rate in each acquisition period under the plurality of preset maximum concurrency numbers.
12. The data acquisition system of any one of claims 7 to 8, wherein the data acquisition system further comprises:
the judging device is used for judging whether the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period;
A shortest time acquisition means for determining a shortest completion time based on a response time of each data item of each acquisition device in the acquisition task in the last acquisition period in the case where the acquisition task in the current acquisition period is the same as the acquisition task in the last acquisition period, and
And the control device is used for stopping the statistics operation of the historical acquisition success rate under the condition that the total time length of the acquisition tasks in the current acquisition period is larger than the preset multiple of the shortest completion time.
13. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the data acquisition method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310409554.6A CN116582773B (en) | 2023-04-17 | 2023-04-17 | Data collection method, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310409554.6A CN116582773B (en) | 2023-04-17 | 2023-04-17 | Data collection method, system and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116582773A CN116582773A (en) | 2023-08-11 |
| CN116582773B true CN116582773B (en) | 2025-09-02 |
Family
ID=87538609
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310409554.6A Active CN116582773B (en) | 2023-04-17 | 2023-04-17 | Data collection method, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116582773B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114374892A (en) * | 2022-01-04 | 2022-04-19 | 烽火通信科技股份有限公司 | Meter reading method and meter reading system based on HPLC |
| CN115495219A (en) * | 2022-09-29 | 2022-12-20 | 中国电信股份有限公司 | Pipeline scheduling method and device and computer readable storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2682879A1 (en) * | 2012-07-05 | 2014-01-08 | Thomson Licensing | Method and apparatus for prioritizing metadata |
| JP6972714B2 (en) * | 2017-07-04 | 2021-11-24 | 富士通株式会社 | Data acquisition programs, equipment, and methods |
| CN111935756B (en) * | 2019-05-13 | 2022-08-26 | 华为技术有限公司 | Data transmission method, device and equipment |
| CN112379994B (en) * | 2021-01-07 | 2021-04-20 | 武汉中原电子信息有限公司 | Multidimensional electricity data acquisition and scheduling method and system |
| JP7037001B1 (en) * | 2021-09-01 | 2022-03-15 | ファナック株式会社 | Control device and data communication system |
| CN115499461A (en) * | 2022-08-23 | 2022-12-20 | 重庆长安汽车股份有限公司 | Data acquisition method and system, cloud server, vehicle and storage medium |
-
2023
- 2023-04-17 CN CN202310409554.6A patent/CN116582773B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114374892A (en) * | 2022-01-04 | 2022-04-19 | 烽火通信科技股份有限公司 | Meter reading method and meter reading system based on HPLC |
| CN115495219A (en) * | 2022-09-29 | 2022-12-20 | 中国电信股份有限公司 | Pipeline scheduling method and device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116582773A (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220138012A1 (en) | Computing Resource Scheduling Method, Scheduler, Internet of Things System, and Computer Readable Medium | |
| CN113886034A (en) | Task scheduling method, system, electronic device and storage medium | |
| CN113867919B (en) | Kubernetes cluster scheduling method, system, equipment and medium | |
| CN117251275B (en) | Multi-application asynchronous I/O request scheduling method, system, equipment and medium | |
| CN116860789A (en) | Data distribution optimization method and distributed database system | |
| CN117785481A (en) | Data center computing resource allocation management system | |
| WO2016188361A1 (en) | Parameter optimization method and device | |
| CN116582773B (en) | Data collection method, system and computer readable storage medium | |
| CN103258389B (en) | The method of self-aided terminal upload file, system and self-aided terminal | |
| CN113190335B (en) | Multi-task scheduling and collecting method of power collecting terminal and power collecting system | |
| CN117395697A (en) | Communication load forecasting method, computer-readable medium | |
| CN111309442B (en) | Method, device, system, medium and equipment for adjusting number of micro-service containers | |
| CN109951330B (en) | Management method for automatically identifying equipment type and communication manager | |
| CN117454043A (en) | Loss early warning method, device, equipment and storage medium for user account | |
| CN116483543A (en) | Task processing method, device, equipment and storage medium | |
| CN116596211A (en) | Data acquisition method, system and computer readable storage medium | |
| CN115915165B (en) | Baseband board resource management method and device | |
| CN113254553A (en) | Web map publishing method and system based on intelligent agent platform | |
| CN112766741A (en) | Query method, system, terminal and storage medium | |
| CN112260869A (en) | Service degradation method, system, terminal device and computer readable storage medium | |
| US20210014123A1 (en) | Dynamic network adaptation | |
| CN119377240B (en) | Method, device and storage medium for updating data in energy internet marketing service system | |
| CN107864096B (en) | Path adjusting method and device | |
| CN119961009B (en) | A cloud-based data processing method and system | |
| CN116955342B (en) | Service data consistency rate verification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |