TWI887894B - Non-transitory computer-readable storage medium and method and apparatus for processing order data - Google Patents

Abstract

The present invention relates to a data verification method for an electronic device, which includes the following steps: generating statistical data of an order based on order data; generating baseline data for verifying the statistical data based on the order data; comparing the incremental values of the baseline data and the statistical data; and verifying the statistical data based on the comparison of the incremental values.

Description

Non-temporary computer-readable storage medium and method and device for processing order data

The embodiment of this specification is related to a method and device for processing order data.

As the use of the Internet becomes more popular, the e-commerce market is also expanding. Especially with the spread of the pandemic, the proportion of people going to physical stores to buy goods has decreased, and the proportion of people using computers or smartphones to buy goods through e-commerce has increased rapidly.

E-commerce companies need to collect and generate statistics such as order volume, page visits, and product clicks in order to analyze and make decisions related to e-commerce services for entities such as users. In order to generate statistics based on order data, all orders of users must be analyzed in real time, but this is quite difficult. Real-time analysis of order data is challenging because the order history of users occupies a large amount of data, and a single order can be created and canceled at any time. Therefore, a method is needed that can analyze and generate statistics from large amounts of data in real time.

In addition, validation of the statistics generated in real time by the data pipeline is also a very important topic. Statistics are updated in real time, and whenever the statistics are updated, users can use the updated statistics. However, if any part of the entire data pipeline fails, the statistics generated by the data pipeline may have data quality problems. For example, there may be errors in the newly released code to implement the data pipeline. Late discovery of such problems may bring more operational problems to e-commerce services. Therefore, it is very important to regularly check the data quality of the statistics to avoid the above situation. However, the storage and processing of aggregated data such as indicators (e.g., SDP visit volume, click volume, total order amount, etc.) is very time-consuming, and it is difficult to confirm the latest data at any time. In addition, data such as user order volume may contain 8TB of data, so a method that can effectively perform data quality checks on massive data is needed.

As a previous document on such data processing methods, there is Korean Patent Publication No. 10-2009-0121754.

The embodiment of the present invention is proposed to solve the above-mentioned problem, and its purpose is to generate statistical data based on real-time order data, and verify the statistical data by comparing the incremental values of the baseline data and the statistical data generated based on the statistical data, so as to effectively perform statistical data analysis and verification on large-capacity data that changes in real time.

In order to achieve the above-mentioned purpose, a data verification method for an electronic device according to an embodiment of the present invention includes the following steps: generating statistical data of an order based on order data; generating baseline data for verifying the statistical data based on the order data; comparing the incremental values of the baseline data and the statistical data; and verifying the statistical data based on the comparison of the incremental values.

According to one embodiment, the order data includes order identifiers and order quantities of more than one order; the statistical data includes the sum of the order quantities.

According to one embodiment, the step of generating the above statistical data further includes the following steps: receiving an order cancellation event; confirming the order quantity canceled according to the order cancellation event; and accumulating the value obtained by multiplying the canceled order quantity by -1 to the above statistical data.

According to one embodiment, the above statistical data is generated by a data pipeline that ensures exactly once processing of the above order data.

According to one embodiment, the data pipeline includes a first pipeline that ensures at least once processing and a second pipeline that ensures exactly once processing, and the output of the first pipeline is input to the second pipeline.

According to one embodiment, the step of generating the above statistical data includes the following steps when R(n+1) representing the n+1th change of the above order data is input to the above first pipeline: retrieving R(n) representing the nth change of the above order data in the cache (retrieving); multiplying the value of the above R(n) by -1; transmitting the above -R(n) and R(n+1) to the above second pipeline; and caching the above R(n+1).

According to one embodiment, the step of generating the above-mentioned statistical data further includes the following steps: generating the above-mentioned statistical data by aggregating the above-mentioned -R(n) and R(n+1) in the above-mentioned second pipeline.

According to one embodiment, the above R(n) includes order ID and order version.

According to one embodiment, the step of generating the above-mentioned baseline data further includes the following steps: based on the above-mentioned order data, verifying the above-mentioned baseline data.

According to one embodiment, the above-mentioned increment value is calculated based on a period set based on the above-mentioned statistical data.

According to an embodiment of the present invention, an electronic device for verifying data includes: a memory storing at least one command; and a processor; the processor is as follows: by executing the at least one command, statistical data of the order is generated based on the order data; based on the order data, baseline data for verifying the statistical data is generated; the incremental values of the baseline data and the statistical data are compared; based on the comparison of the incremental values, the statistical data is verified.

According to an embodiment of the present invention, a non-temporary computer-readable storage medium includes a medium configured to store computer-readable commands. When the computer-readable commands are executed by a processor, the processor executes a data verification method for an electronic device, the method comprising the following steps: generating statistical data of an order based on order data; generating baseline data for verifying the statistical data based on the order data; comparing the incremental values of the baseline data and the statistical data; and verifying the statistical data based on the comparison of the incremental values.

According to an embodiment of the present invention, the electronic device only stores the data that should be reflected in the statistical data of the order in the first database such as the counteraction record, and uses the value obtained by multiplying the amount of canceled orders in the first database by -1 to update the statistical data, thereby being able to calculate and update the statistical data such as the index in real time without having to retrieve all the data in the existing order history.

In addition, according to an embodiment of the present invention, even if the order data sequence flowing in the message queue is abnormal or a conflict occurs in the data processing process in the data pipeline, the electronic device can also reflect the changed order status normally in the final statistical data by executing abnormal processing without being affected by the problem, thereby effectively maintaining the integrity of real-time data processing.

In addition, according to an embodiment of the present invention, if the electronic device performs a data quality check at time T, the accuracy of the data before time T is proved. In order to perform a data quality check at time T+1, the accuracy of the data up to time T+1 can be proved by performing a data quality check only on the increment between time T and time T+1, so that the electronic device can verify the data integrity of all time periods without increasing unnecessary calculations.

100: Electronic devices

110: Processor

120: Memory

FIG1 is a schematic diagram showing an example of the structures of an electronic device according to an embodiment of the present invention.

Figure 2 is an example diagram for illustrating a method for calculating order statistics according to an embodiment of the present invention.

Figures 3a to 3c are exemplary diagrams showing a real-time data pipeline design for generating statistical data of an electronic device according to an embodiment of the present invention.

FIG. 4 shows an example architecture of data flows associated with a baseline generator and a data quality verifier according to an embodiment of the present invention.

Figures 5a and 5b are example diagrams showing the operation of a data quality verifier according to an embodiment of the present invention.

FIG6 is a sequence diagram showing the process of a data verification method for an electronic device according to an embodiment of the present invention.

Figures 7a to 7c are exemplary diagrams used to illustrate designs for generating statistical data according to multiple embodiments of the present invention.

Below, with reference to the attached drawings, the embodiments of the present invention are described in detail.

When describing the embodiments, the description of the technical contents that are well known in the technical field to which the present invention belongs and are not directly related to the present invention will be omitted. The reason is: by omitting redundant descriptions, the main idea of the present invention can be clearly conveyed to avoid confusion of the main idea of the present invention.

For the same reason, some components are exaggerated, omitted or roughly represented in the attached drawings. In addition, the size of each component does not fully reflect the actual size. In each figure, the same or corresponding components are given the same figure mark.

By referring to the embodiments described in detail below in conjunction with the attached drawings, the advantages and features of the present invention and the methods for implementing the same will become clear. However, the present invention is not limited to the embodiments disclosed below, and can also be implemented in various different forms. The embodiments are provided only to make the disclosure of the present invention more complete and to enable technicians in the technical field to which the present invention belongs to fully understand the technical field to which the present invention belongs. The present invention is limited only by the scope of the patent application. Throughout the specification, the same reference symbols refer to the same constituent elements.

At this point, it should be understood that the combination of each block of the processing flow chart and the flow chart can be implemented by computer program instructions. Such computer program instructions can be loaded on a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that such instructions implemented by the processor of the computer or other programmable data processing device will produce a method for implementing the functions described in the blocks of the flow chart. In order to implement the functions in a specific manner, such computer program instructions can be stored in a computer-usable or computer-readable memory that is capable of facing a computer or other programmable data processing device, and the instructions stored in the computer-usable or computer-readable memory can also produce a manufactured item containing the following instruction method, which implements the functions described in the blocks of the flow chart. Computer program instructions may also be loaded onto a computer or other programmable data processing device, thereby executing a series of action steps on the computer or other programmable data processing device to generate a process executed by the computer, and the instructions executed by the computer or other programmable data processing device may also provide steps for executing the functions described in the blocks of the flowchart.

Furthermore, each block may represent a module, a fragment, or a portion of code that includes one or more executable instructions for performing a specific logical function. Furthermore, it should be noted that in some alternative embodiments, the functions described in the blocks may not be executed in sequence. For example, two blocks shown in succession may be executed simultaneously or in reverse order depending on the corresponding functions.

At this time, the term "component" used in this embodiment refers to software or hardware elements such as FPGA or ASIC, and the "component" plays a certain role. However, the "component" is not limited to software or hardware. The "component" can be constructed in a manner of being located in an addressable storage medium, or it can be constructed in a manner of running one or more processors. Therefore, as an example, the "component" includes software elements, object-oriented software elements, class elements, and task elements, processes, functions, attributes, procedures, subroutines, code snippets, drivers, firmware, microcodes, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided in the elements and "components" can be combined into a smaller number of elements and "components", or further separated into additional elements and "components". In addition, these elements and "~" can also be used to activate one or more CPUs in a device or secure multimedia card.

FIG1 is a schematic diagram showing an example of the structures of an electronic device according to an embodiment of the present invention.

Referring to FIG. 1 , the electronic device 100 may include a processor 110 and a memory 120 , and may execute a method for processing order information. The electronic device 100 shown in FIG. 1 only illustrates components related to the present embodiment. Therefore, it is obvious to a person skilled in the art that the electronic device 100 may include other general components in addition to the components shown in FIG. 1 .

The processor 110 is used to control the overall function of the electronic device 100 to process order information. For example, the processor 110 can control the electronic device 100 as a whole by executing a program stored in the memory 120 of the electronic device 100. The processor 110 can be implemented by a central processing unit (CPU), a GPU (graphics processing unit), an AP (application processor), etc. in the electronic device 100, but is not limited thereto.

The memory 120 is hardware for storing various data processed in the electronic device 100. The memory 120 can store data that has been processed and data to be processed in the electronic device 100. In addition, the memory 120 can also store applications, drivers, etc. that can be executed by the electronic device 100. The memory 120 may include RAM (random access memory) such as DRAM (dynamic random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), CD-ROM (Compact Disc Read-Only Memory), Blu-ray or other optical disc memory, HDD (hard disk drive), SSD (solid state drive), or flash memory.

In one embodiment, the electronic device 100 may generate statistics based on order data. In one embodiment, the statistics may include a profile or metric representing characteristics associated with an entity (e.g., user, category, promotion) of an e-commerce service, for example, it may include entity attributes that can be used directly (e.g., membership date), aggregate data of specific event types, predictive indicators, etc. Aggregate data may include behavioral statistics such as logins, clicks, visit time, single detail page (SDP) visits, and transaction statistics such as order times, GMV (gross merchandise volume), order amount, and sum of order quantities. For example, a predictive indicator may include a predictive score related to a user's interest in "iPhone". E-commerce companies need to collect and generate such statistics in order to analyze and make decisions related to e-commerce services.

In order to generate statistics based on order data, all orders of users must be analyzed in real time, which is quite difficult. Real-time analysis of order data is challenging because users' order history occupies a large amount of data and individual orders can be created and canceled at any time.

Specifically, when a user orders one or more items through an e-commerce service, the service server may generate order data representing the user's order information. In one embodiment, the order data may include order identifiers and order quantities of more than one order. After the order data is generated, the user may cancel all or part of one or more items included in the order, and the cancellation of such orders shall be reflected in the user's order profile together with the order creation date. Below, an example of user order creation and cancellation is described in conjunction with Figure 2.

FIG. 2 is an example diagram for illustrating a method for calculating order statistics according to an embodiment of the present invention. Referring to FIG. 2 , examples of order creation and cancellation are represented by several tables of order details 210 and order quantity indicators 220.

For example, if user 1 places two orders 1001 and 1002 on October 1, 2022, and the order quantity of each order is 100 and 200, then the "User Order Quantity" indicator is 300. Then, user 1 cancels part of the items in order 1002, which is equivalent to order quantity 50, on October 2, 2022, so the "User Order Quantity" indicator should change to 250. However, since the real-time order information is the most recent snapshot, we know that the order quantity of order 100 is now 150, but we cannot know that its order quantity was 200 before, which makes it difficult to calculate the change of real-time order quantity. In order to calculate the "user order quantity" indicator, it is usually necessary to access all the user's past order history data and add up the order quantities. However, the order vertical data has a very large amount of data. Based on this, it is not easy to calculate indicators in real time. For example, the current order data set size is 8TB, and it is difficult to calculate indicators such as user order times or order quantity in real time.

To solve this problem, in one embodiment, the electronic device 100 can use the first database to store some records to handle the state change of the order, and can incrementally perform the calculation of the indicators related to the order. For example, when calculating the "user order quantity" indicator, if the user cancels the previous order and the canceled order quantity is 200, the electronic device 100 can determine its counteraction value as -200. The electronic device 100 only needs to add the counteraction value -200 to the user's order quantity, without adding all the user's orders, thereby effectively reducing the amount of data required for calculation. The electronic device 100 stores only the data that should be reflected in the order statistics in the streaming real-time data in the first database such as the counteraction record, and uses the value obtained by multiplying the canceled order quantity in the first database by -1 to update the statistics, so that the index can be calculated in real time without searching all the data in the existing order history. Similarly, in the example shown in the order quantity index 220 of FIG. 2, the "user order quantity" index is generated as 300 on October 1, 2022, and because the user cancels 50 orders, the index is processed as -50 on October 2, 2022. The electronic device 100 can receive the order cancellation event, confirm the canceled order quantity from the order cancellation event, and add the value obtained by multiplying the canceled order quantity by -1 to the statistics to generate the statistics.

Figures 3a to 3c are exemplary diagrams showing a real-time data pipeline design for generating statistical data of an electronic device according to an embodiment of the present invention.

Referring to FIG. 3a, an end-to-end real-time data pipeline architecture 300 is shown for the electronic device 100 to generate statistical data such as indicators 340 from order data 310. The electronic device 100 can establish a data pipeline such as the architecture 300 to automate the process of extracting, changing, combining, validating, and loading real-time streaming data.

In one embodiment, the statistical data may be generated by a data pipeline that ensures exactly once processing of the order data 310. In one embodiment, the data pipeline may include a first pipeline 320 that ensures at least once processing (e.g., a streaming ingestion pipeline) and a second pipeline 330 that ensures exactly once processing (e.g., a real time aggregation pipeline), and the output of the first pipeline may be input to the second pipeline. In real-time data processing, it is very important to ensure the accuracy of the calculation results, and the data pipeline used for accurate calculation may need to embed exactly once processing semantics. Exactly once processing may mean that all data should be reflected exactly once in the final result. For example, if a user cancels an order once, but the cancellation data log of the real-time streaming may be entered repeatedly 10 times, the electronic device 100 needs to delete the duplicate data and only reflect the canceled order once. On the other hand, since the first pipeline 320 only ensures at least one processing but not accurate processing, it can accept duplicate data, and the duplicate data can be deduplicated in the second pipeline 330.

In one embodiment, the second pipeline 330 can ensure at least once processing, deduplication and idempotent write semantics to ensure accurate processing once. At least once processing semantics can ensure that all data are processed, deduplication can ensure that duplicate data in the input is deleted, and idempotent write can ensure that multiple reprocessing of data will not affect the final output result, that is, even if it is calculated multiple times, the result will not change.

In one embodiment, the electronic device 100 may generate a unique identifier that uniquely represents a micro batch of the physical data set to ensure equal writing in the second pipeline 330. In the second pipeline 330, the original input data identifier is no longer used because different computation logics are used to aggregate the data. The unique identifier may be set as a combination of the physical ID, micro batch ID, and pointer ID. The micro batch ID may be generated using a digest of all micro batch starting offsets, and the pointer ID may include unique identifiers of different computation logics. In this way, after the system is restarted or recovered from a conflict, the electronic device 100 reprocesses from the starting offset of the last failed micro-batch, and the micro-batch ID is the same as the last processing, and the data will be overwritten with the same identifier, thereby ensuring equal writing.

In one embodiment, the message queue 350 may include Apache Kafka that can publish, subscribe, store and process real-time streaming order data 310. Apache Kafka is a large-capacity real-time log processing system and a message queueing system that uses a publish/subscribe paradigm to process messages. Apache Kafka is a distributed messaging system with good scalability and high throughput that can ensure reliable and continuous message delivery performance. Kafka applies the producer-consumer problem, uses three system components: producer/consumer/broker, sets topics for large amounts of data, organizes partitions by topic, and stores them in order. Kafka delivers the stored data to consumers in sequence for efficient processing. Kafka is a solution specifically designed to handle large amounts of real-time logs. It can process data with a fault-tolerant, reliable architecture and fast performance in a messaging system whose main goal is to transmit data safely and without loss.

In one embodiment, the electronic device 100 may generate statistics based on the order data by: when R(n+1) representing the n+1th change of the order data is input to the first pipeline 320, R(n) representing the nth change of the order data is retrieved from the cache, the value of R(n) is multiplied by -1, -R(n) and R(n+1) are transmitted to the second pipeline 330, and R(n+1) is cached. The electronic device 100 may aggregate -R(n) and R(n+1) transmitted from the first pipeline 320 to the second pipeline 330 to generate statistics. That is, in the second pipeline 330, the statistics can be calculated as R(0)+[R(1)-R(0)]+...+[R(n)-R(n-1)]+[R(n+1)-R(n)]=R(n+1).

In one embodiment, R(n) may include information of an order ID and an order version. The electronic device 100 may confirm the information of the order ID and order version included in R(n), and confirm the order status (generated or cancelled) corresponding to the order ID or order version to be reflected in the indicator to generate the indicator.

The following is an explanation of the exception handling method. When i) the order data sequence flowing in the message queue is abnormal (disorder); ii) a conflict (crash) occurs during the data processing process in the data pipeline, the changed order status can be normally reflected in the final statistical data without being affected by the problem.

In one embodiment, the electronic device 100 may perform exception processing to ensure that only the latest version of the order data is reflected when a sequence exception occurs in the order source message queue. For example, if R(n+k+m) is received before R(n+k) to R(n+k+m-1) as previous orders, when R(n+k+m) is aggregated in the second pipeline 330, the statistics may be processed as R(0)+[R(1)-R(0)]+...+[R(n+k-1)-R(n+k-2)]+[R(n+k+m)-R(n+k-1)]=R(n+k+m). That is, through the aggregation methods of the second pipeline 330, the electronic device 100 can only reflect the latest order data R(n+k+m) to the statistical data, and discard the information R(n+k) to R(n+k+m-1) as the previous order data, so as not to be affected by the problem, and finally reflect the changed order status in the statistical data.

In one embodiment, the electronic device 100 can perform exception processing to ensure that when a conflict occurs during the data processing process in the data pipeline, the conflict is not affected and the changed order status is reflected in the statistical data. In the first pipeline 320, the data processing steps under normal circumstances can be executed as follows N1 to N5:

N1: Confirm the source message queue and extract useful fields.

N2: Filter disqualified messages.

N3: Convert data into events.

N4: Transmit the converted event to the downstream event message queue.

N5: Commit the source message queue offset.

Among them, submitting the source message queue offset means that it is ready to process the next message in the source.

On the other hand, the electronic device 100 can also perform the following additional steps C1 to C4 on the counteraction record 360:

C1: Receive previously stored events from the database management system (DBMS).

C2: Calculate the counteraction value based on the offsetting record.

C3: Transmits calculated offset values and common events.

C4: Override old events with new events in DBMS.

Among them, additional steps C1 and C2 can be combined with N3. Even if a conflict occurs in N3 or the step before N3, the message in the source message queue will be read and processed again, so it will not be affected by the conflict. Additional step C3 can be executed together with N4.

In one embodiment, the additional step C4 may be executed before N4. That is, the electronic device 100 may transmit the converted event to the downstream event message queue (N4) after overwriting the old event with the new event in the DBMS (C4). At this time, if N4 fails due to a conflict, a conflict may occur in the first pipeline 320 after R(n) is replaced with R(n+1) in the first case. After repair, the electronic device 100 may transmit R(n+1) to the downstream again, which may cause calculation based on R(n)+R(n+1) in the second pipeline 330. On the other hand, if N4 fails due to a conflict, then in the second case, only one of R(n+1) or -R(n) is transmitted downstream, and a conflict may occur in the first pipeline 320. After the repair, if -R(n) is sent, there is no problem, but if only R(n+1) is sent, it may cause calculations based on R(n)+R(n+1) in the second pipeline 330. Ultimately, both the first and second cases may cause the electronic device 100 to perform calculations based on R(n)+R(n+1) in the second pipeline 330, resulting in problems.

In one embodiment, an additional step C4 may be executed after N4. That is, after the electronic device 100 transmits the converted event to the downstream event message queue (N4), the old event is overwritten with the new event in the DBMS (C4). At this time, if N4 partially fails (the first case), only one of R(n+1) or -R(n) is transmitted to the downstream, and a conflict may occur in the first pipeline 320. After repair, R(n+1) and -R(n) will be transmitted, and the second pipeline 330 can deduplicate the previously received records. In addition, if N4 is executed but C4 fails (the second case), only the duplicate records R(n+1) and -R(n) are transmitted to the downstream again. On the other hand, if N4 and C4 are executed but N5 fails (the third case), only the duplicate record R(n+1) is transmitted to the downstream again. Even if duplicate data is transmitted, it will not cause any problem because the second pipeline 330, which is processed accurately once, will remove the duplicates. Therefore, it is better for the electronic device 100 to execute the additional step C4 after N4.

In one embodiment, the DBMS may include Apache Cassandra, an open source distributed No-SQL DBMS. Cassandra can manage large volumes of data across many servers while providing high performance without a single point of failure (SPOF), can support clusters across multiple data centers, and allows low latency operation for all clients.

In one embodiment, the DBMS may include Redis, which is an open source non-relational DBMS with a key-value structure.

In this way, even if the order data sequence flowing in the message queue is abnormal or a conflict occurs in the data processing process in the data pipeline, the electronic device 100 can still reflect the changed order status normally in the final statistical data by executing abnormal processing without being affected by the problem, thereby effectively maintaining the integrity of real-time data processing.

Refer to FIG. 3b, which illustrates a real-time data pipeline architecture 300, data source metadata 371 that can be input into the architecture 300, and indicator metadata 372. In one embodiment, the electronic device 100 can register a data source through the data source metadata 371 and indicate which business fields can be extracted from the data source. In one embodiment, the electronic device 100 can specify the required indicator through the indicator metadata 372 and set the information on the operation and conditions of the indicator (for example, the service type for collecting statistics, the start and end dates for collecting statistics, etc.).

Referring to FIG. 3c, which illustrates the real-time data pipeline architecture 300, exemplary information of the process 381 to 385 for generating indicators is shown. In 381, when a source order event newly enters the message queue 350, the first pipeline 320 can read the event and send it based on the configuration of the data source metadata 371. In 382, the required order event can be processed, the required business fields can be extracted and written to the relevant business message queue topic based on the configuration of the indicator metadata 372. In 383 and 384, the second pipeline 330 can aggregate the required indicators at a fine-grained level according to the indicator metadata 372. In 385, the publisher can obtain the final indicator 340 and push it to the order profile message queue topic.

On the other hand, validation of the statistics generated in real time by the data pipeline is also a very important topic. Statistics are updated in real time, and whenever the statistics are updated, users can use the updated statistics. However, if any part of the entire data pipeline fails, the statistics generated by the data pipeline may have data quality problems. For example, there may be errors in the newly released code to implement the data pipeline. Late discovery of such problems may bring more operational problems to e-commerce services. Therefore, it is very important to regularly check the data quality of the statistics to avoid the above situation. However, the storage and processing of aggregated data such as indicators (e.g., SDP visit volume, click volume, total order amount, etc.) is very time-consuming, and it is difficult to confirm the latest data at any time. In addition, data such as user order volume may contain 8TB of data, so a method that can effectively perform data quality checks on massive data is needed. Below, an embodiment related to statistical data verification will be described in conjunction with Figures 4 and 5.

FIG. 4 shows an example architecture of data flows associated with a baseline generator and a data quality verifier according to an embodiment of the present invention. Referring to FIG. 4 , the diagram includes an architecture 410 of a data pipeline for generating the statistical data described in FIG. 3a and an architecture 420 for verifying the data quality of the statistical data.

In one embodiment, the electronic device 100 can generate baseline data for verifying statistical data based on order data. The electronic device 100 can generate baseline data from the original data (golden truth data) of the order data, and the original data can be extracted from the DBMS dump used by the batch pipeline and the message queue dump used by the real-time pipeline. In one embodiment, the source data can be sampled based on the user's identifier to reduce the computational cost.

In one embodiment, the step of generating baseline data may further include: verifying the baseline data based on the order data. For example, there may be data duplication in the message queue dump used by the real-time pipeline, or conflicts may occur in the process of extracting the original data from the message queue dump, so it is impossible to ensure that the original data is 100% accurate data. Therefore, the electronic device 100 can also perform data quality verification on the baseline data generated based on the order data.

In one embodiment, the electronic device 100 may compare the incremental values of each of the baseline data and the statistical data, and verify the statistical data based on the comparison result. For example, the baseline data and the statistical data may include order quantity, and the incremental value of the baseline data and the statistical data may be calculated as "value (till today)" (i.e., the order quantity until today) minus "value (till yesterday)" (i.e., the order quantity until yesterday), i.e., "value (till today) - value (till yesterday)". In this way, the electronic device 100 can track the incremental value of the data in time units between T and T+1 based on the data value verified at time T, and verify the accuracy of the tracked incremental value, thereby determining whether the data value confirmed at time T+1 is accurate, thereby effectively verifying large-capacity data without having to verify all data from time 0 to time T+1.

Specifically, V(n) represents the calculated index value at time n, B(n) represents the reference index value at time n, and P(n) represents the accuracy of V(n), which can have a value of true or false. That is, if P(n) is true, then V(n)=B(n). The time interval of N can be hours. At this time, the electronic device 100 can prove that P(n) is true at all times n in the following order.

1) First, P(0) = true, where 0 represents the time when data validation starts, which means that all the history of the indicator calculated from the beginning will be validated.

2) For all k>=n+1, it can be shown that if P(k) is true, then P(k+1) is also true. For a particular k, assuming that P(k) is true when n=k, then for k<j<=k+1, comparing B(j) and V(j), when all values in the interval are the same, P(k+1) is also true.

3) If 1) and 2) above are proven to be true, then according to mathematical induction, P(n) is true at all times n.

By using the above mathematical induction method, if the electronic device performs a data quality check at time T, the accuracy of the data before time T is proved. In order to perform a data quality check at time T+1, the accuracy of the data up to time T+1 can be proved by performing a data quality check only on the increment between time T and time T+1, so that the electronic device 100 can verify the data integrity of all time periods without increasing unnecessary calculations.

In one embodiment, the incremental value can be calculated based on a period set based on statistical data. For example, the period for calculating the incremental value (such as one hour, one day, one week) can be adaptively determined based on statistical data (such as the type of ordered goods, the user's order frequency, etc.). In the case of statistical data on the user's order quantity, according to the characteristics of the goods, for the order quantity of goods that the user frequently orders, the period for calculating the incremental value is set to be shorter (such as one day), and for the order quantity of goods that the user does not frequently order, the period for calculating the incremental value is set to be longer (such as one week), so as to effectively reduce the amount of calculation required for data verification of the statistical data.

Figures 5a and 5b are example diagrams showing the operation of a data quality verifier according to an embodiment of the present invention.

Referring to FIG. 5a, a diagram is shown of an architecture 500 including a data quality validator 510. The data quality validator 510 may include the data quality validator (DQ Validator) included in the architecture 420 of FIG. 4. The data quality validator 510 may measure the accuracy of the data by comparing the baseline data generated by the baseline generator with the statistical data obtained from the indicator dump. In one embodiment, the data quality validator 510 may be driven by a Spark Job.

Referring to FIG5b, a code 520 is shown that can check whether the constraint set by the user is satisfied. In the example shown in FIG5b, the data quality validator 510 can set the new order quantity of one day as the check statistic as a key value ("order-fresh-day-amount") through the code 520, and check whether the value is within the range set by the constraint condition (e.g., 0 to 100,000), whether the size is significantly different from the previous day's order quantity (e.g., whether it is within the range of 0.9 to 1.1 times the previous day's order quantity) and the consistency (e.g., consistency 99%). If the comparison result between the statistical data and the baseline data (or its incremental value) indicates that at least one of the above constraints is not met, the data quality verifier 510 may send a notification message to the administrator terminal.

FIG6 is a sequence diagram showing the process of a data verification method for an electronic device according to an embodiment of the present invention. The steps shown in FIG6 can be executed by the electronic device 100 shown in FIG1, and the description repeated with the above content is omitted here.

In step S610, the electronic device may generate statistical data of the order based on the order data. In one embodiment, the statistical data may include a profile or metric representing characteristics associated with an e-commerce service of an entity (such as a user, a category, a promotion), for example, it may include entity attributes that can be used directly (such as a membership date), aggregate data of a specific event type, and predictive indicators. Aggregate data may include behavioral statistics, such as logins, clicks, visit time, single detail page (SDP) visits, and transaction statistics, such as the number of orders, GMV (gross merchandise volume), order amount, and the sum of order quantities. For example, a predictive indicator may include a predictive score related to a user's interest in "iPhone". E-commerce companies need to collect and generate such statistics in order to analyze and make decisions related to e-commerce services.

In step S620, the electronic device may generate baseline data for verifying statistical data based on order data. The electronic device may generate baseline data from raw data of the order data, and the raw data may be extracted from a DBMS dump used by a batch pipeline and a message queue dump used by a real-time pipeline. In one embodiment, the source data may be sampled based on a user's identifier to reduce computational costs.

In step S630, the electronic device may compare the incremental values of the baseline data and the statistical data. In step S640, the electronic device may verify the statistical data based on the comparison of the incremental values. For example, the baseline data and the statistical data may include order quantities, and the incremental values of the baseline data and the statistical data may be calculated as "value (till today)" (i.e., the order quantity until today) minus "value (till yesterday)" (i.e., the order quantity until yesterday), i.e., "value (till today) - value (till yesterday)". In this way, the electronic device 100 can track the incremental value of the data in time units between T and T+1 based on the data value verified at time T, and verify the accuracy of the tracked incremental value, thereby determining whether the data value confirmed at time T+1 is accurate, thereby effectively verifying large-capacity data without having to verify all data from time 0 to time T+1.

Figures 7a to 7c are exemplary diagrams used to illustrate designs for generating statistical data according to multiple embodiments of the present invention.

Referring to FIG. 7a, it illustrates a data pipeline architecture 710 for generating indicators from data sources by an electronic device 100 according to an embodiment of the present invention. Indicators can be divided into today's indicators and indicators before today, where today's indicators can be directly calculated by a stream processing engine (e.g., Spark Streaming application), and indicators before today can be calculated in batches and stored in key-value storage. The final indicator can be generated by combining real-time indicators and batch indicators. For state-variable source data such as order data, the electronic device 100 can use offset processing to record state changes. As shown in FIG. 7a, a database 711 for recording the latest status record may be added. In one embodiment, the database may include a database managed by a Cassandra database management system that supports key value reading and writing. Taking orders as an example, the data pipeline may store all newly processed source order events in the database 711 and replace old events (if any). For all received source order events, if no new order is generated, the pipeline obtains the previous record of the order to calculate the offset value. For example, if an order is canceled, the pipeline may transmit the value obtained by multiplying the previous order amount by -1 to the downstream.

Refer to FIG. 7b, which illustrates a data pipeline architecture 720 for generating indicators from a data source in an electronic device 100 according to an embodiment of the present invention. As shown in FIG. 7b, a metric calculation module 721 may first store business event details and then generate required indicators based on the stored detailed events. The feature of the architecture 720 is that detailed events are aggregated during recalculation in the metric calculation module, which has the advantage of speeding up the metric calculation speed and reducing the storage cost, but has the disadvantage of increasing the complexity of implementation and making it difficult to add new indicators beyond the defined aggregation range.

Referring to FIG. 7c, it illustrates a data pipeline architecture 730 for generating indicators from data sources by an electronic device 100 according to an embodiment of the present invention. As shown in FIG. 7c, an indicator calculation module 731 can aggregate business events into a pre-aggregated format at a fine-grained level and its data is first stored. Then, the indicator calculation module 731 can send a notification to a real-time publisher 732 and allow the real-time publisher 732 to aggregate the pre-aggregated data into a final indicator. For state-variable source data such as order data, the electronic device 100 can use offset processing to change the state of the source event. Architecture 730 stores detailed business events in the database 733 of the data conversion module instead of in the indicator calculation module, because the offsetting events generated can be processed as general business events.

On the other hand, the preferred embodiments of the present invention are disclosed in this specification and the attached drawings. Although specific terms are used, these terms are only used in a general sense to facilitate the description of the technical content of the present invention and to help understand the invention, and are not intended to limit the scope of the present invention. In addition to the embodiments disclosed herein, it is obvious to ordinary technicians in the technical field to which the present invention belongs that other modifications based on the technical ideas of the present invention can be realized.

S610～S640: Step

Claims (10)

A data verification method for an electronic device, comprising the following steps: Based on order data, generating statistical data of an order; Based on the order data, generating baseline data for verifying the statistical data; Comparing the incremental values of the baseline data and the statistical data; and Verifying the statistical data based on the comparison of the incremental values, wherein the statistical data is generated by a data pipeline that ensures accurate one-time processing of the order data, wherein the data pipeline includes a first pipeline that ensures at least one-time processing, and a second pipeline that ensures accurate one-time processing, wherein the output of the first pipeline includes duplicate data related to the order data, and wherein the duplicate data is deduplicated in the second pipeline. The data verification method of the electronic device as claimed in claim 1, wherein the order data includes order identifiers and order quantities of more than one order; The statistical data includes the sum of the order quantities. The data verification method of the electronic device as claimed in claim 2, wherein the step of generating the above statistical data further includes the following steps: Receiving an order cancellation event; Confirming the number of orders canceled according to the above order cancellation event; and Adding the value obtained by multiplying the above canceled order number by -1 to the above statistical data. The data verification method of the electronic device of claim 1, wherein the step of generating the above-mentioned statistical data includes the following steps when R(n+1) representing the n+1th change of the above-mentioned order data is input into the above-mentioned first pipeline: Retrieving R(n) representing the nth change of the above-mentioned order data in the cache; Multiplying the value of the above-mentioned R(n) by -1; Transmitting the above-mentioned -R(n) and R(n+1) to the above-mentioned second pipeline; and Cache the above-mentioned R(n+1). The data verification method of the electronic device as claimed in claim 4, wherein the step of generating the above-mentioned statistical data further includes the following steps: The above-mentioned statistical data is generated by aggregating the above-mentioned -R(n) and R(n + 1) in the above-mentioned second pipeline. A data verification method for an electronic device as claimed in claim 4, wherein the above R(n) includes an order ID and an order version. In the data verification method of electronic device as claimed in claim 1, the step of generating the above-mentioned baseline data further includes the following steps: Based on the above-mentioned order data, verify the above-mentioned baseline data. As in the data verification method of the electronic device of claim 1, the above-mentioned increment value is calculated based on a period set based on the above-mentioned statistical data. An electronic device for verifying data, comprising: a memory storing at least one command; and a processor; The processor is as follows: by executing the at least one command, statistical data of the order is generated based on the order data; based on the order data, baseline data for verifying the statistical data is generated; the incremental values of the baseline data and the statistical data are compared; based on the comparison of the incremental values, the statistical data is verified, wherein the statistical data is generated by a data pipeline that ensures accurate one-time processing of the order data, wherein the data pipeline includes a first pipeline that ensures at least one-time processing, and a second pipeline that ensures accurate one-time processing, wherein the output of the first pipeline includes duplicate data related to the order data, and wherein the duplicate data is deduplicated in the second pipeline. A non-temporary computer-readable storage medium, comprising a medium configured to store computer-readable commands, When the computer-readable commands are executed by a processor, the processor executes a data verification method for an electronic device, the method comprising the following steps: Based on order data, statistical data of an order is generated; Based on the order data, baseline data for verifying the statistical data is generated; Comparing the incremental values of the baseline data and the statistical data; and Verifying the statistical data based on the comparison of the incremental values, Wherein the statistical data is generated by a data pipeline that ensures accurate one-time processing of the order data, The data pipeline includes a first pipeline for ensuring at least one processing and a second pipeline for ensuring exactly one processing, wherein the output of the first pipeline includes duplicate data related to the order data, and wherein the duplicate data is deduplicated in the second pipeline.