HK40050593B - Data sampling method, device and computer readable storage medium - Google Patents

Description

A data sampling method, apparatus, and computer-readable storage medium

Technical Field

This application relates to the field of data processing technology, and more particularly to a data sampling method, apparatus, and computer-readable storage medium.

Background Technology

With the continuous development of computer networks, the amount of data generated in computer networks is increasing day by day. When it is necessary to perform big data statistical analysis on the full amount of data in computer networks, the first step is to sample and statistically analyze the full amount of data. Then, the data obtained from the sampling and statistical analysis can be used to characterize the full amount of data. For example, the data characteristics of the data obtained from the sampling and statistical analysis can be used as the data characteristics of the full amount of data.

In existing technologies, when data sampling is required for data generated within a certain time period, it is usually necessary to pre-read all the data within that time period and then sample the read data within that time period using a set sampling rate. Because existing technologies require pre-reading all the data within that time period, when the amount of data within that time period is too large, it places high demands on the system capacity.

Summary of the Invention

This application provides a data sampling method, apparatus, and computer-readable storage medium that can save system capacity when sampling data.

This application provides a data sampling method, including:

The target type data is sampled at the first sampling rate within the first time window to obtain the target sampled data;

The target sampling data and historical sampling data in the sampling database are merged to obtain merged sampling data. The third sampling rate of the merged sampling data is determined based on the first sampling rate and the second sampling rate of the historical sampling data. The historical sampling data is the target type sampling data obtained by sampling the target type data before obtaining the target sampling data. The second sampling rate is the sampling rate of the historical sampling data relative to the target type data. The third sampling rate is the sampling rate of the merged sampling data relative to the target type data.

When the number of merged sampled data exceeds the sampling number threshold, the merged sampled data is sampled based on the adaptive sampling parameter to obtain updated historical sampled data. The sampling number threshold is related to the target type of data. The adaptive sampling parameter is used to control the number of merged sampled data within the sampling number threshold.

Replace the historical sampling data in the sampling database with the updated historical sampling data.

The methods also include:

Based on the adaptive sampling parameters and the third sampling rate, a fourth sampling rate is determined so that the fourth sampling rate can be used as the first sampling rate to sample the target type data in the second time window. The fourth sampling rate is the sampling rate of the updated historical sampling data relative to the target type data, and the second time window is the next time window after the first time window.

This involves merging the target sampling data and historical sampling data in the sampling database to obtain merged sampling data, including:

When the first sampling rate is greater than the second sampling rate, the ratio between the second sampling rate and the first sampling rate is determined as the first ratio sampling rate.

Based on the first ratio sampling rate, the target sampling data is sampled to obtain the sampled target sampling data;

The target sampling data and historical sampling data after sampling are determined as the merged sampling data.

This involves merging the target sampling data and historical sampling data in the sampling database to obtain merged sampling data, including:

When the first sampling rate is less than the second sampling rate, the ratio between the first sampling rate and the second sampling rate is determined as the second ratio sampling rate;

Based on the second ratio sampling rate, historical sampling data is sampled to obtain sampled historical sampling data;

The historical sampling data and the target sampling data after sampling are determined as the merged sampling data;

When the first sampling rate equals the second sampling rate, the target sampling data and historical sampling data are determined as the merged sampling data.

Specifically, the target type data is sampled at a first sampling rate within the first time window to obtain target sampled data, including:

The second sampling rate of the historical sampling data is determined as the first sampling rate;

Within the first time window, the target type data is sampled using the first sampling rate to obtain the target sampled data;

Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

The target sampling data and historical sampling data are determined as the merged sampling data.

When the number of merged sampled data exceeds a sampling threshold, the merged sampled data is sampled based on adaptive sampling parameters to obtain updated historical sampled data, including:

When the number of merged sampled data exceeds the sampling quantity threshold, obtain the data identifier string corresponding to the merged sampled data;

Map the data identifier string to the uniform sampling space to obtain the hash value corresponding to the data identifier string;

Based on the third sampling rate, adaptive sampling parameters, and hash value, the merged sampled data is sampled to obtain updated historical sampled data.

Specifically, based on the third sampling rate, adaptive sampling parameters, and hash values, the merged sampled data is sampled to obtain updated historical sampled data, including:

The fourth sampling rate is obtained based on the third sampling rate and the adaptive sampling parameters;

Based on the fourth sampling rate and the hash value, the merged sampled data is sampled to obtain updated historical sampled data; the fourth sampling rate is the sampling rate of the updated historical sampled data relative to the target type of data.

This also includes:

When the number of merged sampled data is less than or equal to the sampling number threshold, the historical sampled data in the sampling database is replaced with the merged sampled data.

The time window preceding the first time window is the third time window; the first time window and the third time window have an intersection time window; historical sampling data is the sampling data obtained by sampling the target type data within the intersection time window of the third time window;

The target type data is sampled at the first sampling rate within the first time window to obtain the target sampled data, including:

Within the time window excluding the intersection time window of the first time window, the target type data is sampled at the first sampling rate to obtain the target sampled data;

Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain merged sampling data, including:

When the number of historical sampled data is less than the merging threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampled data is deleted, and the target sampled data is determined as the merged sampled data.

Specifically, the target type data is sampled at a first sampling rate within the first time window to obtain target sampled data, including:

Read data of the target type from the data stream in the first time window, parse the fields of the read target type data to obtain the initial parsed data;

The filtered data is obtained by filtering multiple fields of information in the initial parsed data based on the filtering mechanism.

Based on the vocabulary association mechanism, related field information is added to the filtered and parsed data to obtain sampled business data;

The target sampled data is obtained by sampling the sampled business data at the first sampling rate.

Specifically, the target type data is sampled at a first sampling rate within the first time window to obtain target sampled data, including:

Within the first time window, the target type of data is sampled using the first thread at the first sampling rate to obtain the target sampled data;

Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

The target sampling data and historical sampling data in the sampling database are merged by a second thread to obtain merged sampling data.

This application provides a data sampling device, including:

The sampling module is used to sample data of the target type at a first sampling rate within a first time window to obtain target sampled data;

The merging module is used to merge the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data. The third sampling rate of the merged sampling data is determined based on the first sampling rate and the second sampling rate of the historical sampling data. The historical sampling data is the target type sampling data obtained by sampling the target type data before obtaining the target sampling data. The second sampling rate is the sampling rate of the historical sampling data relative to the target type data. The third sampling rate is the sampling rate of the merged sampling data relative to the target type data.

The adaptive sampling module is used to sample the merged sampled data based on adaptive sampling parameters when the number of merged sampled data exceeds the sampling number threshold, so as to obtain updated historical sampled data. The sampling number threshold is related to the target type of data, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling number threshold.

The replacement module is used to replace historical sampling data in the sampling database with updated historical sampling data.

The data sampling device is also used for:

Based on the adaptive sampling parameters and the third sampling rate, a fourth sampling rate is determined so that the fourth sampling rate can be used as the first sampling rate to sample the target type data in the second time window. The fourth sampling rate is the sampling rate of the updated historical sampling data relative to the target type data, and the second time window is the next time window after the first time window.

The merging module includes:

The first ratio determination unit is used to determine the ratio between the second sampling rate and the first sampling rate as the first ratio sampling rate when the first sampling rate is greater than the second sampling rate.

The first ratio sampling unit is used to sample the target sampling data according to the first ratio sampling rate to obtain the sampled target sampling data.

The first merging unit is used to determine the target sampled data and historical sampled data as merged sampled data.

The merging module includes:

The second ratio determination unit is used to determine the ratio between the first sampling rate and the second sampling rate as the second ratio sampling rate when the first sampling rate is less than the second sampling rate.

The second ratio sampling unit is used to sample the historical sampling data according to the second ratio sampling rate to obtain the sampled historical sampling data.

The second merging unit is used to determine the merged sampled data from the sampled historical sampled data and the target sampled data.

The third merging unit is used to determine the target sampled data and historical sampled data as merged sampled data when the first sampling rate is equal to the second sampling rate.

The sampling module includes:

A sampling rate determination unit is used to determine the second sampling rate of historical sampling data as the first sampling rate;

A window sampling unit is used to sample data of the target type at a first sampling rate within a first time window to obtain target sampled data.

The merging module is also specifically used for:

The target sampling data and historical sampling data are determined as the merged sampling data.

The adaptive sampling module includes:

The string acquisition unit is used to acquire the data identifier string corresponding to the merged sampled data when the number of merged sampled data exceeds the sampling number threshold.

The mapping unit is used to map the data identifier string to the uniform sampling space to obtain the hash value corresponding to the data identifier string;

The merged sampling unit is used to sample the merged sampling data based on the third sampling rate, adaptive sampling parameters, and hash value to obtain updated historical sampling data.

The merging sampling unit includes:

The adaptive sampling rate subunit is used to obtain the fourth sampling rate based on the third sampling rate and the adaptive sampling parameters;

The merged sampling subunit is used to sample the merged sampled data based on the fourth sampling rate and the hash value to obtain updated historical sampled data; the fourth sampling rate is the sampling rate of the updated historical sampled data relative to the target type of data.

Specifically, the data sampling device is also used for:

When the number of merged sampled data is less than or equal to the sampling number threshold, the historical sampled data in the sampling database is replaced with the merged sampled data.

The time window preceding the first time window is the third time window; the first time window and the third time window have an intersection time window; historical sampling data is the sampling data obtained by sampling the target type data within the intersection time window of the third time window;

The sampling module is also specifically used for:

Within the time window excluding the intersection time window of the first time window, the target type data is sampled at the first sampling rate to obtain the target sampled data.

The merging module is also specifically used for:

When the number of historical sampled data is less than the merging threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampled data is deleted, and the target sampled data is determined as the merged sampled data.

The sampling module includes:

The data reading unit is used to read data of the target type from the data stream in the first time window, and to parse the fields of the read target type data to obtain the initial parsed data;

The filtering unit is used to filter multiple fields of information in the initial parsed data based on the filtering mechanism to obtain filtered parsed data;

The association unit is used to add association field information to the filtered and parsed data based on the vocabulary association mechanism to obtain sampled business data.

The target sampling unit is used to sample the sampling service data at a first sampling rate to obtain target sampling data.

Specifically, the sampling module is also used for:

Within the first time window, the target type of data is sampled using the first thread at the first sampling rate to obtain the target sampled data;

The merging module is also specifically used for:

The target sampling data and historical sampling data in the sampling database are merged by a second thread to obtain merged sampling data.

This application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method as described in one aspect of this application.

This application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the above-described aspect.

This application can sample target type data at a first sampling rate within a first time window to obtain target sampled data; merge the target sampled data and historical sampled data in the sampling database to obtain merged sampled data. The third sampling rate of the merged sampled data is determined based on the first sampling rate and the second sampling rate of the historical sampled data. The historical sampled data refers to the target type sampled data obtained before obtaining the target sampled data. When the number of merged sampled data exceeds a sampling quantity threshold, the merged sampled data is sampled based on adaptive sampling parameters to obtain updated historical sampled data. The sampling quantity threshold is related to the target type data, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling quantity threshold. The historical sampled data in the sampling database is then replaced with the updated historical sampled data. Therefore, the method proposed in this application can further sample the data when the number of merged sampled data exceeds the sampling quantity threshold. This ensures that the number of sampled data stored in the sampling database is always controlled within the sampling quantity threshold range during or after sampling, thereby saving system capacity and achieving adaptive adjustment of the sampling rate.

Attached Figure Description

To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

Figure 1 is a schematic diagram of a system architecture provided in this application;

Figure 2a is a schematic diagram of a data sampling scenario provided in this application;

Figure 2b is a schematic diagram of a thread sampling scenario provided in this application;

Figure 3 is a flowchart illustrating a data sampling method provided in this application;

Figure 4 is a schematic diagram of a time window scenario provided in this application;

Figure 5 is a schematic diagram of a scenario for obtaining merged sampled data provided in this application;

Figure 6 is a schematic diagram of a process for obtaining updated historical sampling data provided in this application;

Figure 7 is a schematic diagram of another process for obtaining updated historical sampling data provided in this application;

Figure 8 is a schematic diagram of another data sampling scenario provided in this application;

Figure 9 is a schematic diagram of the structure of a data sampling device provided in this application;

Figure 10 is a structural schematic diagram of a computer device provided in this application.

Detailed Implementation

The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

Please refer to Figure 1, which is a schematic diagram of a system architecture provided in this application. As shown in Figure 1, the system architecture includes device cluster 400a, server 100, and device cluster 400b. Device cluster 400a may include multiple terminal devices, specifically terminal devices 200a, 200b, and 200c. These terminal devices may be mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), or wearable devices (e.g., smartwatches, smart bracelets). Device cluster 400b may include multiple sampling devices, which may be servers; therefore, these multiple sampling devices specifically include servers 300a, 300b, and 300c. The terminal devices in device cluster 400a can generate data for sampling, and server 100 can store the data generated by the terminal devices in device cluster 400a. The sampling devices in device cluster 400b can obtain the data to be sampled from server 100 and perform sampling on the obtained data.

Please refer to Figure 2a, which is a schematic diagram of a data sampling scenario provided in this application. As shown in Figure 2a, the device cluster 106a includes multiple terminal devices (device cluster 106a is equivalent to device cluster 400a in Figure 1 above). These terminal devices can all be used to respond to user operation commands and generate corresponding data. For example, a terminal device can respond to a user's operation command to place an order for a certain product and generate corresponding order information. This order information is the corresponding data generated by the terminal device based on the user's operation command. The data generated by the terminal device can not only be one type of data (such as order type data), but can also include multiple types of data, such as click records of a product being clicked by the user. Therefore, it can be understood that a large amount of data can be obtained through the multiple terminal devices in the device cluster 106a, and this large amount of data can include multiple types of data. The data generated by the aforementioned terminal device cluster 106a can be data from data set 100a. Data set 100a includes multiple data entries, specifically n data entries. The value of n is determined based on the actual application scenario and is not limited thereto; for example, n could be 10,000 or 100 million. A single data entry in data set 100a can refer to a single order information entry generated above, or it could refer to a click record of a product being clicked by a user. Data set 100a can be stored in server 100 as shown in Figure 1.

This application primarily provides a data sampling method. The sampled data can be generated by multiple terminal devices in the aforementioned device cluster 106a. In other words, this application uses sampling n data items in a dataset 100a as an example to illustrate the method provided. Multiple threads can be used to simultaneously read and sample the n data items in the dataset 100a. It is understood that these multiple threads can be threads within sampling devices, and multiple sampling devices can be provided to read and sample the data in the dataset 100a. The sampling device can be a server, and a single sampling device can also include multiple threads. Therefore, the sampling results obtained by the multiple threads in all sampling devices need to be merged. Specifically, firstly, for a single sampling device, the sampling results obtained by its multiple threads need to be merged to obtain the sampling result corresponding to that sampling device. Next, the sampling results corresponding to each sampling device need to be merged (each sampling device can be understood as a higher-level thread), ultimately obtaining the sampling result corresponding to all data in the dataset 100a. Here, we will use three threads (as shown in Figure 2a, threads s1, s2, and s3) to read and sample data from dataset 100a as an example. These three threads can be three different threads within the same sampling device, or threads located in different sampling devices. For example, these three threads could be threads within the sampling devices of device cluster 400b in Figure 1. In practice, more threads (from multiple sampling devices) can be used to read and sample data from dataset 100a. The number of threads and the number of sampling devices depend on the actual application scenario and are not limited thereto.

Each of threads s1, s2, and s3 includes multiple data pools, each used to store different types of data sampled by the thread. Specifically, thread s1 includes data pools a1, b1, and c1; thread s2 includes data pools a2, b2, and c2; and thread s3 includes data pools a3, b3, and c3. It should be noted that data pools a1, a2, and a3 store data of the same type sampled from data set 100a. Data pools b1, b2, and b3 also store data of the same type sampled from data set 100a. Data pools c1, c2, and c3 also store data of the same type sampled from data set 100a. Assume that data pools a1, a2, and a3 are used to store data of the first type, data pools b1, b2, and b3 are used to store data of the second type, and data pools c1, c2, and c3 are used to store data of the third type. Threads s1, s2, and s3 can all simultaneously read data from data set 100a, and sample the read data during the reading process, i.e., they sample while reading.

It should be noted that the data in the aforementioned data set 100a can be a data stream. That is, during the process of threads 1, 2, and 3 reading data from data set 100a and sampling the read data, data generated by the device cluster 106a can be continuously added to data set 100a. In other words, the sampling method provided in this application can be real-time sampling; that is, the sampled data in this application can be data generated in real-time and read and sampled in real-time by multiple threads. Each piece of data in data set 100a can include multiple fields. Each thread (including threads s1, s2, and s3) can identify and classify the read data using the multiple fields corresponding to each piece of data. For example, each thread can classify the read data into income type data, age type data, click type data, or order type data, etc. Here, it is assumed that there are three types of data: the first type (e.g., income type data), the second type (e.g., age type data), and the third type (e.g., order type data). Thread 1 can sample the first type of data read, and the sampling result is the first type of data ultimately stored in data pool a1, let's call it sampling result A1. Thread 1 can sample the second type of data read, and the sampling result is the second type of data ultimately stored in data pool b1, let's call it sampling result B1. Thread 1 can sample the third type of data read, and the sampling result is the third type of data ultimately stored in data pool c1, let's call it sampling result C1. Thread 2 can sample the first type of data read, and the sampling result is the first type of data ultimately stored in data pool a2, let's call it sampling result A2. Thread 2 can sample the second type of data read, and the sampling result is the second type of data ultimately stored in data pool b2, let's call it sampling result B2. Thread 2 can sample the third type of data read, and the sampling result is the third type of data ultimately stored in data pool c2, let's call it sampling result C2. Thread 3 can sample the first type of data read, and the sampling result is the first type of data ultimately stored in data pool a3, let's call it sampling result A3. Thread 3 can sample the second type of data read, and the sampling result is the second type of data ultimately stored in data pool b3, let's call it sampling result B3. Thread 3 can sample the third type of data read, and the sampling result is the third type of data ultimately stored in data pool c3, let's call it sampling result C3. Therefore, it can be understood that each type of data corresponds to a sampling result, and the sampling process between different types of data is independent of each other. The sampling result for the first type of data is the result obtained by merging the above sampling results A1, A2, and A3, as shown in sampling result 101a in Figure 2a; the sampling result for the second type of data is the result obtained by merging the above sampling results B1, B2, and B3, as shown in sampling result 102a in Figure 2a; and the sampling result for the third type of data is the result obtained by merging the above sampling results C1, C2, and C3, as shown in sampling result 103a in Figure 2a. The sampling results 101a for the first type of data, 102a for the second type of data, and 103a for the third type of data can be collectively referred to as sampling result 104a obtained by sampling the data in the above data set 100a.

Sampling result 104a includes sampling results 101a, 102a, and 103a. As shown in Figure 2a, in sampling result 104a, sampling result 101a specifically obtained three data points of the first type (first type data 1, first type data 2, and first type data 3), and these three data points of the first type were sampled at a sampling rate of 1/24 (i.e., the probability of each data point being sampled and retained is 1/24). Sampling result 102a specifically obtained four data points of the second type (second type data 1, second type data 2, second type data 3, and second type data 4), and these four data points of the second type were sampled at a sampling rate of 1/8 (i.e., the probability of each data point being sampled and retained is 1/8). Sampling result 103a specifically involves sampling five data points of the third type (namely, third type data 1, third type data 2, third type data 3, third type data 4, and third type data 5), obtained at a sampling rate of 1/4 (i.e., each data point has a 1/4 probability of being sampled and remaining). The sampling rate for each data type is adaptively adjusted and changed during the sampling process based on the corresponding sampling quantity threshold. The adjustment process of the sampling rate can be seen in the embodiment corresponding to Figure 2b below. It should be noted that in practical applications, sampling results 101a, 102a, and 103a may include more sampled data (e.g., 100,000 data points, 1 million data points, etc.). Here, only three, four, and five data points are used as examples. The data features of the three first-type data included in the sampling result 101a obtained above can be used to characterize the data features of all first-type data in the dataset 100a; the data features of the four second-type data included in the sampling result 102a obtained above can be used to characterize the data features of all second-type data in the dataset 100a; and the data features of the five third-type data included in the sampling result 103a obtained above can be used to characterize the data features of all third-type data in the dataset 100a.

For example, assuming the first type of data mentioned above represents income data for each citizen, when it is necessary to statistically analyze the per capita income of citizens in each region (here, we assume it includes region 1, region 2, and region 3), we can perform statistical calculations on the first type of data in the sampling result 101a (i.e., calculate the per capita income of citizens in each region), resulting in a per capita income of 3000 yuan for region 1, 6000 yuan for region 2, and 8000 yuan for region 3. The process of performing statistical calculations on the first type of data in sampling result 101a can be executed by a separate device specifically designed for statistical calculations, or it can be executed by the sampling device containing any of the threads s1, s2, and s3 mentioned above. The device performing the statistical calculations depends on the actual application scenario and is not restricted in this regard. After statistical calculations are performed on the first type of data in sampling result 101a to obtain the calculation results (i.e., per capita income of 3000 yuan for region 1, 6000 yuan for region 2, and 8000 yuan for region 3), the device that obtained the calculation results can send them to user terminal 105a. User terminal 105a can display the calculation results on its terminal page. User terminal 105a can be a terminal held by an administrator or a terminal containing an administrator account. The administrator can be an authenticated user with data management permissions; for example, the administrator can be a senior manager of a government agency who can access citizens' income information. As shown in Figure 2a, the calculation results displayed on user terminal 105a are: "Statistical analysis results of income distribution: Region 1: per capita income 3000 yuan; Region 2: per capita income 6000 yuan; Region 3: per capita income 8000 yuan."

In real-world applications, the total amount of data collected for each type is typically enormous. Therefore, the above method can be used to sample each type of data first. Later, when statistical calculations are needed on the data features of each type of data in dataset 100a, the data features of each sampled type can be directly calculated. This method reduces the amount of data involved in the calculation, thus reducing the computational load and complexity of calculating the data features for each type of data. Furthermore, because the sampling process for each type of data ensures equal probability of sampling each data point, the data features of the sampled data can be used to characterize the data features of all data.

The method provided in this application allows setting a sampling quantity threshold for each type of sampled data. This threshold can be understood as the capacity of each data pool. Therefore, each data pool can sample the read data according to its corresponding sampling quantity threshold, ensuring that the final number of sampled data in each data pool does not exceed the corresponding sampling quantity threshold. Specifically, the sampling quantity thresholds for different data pools in multiple threads corresponding to a type of sampled data are the same. For example, setting the sampling quantity threshold for the first type of data to 50 indicates that the final number of first type of data to be sampled is less than or equal to 50. Therefore, the sampling quantity thresholds for data pools a1, a2, and a3 are all 50. After merging the sampling results A1 (for data pool a1), A2 (for data pool a2), and A3 (for data pool a3), the final total number of first type of data will also be less than or equal to 50. For example, setting the sampling threshold for the second type of data to 100 entries means that the final number of second-type data samples is less than or equal to 100. Therefore, the sampling threshold for data pools b1, b2, and b3 is also 100. Then, after merging the sampling results B1 (for data pool b1), B2 (for data pool b2), and B3 (for data pool b3), the total number of second-type data obtained will also be less than or equal to 100. Similarly, setting the sampling threshold for the third type of data to 150 entries means that the final number of third-type data samples is less than or equal to 150. Therefore, the sampling threshold for data pools c1, c2, and c3 is also 150. Then, after merging the sampling results C1 (for data pool c1), C2 (for data pool c2), and C3 (for data pool c3), the total number of third-type data obtained will also be less than or equal to 150. For details on how the above threads sample each type of data, and the merging process between sampling results of the same type of data between different threads, please refer to the description in the embodiment corresponding to Figure 2b below.

Please refer to Figure 2b, which is a schematic diagram of a thread sampling scenario provided in this application. The embodiment described in Figure 2b is a process of how a thread samples a certain type of data it reads. In other words, the embodiment described in Figure 2b is a specific explanation of the sampling principle corresponding to a data pool (corresponding to the sampling of a certain type of data) in a thread. It can be understood that the sampling principle corresponding to each data pool (used by different threads to store different types of data pools) is the same and independent. As shown in Figure 2b, data pool 100g is assumed to be any data pool in a certain thread of a sampling device used to sample data (for example, data pool 100g can be any one of data pools a1, a2, a3, b1, b2, b3, c1, c2, and c3 in Figure 2a). Here, it is assumed that data pool 100g is data pool b3 in the aforementioned thread s3. Data pool 100g can store the second type of data read and sampled by thread s3 from the data layer. This can be understood as the data set 100a in Figure 2a being stored in the data layer. Thread s3 can simultaneously read and sample data from the data layer. This means that thread s3 continuously reads data from the data layer and stores it in data pool 100g. The data pool can filter the stored data (this filtering process is the sampling process). During filtering, some data is retained, while some is discarded. In other words, the method provided in this application does not require reading all the data before sampling; instead, it can sample while reading (based on a set sampling data threshold), i.e., filtering while reading. This limits the amount of data stored in the system to within the sampling data threshold, thus saving system capacity and improving sampling efficiency.

As shown in Figure 2b, assuming the sampling threshold for data pool 100g is 5, initially, thread s3 will read data from the data layer with a probability of "1". Before the number of read data exceeds the sampling threshold of 5, thread s3 can store all the read data (here referring to a specific type of data, as one data pool corresponds to storing one type of data) in data pool 100g. That is, when thread s3 reads data from the 1st to the 5th data item, it can store all 5 read data items in data pool 100g. At this time, the data stored in data pool 100g can be 5 data items from data set 101g. When thread s3 reads the 6th data item, the data stored in data pool 100g can be 6 data items from data set 102g. At this point, since the sampling threshold for data pool 100g is 5, and the number of stored data items is 6, exceeding the sampling threshold of 5, it is necessary to sample (i.e., filter) these 6 stored data items. By sampling these 6 data items, some of the data can be discarded. These 6 data points can be sampled at a sampling rate of 1/2. After sampling, the data stored in data pool 100g can be 3 data points from data set 103g.

Next, when reading new data, it is read at a sampling rate of 1 (i.e., no sampling is performed on the new data). Since the newly read data needs to be added to data pool 100g, that is, the newly read data (i.e., assuming that after obtaining data set 103g, the newly read data is data 105g) needs to be merged with the existing data set 103g in data pool 100g, the sampling rate between the data needs to be unified to ensure that each data point is sampled with equal probability. Specifically, since the data in data set 103g has already been sampled at a sampling rate of 1/2, data 105g also needs to be sampled at a sampling rate of 1/2. The process of sampling the data in data set 102g and data 105g at a sampling rate of 1/2 can be found in the description of step S102 in the embodiment corresponding to Figure 3 below. Suppose that after sampling data 105g at a sampling rate of 1/2, we obtain data set 104g. This means that the data stored in data pool 100g here is the data from data set 104g. In other words, data 105g was stored in addition to the data from data set 103g. At this point, all the data in data set 104g was obtained by sampling at a sampling rate of 1/2.

Next, thread s3 will continuously read data, and for each data point it reads, it will unify the sampling rate with the data in dataset 104g. That is, thread s3 will sample each data point at a sampling rate of 1/2. During this sampling rate unification process, some data read and sampled by thread s3 will be added to data pool 100g, while some data read and sampled by thread s3 will be filtered (discarded). When the data in data pool 100g again exceeds the sampling threshold of 5, the data stored in data pool 100g can then be the 6 data points from dataset 106g. Since all 6 data points in dataset 106g were sampled at a sampling rate of 1/2, thread s3 can again sample the data in dataset 106g at the 1/2 sampling rate. After sampling the data in dataset 106g at a sampling rate of 1/2, dataset 107g is obtained. At this point, the data stored in data pool 100g consists of three data points from dataset 107g, which were sampled from dataset 106g at a sampling rate of 1/2. It can be understood that the data in dataset 107g were all sampled at a sampling rate of 1/4 (because it underwent two sampling operations at a sampling rate of 1/2).

Subsequently, thread s3 can continue to read new data. Similarly, thread s3 will unify the sampling rate of each new data read with the data in data set 107g; that is, thread s3 will sample each new data read with a 1/4 probability. During this unification process, data pool 100g will retain some new data read and sampled by thread s3, while discarding some. When the number of data stored in data pool 100g exceeds the sampling threshold again (i.e., when the number of data stored in data pool 100g reaches 6), at this time, all 6 data stored in data pool 100g are sampled at a 1/4 sampling rate. Thread s3 can then sample the data stored in data pool 100g again at a 1/2 sampling rate. After sampling, the number of data stored in data pool 100g can again be 3, and these 3 data are sampled at a 1/8 sampling rate (because they were sampled 3 times at a 1/2 sampling rate). Afterwards, thread s3 can continuously read new data and repeat the steps described above on the new data to obtain the sampling results.

Finally, when thread s3 stops reading new data (i.e., sampling is complete), data pool 100g can store no more than the sampling quantity threshold of 6, thus obtaining the final sampling result. For example, the data stored in the final data pool 100g can be 5 data items from data set 108g. The sampling rate corresponding to these 5 data items is determined by the quantity of that type of data stored in data pool 100g read by thread s3 and the sampling quantity threshold corresponding to data pool 100g. The sampling result corresponding to data pool 100g is the 5 data items contained in data set 108g. Using the same process described above, we can obtain the sampling results A1 corresponding to data pool a1, B1 corresponding to data pool b1, C1 corresponding to data pool c1, A2 corresponding to data pool a2, B2 corresponding to data pool b2, C2 corresponding to data pool c2, A3 corresponding to data pool a3, B3 corresponding to data pool b3, and C3 corresponding to data pool c3 in Figure 2a.

When merging the sampling results obtained from multiple threads, the merging principle is the same as that for thread s3. Specifically, when multiple threads exist, the sampling results obtained from each thread can be merged pairwise (merging results corresponding to the same data type). For example, when there are threads 1, 2, 3, and 4, the sampling results of thread 1 and thread 2 can be merged to obtain merged result 1; the sampling results of thread 3 and thread 4 can be merged to obtain merged result 2. Then, merged result 1 and merged result 2 can be merged to obtain the total sampling result for the four threads (including threads 1, 2, 3, and 4). Optionally, the merging method can also be as follows: the sampling results of thread 1 and thread 2 can be merged to obtain merged result 1. Merged result 1 can be merged with the sampling result of thread 3 to obtain merged result 3. Merged result 3 can be merged with the sampling result of thread 4 to obtain merged result 4. This merged result 4 is the total sampling result for the four threads.

For example, when merging the sampling results of thread 1 and thread 2, the sampling rates of the two results first need to be unified. Assuming the sampling rate for multiple data points in thread 1's sampling result is 1/8, and the sampling rate for multiple data points in thread 2's sampling result is 1/16, then the data points in thread 1's sampling result need to be sampled again at a sampling rate of 1/2, so that the sampling rates of both thread 1 and thread 2's results are unified to 1/16 (because 1/8 * 1/2 = 1/16). After unifying the sampling rate for multiple data points in thread 1's sampling result to 1/16, some data points in thread 1's sampling result will be discarded. The sampling result of thread 1 with a unified sampling rate of 1/16 is called the unified sampling result. This unified sampling result can be directly merged with the sampling result of thread 2 to obtain the merged sampling result. The merged sampling result includes multiple data points (specifically, data from the unified sampling result and data from the sampling result of thread 2), all with a sampling rate of 1/16. Next, the number of data points in the merged sampling result is determined. If the number of data points exceeds the sampling threshold, the data in the merged sampling result needs to be sampled again at a sampling rate of 1/2. After sampling, the final merged result for thread 1 and thread 2 is obtained. It can be understood that the sampling rate for the data in this merged result is 1/32 (i.e., 1/16 * 1/2 = 1/32). If the number of data points in the above merged sampling result does not exceed the sampling threshold, this merged sampling result can be directly used as the final merged result for thread 1 and thread 2. When merging sampling results from multiple threads, the merging principle is the same as above: first, the sampling rate is unified; after unification, it is determined whether the number of data points to be merged exceeds the sampling threshold. If it does, sampling is performed again with a probability of 1/2 to obtain the final sampling result, ensuring that the sampling rates for multiple data points in the final merged result are the same. If the number of data points does not exceed the limit, the merged data can be used directly as the final merge result.

The method provided in this application supports parallel sampling of the read data during the reading of large amounts of data, i.e., filtering while reading. This allows the amount of data stored in the data pool to be controlled within a sampling threshold, saving system capacity and improving sampling efficiency. Simultaneously, during data sampling, the sampling rate of the read data can be adaptively adjusted according to the sampling threshold corresponding to the data pool, making it suitable for sampling data in the form of data streams. That is, even when the specific amount of data to be read is unknown, the sampling rate of the read data can be adaptively adjusted according to the final amount of data to be sampled, ensuring that each piece of data is sampled with equal probability (meaning that data of the same type is sampled with the same sampling rate, and different types of data can correspond to different sampling rates), thus improving sampling accuracy.

Please refer to Figure 3, which is a flowchart illustrating a data sampling method provided in this application. As shown in Figure 3, the method may include:

Step S101: Sample the target type data at a first sampling rate in the first time window to obtain target sampled data;

Specifically, the data sampling process involved in this application needs to be explained first. The data to be sampled in this application can be arbitrary data, and multiple sampling devices can be used to read and sample the data. Each sampling device can also include multiple threads. Therefore, the final sampling result is as follows: First, each sampling device needs to merge the sampling results corresponding to its multiple threads to obtain the sampling result corresponding to each sampling device. Second, the sampling results corresponding to each sampling device need to be merged to obtain the final sampling result. The execution subject in this embodiment can be a sampling system, which includes multiple threads (which can be multiple threads corresponding to multiple sampling devices). The method provided in this application can be applied to scenarios where data in a data stream (which can be understood as data being generated in real time, and the process of sampling the data also being performed in real time) is sampled. In this scenario, there can be multiple time windows, and each time window can correspond to a sampling result. The sampling result corresponding to a certain time window is the result obtained by sampling all the data generated in that time window.

For example, suppose we need to read and sample data generated one hour ago (or 24 hours, or other set durations), and sample every 10 minutes to obtain a sampling result. Then, we can understand that each time window corresponds to a window time of one hour (60 minutes), the start time of two adjacent time windows differs by 10 minutes, and the end time of two adjacent time windows also differs by 10 minutes. Please refer to Figure 4, which is a schematic diagram of a time window scenario provided in this application. As shown in Figure 4, the width of the time window can be understood as the length of the time period corresponding to the time window. Since the length of the time period corresponding to the time window (e.g., one hour) is usually longer than the time difference between the start times of two adjacent time windows (e.g., 10 minutes), there is usually an intersection time window between two adjacent time windows. This intersection time window refers to the time window corresponding to the same time period in the two time windows. As shown in Figure 4, assuming each time window is 1 hour long, the lengths of time windows 100f, 101f, 102f, 103f, and 104f all represent time periods of 1 hour. Time windows 100f and 101f are two adjacent time windows, and they have an intersection time window 1 (the intersection time window described in this application refers to the intersection of the later time window with the earlier time window). Time window 100f is the first time window; therefore, there are no other time windows before it. The purpose of time window 100f is to read and sample data generated within the hour preceding the current time. For example, if reading starts at 13:00, the data read and sampled in time window 100f is the data generated between 12:00 and 13:00. Since each piece of data has a corresponding data generation timestamp, which represents the generation time of each piece of data, when reading data in each time window, the data generated in the corresponding time window can be read according to the data generation timestamp of each piece of data.

The data within each time window can be read and sampled simultaneously and in parallel using multiple threads (which can reside on multiple sampling devices). Time window 101f also includes time windows that do not intersect with time window 100f, i.e., non-intersecting time windows 1. The sum of intersecting time windows 1 and non-intersecting time windows 1 equals time window 101f. Intersecting time windows 1 can be referred to as the intersecting time window corresponding to time window 101f, and non-intersecting time windows 1 can be referred to as the non-intersecting time window corresponding to time window 101f. Similarly, time window 101f and time window 102f also have an intersecting time window 2. Time window 102f also includes time windows that do not intersect with time window 101f, i.e., non-intersecting time windows 2. The sum of intersecting time windows 2 and non-intersecting time windows 2 equals time window 102f. Intersecting time windows 2 can be referred to as the intersecting time window corresponding to time window 102f, and non-intersecting time windows 2 can be referred to as the non-intersecting time window corresponding to time window 102f. Time windows 102f and 103f also intersect with time window 3. Time window 103f also includes time windows that do not intersect with time window 102f, i.e., non-intersecting time windows 3. The sum of intersecting time windows 3 and non-intersecting time windows 3 gives time window 103f. Intersecting time windows 3 can be called the intersecting time windows corresponding to time window 103f, and non-intersecting time windows 3 can be called the non-intersecting time windows corresponding to time window 103f. Similarly, time windows 103f and 104f also intersect with time window 4. Time window 104f also includes time windows that do not intersect with time window 103f, i.e., non-intersecting time windows 4. The sum of intersecting time windows 4 and non-intersecting time windows 4 gives time window 104f. Intersecting time windows 4 can be called the intersecting time windows corresponding to time window 104f, and non-intersecting time windows 4 can be called the non-intersecting time windows corresponding to time window 104f.

Since the sampling process for each time window is sequential, the sampling of data in the next time window is only performed after the sampling results of the previous time window have been obtained. Therefore, data whose generation timestamps fall within the intersection time window with the next time window can be directly used as part of the sampling results for the next time window. Thus, the next time window only needs to process new data generated within time windows that do not intersect with the previous time window (i.e., non-intersection time windows), without needing to re-read and sample data generated within the intersection time windows. This improves the efficiency of data sampling within a time window. For example, the sampling results for time window 100f are obtained by reading and sampling all data from the current time to the previous hour. After reading and sampling the data generated in time window 100f to obtain the corresponding sampling results, the sampling process for time window 101f can then proceed. Assuming there is a 10-minute difference between the start times of time window 100f and time window 101f, when performing the sampling process corresponding to time window 101f, it is only necessary to read and sample the new data generated in the non-intersecting time window 1 (10 minutes). After reading and sampling the data generated in the non-intersecting time window 1, the data whose data generation timestamps are in the intersecting time window 1 (50 minutes) can be directly taken from the sampling results corresponding to time window 100f and merged with the sampling results obtained after reading and sampling the data generated in the non-intersecting time window 1 to obtain the sampling result corresponding to time window 101f.

Since multiple threads read and sample data in parallel within each time window, each thread reads and samples the data generated in that time window during the sampling process (the data read by each thread will be different, and the amount of data read and sampled by each thread will also vary; as mentioned above, the data read here is the new data generated in the non-intersecting time window corresponding to the previous time window). Finally, the sampling result corresponding to that time window is obtained by merging the sampling results of each thread and merging the sampled data obtained in the intersecting time window corresponding to the previous time window.

In this process, the data reading and sampling processes are identical for each thread. Therefore, this explanation uses the first thread in the sampling system as an example to illustrate how data is read and sampled. The first thread can exist in any sampling device and is used for reading and sampling data. The data sampled by the first thread can be stored in a sampling database. Furthermore, the process of reading and sampling different types of data is the same for each thread. Therefore, this explanation uses the first thread sampling data of the target type (which can be any type of data being sampled) as an example. The first thread can also have multiple sampling databases; one sampling database corresponds to one type of sampled data, meaning one sampling database stores the sampled data obtained from a specific type of sampling. Therefore, the target type of data also corresponds to a sampling database, which stores the sampled data obtained by sampling the target type of data. The sampling databases used in the following processes all refer to the sampling database corresponding to the target type of data.

Specifically, data already existing in the sampling database before the target sampling data is acquired can be referred to as historical sampling data. This historical sampling data can be the sampling data obtained by the first thread after sampling the target type of data before acquiring the target sampling data. This historical sampling data can be obtained by the first thread itself or sent to it by other threads. Furthermore, the historical sampling data in the sampling database can also be empty.

The process of acquiring target transaction data can be as follows: Each sampled data item can include multiple fields. After the first thread reads the original business data (including target type data and other types of data) from the data stream (which can be in the underlying data layer), it first performs field parsing on the original business data. After parsing, multiple fields of each piece of original business data can be obtained. This parsed original business data can be called the initial parsed data (including the initial parsed data corresponding to the target type data). The first thread can filter the multiple fields in the initial parsed data through a filtering mechanism. That is, filtering rules can be preset, indicating which fields in the initial parsed data need to be retained and which fields need to be filtered out (i.e., deleted). This filtering operation is optional; multiple fields in the initial parsed data can be filtered or not, retaining all fields. The data obtained after filtering the initial parsed data can be called the filtered parsed data (including the filtered parsed data corresponding to the target type data). Furthermore, related field information can be added to the filtered parsed data through a thesaurus association mechanism. This means that the thesaurus to be associated can be pre-defined, and the field information from that thesaurus can be added to the filtered parsed data. For example, if the filtered parsed data includes personal income information, then through thesaurus association, fields such as personal birthplace can be added. Another example is when the filtered parsed data includes personal marital status information; then through thesaurus association, fields such as parents' names can be added. This thesaurus association operation is optional; additional fields can be added to the filtered parsed data, or not. The filtered parsed data after the thesaurus association operation can be called sampled business data (including sampled business data corresponding to the target type of data). This sampled business data is the data processed by the first thread from the data layer after reading the original business data. Subsequently, sampling operations can be performed on this sampled business data.

The first thread can identify the data type of the read data by sampling multiple fields in the business data, thus identifying the target type of sampled business data. The first thread can sample the identified target type of data to obtain target sampled data. This target sampled data can be obtained in real-time within a first time window, which can be any time window during the sampling process. In the first thread, each type of data (including the target type of data) corresponds to a sampling result. The sampled data obtained from this sampling result is the data ultimately stored in the sampling database corresponding to each type of data. The aforementioned target sampled data can also be obtained by the first thread sampling the read data at a first sampling rate (which may vary depending on the actual sampling scenario). For example, when new data is read, the target sampled data could also be the new data read by the first thread at a sampling rate of 1 (i.e., a first sampling rate of 1).

This application may include two sampling scenarios, in which the method of acquiring the target sampled data differs. The first sampling scenario involves each thread reading and sampling data of the target type within a specific time window. For example, taking the process of the first thread sampling the target type data (which may be raw data read from the data stream) at a first sampling rate within the first time window to obtain the target sampled data: the first thread will read and sample data from the data layer simultaneously. At this stage, a time point t1 can be introduced. The historical sampled data in the sampling database can be the target type sampled data obtained by the first thread before time point t1. After time point t1, the first thread can continue to read new data, which can be referred to as the target sampled data (in this special case, the first sampling rate of the target sampled data is 1, meaning the target sampled data is target type sampled data read with a probability of 1). Since the first thread continuously reads new data in the first sampling scenario, there are historical sampling data and target sampling data at each moment when the first thread reads new data (e.g., time t2). The historical sampling data is the target type data sampled before time t2, and the target sampling data is the new target type sampling data read after time t2.

The second sampling scenario described above is illustrated using a first and third time window as an example. The first and third time windows are two adjacent time windows, with the first and second time windows having the same window size. The first and third time windows have an intersection window; the third time window is the preceding time window of the first time window. In other words, the start time of the first time window is later than the start time of the third time window, and the end time of the first time window is also later than the end time of the third time window. For example, if a sampling operation is performed every 10 minutes, then the time difference between two adjacent time windows can be 10 minutes. That is, the time difference between the start time of the first time window and the start time of the second time window can be 10 minutes, and the time difference between the end time of the first time window and the end time of the second time window can also be 10 minutes.

In this sampling scenario, the historical sampling data in the sampling database can be the sampling data obtained by real-time sampling of the target type data within the intersection time window between the third time window and the first time window during the third time window. In other words, the historical sampling data is the sampling data sampled within the third time window, with the generation time within the intersection time window of the first and third time windows. The target sampling data, on the other hand, is the data of the target type sampled at the first sampling rate within the first time window, excluding the intersection time window with the third time window.

Step S102: Merge the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data. The third sampling rate of the merged sampling data is determined based on the first sampling rate and the second sampling rate of the historical sampling data. The historical sampling data is the target type sampling data obtained by sampling the target type data before obtaining the target sampling data. The second sampling rate is the sampling rate of the historical sampling data relative to the target type data. The third sampling rate is the sampling rate of the merged sampling data relative to the target type data.

Specifically, the first thread can merge the target sampled data and the historical sampled data in the sample database to obtain merged sampled data. Specifically, when the historical sampled data and the target sampled data are sampled data from the first sampling scenario described above, the historical sampled data is obtained by the first thread sampling the target type data at a certain sampling rate. Assuming the historical sampled data has a second sampling rate, this second sampling rate is the sampling rate of the historical sampled data for the target type data; in other words, this second sampling rate is the sampling rate for the original business data (unsampled) corresponding to the historical sampled data. The target sampled data has a first sampling rate of 1. In fact, in this sampling scenario, the target sampled data has not yet been sampled; it has only been read. Therefore, the first thread needs to unify the sampling rates of the target sampled data and the historical sampled data to ensure that the target type data is sampled at the same sampling rate. In this sampling phase, the first sampling rate will definitely be greater than or equal to the second sampling rate (in this application, the range of the sampling rate is 0-1 for illustration). Therefore, the ratio between the second sampling rate and the first sampling rate can be obtained (this ratio can be called the first ratio sampling rate; the range of the first ratio sampling rate is also between 0 and 1, and this ratio is obtained by using the smaller sampling rate as the dividend and the larger sampling rate as the divisor). The target sampling data can be sampled using this ratio, and the sampled target sampling data will have the same sampling rate as the current historical sampling data. The sampled target sampling data and the current historical sampling data can be directly merged to obtain merged sampling data. The sampling rate of the merged sampling data can be called the third sampling rate; therefore, at this time, the third sampling rate of the merged sampling data is equal to the second sampling rate. The third sampling rate is also relative to the sampling rate of the target type of data; that is, the third sampling rate is the sampling rate for the original business data (which has not been sampled) corresponding to the merged sampling data.

In this application, the sampling rate can be any power of 1/2 (explained in step S103 below), for example, the sampling rate can be 1, ... For example, if the first sampling rate is 1/4, then the first ratio sampling rate is 1/4 divided by 1, which equals 1/4. The target sampled data can be sampled at a sampling rate of 1/4. Specifically, the sampling method is as follows: since each data point participating in the sampling has a data identifier (i.e., data ID), which can be a string composed of letters and numbers, the data identifier of each data point can be called a data identifier string. Therefore, each target sampled data point corresponds to a data representation string. Each target transaction data point can be mapped to a uniform sampling space using its data identifier string. In this uniform sampling space, each target transaction data point can be sampled at the same sampling rate (here, 1/4). Mapping each target transaction data point to the uniform sampling space can be achieved by calculating the hash value corresponding to the data identifier string of each target transaction data point. The target sampling data is sampled at a sampling rate of 1/4. This means that the hash value corresponding to each target sampling data point is divided by 4. If the division is exact (i.e., the remainder is 0), the corresponding target sampling data is stored as the sampled data. If the division is not exact (i.e., the remainder is not 0), the corresponding target sampling data is discarded. After sampling the target data through this process, the sampled target sampling data is obtained. In this sampling scenario, the sampled target sampling data and historical sampling data can be directly merged to obtain the merged sampling data.

In the first sampling scenario described above, the second sampling rate of historical sampling data can be directly obtained as the first sampling rate (even if the first sampling rate equals the second sampling rate). When reading new target type data, this first sampling rate is used to sample the read target type data to obtain the target sampled data. Therefore, the sampling rate of the historical sampling data and the sampling rate of the target sampled data are now consistent, allowing for direct merging of the historical and target sampled data to obtain merged sampled data. This method improves the efficiency of merging target and historical sampled data.

When the historical and target sampled data fall under the second sampling scenario described above: the historical sampled data was obtained within the intersection of the third and first time windows during the third time window, while the target sampled data was obtained within the first time window, excluding the intersection with the third time window. Similarly, in this second sampling scenario, the target and historical sampled data also need to be merged. First, the sampling rates of the target and historical sampled data need to be unified. In this case, the difference between the first and second sampling rates depends on the actual sampling scenario. When the first sampling rate corresponding to the target sampled data is less than the second sampling rate corresponding to the historical sampled data, the sampling rate of the historical sampled data needs to be unified from the second sampling rate to the first sampling rate. For example, if the first sampling rate is 1/16 and the second sampling rate is 1/4, it means that the historical sampled data was obtained using a sampling rate of 1/4, and the target sampled data was obtained using a sampling rate of 1/16. In theory, the historical sampling data needs to be sampled again at a sampling rate of 1/4 (because the first sampling rate of 1/16 divided by the second sampling rate of 1/4 equals 1/4, and the ratio between the first sampling rate and the second sampling rate can be called the second ratio sampling rate).

Specifically, when sampling historical data using the second sampling rate, the hash value corresponding to the data identifier string of the historical sample data is divided by 4. If the remainder is 0, the historical sample data is stored (the remainder is shown as 0 here because the historical sample data is stored in the sampling database). If the remainder is not 0, the historical sample data is discarded. Since the historical sample data is obtained by sampling at a 1/4 sampling rate, the result of dividing the hash value corresponding to the data identifier string of the historical sample data by 4 can be called the first sampling hash value. If the sampling system stores this first sampling hash value when it obtains the historical sample data, then when sampling the historical sample data again at a 1/4 sampling rate, this first sampling hash value can be divided by 4. If the remainder is 0, the corresponding historical sample data is stored; if the remainder is not 0, the corresponding historical sample data is discarded. If the sampling system does not store the aforementioned first sampling hash value, then the step in step box 100b (i.e., uniform sampling rate) can be executed. This involves directly sampling the historical sampling data using the first sampling rate of 1/16. Specifically, the hash value corresponding to the data identifier string of the historical sampling data is divided by 16. If the remainder is 0, the corresponding historical sampling data is stored; otherwise, it is discarded. Through this process, the sampled historical sampling data can be obtained.

Since there are usually multiple historical sampling data sets, resampling these historical data sets can filter out some of the original historical data. The sampled historical data and the target data can then be directly merged to obtain the merged sampling data.

Similarly, in the second sampling scenario, if the first sampling rate is greater than the second sampling rate, the sampling rate of the target sampled data needs to be unified from the first sampling rate to the second sampling rate. For example, if the first sampling rate is 1/4 and the second sampling rate is 1/16, the target sampled data needs to be sampled again by 1/4. As described above, the process of sampling the target sampled data again by 1/4 can be as follows: divide the hash value corresponding to the data identifier string of the target sampled data directly by 16. If the remainder is 0, store the corresponding target sampled data; if the remainder is not 0, discard the corresponding target sampled data. Alternatively, if, after sampling the target sampled data using the first sampling rate, the hash value corresponding to the data identifier string of the target sampled data is stored divided by 4, then when sampling the target sampled data again by 1/4, this value can be divided by 4 again. If the remainder is 0, store the corresponding target sampled data; if the remainder is not 0, discard the corresponding target sampled data. Since there are usually multiple target sample data points, re-sampling the target sample data at a 1/4 sampling rate can filter out some of the target sample data. This process yields the sampled target sample data. Similarly, the sampled target sample data and historical sample data can be directly merged to obtain merged sample data. When the first sampling rate equals the second sampling rate, the target sample data and merged sample data can be directly merged to obtain merged sample data.

Optionally, in the second sampling scenario, if the number of historical sampled data is less than the merging threshold (which can be set manually, for example, to 10,000 data points), and the ratio between the second sampling rate corresponding to the historical sampled data and the first sampling rate corresponding to the target sampled data (second sampling rate divided by first sampling rate) is less than the ratio threshold (which can be set manually, for example, 1/512), then the historical sampled data can be discarded, and only the target sampled data can be retained. In this case, it is necessary to start accumulating sampled data again from the non-intersecting time window corresponding to the second time window, discarding all previously sampled data. This method can avoid the problem of insufficient data and low sampling rate in the previous time window affecting the sampling process in subsequent time windows, such as causing the number of sampled data to be too small and the sampling rate to be too low.

As can be seen from the above process, the third sampling rate corresponding to the obtained merged sampled data is actually the minimum value between the first sampling rate and the second sampling rate.

Please refer to Figure 5, which is a schematic diagram of a scenario for obtaining merged sampled data provided in this application. As shown in Figure 5, there is an intersection between the first time window 101e and the third time window 100e. The first time window 101e also includes non-intersecting time windows that do not intersect with the third time window 100e. By sampling all the data generated in the third time window 100e, a sampling result 102e is obtained. As shown in Figure 5, the sampling result 102e includes sampled data 1, sampled data 2, sampled data 3, sampled data 4, sampled data 5, and sampled data 6. Each sampled data in the sampling result 102e corresponds to a data generation timestamp, which represents the generation time of the sampled data. Specifically, the data generation timestamp for sample data 1 is data generation timestamp 1, the data generation timestamp for sample data 2 is data generation timestamp 2, the data generation timestamp for sample data 3 is data generation timestamp 3, the data generation timestamp for sample data 4 is data generation timestamp 4, the data generation timestamp for sample data 5 is data generation timestamp 5, and the data generation timestamp for sample data 6 is data generation timestamp 6.

The process of obtaining historical sampling data from sampling result 102e can be as follows: the data generation timestamp 1 corresponding to the above sampling data 1 can be recorded as t1, the data generation timestamp 2 corresponding to the above sampling data 2 can be recorded as t2, the data generation timestamp 3 corresponding to the above sampling data 3 can be recorded as t3, the data generation timestamp 4 corresponding to the above sampling data 4 can be recorded as t4, the data generation timestamp 5 corresponding to the above sampling data 5 can be recorded as t5, and the data generation timestamp 6 corresponding to the above sampling data 6 can be recorded as t6. As shown in the timestamp schematic box 104e in Figure 5, the above data generation timestamps t4 and t6 are within the non-intersecting time window corresponding to the third time window 100e, while the above data generation timestamps t1, t2, t3, and t5 are within the intersecting time window corresponding to the third time window 100e. Since the timestamps of the data generation in the third window of the historical sampling data fall within the intersection time window, the aforementioned sampling data 1, sampling data 2, sampling data 3, and sampling data 5 can be considered as historical sampling data. In other words, the sampling data in dataset 105e is the historical sampling data. Furthermore, the sampling rate corresponding to each historical sampling data in dataset 105e is the second sampling rate; that is, the sampling rates corresponding to sampling data 1, sampling data 2, sampling data 3, and sampling data 5 are the second sampling rate.

More specifically, during the sampling process corresponding to the first time window 101e, the sampling result obtained by sampling the data generated within the non-overlapping time windows can be sampling result 103e. In this case, the sampled data in sampling result 103e can be used as the aforementioned target sampled data. As shown in Figure 5, assume that sampling result 103e includes sampled data 7, sampled data 8, and sampled data 9. Furthermore, the sampling rates corresponding to sampled data 7, sampled data 8, and sampled data 9 are the first sampling rate. If the final sampling result within the first time window 101e is required, the aforementioned target sampled data and historical sampled data need to be merged. During merging, the sampling rates of the target sampled data and historical sampled data need to be unified. As shown in data set 106e in Figure 5, after unifying the sampling rates between the target sampled data and historical sampled data, sampled data 1, sampled data 7, sampled data 8, and sampled data 9 remain. These sampled data 1, sampled data 7, sampled data 8, and sampled data 9 are the merged sampled data. Furthermore, the sampling rates corresponding to sampled data 1, sampled data 7, sampled data 8, and sampled data 9 are the third sampling rate (this third sampling rate can be equal to the first sampling rate mentioned above), that is, the sampling rate of the merged sampled data is the third sampling rate.

It is understandable that the final sampling result for the target type of data obtained within each time window can be obtained by merging the target type of sampled data obtained by multiple threads within each time window. The merging of the sampling results for the target type of data from each thread can be performed by any one or more threads. Therefore, the target transaction data obtained by sampling the target type of data at the first sampling rate in the first time window can be obtained by the first thread. The first thread can then pass the sampled target transaction data to the second thread in the sampling system, which will merge the target sampled data with the historical sampled data to obtain the merged sampled data. Thus, it can be seen that the sampling process is completed collaboratively by multiple threads in the sampling system, ensuring sampling efficiency.

Step S103: When the number of merged sampled data is greater than the sampling number threshold, the merged sampled data is sampled based on the adaptive sampling parameter to obtain updated historical sampled data. The sampling number threshold is related to the target type of data. The adaptive sampling parameter is used to control the number of merged sampled data within the sampling number threshold.

Specifically, when the number of merged sampled data points exceeds the sampling number threshold, the merged sampled data can be sampled again using adaptive sampling parameters to obtain updated historical sampled data. Different types of data can correspond to different sampling number thresholds. The sampling number threshold for a certain type of data indicates that the number of sampled data points of that type obtained in the final sampling will not exceed (be less than or equal to) its corresponding sampling number threshold. Therefore, the target type of data also corresponds to a sampling number threshold.

The aforementioned adaptive sampling parameter is used to control the number of merged sampled data within a sampling quantity threshold. This adaptive sampling parameter can be set to 1/2. The reason is as follows: Let's take the sampling process corresponding to the target type of data as an example. Since the number of sampled data points is controlled within the sampling quantity threshold corresponding to the target type during the sampling process, both the number of target sampled data and the number of historical sampled data will be less than the sampling quantity threshold. Assuming the sampling quantity threshold for the target type of data is M, we can see that after merging the target transaction data and historical sampled data (either merging after sampling or merging directly), the number of merged sampled data points will be less than or equal to twice the sampling quantity threshold (i.e., less than 2M), because the sum of two values less than or equal to M will always be less than or equal to 2M. Therefore, setting the adaptive sampling parameter to 1/2 and then sampling the merged sampled data by 1/2 using this adaptive sampling parameter will result in a merged sampled data point number less than M. In this process, the merged sampled data is sampled again at a sampling rate of 1/2. This means that half of the sampled data in the merged sampled data is filtered out, and half of the sampled data in the merged sampled data is retained to obtain the updated historical sampled data (i.e., the merged sampled data after sampling). As can be seen from the above, the number of updated historical sampled data will definitely be less than the sampling number threshold M.

The sampling rate for updating historical sampled data can be referred to as the fourth sampling rate. This fourth sampling rate is the sampling rate of the updated historical sampled data relative to the target type of data. In other words, it is the sampling rate of the updated historical sampled data relative to the corresponding original business data (which has not been sampled). This fourth sampling rate is equal to the product of the third sampling rate of the merged sampled data and the adaptive sampling parameter. For example, if the third sampling rate is 1/8 and the adaptive sampling parameter is 1/2, then the fourth sampling rate is 1/16. Updated historical sampled data can be obtained by sampling the merged sampled data using the fourth sampling rate. For example, when the fourth sampling rate is 1/16, the hash value corresponding to the data identifier string of the merged sampled data can be divided by 16. If the remainder is 0, the corresponding merged sampled data is retained; if the remainder is not 0, the corresponding merged sampled data is discarded.

Furthermore, the updated historical sampling data obtained above can be the sampling result corresponding to the first time window. If the second time window is the next time window after the first time window, the sampling data obtained from the intersection time window between the first and second time windows in the updated historical sampling data can be used as part of the sampling data in the sampling result corresponding to the second time window. Therefore, for the sampling process within the second time window, the historical sampling data in the second time window can be the sampling data obtained in the first time window within the intersection time window between the first and second time windows. The target sampling data in the second time window can be the target type sampling data obtained in the second time window excluding the intersection time window with the first time window. Therefore, when reading new target type data in the second time window excluding the intersection time window with the first time window, and sampling the read target type data, the fourth sampling rate (the sampling rate of the updated historical sampling data in the first time window) can be directly used as the first sampling rate in the second time window to sample the read new target type data to obtain the target sampling data. In this way, within the second time window, since the target sampled data obtained from the sampling has the same sampling rate as the historical sampled data within the second time window, they can be directly merged to obtain the merged sampled data within the second time window. This improves the merging efficiency of the target and historical sampled data within the second time window, while saving system resources because it eliminates the need to sample the target or historical sampled data before merging. Furthermore, this method achieves adaptive adjustment of the sampling rate. Subsequently, the updated historical sampled data within the second time window can be obtained from the merged sampled data within the second time window; the updated historical sampled data within the second time window can be the sampling results corresponding to the second time window.

Step S104: Replace the historical sampling data in the sampling database with the updated historical sampling data;

Specifically, the historical sampling data in the aforementioned sampling database can be replaced with the updated historical sampling data obtained above. The updated historical sampling data in the sampling database at this point is the final sampling data obtained after merging the target sampling data and the historical sampling data.

If the number of merged sampled data is less than or equal to the sampling quantity threshold, there is no need to sample the merged sampled data again. The historical sampled data in the sampling database can be directly replaced with the merged sampled data. In this case, the merged sampled data in the sampling database is the final sampled data obtained by merging the target sampled data and the historical sampled data. Therefore, the method provided in this application can ensure that the number of sampled data ultimately stored in the sampling database is controlled within the sampling quantity threshold by using the sampling quantity threshold and adaptive sampling parameters. Furthermore, the sampling rate of the sampled data also adaptively changes according to the corresponding sampling quantity threshold and adaptive sampling parameters during this process.

Please refer to Figure 6, which is a flowchart illustrating one method for obtaining updated historical sampling data provided in this application. As shown in Figure 6, when the first sampling rate corresponding to the target sampling data is less than the second sampling rate corresponding to the historical sampling data, step 100b can be executed, that is, the historical sampling data can be sampled using the first sampling rate to obtain the sampled historical sampling data. Then, step 101b can be executed, that is, the sampled historical sampling data and the target sampling data are merged to obtain merged sampling data. When the total number of merged sampling data exceeds the sampling quantity threshold, step 102b (capacity control, i.e., controlling the amount of data stored in the sampling database through the sampling quantity threshold) can be executed, that is, the merged sampling data can be sampled using a sampling rate increment coefficient. In this application, the sampling rate increment coefficient is 1/2, that is, the merged sampling data can be sampled again by 1/2 to obtain updated historical sampling data. Please refer to Figure 7, which is another flowchart illustrating another method for obtaining updated historical sampling data provided in this application. As shown in Figure 7, when the first sampling rate corresponding to the target sampling data is greater than the second sampling rate corresponding to the historical sampling data, step 100c can be executed, that is, the target sampling data can be sampled using the second sampling rate to obtain the sampled target sampling data. Then, step 101c can be executed, that is, the sampled target sampling data and the historical sampling data are merged to obtain merged sampling data. When the total number of merged sampling data is less than or equal to the sampling quantity threshold, step 102b can be executed, that is, the merged sampling data is directly used as the updated historical sampling data.

As described above, each time window corresponds to a sampling result. If the window length is 60 minutes, the sampling result for each time window is the result obtained by sampling the data generated within that 60-minute time period. Subsequent time windows can directly use the data sampled from the intersection time windows of the previous time window. This data, plus the data included in the sampling results obtained from the corresponding non-intersection time windows of the subsequent time window, constitutes the sampling result for that subsequent time window. The sampling results obtained from each time window can be output and displayed on the user terminal. Each time window corresponds to a sampling state (represented by the sampling results). In this way, the changes in the sampling results obtained from previous and subsequent time windows can be visually observed as the time window progresses. The output sampling results for each time window can include sampling results corresponding to various data types, as well as the number of data points obtained from sampling each type of data and the sampling rate for each type of data. Different types of data can correspond to different sampling rates and sampling number thresholds, but data of the same type must be sampled with equal probability (i.e., equal sampling rate, that is, the same sampling rate).

Please refer to Figure 8, which is a schematic diagram of another data sampling scenario provided in this application. As shown in Figure 8, the computing layer includes multiple sampling devices, each of which can include multiple threads. These multiple sampling devices can read the raw business data in the data layer, that is, the data provided by the data layer for sampling. The data layer can provide several types of data to be sampled, specifically including click logs, exposure logs, performance logs, advertising keyword lists, order keyword lists, resource keyword lists, etc. The aforementioned multiple sampling devices can sequentially read and sample the data in the data layer according to the time sequence of each time window (including time window 1, time window 2, time window 3, ..., time window n). After outputting the sampling results corresponding to the previous time window, the sampling process corresponding to the next time window can be performed. Time windows can overlap. For example, if the window length is 60 minutes (1 hour) and the time window shift is 10 minutes (meaning global sampling occurs every 10 minutes), then the sampling period for time window 1 could be 12:00-13:00 (i.e., the data sampled was generated between 12:00 and 13:00), the sampling period for time window 2 could be 12:10-13:10, the sampling period for time window 3 could be 12:20-13:20, and so on. The overlap between time window 1 and time window 2 is 12:10-13:00. By sampling the data generated within each time window, the sampling result for each time window can be obtained. The process of the sampling device reading and sampling the raw business data from the data layer can be roughly as follows: First, the sampling device can perform format parsing on the raw business data read from the data layer (i.e., data access format parsing, which means parsing out the field information in the raw business data). Next, filters can be used to filter multiple fields in the parsed raw business data, removing unnecessary fields (i.e., data filtering field selection, which can be done using preset filtering rules). Then, aggregation functions are used to pre-aggregate the read data, grouping data of the same type together. Next, alexicon join operations can be used to add necessary fields to the multiple fields in the parsed raw business data (addition can be done according to preset rules). Following this, logical calculations are performed, which involves sampling the data after field filtering and alexicon joins. The output is the sampling result, with each time window corresponding to one sampling result. The final sampling results can be stored in a storage layer. The storage layer can contain structures such as distributed file systems (e.g., HDFS), log-based open-source databases (e.g., Redis), relational databases (e.g., MySQL), and structured databases (e.g., HBase).

This application can sample target type data at a first sampling rate within a first time window to obtain target sampled data; merge the target sampled data and historical sampled data in the sampling database to obtain merged sampled data. The third sampling rate of the merged sampled data is determined based on the first sampling rate and the second sampling rate of the historical sampled data. The historical sampled data refers to the target type sampled data obtained before obtaining the target sampled data. When the number of merged sampled data exceeds a sampling quantity threshold, the merged sampled data is sampled based on adaptive sampling parameters to obtain updated historical sampled data. The sampling quantity threshold is related to the target type data, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling quantity threshold. The historical sampled data in the sampling database is then replaced with the updated historical sampled data. Therefore, the method proposed in this application can further sample the data when the number of merged sampled data exceeds the sampling quantity threshold. This ensures that the number of sampled data stored in the sampling database is always controlled within the sampling quantity threshold range during or after sampling, thereby saving system capacity and achieving adaptive adjustment of the sampling rate.

Please refer to Figure 9, which is a schematic diagram of the structure of a data sampling device provided in this application. This data sampling device can be a computer program (including program code) running on a computer device, for example, the data sampling device is an application software, and the data sampling device can be used to execute the corresponding steps in the method provided in the embodiments of this application. As shown in Figure 9, the data sampling device 1 may include: a sampling module 11, a merging module 12, an adaptive sampling module 13, and a replacement module 14;

Sampling module 11 is used to sample data of the target type at a first sampling rate in a first time window to obtain target sampled data;

The merging module 12 is used to merge the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data. The third sampling rate of the merged sampling data is determined based on the first sampling rate and the second sampling rate of the historical sampling data. The historical sampling data is the target type sampling data obtained by sampling the target type data before obtaining the target sampling data. The second sampling rate is the sampling rate of the historical sampling data relative to the target type data. The third sampling rate is the sampling rate of the merged sampling data relative to the target type data.

The adaptive sampling module 13 is used to sample the merged sampled data based on the adaptive sampling parameters when the number of merged sampled data is greater than the sampling number threshold, so as to obtain updated historical sampled data. The sampling number threshold is related to the target type of data, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling number threshold.

Replacement module 14 is used to replace historical sampling data in the sampling database with updated historical sampling data.

The specific functional implementation of the sampling module 11, merging module 12, adaptive sampling module 13 and replacement module 14 can be found in steps S101-S104 of the embodiment corresponding to Figure 3, and will not be repeated here.

The data sampling device 1 is also used for:

Based on the adaptive sampling parameters and the third sampling rate, a fourth sampling rate is determined so that the fourth sampling rate can be used as the first sampling rate to sample the target type data in the second time window. The fourth sampling rate is the sampling rate of the updated historical sampling data relative to the target type data, and the second time window is the next time window after the first time window.

The merging module 12 includes: a first ratio determination unit 121, a first ratio sampling unit 122, and a first merging unit 123;

The first ratio determination unit 121 is used to determine the ratio between the second sampling rate and the first sampling rate as the first ratio sampling rate when the first sampling rate is greater than the second sampling rate;

The first ratio sampling unit 122 is used to sample the target sampling data according to the first ratio sampling rate to obtain the sampled target sampling data.

The first merging unit 123 is used to determine the target sampling data and historical sampling data after sampling as merged sampling data.

The specific functional implementation of the first ratio determination unit 121, the first ratio sampling unit 122, and the first merging unit 123 can be found in steps S101-S104 of the embodiment corresponding to Figure 3, and will not be repeated here.

The merging module 12 includes: a second ratio determination unit 124, a second ratio sampling unit 125, a second merging unit 126, and a third merging unit 127.

The second ratio determination unit 124 is used to determine the ratio between the first sampling rate and the second sampling rate as the second ratio sampling rate when the first sampling rate is less than the second sampling rate.

The second ratio sampling unit 125 is used to sample historical sampling data according to the second ratio sampling rate to obtain the sampled historical sampling data.

The second merging unit 126 is used to determine the sampled historical sampling data and the target sampling data as the merged sampling data;

The third merging unit 127 is used to determine the target sampling data and historical sampling data as merged sampling data when the first sampling rate is equal to the second sampling rate.

The specific functional implementation of the second ratio determination unit 124, the second ratio sampling unit 125, the second merging unit 126, and the third merging unit 127 can be found in step S102 of the embodiment corresponding to Figure 3, and will not be repeated here.

The sampling module 11 includes: a sampling rate determination unit 111 and a window sampling unit 112;

The sampling rate determination unit 111 is used to determine the second sampling rate of historical sampling data as the first sampling rate;

The window sampling unit 112 is used to sample the target type data using a first sampling rate within a first time window to obtain target sampled data;

The merging module is also specifically used for:

The target sampling data and historical sampling data are determined as the merged sampling data.

The specific functional implementation of the sampling rate determination unit 111 and the window sampling unit 112 can be found in step S101 of the embodiment corresponding to Figure 3, and will not be repeated here.

The adaptive sampling module 13 includes: a string acquisition unit 131, a mapping unit 132, and a merging sampling unit 133;

The string acquisition unit 131 is used to acquire the data identifier string corresponding to the merged sampled data when the number of merged sampled data is greater than the sampling number threshold.

The mapping unit 132 is used to map the data identifier string to the uniform sampling space to obtain the hash value corresponding to the data identifier string;

The merging sampling unit 133 is used to sample the merged sampling data according to the third sampling rate, the adaptive sampling parameter and the hash value to obtain updated historical sampling data.

The specific functional implementation of the string acquisition unit 131, the mapping unit 132 and the merging sampling unit 133 can be found in step S103 of the embodiment corresponding to Figure 3, and will not be repeated here.

The merging sampling unit 133 includes: an adaptive sampling rate subunit 1331 and a merging sampling subunit 1332;

The adaptive sampling rate subunit 1331 is used to obtain the fourth sampling rate based on the third sampling rate and the adaptive sampling parameters;

The merged sampling subunit 1332 is used to sample the merged sampled data based on the fourth sampling rate and the hash value to obtain updated historical sampled data; the fourth sampling rate is the sampling rate of the updated historical sampled data relative to the target type of data.

The specific functional implementation of the adaptive sampling rate subunit 1331 and the merging sampling subunit 1332 can be found in step S103 of the embodiment corresponding to Figure 3, and will not be repeated here.

Specifically, the data sampling device 1 is also used for:

When the number of merged sampled data is less than or equal to the sampling number threshold, the historical sampled data in the sampling database is replaced with the merged sampled data.

The time window preceding the first time window is the third time window; the first time window and the third time window have an intersection time window; historical sampling data is the sampling data obtained by sampling the target type data within the intersection time window of the third time window;

Sampling module 12 is also specifically used for:

Within the time window excluding the intersection time window of the first time window, the target type data is sampled at the first sampling rate to obtain the target sampled data.

Therefore, module 12, in particular, is also used for:

When the number of historical sampled data is less than the merging threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampled data is deleted, and the target sampled data is determined as the merged sampled data.

The sampling module 11 includes: a data reading unit 113, a filtering unit 114, an association unit 115, and a target sampling unit 116.

Data reading unit 113 is used to read data of target type from the data stream in the first time window, and to parse the fields of the read target type data to obtain initial parsed data;

Filtering unit 114 is used to filter multiple fields of information in the initial parsed data based on the filtering mechanism to obtain filtered parsed data;

The association unit 115 is used to add association field information to the filtered parsed data based on the vocabulary association mechanism to obtain the sampled business data;

The target sampling unit 116 is used to sample the sampling service data at a first sampling rate to obtain target sampling data.

The specific functional implementation of the data reading unit 113, the filtering unit 114, the association unit 115 and the target sampling unit 116 can be found in step S101 of the embodiment corresponding to Figure 3, and will not be repeated here.

The sampling module 11 is further used for:

Within the first time window, the target type of data is sampled using the first thread at the first sampling rate to obtain the target sampled data;

Therefore, module 12, in particular, is also used for:

The target sampling data and historical sampling data in the sampling database are merged by a second thread to obtain merged sampling data.

This application can sample target type data at a first sampling rate within a first time window to obtain target sampled data; merge the target sampled data and historical sampled data in the sampling database to obtain merged sampled data. The third sampling rate of the merged sampled data is determined based on the first sampling rate and the second sampling rate of the historical sampled data. The historical sampled data refers to the target type sampled data obtained before obtaining the target sampled data. When the number of merged sampled data exceeds a sampling quantity threshold, the merged sampled data is sampled based on adaptive sampling parameters to obtain updated historical sampled data. The sampling quantity threshold is related to the target type data, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling quantity threshold. The historical sampled data in the sampling database is then replaced with the updated historical sampled data. Therefore, the method proposed in this application can further sample the data when the number of merged sampled data exceeds the sampling quantity threshold. This ensures that the number of sampled data stored in the sampling database is always controlled within the sampling quantity threshold range during or after sampling, thereby saving system capacity and achieving adaptive adjustment of the sampling rate.

Please refer to Figure 10, which is a schematic diagram of the structure of a computer device provided in this application. As shown in Figure 10, the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. As shown in Figure 10, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

In the computer device 1000 shown in Figure 10, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the data sampling method described in the corresponding embodiment of Figure 3 above. It should be understood that the computer device 1000 described in this application can also execute the data sampling device 1 described in the corresponding embodiment of Figure 9 above, and will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here.

Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program executed by the aforementioned data sampling device 1. The computer program includes program instructions, which, when executed by the processor, enable the execution of the data sampling method described in the embodiment corresponding to Figure 3 above. Therefore, this description will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims (15)

1. A data sampling method, characterized in that it comprises: The target type data is sampled at the first sampling rate within the first time window to obtain the target sampled data; The target sampling data and historical sampling data in the sampling database are merged to obtain merged sampling data. The third sampling rate of the merged sampling data is determined based on the first sampling rate and the second sampling rate of the historical sampling data. The historical sampling data is the target type sampling data obtained by sampling the target type of data before obtaining the target sampling data. The second sampling rate is the sampling rate of the historical sampling data relative to the target type of data. The third sampling rate is the sampling rate of the merged sampling data relative to the target type of data. When the number of merged sampled data exceeds the sampling number threshold, the merged sampled data is sampled based on adaptive sampling parameters to obtain updated historical sampled data. The sampling number threshold is related to the data of the target type, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling number threshold. Replace the historical sampling data in the sampling database with the updated historical sampling data. 2. The method according to claim 1, characterized in that the method further comprises: A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate, so that the fourth sampling rate is used as the first sampling rate to sample the target type data in a second time window, wherein the fourth sampling rate is the sampling rate of the updated historical sampling data relative to the target type data, and the second time window is the next time window after the first time window. 3. The method according to claim 1, characterized in that, merging the target sampling data and historical sampling data in the sampling database to obtain merged sampling data includes: When the first sampling rate is greater than the second sampling rate, the ratio between the second sampling rate and the first sampling rate is determined as the first ratio sampling rate; Based on the first ratio sampling rate, the target sampling data is sampled to obtain the sampled target sampling data; The target sampled data and the historical sampled data are determined as the merged sampled data. 4. The method according to claim 1, characterized in that, merging the target sampling data and historical sampling data in the sampling database to obtain merged sampling data includes: When the first sampling rate is less than the second sampling rate, the ratio between the first sampling rate and the second sampling rate is determined as the second ratio sampling rate; Based on the second ratio sampling rate, the historical sampling data is sampled to obtain the sampled historical sampling data; The sampled historical data and the target sampled data are determined as the merged sampled data; When the first sampling rate is equal to the second sampling rate, the target sampling data and the historical sampling data are determined as the merged sampling data. 5. The method according to claim 1, characterized in that, the step of sampling the target type data at a first sampling rate within a first time window to obtain target sampled data includes: The second sampling rate of the historical sampling data is determined as the first sampling rate; Within the first time window, the target type of data is sampled using the first sampling rate to obtain the target sampled data; Then, merging the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data includes: The target sampled data and the historical sampled data are determined as the merged sampled data. 6. The method according to claim 1, characterized in that, when the number of merged sampled data is greater than a sampling quantity threshold, sampling the merged sampled data based on adaptive sampling parameters to obtain updated historical sampled data includes: When the number of merged sampled data exceeds the sampling quantity threshold, the data identifier string corresponding to the merged sampled data is obtained; The data identifier string is mapped to a uniform sampling space to obtain the hash value corresponding to the data identifier string; The merged sampled data is sampled according to the third sampling rate, the adaptive sampling parameter, and the hash value to obtain the updated historical sampled data. 7. The method according to claim 6, characterized in that, the step of sampling the merged sampled data according to the third sampling rate, the adaptive sampling parameter, and the hash value to obtain the updated historical sampled data includes: The fourth sampling rate is obtained based on the third sampling rate and the adaptive sampling parameters; Based on the fourth sampling rate and the hash value, the merged sampled data is sampled to obtain the updated historical sampled data; the fourth sampling rate is the sampling rate of the updated historical sampled data relative to the target type of data. 8. The method according to claim 1, characterized in that the method further comprises: When the number of merged sampled data is less than or equal to the sampling quantity threshold, the historical sampled data in the sampling database is replaced with the merged sampled data. 9. The method according to claim 1, wherein the previous time window of the first time window is a third time window; the first time window and the third time window have an intersection time window; and the historical sampling data is sampling data obtained by sampling data of the target type within the intersection time window of the third time window; The step of sampling data of the target type at a first sampling rate within a first time window to obtain target sampled data includes: Within the time window other than the intersection time window of the first time window, the data of the target type is sampled at the first sampling rate to obtain the target sampled data; Then, merging the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data includes: When the number of historical sampled data is less than the merging quantity threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampled data is deleted, and the target sampled data is determined as the merged sampled data. 10. The method according to claim 1, wherein sampling the target type data at a first sampling rate within a first time window to obtain target sampled data comprises: In the first time window, data of the target type is read from the data stream, and the fields of the read data of the target type are parsed to obtain initial parsed data; The initial parsed data is filtered based on a filtering mechanism to obtain filtered parsed data. Based on the vocabulary association mechanism, related field information is added to the filtered and parsed data to obtain sampled business data; The target sampled data is obtained by sampling the sampled service data at the first sampling rate. 11. The method according to claim 1, wherein sampling the target type data at a first sampling rate within a first time window to obtain target sampled data comprises: Within the first time window, the target type of data is sampled by the first thread at the first sampling rate to obtain the target sampled data; Then, merging the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data includes: The target sampling data and the historical sampling data in the sampling database are merged by a second thread to obtain the merged sampling data. 12. A data sampling device, characterized in that it comprises: The sampling module is used to sample data of the target type at a first sampling rate within a first time window to obtain target sampled data; The merging module is used to merge the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data. The third sampling rate of the merged sampling data is determined based on the first sampling rate and the second sampling rate of the historical sampling data. The historical sampling data is the target type sampling data obtained by sampling the target type data before obtaining the target sampling data. The second sampling rate is the sampling rate of the historical sampling data relative to the target type data. The third sampling rate is the sampling rate of the merged sampling data relative to the target type data. The adaptive sampling module is used to sample the merged sampled data based on adaptive sampling parameters when the number of merged sampled data exceeds the sampling number threshold, so as to obtain updated historical sampled data. The sampling number threshold is related to the target type of data, and the adaptive sampling parameters are used to control the number of merged sampled data within the sampling number threshold. The replacement module is used to replace historical sampling data in the sampling database with updated historical sampling data. 13. The apparatus according to claim 12, wherein the apparatus is further configured to: A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate, so that the fourth sampling rate is used as the first sampling rate to sample the target type data in a second time window, wherein the fourth sampling rate is the sampling rate of the updated historical sampling data relative to the target type data, and the second time window is the next time window after the first time window. 14. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as claimed in any one of claims 1-11. 15. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program comprising program instructions, which, when executed by a processor, perform the method as described in any one of claims 1-11.