WO2025065330A1 - Model monitoring method and apparatus, monitoring entity, and storage medium - Google Patents

Abstract

Embodiments of the present application relate to the technical field of networks, and provide a model monitoring method and apparatus, a monitoring entity, and a storage medium. The method is applied to a monitoring entity, and comprises: obtaining first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of a data processing model, and the first measurement data is input data of the data processing model; obtaining training data of the data processing model; determining a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs; and determining a monitoring result of the data processing model on the basis of the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold. Applying the technical solution provided in the embodiments of the present application can improve the data processing accuracy of the data processing model in actual production.

Description

A model monitoring method, device, monitoring entity and storage medium Technical Field

The present application relates to the field of network technology, and in particular to a model monitoring method, device, monitoring entity and storage medium.

Background Art

On the Internet, data processing models based on artificial intelligence/machine learning (AI/ML) are often used to process the measurement values of reference signals. For example, the positioning reference signals are measured using a positioning model based on AI/ML, and the measurement values are processed to obtain the predicted position of the user equipment (UE) and complete the positioning.

The data processing model can improve the data processing efficiency. However, there are differences between the input data of the data processing model obtained in actual production (i.e., the measurement value of the reference signal) and the training data of the data processing model, which leads to a decrease in the data processing accuracy of the data processing model in actual production.

Summary of the invention

The purpose of the embodiments of the present application is to provide a model monitoring method, device, monitoring entity and storage medium to improve the data processing accuracy of the data processing model in actual production. The specific technical solution is as follows:

In a first aspect, an embodiment of the present application provides a model monitoring method, which is applied to a monitoring entity, and the method includes:

Acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, the first entity and the second entity being located in a service area of a data processing model, and the first measurement data being input data of the data processing model;

Obtaining training data for the data processing model;

Determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs;

The monitoring result of the data processing model is determined according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

In a second aspect, an embodiment of the present application provides a model monitoring device, which is applied to a monitoring entity, and the device includes:

A first acquisition module, configured to acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of a data processing model, and the first measurement data is input data of the data processing model;

A second acquisition module, used to acquire training data of the data processing model;

A first determination module, configured to determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs;

The second determination module is used to determine the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

In a third aspect, an embodiment of the present application provides a monitoring entity, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

The memory is used to store computer programs;

The processor is used to implement any of the above-mentioned method steps when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the method steps described above is implemented.

Beneficial effects of the embodiments of the present application:

In the technical solution provided by the embodiment of the present application, the monitoring entity clusters the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model into two clusters, namely, the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, and compares the Euclidean distance between the center point of the first cluster and the center point of the second cluster with the preset monitoring threshold, so as to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring result. The difference between the input data of the data processing model obtained in actual production and the training data of the data processing model indicates that the environmental data is offset and the data processing model cannot accurately process the input data in actual production. Using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

Figure 1(a) is a schematic diagram of the first structure of the AI/ML assisted positioning framework;

Figure 1(b) is a schematic diagram of the second structure of the AI/ML-assisted positioning framework;

FIG2 is a schematic diagram of a structure of an AI/ML direct positioning framework;

FIG3 is a schematic diagram of a first flow chart of a model monitoring method provided in an embodiment of the present application;

FIG4 is a schematic diagram of measurement configuration information provided in an embodiment of the present application;

FIG5 is a detailed schematic diagram of step S33 provided in an embodiment of the present application;

FIG6 is a detailed schematic diagram of step S34 provided in an embodiment of the present application;

FIG7 is a flow chart of a model updating method provided in an embodiment of the present application;

FIG8 is a flow chart of a model monitoring capability request/providing process provided in an embodiment of the present application;

FIG9 is a schematic diagram of a second flow chart of the model monitoring method provided in an embodiment of the present application;

10-16 are schematic diagrams of a model monitoring scenario provided in an embodiment of the present application;

17 to 19 are schematic diagrams of the distribution of measurement data after dimensionality reduction using the t-SNE algorithm provided in an embodiment of the present application;

FIG20 is a schematic diagram of a structure of a model monitoring device provided in an embodiment of the present application;

FIG. 21 is a schematic diagram of a structure of a monitoring entity provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical scheme, and advantages of the present application more clearly understood, the present application is further described in detail with reference to the accompanying drawings and examples. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present application.

In the Internet, AI/ML-based data processing models are often used to process the measurement values of reference signals. For example, the positioning reference signals are measured using AI/ML-based positioning models to obtain the measurement values, which are then processed to obtain the predicted position of the UE and complete the positioning.

Take the scenario of using AI/ML models for positioning enhancement in 5G NR (5th Generation Mobile Communication Technology New Radio) positioning as an example. In the scenario of using AI/ML for positioning enhancement in 5G NR positioning, the positioning model is an AI/ML model, that is, a data processing model. The measurement value input to the positioning model is the measurement data, and the predicted position output by the positioning model is the processing result of the data processing model. In the scenario of using AI/ML for positioning enhancement in 5G NR positioning, the UE or gNB (gNodeB, 5G base station) measures PRS (Positioning Reference Signal) or SRS (Sounding Reference Signal) to obtain the measurement values of channel-related information such as CIR (Channel Impulse Response) and PDP (Power Delay Profile). The positioning model outputs the position prediction of the UE based on the measurement value, thereby realizing the positioning function of 5G NR.

According to the standardization discussion of 3GPP TSG RAN1 (3rd Generation Partnership Project Technical Specification Group Radio Access Network Work Group 1) on 5G NR AI/ML positioning, the positioning model framework is mainly divided into the following two categories.

1) AI/ML-assisted positioning framework.

The AI/ML assisted positioning framework is shown in Figure 1(a) and Figure 1(b). In AI/ML assisted positioning, AI/ML is combined with traditional NR positioning methods such as TDOA (Time Difference Of Arrival), and CIR and other measurement values are input into the positioning model, as shown in Figure 1(a) and Figure 1(b), and the measurement value 1-measurement value N is input into the AI/ML model. The positioning model outputs intermediate quantities such as TOA (Time Of Arrival) of the positioning process, as shown in Figure 1(a) and Figure 1(b), as shown in Figure 1(a) and Figure 1(b), and intermediate quantity 1-intermediate quantity N. The intermediate quantity is input into the LMF (Location Management Function) end, and the LMF end performs UE positioning solution according to the principle of the traditional NR positioning method to obtain the predicted position of the UE, that is, the positioning result.

2) AI/ML direct positioning framework.

The AI/ML direct positioning framework is shown in Figure 2. In AI/ML direct positioning, the positioning model receives inputs such as CIR and other measurements, such as The measurement values 1-N shown in Figure 2 are input into the AI/ML model, and the positioning model directly outputs the predicted position of the UE, that is, the positioning result.

The data processing model uses a specific data set collected at a certain moment as a training set. When the data processing model is deployed to production, there are often differences between the original data included in the training set and the dynamic measurement data in the production environment. This difference may cause the performance of the data processing model to gradually decline over time. To address this issue, the entity needs to monitor the data processing model during its operation, and retrain the data processing model and update the network parameters of the data processing model when it is detected that the data processing model is unavailable, so as to avoid the accuracy of the processing results of the data processing model from decreasing over time, such as avoiding the positioning accuracy of the positioning model from decreasing over time.

The typical data processing model generation process includes: process 1, data collection; process 2, model training and testing. Therefore, at the functional level, model monitoring can be designed around the above process, such as detecting changes in input data distribution and model concept drift.

1) Detection of changes in input data distribution. When a data processing model receives new measurement data that is significantly different from the training set, the performance of the data processing model may degrade. Therefore, it is crucial to provide early warning of changes in the characteristics of the data processing model and the data distribution predicted by the data processing model. The model monitoring design can monitor changes in the input data distribution. When the input data distribution during the deployment of the data processing model is significantly different from the data distribution of the training set of the data processing model, it can be indirectly determined that the performance of the data processing model has degraded, and further operations such as updating the data processing model can be performed to improve the performance of the data processing model.

2) Model drift detection. When a data processing model is applied to production, the intrinsic characteristics of the production environment may evolve over time, which causes the output of the data processing model to deviate from the actual situation of the production environment, thereby causing the performance of the data processing model to degrade. Therefore, it is necessary to continuously monitor the effectiveness of the data processing model. If the true value of the output of the data processing model can be obtained by other means, the model monitoring design can use the collected real input data and true values as a test set during the deployment of the data processing model to test the data processing model and directly measure the performance of the data processing model to determine whether the performance of the data processing model has degraded, and perform subsequent data processing model updates and other operations.

Taking the scenario of using AI/ML models for positioning enhancement in 5G NR positioning as an example, two major types of model monitoring methods are proposed for the model monitoring problem. These two types of methods are:

1) Model monitoring method based on true value labels (or label estimates). In this method, the monitoring entity can obtain the true position of the UE corresponding to the measured value. For example, using a PRU (Positioning Reference Unit) with a known position or UE feedback of the true position, or using other positioning methods to generate a positioning result with higher accuracy. The positioning model is directly tested and evaluated using the true value label to determine whether the performance of the positioning model has degraded.

2) Model monitoring method without labels: In this method, the monitoring entity only relies on the statistical features of the model input data or output data to monitor the positioning model.

With the development of semi-supervised learning, a technique called adversarial verification has been widely used to solve the problem of model overfitting caused by the inconsistent distribution of training sets and test sets used by data processing models.

In adversarial verification, the original labels of the training set data and test set data used by the data processing model are deleted, and the training set data and test set data are re-labeled, such as all the labels of the training set data are 0, and all the labels of the test set data are 1; then, the re-labeled training set data and test set data are merged into the same data set, and a new training set and a new test set are generated based on this data set. After that, a binary classifier is trained using the data of the new training set, and the performance of the classifier is tested on the data of the new test set. If the classifier can well distinguish the data of the original training set and the data of the original test set in the new test set, it is determined that the data distribution of the original training set and the original test set has a significant difference; if the classifier cannot accurately distinguish the data of the original training set and the data of the original test set in the new test set, it is determined that the data distribution of the original training set and the original test set is similar.

Using this adversarial verification method can also help solve the problem of model monitoring that does not use true value labels. In model monitoring, the dataset used during data processing model training can be regarded as the original training set, and the real data of the production environment obtained during model deployment can be regarded as the original test set in adversarial verification. The principle of adversarial verification can then be used to effectively determine whether there is a difference between the data used during model training and the real data of the environment, thereby effectively detecting changes in the input data distribution of the data processing model.

Adversarial verification is a semi-supervised machine learning method. In the 5G NR model monitoring scenario, every time the monitoring entity needs to obtain the monitoring results of the model, it is necessary to annotate the training set of the data processing model and the real measurement data, and train a binary classifier to determine whether there is a change in data distribution by evaluating the performance of the binary classifier. After the monitoring entity generates the monitoring results, the change cannot be reused. The next time the model is monitored, another binary classifier needs to be retrained based on the newly collected real measurement data. Therefore, in practice, In practical applications, model monitoring methods based on adversarial verification may lead to waste of computing resources and time due to complex classifiers.

In order to improve the data processing accuracy of the data processing model in actual production, the embodiment of the present application provides a model monitoring method, which is applied to a monitoring entity. The monitoring entity can be a base station (gNB), a terminal (UE) or a management entity, or other electronic devices capable of model monitoring, without limitation. The management entity can be an LMF terminal or other device with management functions. The model monitoring method provided in the embodiment of the present application can be applied to the scenario of using AI/ML for positioning enhancement in 5G NR positioning. In this case, the data processing model is a positioning model and the management entity is an LMF terminal.

In the model monitoring method provided in the embodiment of the present application, the monitoring entity maps the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model to a low-dimensional space through dimensionality reduction processing and clusters them into two clusters, i.e., the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, so that the relative relationship between the high-dimensional data can be retained as much as possible, and it can be visualized and classified in the low-dimensional space. Based on this, the Euclidean distance between the center point of the first cluster and the center point of the second cluster is compared with the preset monitoring threshold, and it can be determined whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and the corresponding monitoring result is obtained. There is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, indicating that the environmental data is offset and the data processing model cannot accurately process the input data in actual production. Using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, which improves the data processing accuracy of the data processing model in actual production.

The model monitoring method provided in the embodiments of the present application is described in detail below through specific examples.

Refer to Figure 3, which is a first flow chart of the model monitoring method provided in an embodiment of the present application, which is applied to a monitoring entity. The method includes the following steps.

Step S31, obtaining first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, the first entity and the second entity are located in a service area of a data processing model, and the first measurement data is input data of the data processing model.

Step S32, obtaining training data for the data processing model.

Step S33: determine the first cluster to which the first measurement data belongs and the second cluster to which the training data belongs.

Step S34, determining the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

In the technical solution provided by the embodiment of the present application, the monitoring entity clusters the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model into two clusters, namely, the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, and compares the Euclidean distance between the center point of the first cluster and the center point of the second cluster with the preset monitoring threshold, so as to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring result. The difference between the input data of the data processing model obtained in actual production and the training data of the data processing model indicates that the environmental data is offset and the data processing model cannot accurately process the input data in actual production. Using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

In addition, in the technical solution provided in the embodiment of the present application, there is no need to train a classifier each time the model is monitored, which to a certain extent alleviates the problem of waste of computing resources and time in existing semi-supervised learning.

In the above step S31, the data processing model can be deployed on the monitoring entity or on other entities, such as the fifth entity. The data processing model can be a positioning model or other models. The positioning model is used as an example for explanation in the following, and it does not play a limiting role. The monitoring period is the period during which the monitoring entity monitors the data processing model once, and the duration of the monitoring period can be set according to actual conditions. The service area of the data processing model (referred to as the service area) can be one or multiple adjacent cells.

The first entity may be a physical device such as a UE or a gNB in the service area, and the second entity may also be a physical device such as a UE or a gNB in the service area. The first entity and the second entity are entities that transmit reference signals to each other. For example, when the first entity is a UE, the second entity is a gNB, and when the first entity is a gNB, the second entity is a UE. The first reference signal is an uplink reference signal and/or a downlink reference signal transmitted between the first entity and the second entity, such as SRS, CSI-RS (Channel State Information-Reference Signal), PRS, SSB (Synchronization Signal Block), DMRS (Demodulation Reference Signal), PTRS (Phase Tracking Reference Signal), etc. The first measurement number The data is data obtained by measuring the first reference signal, such as CIR, PDP, etc. Both the first entity and the second entity can serve as entities for measuring the first reference signal.

When a monitoring cycle is reached, the monitoring cycle reached is the current monitoring cycle, and the monitoring entity can obtain the first reference signal in the service area within the current monitoring cycle, measure the first reference signal, and obtain the first measurement data; other entities can also measure the first reference signal to obtain the first measurement data, and the monitoring entity directly obtains the first measurement data from other entities. Here, the first measurement data obtained by the monitoring entity is the real measurement data of the production environment.

In an embodiment of the present application, when performing model monitoring, the monitoring entity may obtain first measurement data of a first reference signal from the first entity or the second entity. The first measurement data does not carry a true value label, such as when the data processing model is a positioning model, the true positions of the first entity and the second entity are unknown.

In the embodiment of the present application, the monitoring entity may be the first entity or the second entity, or may be a management entity. Depending on the location of the monitoring entity, there are two situations.

1) When the monitoring entity is the first entity or the second entity, taking the monitoring entity being the first entity as an example, the monitoring entity can implement the above step S31 in the following three ways.

a. Receive a first reference signal sent by a second entity in a current monitoring period; measure the first reference signal to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal.

In an embodiment of the present application, the first reference signal may be a reference signal broadcast by the second entity to the first entity, or may be a reference signal unicast by the second entity to the first entity. The first entity is responsible for measuring the reference signal. In the current monitoring period, the second entity sends the first reference signal to the first entity; the first entity (i.e., the monitoring entity) receives the first reference signal sent by the second entity, and measures the first reference signal to obtain a first measurement result including first measurement data.

In the embodiment of the present application, one of the first entity and the second entity is a base station and the other entity is a terminal. When the first entity is a terminal and the second entity is a base station, the first reference signal may be a PRS, a CSI-RS, a SRS, a SSB, a DMRS, or a PTRS; when the first entity is a base station and the second entity is a terminal, the first reference signal is an SRS.

In some embodiments, the first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal. To ensure accurate identification of the first reference signal and measurement of the first reference signal, the base station sends the first configuration information to the terminal; the terminal sends or receives the first reference signal according to the first configuration information.

In some embodiments, the first configuration information is configuration information sent by the base station to the terminal according to a first request sent by the third entity, and the first request instructs the base station to send the first configuration information to the terminal. In order to accurately control the sending of the first configuration information, when entering a monitoring cycle, the third entity (such as a management entity) sends a first request to the base station; the base station sends the first configuration information to the terminal according to the first request sent by the third entity, and then the terminal sends or receives the first reference signal according to the first configuration information.

The first measurement data may be measurement data of the first reference signal obtained by the monitoring entity according to the second configuration information sent by the third entity. That is, the third entity sends the second configuration information to the second entity, and the second configuration information is used to specify the relevant configuration of the model monitoring. The second entity obtains the first measurement data according to the second configuration information to complete the model monitoring.

The second configuration information may include at least one of the following: monitoring cycle information, measurement configuration information, monitoring algorithm and preset monitoring threshold. The measurement configuration information, monitoring algorithm and preset monitoring threshold may be represented by a number of bits.

a1) Monitoring cycle information. The monitoring cycle information may include a cycle unit and a number of bits. The cycle unit may be seconds, minutes, hours, etc. The cycle unit and the number of bits together represent the length of the monitoring cycle, which may be determined based on the actual system requirements and algorithm time consumption. For example, when the monitoring cycle is in hours, 5 bits may be used to represent the monitoring cycle length of 1 to 24 hours. In order to save bits, different cycle options may also be agreed upon. For example, when the cycle unit is times/day, 2 bits may be used to represent 4 monitoring cycle options. The cycle options may be defined as: {00: 1 time/day (i.e., 24-hour cycle); 01: 4 times/day (i.e., 6-hour cycle); 10: 8 times/day (i.e., 3-hour cycle); 11: 24 times/day (i.e., 1-hour cycle)}, that is, when the number of bits is 00, it indicates that the monitoring cycle is 24 hours; when the number of bits is 01, it indicates that the monitoring cycle is 6 hours, and so on. The monitoring entity may determine the monitoring cycle for model monitoring through the cycle unit and the number of bits.

a2) Measurement configuration information. The measurement configuration information may include the measurement cycle length, the measurement time slice length, and the measurement frequency. The measurement cycle length is the measurement data collection cycle for model monitoring, the measurement time slice length is the continuous time length of each data collection measurement value, and the measurement frequency is the total number of measurements for this data collection. In the embodiment of the present application, the measurement cycle length is greater than the measurement time slice length. The relationship between them is shown in Figure 4.

In an embodiment of the present application, a parameter set can be used to represent different measurement configuration sets, which can be set specifically according to actual needs. For example, 2 bits are used to represent 4 parameter selections: {00: (measurement cycle length: 512; slice length: 256; measurement frequency: 4 times); 01: (measurement cycle length: 256; slice length: 128; measurement frequency: 4 times); 10: (measurement cycle length: 512; slice length: 256; measurement frequency: 8 times); 11: (measurement cycle length: 256; slice length: 128; measurement frequency: 8 times)}, that is, when the number of bits corresponding to the configuration information is 00, it means that the data collection cycle is 512, the continuous time length of each collection is 256, the number of collections is 4 times, and so on. The first entity obtains the first measurement data according to the measurement configuration information.

a3) Monitoring algorithm. Considering that the model monitoring method can support different monitoring algorithms, it is necessary to specify the type of algorithm used by the monitoring entity. The number of bits used is determined according to the total number of optional algorithms, and the correspondence between the number of bits and the monitoring algorithm is preset. For example, 2 bits are used to represent 4 monitoring algorithms: {00: monitoring algorithm based on t-SNE; 01: model monitoring algorithm based on adversarial verification; 10: model monitoring algorithm based on KS test; 11: model monitoring algorithm based on autoencoder}. For example, when the number of bits corresponding to the monitoring algorithm is 00, the monitoring entity uses a monitoring algorithm based on t-SNE for model monitoring.

a4) Preset monitoring threshold. Since different monitoring algorithms are used for model monitoring, the output types are different. Therefore, it is necessary to specify the monitoring threshold or the relevant data of the threshold auxiliary calculation in the current situation according to the monitoring algorithm, and determine the number of bits used according to the threshold type or auxiliary data type of different monitoring algorithms. For example, in the monitoring algorithm based on t-SNE designed in the embodiment of the present application, the ratio of the Euclidean distance of the center points of the two clusters to the sum of the radii of the two clusters is designed as the discriminant, then 4 bits can be used to represent the ten ratio options [0.1, 0.2, ... 0.9, 1] as the threshold for determining data drift: {0000: 0.1; 0001: 0.2; ... 1000: 0.9; 1001: 1}. For example, when the number of bits corresponding to the preset monitoring threshold is 0000, the preset monitoring threshold is 0.1, indicating that when the ratio of the Euclidean distance of the center points of the two clusters to the sum of the radii of the two clusters is greater than 0.1, it can be determined as data drift.

b. Sending a first reference signal to the second entity within a current monitoring period; and receiving a second measurement result sent by the second entity, where the second measurement result includes first measurement data of the first reference signal.

In the embodiment of the present application, the first reference signal may be a reference signal broadcast by the first entity to the second entity, or may be a reference signal unicast by the first entity to the second entity. The second entity is responsible for measuring the reference signal.

During the current monitoring cycle, the first entity (i.e., the monitoring entity) sends a first reference signal to the second entity; the second entity receives the first reference signal, measures the first reference signal, obtains a second measurement result including the first measurement data, and sends the second measurement result to the monitoring entity; the monitoring entity obtains the first measurement data by receiving the second measurement result.

In the embodiment of the present application, one of the first entity and the second entity is a base station and the other entity is a terminal. When the first entity is a terminal and the second entity is a base station, the first reference signal is SRS; when the first entity is a base station and the second entity is a terminal, the first reference signal may be PRS, CSI-RS, SRS, SSB, DMRS, or PTRS, etc.

In some embodiments, the first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal. To ensure accurate identification of the first reference signal and measurement of the first reference signal, the base station sends the first configuration information to the terminal; the terminal sends or receives the first reference signal according to the first configuration information.

In some embodiments, the first configuration information is configuration information sent by the base station to the terminal according to a first request sent by the third entity, and the first request instructs the base station to send the first configuration information to the terminal. In order to accurately control the sending of the first configuration information, when entering a monitoring cycle, the third entity (such as a management entity) sends a first request to the base station; the base station sends the first configuration information to the terminal according to the first request sent by the third entity, and then the terminal sends or receives the first reference signal according to the first configuration information.

The first measurement data may be the measurement data of the first reference signal obtained by the monitoring entity according to the second configuration information sent by the third entity. That is, the third entity (management entity) sends the second configuration information to the first entity, and the second configuration information is used to specify the relevant configuration of the model monitoring. The first entity obtains the first measurement data from the second entity according to the second configuration information to complete the model monitoring. The specific form of the second configuration information can be found in the description in a above.

c. Send a second request to a third entity, where the third entity stores first measurement data of a first reference signal between the first entity and the second entity, and the second request instructs the third entity to send the first measurement data to the monitoring entity; receive a third measurement result corresponding to the first request sent by the third entity, where the third measurement result includes the first measurement data of the first reference signal between the first entity and the second entity.

In the implementation of this application, the first entity may be a terminal or a base station, and the third entity may be a management entity, such as an LMF terminal. Request for measurement results. The third entity (such as the LMF end) stores the existing measurement data. In the current monitoring cycle, the first entity (i.e., the monitoring entity) directly sends a second request to the LMF end, requesting to obtain the first measurement data already in the LMF end; the LMF end sends a third measurement result including the first measurement data to the first entity according to the second request; the monitoring entity obtains the first measurement data by receiving the third measurement result. In an embodiment of the present application, the situation where the LMF end may have existing measurement data is taken into consideration to facilitate the rapid acquisition of measurement data.

The first measurement data may be the measurement data of the first reference signal obtained by the monitoring entity according to the second configuration information sent by the third entity. That is, the third entity (management entity) sends the second configuration information to the first entity, and the second configuration information is used to specify the relevant configuration of the model monitoring. The first entity obtains the first measurement data from the third entity according to the second configuration information to complete the model monitoring. The specific form of the second configuration information can be found in the description in a above.

The model monitoring method when the monitoring entity is the second entity is similar to the model monitoring method when the monitoring entity is the first entity. Please refer to the description of the above situations a-c, and will not be repeated here.

2) When the monitoring entity is a third entity, the monitoring entity can implement the above step S31 through the following steps: obtain the first measurement data of the first reference signal between the first entity and the second entity in the current monitoring period from the first target entity, and the first target entity is the entity between the first entity and the second entity that measures the first reference signal.

In an embodiment of the present application, the third entity may be a management entity, such as an LMF end; when the first entity is a terminal, the second entity is a base station, the first target entity is a terminal, and the first reference signal may be PRS, CSI-RS, SRS, SSB, DMRS, or PTRS, etc.; when the first entity is a base station, the second entity is a terminal, and the first target entity is a base station, the first reference signal is SRS.

In the current monitoring cycle, a first reference signal is transmitted between the first entity and the second entity, and one of the first entity and the second entity measures the first reference signal to obtain first measurement data, and the entity performing the measurement is the first target entity. The monitoring entity (i.e., the third entity) obtains the first measurement data obtained by measurement from the first target entity.

In the embodiment of the present application, the third entity stores the second configuration information. The monitoring entity (ie, the third entity) obtains the first measurement data of the first reference signal according to the stored second configuration information. The specific form of the second configuration information can be found in the description in a above.

In the technical solution provided in the embodiment of the present application, the monitoring entity obtains first measurement data based on the first reference signal between the first entity and the second entity. The first measurement data is the data that needs to be input into the data processing model in actual production, reflecting the data distribution in actual production within the current monitoring cycle, so as to facilitate accurate model monitoring without true value labels.

In the above step S32, the training data is the data in the training set of the data processing model, such as CIR, PDP, etc. The monitoring entity obtains the training data when training the data processing model. When the data processing model is generated on the monitoring entity side, the monitoring entity can directly obtain the training data from the monitoring entity locally; the monitoring entity can also receive the training data from other entities containing training data, such as the entity that generates the data processing model or the LMF end. The method of obtaining the training data is not limited here.

The monitoring entity stores the training data of the currently deployed data processing model and the real measurement data during the deployment of the data processing model through the above steps S31 and S32, and collects the data required for model monitoring. The execution order of the above steps S31 and S32 is not limited.

In the above step S33, the monitoring entity can use monitoring algorithms such as t-SNE (t-Distributed Stochastic Neighbor Embedding) to process the collected first measurement data and training data to obtain clustering clusters corresponding to the first measurement data and the training data, namely, the first cluster and the second cluster.

In the above step S34, the preset monitoring threshold is a parameter for evaluating the monitoring result, which is used to determine whether the data processing model is available. The preset monitoring threshold can be set according to the actual situation. The monitoring entity calculates the Euclidean distance between the center point of the first cluster and the center point of the second cluster, and determines the monitoring result of the data processing model according to the relationship between the Euclidean distance and the preset monitoring threshold. For example, if the Euclidean distance is greater than the preset monitoring threshold, the monitoring result indicates that the data processing model is unavailable; if the Euclidean distance is less than or equal to the preset monitoring threshold, the monitoring result indicates that the data processing model is available.

In some embodiments, referring to FIG. 5 , which is a detailed schematic diagram of the above step S33 provided in an embodiment of the present application, the above step S33 may include the following steps.

Step S51: Perform dimensionality reduction processing on the first measurement data and the training data according to a preset monitoring algorithm to obtain dimensionality reduced data.

Step S52: clustering the dimension-reduced data to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs.

In the technical solution provided in the embodiment of the present application, for the high-dimensional first measurement data and training data, before clustering the data, monitoring The control entity can reduce the dimension of the data through the preset monitoring algorithm. After mapping the high-dimensional measurement data and training data to the low-dimensional space, the data processing amount of the model monitoring method is reduced and the efficiency of the model monitoring is improved. At the same time, the relative relationship between the measurement data and the training data is retained, ensuring the accuracy of clustering the reduced-dimensional data.

In the above step S51, the preset monitoring algorithm may be a t-SNE algorithm or other dimensionality reduction algorithms, such as the various monitoring algorithms included in the above second configuration information, which are not limited here. The specific calculation process of the t-SNE algorithm will be described in detail later and will not be described in detail here. The monitoring entity reduces the dimension of the acquired first measurement data and training data into two-dimensional plane points according to the preset monitoring algorithm to obtain reduced dimension data.

In the above step S52, the monitoring algorithm clusters the dimension-reduced data, that is, clusters the points on the two-dimensional plane of the training data from the training set and the first measurement data from the environmental measurement, respectively, to obtain the first cluster to which the first measurement data belongs and the second cluster to which the training data belongs.

In some embodiments, the monitoring entity may pre-process the acquired first measurement data and training data. The above step S51 may be implemented by the following steps: converting the first measurement data and training data into intermediate data matching the data processing model; performing dimensionality reduction processing on the intermediate data according to a preset monitoring algorithm to obtain dimensionality reduction data.

In an embodiment of the present application, the monitoring entity may pre-process the collected first measurement data and training data to obtain intermediate data that matches the subsequent preset monitoring algorithm input. For example, the preset monitoring algorithm (such as the t-SNE algorithm) requires PDP as the input data of the positioning model, and the first measurement data and training data are CIR, then the data needs to be processed to convert CIR into PDP to achieve preliminary data compression, while ensuring that the data monitored by the model is consistent with the input data of the positioning model, and ensuring the accuracy of the subsequent model monitoring results.

In some embodiments, referring to FIG. 6 , which is a detailed schematic diagram of the above step S34 provided in an embodiment of the present application, the above step S34 may include the following steps.

Step S61, calculate the ratio of the first distance to the second distance to obtain a monitoring value, where the first distance is the Euclidean distance between the center point of the first cluster and the center point of the second cluster, and the second distance is the sum of the radius of the first cluster and the radius of the second cluster. If the monitoring value is greater than the preset monitoring threshold, execute step S62; if the monitoring value is less than or equal to the preset monitoring threshold, execute step S63.

Step S62: determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable.

Step S63: determining a second monitoring result of the data processing model, where the second monitoring result indicates that the data processing model is available.

In the technical solution provided by the embodiment of the present application, based on the obtained first cluster and the second cluster, combined with the preset monitoring threshold, a conclusion is judged through calculation and comparison, and the model monitoring result is output. For example, the ratio of the Euclidean distance between the center points of the two clusters to the sum of the cluster radii is calculated, and compared with the preset monitoring threshold generated, stored or received locally, to judge the monitoring result of the data processing model, thereby realizing the quantitative processing of model monitoring.

In the above step S61, after the monitoring entity obtains the two clusters of the first cluster and the second cluster, it calculates the Euclidean distance between the center point of the first cluster and the center point of the second cluster as the first distance, calculates the sum of the radius of the first cluster and the radius of the second cluster as the second distance, and then calculates the ratio of the first distance to the second distance, and uses the ratio as the monitoring value. For example, if the first distance is R, the radius of the first cluster is R1, and the radius of the second cluster is R2, then the monitoring value is R/(R1+R2).

The monitoring entity determines the monitoring result of the data processing model according to the size relationship between the monitoring value and the preset monitoring threshold. When the monitoring value is greater than the preset monitoring threshold, the monitoring entity executes the above step S62 to determine that the monitoring result of the data processing model is the first monitoring result, that is, the environmental data is offset and the data processing model is unavailable; when the monitoring value is less than or equal to the preset monitoring threshold, the monitoring entity executes the above step S63 to determine that the monitoring result of the data processing model is the second monitoring result, that is, the environmental data is not offset and the data processing model is still applicable.

In some embodiments, the monitoring entity may also determine the monitoring result of the data processing model based on the first distance. The monitoring entity may implement the above step S34 by the following steps: if the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, then determine the first monitoring result of the data processing model, and the first monitoring result indicates that the data processing model is unavailable; if the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, then determine the second monitoring result of the data processing model, and the second monitoring result indicates that the data processing model is available.

In the embodiment of the present application, the monitoring entity compares the Euclidean distance (i.e., the first distance) between the center point of the first cluster and the center point of the second cluster. The relationship between the first distance and the preset monitoring threshold is that when the first distance is greater than the preset monitoring threshold, the monitoring entity determines that the monitoring result of the data processing model is the first monitoring result, and the data processing model is unavailable; when the monitoring value is less than or equal to the preset monitoring threshold, the monitoring entity determines that the monitoring result of the data processing model is the second monitoring result, and the data processing model is still applicable. By applying the technical solution provided in the embodiment of the present application, the first distance is compared with the preset monitoring threshold to obtain the model monitoring result, which can reduce the amount of calculation and improve the efficiency of model monitoring.

In some embodiments, when the data processing model is deployed on a monitoring entity, after obtaining the monitoring results, the monitoring entity can use the monitoring results to determine whether to update the data processing model.

When the data processing model is deployed on other entities (such as the fifth entity), after obtaining the monitoring result of the data processing model, the monitoring entity can also send the monitoring result to the fifth entity. After obtaining the monitoring result, the fifth entity can use the monitoring result to determine whether to update the data processing model. Among them, the fifth entity is the entity that deploys the data processing model, which can be a gNB, UE and other entities. The monitoring entity broadcasts the monitoring result, transmits the model monitoring conclusion, and completes the update of the data processing model, so as to subsequently improve the accuracy of the data processing model.

Based on the above model monitoring method, when the data processing model is deployed on the monitoring entity, the embodiment of the present application also provides a model updating method, see Figure 7, which is a flow chart of the model updating method provided by the embodiment of the present application, applied to the monitoring entity. The above model updating method includes the following steps.

Step S71, when the monitoring result of the data processing model indicates that the data processing model is unavailable, obtain second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, the fourth entity is located in a service area, and the processing result of the data processing model corresponding to the fourth entity is known, and the second target entity is the entity that sends the second reference signal between the first entity and the second entity.

Step S72: updating the data processing model according to the second measurement data.

In the technical solution provided in the embodiment of the present application, when the monitoring result of the data processing model is the first monitoring result, that is, when the monitoring entity determines that the data processing model is unavailable, the monitoring entity needs to collect data with true value labels and update the current data processing model to ensure the availability of the data processing model.

In the above step S71, the fourth entity may also be a physical device such as a UE, a gNB, or a PRU in the service area. The second target entity and the fourth entity are entities that transmit reference signals to each other. The second reference signal is a reference signal transmitted between the second target entity and the fourth entity. The fourth entity can be used as an entity for measuring the second reference signal.

The monitoring entity obtains second measurement data obtained by measuring the second reference signal, and a plurality of second measurement data constitute an auxiliary data set, and each second measurement data carries a true value label. For example, when the data processing model is a positioning model, the true position of the fourth entity is known. To ensure the accuracy of model training, the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold.

In the above step S72, the monitoring entity obtains the second measurement data carrying the true value label, and retrains the current data processing model according to the second measurement data to update the network parameters of the data processing model.

In some embodiments, the monitoring entity may obtain a fourth measurement result from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity. That is, the monitoring entity obtains the second measurement data from the third entity to update the model.

In the embodiment of the present application, when the monitoring entity is the first entity, the monitoring entity may obtain the fourth measurement result from the third entity in an active or passive manner.

1) Passively obtaining the fourth measurement result: the third entity periodically sends the fourth measurement result to the monitoring entity.

2) Actively obtain the fourth measurement result: the monitoring entity sends a third request to the third entity, where the third request instructs the third entity to send the second measurement data to the monitoring entity; and receives the fourth measurement result corresponding to the third request sent by the third entity.

The monitoring entity actively sends a third request to the third entity. After receiving the third request, the third entity sends a fourth measurement result to the monitoring entity.

To ensure the accuracy of the updated data processing model, the third request includes a minimum number of samples. In this case, the number of second measurement data included in the fourth measurement result is greater than or equal to the minimum number of samples. The minimum number of samples is expressed in units of quantity and bits, and may also be expressed in other forms, which are not limited.

The third request is sent by the monitoring entity. When the monitoring entity detects data drift through the model monitoring algorithm, or determines that the data processing model needs to be updated, the third request is a signaling for requesting the third entity to update the auxiliary data set (i.e., the second measurement data). The auxiliary information requested by the monitoring entity may include, but is not limited to: the minimum number of samples of the auxiliary data set.

Considering that the migration training of the original data processing model or the retraining of the new data processing model has certain requirements on the size of the data set, a data set that is too small may lead to performance degradation. Therefore, when requesting an auxiliary data set, there should be requirements for the size of the minimum number of samples. The number of bits used can be determined based on the number of samples or the actual size of the data set. For example, if 1000 is used as a unit, 8 bits can be used to represent 1000 to 256000 samples. For example, when the minimum number of samples is 00000001, it means that the minimum number of samples supported by the monitoring entity is 1000 samples.

In some embodiments, after sending the third request to the third entity, the monitoring entity may also receive a first response or a second response corresponding to the third request sent by the third entity, wherein the first response indicates that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicates that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity. After the monitoring entity receives the first response, the monitoring entity performs the step of receiving a fourth measurement result corresponding to the third request sent by the third entity.

The response signaling is sent by the third entity. When the third entity receives the third request sent by the monitoring entity, it preliminarily evaluates whether it is currently capable of generating sufficient measurement data and sending it to the monitoring entity. If it is capable of sending, it replies with a first response, otherwise it replies with a second response. To save bandwidth, the response signaling can occupy 1 bit. For example, when the response signaling is 1, it indicates the first response, and when the response signaling is 0, it indicates the second response.

In an embodiment of the present application, the third entity sends a response corresponding to the third request to the monitoring entity, so that the monitoring entity can decide whether to wait for receiving the fourth measurement result or to perform the next round of model monitoring, thereby avoiding the monitoring entity from continuously waiting for receiving the fourth measurement result when it cannot receive the fourth measurement result.

In the embodiment of the present application, when the monitoring entity is a third entity, the monitoring entity may directly obtain the fourth measurement result locally.

In some embodiments, the fourth measurement result may be a measurement result obtained by the third entity from the fourth entity.

When the evaluation result of the third entity is that it is capable of generating sufficient measurement data, the second measurement data is measurement data sent by the fourth entity to the third entity according to the third configuration information of the second reference signal sent by the third entity. The third entity sends the third configuration information to a fourth entity such as a PRU or UE with known location information, and the fourth entity obtains the second measurement data according to the third configuration information of the second reference signal sent by the third entity, and feeds back a fifth measurement result including the second measurement data to the fourth entity.

The third configuration information may include at least one of the following: measurement-related information of the second reference signal and an identifier of the monitoring entity.

1) Measurement related information of the second reference signal, where some IEs (information elements) and processes in LPP are multiplexed, such as NR-On-Demand-DL-PRS-Configurations, etc. The measurement related information is used to configure PRS measurement related information to the UE or PRU that generates the auxiliary data set.

2) The identification of the monitoring entity, that is, the entity ID carrying the data processing model. Considering that the UE and PRU that generate the auxiliary data set can directly send the auxiliary data set to the monitoring entity after the collection is completed, it is necessary to identify the monitoring entity. Existing identifications can be reused here, for example, the TMSI (Temporary Mobile Subscriber Identity) used by the UE.

In some embodiments, the second measurement data may include at least one of the following: a measurement value, a true value label corresponding to the measurement value, and a data quality corresponding to the true value label. The measurement value, the true value label, and the data quality are represented by the number of bits.

The second measurement data is sent by a fourth entity such as a UE or a PRU that assists in generating a data set, and the fourth entity transmits the auxiliary data set or the auxiliary measurement data to the monitoring entity.

1) Measurement values, such as CIR, PDP, etc., can be encoded and represented according to the specific data format of the high-dimensional data, and the number of bits used is determined according to the amount of data to be transmitted.

2) The true value label corresponding to the measurement value, for example, the terminal position or TOA prediction value corresponding to a set of CIR data (including CIR measurement values of different base stations and at different times) can be determined according to the specific data format. The number of bits used can be determined according to the specific data format. If the decimeter level (0.1m) accuracy is used, 17 bits can be used to represent the distance or coordinate variable in the range of 0 to 13km.

3) The data quality corresponding to the true value label, for example, the terminal position error range, if the accuracy is in decimeter level, 6 bits can be used to represent the absolute value of the deviation of 0 to 5 meters. The data quality corresponding to the true value label here can be used to evaluate the accuracy of the measurement data obtained by the fourth entity.

In the embodiment of the present application, if the fourth entity transmits only auxiliary measurement data, the second measurement data may only include measurement values. In this case, the fourth entity may also be used as the first entity or the second entity for model monitoring.

In some embodiments, when the monitoring entity is not a management entity, the monitoring entity may also send monitoring capability information to the third entity, where the monitoring capability information includes at least one of the following: the maximum number of samples supported by the monitoring entity. The maximum number of samples can be represented by bits. For example, if 1000 is used as a unit, 8 bits can be used to represent 1000 to 256000 samples. For example, when the maximum number of samples is 00000001, it means that the maximum number of samples supported by the monitoring entity is 1000 samples.

The model monitoring capability request/providing process shown in Figure 8, wherein the monitoring entity is UE/gNB and the third entity is the LMF end. Before the monitoring entity sends the monitoring capability information to the third entity, the monitoring entity may also receive a fourth request sent by the third entity, the fourth request indicating the acquisition of the monitoring capability information, that is, requesting the monitoring capability signaling; according to the fourth request, the monitoring entity sends the monitoring capability information to the third entity, that is, sends the monitoring capability signaling carrying the monitoring capability information to the third entity. By sending the monitoring capability information to the third entity, the monitoring entity provides the third entity with the processing or storage capabilities it supports to assist in the subsequent auxiliary data transmission.

The following is a detailed introduction to the t-SNE algorithm. The basic idea of the SNE algorithm is to map data points to probability distributions. The main steps include three steps:

1) SNE is based on the similarity between high-dimensional data. The similarity can be measured using the Euclidean distance between a sample point and other sample points. The Euclidean distance is used to construct a Gaussian conditional probability distribution. The characteristic of this probability distribution is that for a sample point, sample points similar to it have a higher probability of being selected, while sample points dissimilar to it have a lower probability of being selected. The Gaussian conditional probability distribution between high-dimensional sample points _xi and _xj constructed by SNE is shown below:

Among them, _xi , _xj , and _xk represent high-dimensional sample points, pj _|i represents the Gaussian conditional probability distribution between high-dimensional sample points _xi and _xj , that is, the probability that the high-dimensional sample point _xi will choose _xj as its neighbor, exp(·) represents the exponential function, ||·|| represents the modulus function, _σi represents the standard deviation of the Gaussian distribution centered on _xi , and Σ(·) represents the summation function.

For the same sample point, the conditional probability is 0, that is, p _i|i = 0. For each sample point x _i , the Gaussian distribution constructed by other sample points relative to the sample point x _i has a standard deviation σ _i corresponding to the sample point. Under different data distributions, the initialization of σ _i first defines the perplexity, and then determines the σ _i value corresponding to the perplexity through the binary search method. The perplexity is defined using the entropy of the constructed distribution:

in,

Perp(P _i ) represents perplexity, P _i represents the Gaussian distribution constructed by the relative distance between the i-th sample point and other sample points (Euclidean distance is used in the formula), H(P _i ) represents the cross entropy of the center point P _i in binary measurement, p _j|i represents the Gaussian conditional probability distribution between high-dimensional sample points _xi and _xj , and log(·) represents the logarithmic function.

In practical applications, the value of perplexity is determined by the user. SNE is robust to the value of perplexity and generally chooses a value between 5 and 50.

2) SNE constructs a probability distribution of mapping points in low-dimensional space, under which each low-dimensional space data point corresponds to a data point in the original high-dimensional space. The conditional probability distribution between low-dimensional sample points y _i and y _j constructed by SNE is as follows:

Among them, qj _|i is the conditional probability distribution between low-dimensional sample points _yi and _yj , and _yi , _yj and _yk represent low-dimensional sample points.

Likewise, for the same sample point, q _i|i =0.

3) Theoretically, when the dimensionality reduction effect is good enough, the distributions of the two constructs should be the same, that is, p _j|i = q _j|i . Therefore, the objective function is constructed using KL-divergence, and the optimization goal is to minimize the sum of the KL-divergence between all sample points in the high-dimensional space and all sample points in the low-dimensional space. The constructed objective function is as follows:

Among them, Cost represents the objective function, that is, the loss function, and _Pi represents the relative distance between the i-th sample point and other samples (the formula uses Euclidean distance), _Pi is represented by the probability distribution of pj _|i {p1 _|i , p2 _|i …}; _Qi represents the Gaussian distribution constructed from the relative distance between the i-th sample point and other sample points on the 2D mapping plane, _Qi is represented by the probability distribution of qj _|i {q1 _|i , q2 _|i …}, i=1,…,n.

The basic steps of the t-SNE algorithm are similar to those of the SNE algorithm. In order to optimize the defects of the SNE algorithm (such as crowding), t-SNE uses a joint distribution to construct the relationship between sample points, and uses a heavier-tailed t distribution to construct the distribution of low-dimensional space mapping points:

Among them, _qij represents the joint probability distribution, _yi , _yj , _yk and _yl represent low-dimensional sample points.

For the joint distribution of high-dimensional data, a symmetric conditional distribution is used:

Among them, p _ij represents the joint probability distribution, and n represents the number of sample points.

The corresponding objective function is:

Among them, P represents the joint probability distribution of the relative distances between all sample points; Q represents the joint probability distribution of the relative distances of all sample points on the two-dimensional mapping plane.

This application uses the t-SNE algorithm principle described above and applies this principle to the detection of input data distribution drift in model monitoring. In the 5G positioning scenario, the input of the ML model is generally high-dimensional data such as CIR and PDP obtained by measurement. In order to detect whether the distribution of the measured data received during the model deployment process and the distribution of the measured data used in model training have drifted, the training set data can be spliced with the new measurement data and subjected to t-SNE dimensionality reduction processing. The low-dimensional space mapping points from the training set data and the new data after dimensionality reduction are clustered separately. By judging the relative relationship between the Euclidean distance between the center points of the two clusters and the cluster radius, it is measured whether there is a more obvious offset between the training set and the test set.

The model monitoring method provided in the embodiment of the present application is described in detail below in conjunction with Figures 9 to 16.

The process of the model monitoring method shown in Figure 9 includes: 1) data collection step; 2) data processing step; 3) t-SNE-based model monitoring algorithm; 4) model monitoring result determination; 5) model monitoring result broadcasting.

The monitoring entity collects measurement data (i.e., measurement values) and training data, and preprocesses the data to obtain intermediate quantities (i.e., intermediate data). The monitoring entity uses a t-SNE-based model monitoring algorithm to reduce the dimension of the data to obtain a two-dimensional point set, and performs clustering calculations on the two-dimensional point set data to obtain cluster clusters corresponding to the measurement data and training data, i.e., the first cluster and the second cluster, respectively. The monitoring entity determines the monitoring results based on the cluster center point and cluster radius, and broadcasts the model monitoring results to other entities that deploy the data processing model to complete the model monitoring.

Taking the scenario of using AI/ML models for positioning enhancement in 5G NR positioning as an example, in 5G NR positioning, the positioning framework includes UE, gNB, LMF, PRU, etc. The positioning model is referred to as the ML model.

Depending on the entity where the monitoring entity is located, the model monitoring scenarios are divided into: 1) the scenario where the monitoring entity is deployed on the UE as shown in Figures 10 to 12; 2) the scenario where the monitoring entity is deployed on the gNB as shown in Figures 13 to 15; 3) the scenario where the monitoring entity is deployed on the LMF end as shown in Figure 16.

Each scenario is described below.

Scenario 1) is suitable for UE-based AI/ML positioning using the UE-side ML model, using the AI/ML model direct or assisted positioning framework, and using the UE-side ML model based on the LMF end or other UE-assisted AI/ML positioning, using the AI/ML model assisted positioning framework. In the embodiment of the present application, scenario 1) can be subdivided into 3 sub-scenarios 11)-13).

In sub-scenario 11), the UE measures the RS (Reference Signal) from the gNB, that is, the first reference signal, which is a downlink reference signal, and obtains the measurement value required for model monitoring (that is, the first measurement data) through measurement.

As shown in FIG10 , the LMF sends a monitoring configuration signaling, such as monitoring configuration, to the UE, and the monitoring configuration signaling carries the second configuration information. In each monitoring period, the LMF sends a reference signal configuration request signaling (such as RS configuration request), i.e., a first request, to the gNB. The gNB sends a reference signal configuration (RS configuration) to the UE according to the reference signal configuration request signaling. The RS is sent to the UE, and the reference signal configuration is the first configuration information. The UE measures the RS (such as RS measurement), obtains a measurement result, and performs model monitoring according to the measurement data included in the measurement result.

If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the UE sends a request assistance signaling (request assistance) to the LMF end, that is, the third request, and the LMF end feeds back a request acceptance signaling or a request rejection signaling (request acceptance/rejection) to the UE based on its own capabilities. The request acceptance signaling is the first response, and the request rejection signaling is the second response.

After the UE receives the feedback request acceptance signaling (i.e., acceptance), the LMF sends an assistance configuration signaling to the UEs/PRUs (i.e., the fourth entity) at the known location. The assistance configuration signaling carries the third configuration information. The UEs/PRUs feed back the assistance measurement result signaling to the LMF. The assistance measurement result signaling carries the second measurement data of the second reference signal. The LMF receives the feedback assistance measurement result signaling sent by the UEs/PRUs, forms an assistance dataset, and sends it to the UE, which performs model updating.

In sub-scenario 12), the gNB measures the RS from the UE, that is, the first reference signal. At this time, the first reference signal is an uplink reference signal, such as SRS. The gNB obtains the measurement value data (that is, the first measurement data) required for model monitoring through measurement and transmits it back to the UE.

As shown in Figure 11, the LMF sends a monitoring configuration signaling to the UE, and the monitoring configuration signaling carries the second configuration information. In each monitoring period, the LMF sends a reference signal configuration request signaling (such as SRS configuration request) to the gNB, that is, the first request. The gNB sends a reference signal configuration (SRS configuration) to the UE according to the reference signal configuration request signaling, and receives the SRS sent by the UE. The gNB measures the SRS, obtains the measurement result, and sends the measurement result to the UE, and the UE performs model monitoring according to the measurement data included in the measurement result.

If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the UE sends a request for assistance signaling, that is, a third request, to the LMF end. The LMF end feeds back a request for acceptance signaling or a request for rejection signaling to the UE according to its own capabilities.

After the UE is fed back with the request acceptance signaling, the LMF end sends an auxiliary configuration signaling to the UEs/PRUs (i.e., the fourth entity) at a known location, and the auxiliary configuration signaling carries the third configuration information. The UEs/PRUs feed back the auxiliary measurement result signaling to the LMF end, and the auxiliary measurement result signaling carries the second measurement data of the second reference signal. The LMF end receives the feedback auxiliary measurement result signaling sent by the UEs/PRUs, forms an auxiliary data set, and sends it to the UE, and the UE updates the model.

Sub-scenario 13), considering that the LMF end may have existing measurement values, the UE directly requests the measurement data (ie, the first measurement data) from the LMF end.

As shown in FIG12 , the LMF sends a monitoring configuration signaling to the UE, and the monitoring configuration signaling carries the second configuration information. In each monitoring period, the UE sends a request measurement result signaling (request measurement result), i.e., the second request, to the LMF. The LMF sends a measurement result signaling to the UE, and the measurement result signaling carries the measurement result, and the UE monitors the model according to the measurement data included in the measurement result.

If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the UE sends a request for assistance signaling, that is, a third request, to the LMF end. The LMF end feeds back a request for acceptance signaling or a request for rejection signaling to the UE according to its own capabilities.

After the UE is fed back with the request acceptance signaling, the LMF end sends an auxiliary configuration signaling to the UEs/PRUs (i.e., the fourth entity) at a known location, and the auxiliary configuration signaling carries the third configuration information. The UEs/PRUs feed back the auxiliary measurement result signaling to the LMF end, and the auxiliary measurement result signaling carries the second measurement data of the second reference signal. The LMF end receives the feedback auxiliary measurement result signaling sent by the UEs/PRUs, forms an auxiliary data set, and sends it to the UE, and the UE updates the model.

Scenario 2) is suitable for base station-assisted AI/ML positioning using the gNB-side ML model, using an AI/ML model-assisted positioning framework. In the embodiment of the present application, scenario 2) can be subdivided into three sub-scenarios 21)-23).

In sub-scenario 21), the gNB measures the RS from the UE, that is, the first reference signal. At this time, the first reference signal is an uplink reference signal, such as SRS. The gNB obtains the measurement value data (that is, the first measurement data) required for model monitoring through measurement.

As shown in Figure 13, the LMF sends a monitoring configuration signaling to the gNB, and the monitoring configuration signaling carries the second configuration information. In each monitoring period, the LMF sends a reference signal configuration request signaling, i.e., a first request, to the gNB. The gNB sends a reference signal configuration to the UE according to the reference signal configuration request signaling, and receives the SRS sent by the UE. The gNB measures the SRS, obtains the measurement result, and performs model monitoring according to the measurement data included in the measurement result.

If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the gNB sends a request to the LMF end Auxiliary signaling, i.e., the third request. The LMF end feeds back a request to accept signaling or a request to reject signaling to the gNB based on its own capabilities.

After the request acceptance signaling is fed back to the gNB, the LMF sends an auxiliary configuration signaling to the UEs/PRUs (i.e., the fourth entity) at the known location, and the auxiliary configuration signaling carries the third configuration information. The UEs/PRUs feed back the auxiliary measurement result signaling to the LMF, and the auxiliary measurement result signaling carries the second measurement data of the second reference signal. The LMF receives the feedback auxiliary measurement result signaling sent by the UEs/PRUs, forms an auxiliary data set, and sends it to the gNB, and the gNB updates the model.

In sub-scenario 22), the UE measures the RS from the gNB, i.e., the first reference signal. At this time, the first reference signal is a downlink reference signal. The UE obtains the measurement value data (i.e., the first measurement data) required for model monitoring through measurement and transmits it back to the gNB.

As shown in Figure 14, the LMF sends a monitoring configuration signaling to the gNB, and the monitoring configuration signaling carries the second configuration information. In each monitoring period, the LMF sends a reference signal configuration request signaling, i.e., a first request, to the gNB. The gNB sends a reference signal configuration to the UE according to the reference signal configuration request signaling, and sends the RS to the UE. The UE measures the RS, obtains the measurement result, and sends the measurement result to the gNB, and the gNB performs model monitoring according to the measurement data included in the measurement result.

If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the gNB sends a request for assistance signaling to the LMF, that is, the third request. The LMF feeds back a request to accept signaling or a request to reject signaling to the gNB based on its own capabilities.

After the request acceptance signaling is fed back to the gNB, the LMF sends an auxiliary configuration signaling to the UEs/PRUs (i.e., the fourth entity) at the known location, and the auxiliary configuration signaling carries the third configuration information. The UEs/PRUs feed back the auxiliary measurement result signaling to the LMF, and the auxiliary measurement result signaling carries the second measurement data of the second reference signal. The LMF receives the feedback auxiliary measurement result signaling sent by the UEs/PRUs, forms an auxiliary data set, and sends it to the gNB, and the gNB updates the model.

Sub-scenario 23), considering that the LMF may have existing measurement values, the gNB can directly request the measurement data (i.e., the first measurement data) from the LMF.

As shown in Figure 15, the LMF sends a monitoring configuration signaling to the gNB, and the monitoring configuration signaling carries the second configuration information. In each monitoring period, the gNB sends a measurement result request signaling, i.e., the second request, to the LMF. The LMF sends a measurement result signaling to the gNB, and the gNB monitors the model according to the measurement data included in the measurement result.

If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the gNB sends a request for assistance signaling to the LMF, that is, the third request. The LMF feeds back a request to accept signaling or a request to reject signaling to the gNB based on its own capabilities.

After the request acceptance signaling is fed back to the gNB, the LMF sends an auxiliary configuration signaling to the UEs/PRUs (i.e., the fourth entity) at the known location, and the auxiliary configuration signaling carries the third configuration information. The UEs/PRUs feed back the auxiliary measurement result signaling to the LMF, and the auxiliary measurement result signaling carries the second measurement data of the second reference signal. The LMF receives the feedback auxiliary measurement result signaling sent by the UEs/PRUs, forms an auxiliary data set, and sends it to the gNB, and the gNB updates the model.

Scenario 3) is suitable for LMF-based or UE-assisted AI/ML positioning using the LMF side ML model, direct positioning framework using the AI/ML model, and base station-assisted AI/ML positioning using the LMF side ML model, direct positioning framework using the AI/ML model.

As shown in Figure 16, the UE/gNB sends the measurement result of the reference signal (RS/SRS measurement result) to the LMF end, and the LMF end performs model monitoring according to its own second configuration information and the measurement data carried by the received measurement result. The process of the UE/gNB obtaining the measurement data of the reference signal can be referred to the relevant description of Figures 10 to 15 above, which will not be repeated here.

If the model monitoring result indicates that the ML model is unavailable, the LMF sends auxiliary configuration signaling to the UEs/PRUs at known locations. The UEs/PRUs feed back auxiliary measurement result signaling to the LMF, and the LMF receives the feedback auxiliary measurement result signaling sent by the UEs/PRUs, forms an auxiliary data set, and updates the model based on the measurement data.

A model monitoring method based on t-SNE data dimensionality reduction proposed in the embodiment of the present application can be used in monitoring scenarios without true value labels, and can effectively monitor whether there is a large difference between the distribution of model training sets and true measurement values. The t-SNE algorithm is a new algorithm that is further optimized and designed based on the traditional SNE algorithm to solve the congestion problem. t-SNE is mostly used for the visualization of high-dimensional data. The t-SNE algorithm can be used to map high-dimensional data to low-dimensional space while retaining the relative relationship between high-dimensional data as much as possible. Finally, the relative relationship between the mapping points in two-dimensional or one-dimensional space is used to characterize the relative relationship between high-dimensional data.

The distribution of the measured data after dimensionality reduction using the t-SNE algorithm can be seen in Figures 17 to 19. In Figures 17 to 19, the right sub-figures are the geographical location distribution of the CIR samples, and their horizontal and vertical coordinates represent the positions in the X and Y directions, respectively, in meters. The left sub-figures are the distribution of the CIR samples after dimensionality reduction by t-SNE on the two-dimensional XY mapping plane, and their horizontal and vertical coordinates represent the coordinate values in the X and Y directions, respectively. The sample points of two colors represent the measurement data and training data respectively.

Figure 17 shows the t-SNE dimensionality reduction and k-means clustering results of sample points in the same distribution area. Figure 18 shows the t-SNE dimensionality reduction and k-means clustering results of sample points in adjacent distribution areas. Figure 19 shows the t-SNE dimensionality reduction and k-means clustering results of sample points in distribution areas that are a certain distance apart.

The t-SNE model monitoring algorithm is used to detect the model input data. As the actual distribution of the two sets of data continues to shift, the discrimination of the clustering results after t-SNE dimensionality reduction continues to expand. Therefore, the t-SNE algorithm can be used to process the model input to reflect the differences in the geographical distribution characteristics of the data, so as to further determine whether the distribution of the model input data has drifted relative to the training set, and to achieve effective model monitoring that does not rely on the true value label. Moreover, in the actual use of the model, there is no need to train the network, only to pass the threshold to complete the model monitoring function. The computational processing complexity is greatly reduced.

In the technical solution provided in the embodiment of the present application, for the problem of model monitoring without truth value labels in 5G NR AI positioning scenarios, the embodiment of the present application provides a method and workflow for evaluating model performance using ML model input data when there are no truth value labels by designing a model monitoring algorithm without truth value labels and a model monitoring implementation process, and to a certain extent alleviates the waste of computing resources and time caused by the existing semi-supervised learning to re-label data and repeatedly train the network parameters of the ML model.

Corresponding to the above-mentioned model monitoring method, the embodiment of the present application further provides a model monitoring device, as shown in FIG20 , which is applied to a monitoring entity, and the device includes:

A first acquisition module 201 is used to acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of a data processing model, and the first measurement data is input data of the data processing model;

A second acquisition module 202, used to acquire training data of the data processing model;

A first determination module 203, configured to determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs;

The second determination module 204 is used to determine the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

In some embodiments, when the monitoring entity is a first entity, the first acquisition module 201 is specifically configured to:

receiving a first reference signal sent by a second entity within a current monitoring period;

The first reference signal is measured to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal.

In some embodiments, when the first entity is a terminal, the second entity is a base station, and the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS;

When the first entity is a base station, the second entity is a terminal, and the first reference signal is an SRS;

The first measurement data includes CIR and PDP.

In some embodiments, when the monitoring entity is a first entity, the first acquisition module 201 is specifically configured to:

Sending a first reference signal to the second entity within a current monitoring period;

A second measurement result sent by the second entity is received, where the second measurement result includes first measurement data of the first reference signal.

In some embodiments, when the first entity is a terminal, the second entity is a base station, and the first reference signal is an SRS;

When the first entity is a base station, the second entity is a terminal, and the first reference signal is a PRS, a CSI-RS, an SRS, a SSB, a DMRS, or a PTRS;

The first measurement data includes CIR and PDP.

In some embodiments, the first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates time-frequency resources occupied by the first reference signal.

In some embodiments, the first configuration information is configuration information sent by the base station to the terminal according to a first request sent by a third entity, and the first request instructs the base station to send the first configuration information to the terminal.

In some embodiments, when the monitoring entity is a first entity, the first acquisition module 201 is specifically configured to:

Sending a second request to a third entity, wherein the third entity stores a first reference signal between the first entity and the second entity a measurement data, wherein the second request instructs the third entity to send the first measurement data to the monitoring entity;

A third measurement result corresponding to the second request sent by the third entity is received, where the third measurement result includes the first measurement data.

In some embodiments, the first entity is a terminal or a base station, and the third entity is a management entity;

The first measurement data includes CIR and PDP.

In some embodiments, when the monitoring entity is a third entity, the first acquisition module 201 is specifically configured to:

First measurement data of a first reference signal between the first entity and the second entity in a current monitoring period is acquired from a first target entity, where the first target entity is an entity between the first entity and the second entity that measures the first reference signal.

In some embodiments, the third entity is a management entity;

When the first entity is a terminal, the second entity is a base station, and the first target entity is the terminal, the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS;

When the first entity is a base station, the second entity is a terminal, and the first target entity is the base station, the first reference signal is an SRS;

The first measurement data includes channel CIR and PDP.

In some embodiments, the first measurement data is first measurement data of the first reference signal obtained by the monitoring entity according to second configuration information sent by a third entity.

In some embodiments, the second configuration information includes at least one of the following: monitoring cycle information, measurement configuration information, a monitoring algorithm, and the preset monitoring threshold.

In some embodiments, the monitoring cycle information includes a cycle unit and a bit number.

In some embodiments, the measurement configuration information includes a measurement cycle length, a measurement time slice length, and a measurement frequency.

In some embodiments, the measurement configuration information, the monitoring algorithm and the preset monitoring threshold are represented by a number of bits.

In some embodiments, the first determining module 203 is specifically configured to:

According to a preset monitoring algorithm, performing dimensionality reduction processing on the first measurement data and the training data to obtain dimensionality reduced data;

The dimension-reduced data is clustered to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs.

In some embodiments, the first determining module 203 is specifically configured to:

converting the first measurement data and the training data into intermediate data matching the data processing model;

According to a preset monitoring algorithm, the intermediate data is subjected to dimensionality reduction processing to obtain dimensionality reduced data.

In some embodiments, the preset monitoring algorithm is a t-SNE algorithm.

In some embodiments, the second determining module 204 is specifically configured to:

Calculating a ratio of a first distance to a second distance to obtain a monitoring value, wherein the first distance is a Euclidean distance between a center point of the first cluster and a center point of the second cluster, and the second distance is a sum of a radius of the first cluster and a radius of the second cluster;

If the monitoring value is greater than the preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable;

If the monitoring value is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

In some embodiments, the second determining module 204 is specifically configured to:

If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable;

If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

In some embodiments, when the data processing model is deployed on the monitoring entity, the apparatus further comprises:

a third acquisition module, configured to, when the monitoring result indicates that the data processing model is unavailable, acquire second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, wherein the fourth entity is located in the service area, and a processing result of the data processing model corresponding to the fourth entity is known, and the second target entity is the first entity and the second entity sending the second measurement data of the second reference signal. an entity of a second reference signal;

An updating module is used to update the data processing model according to the second measurement data.

In some embodiments, the third acquisition module is specifically used to:

A fourth measurement result is obtained from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity.

In some embodiments, when the monitoring entity is the first entity, the third acquisition module is specifically used to:

Sending a third request to a third entity, wherein the third request instructs the third entity to send the second measurement data to the monitoring entity;

A fourth measurement result corresponding to the third request sent by the third entity is received.

In some embodiments, the third request includes a minimum number of samples, and the fourth measurement result includes a number of second measurement data that is greater than or equal to the minimum number of samples.

In some embodiments, the third acquisition module is further used to:

receiving a first response or a second response corresponding to the third request sent by the third entity, the first response indicating that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicating that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity;

After receiving the first response, the step of receiving a fourth measurement result corresponding to the third request sent by the third entity is performed.

In some embodiments, the minimum number of samples is expressed in units of quantity and bits.

In some embodiments, the second measurement data is measurement data sent by the fourth entity to the third entity according to third configuration information of the second reference signal sent by the third entity.

In some embodiments, the third configuration information includes at least one of the following: measurement-related information of the second reference signal and an identifier of a monitoring entity.

In some embodiments, the second measurement data includes at least one of the following: a measurement value, a true value label corresponding to the measurement value, and a data quality corresponding to the true value label.

In some embodiments, the measurement value, the true value label, and the data quality are represented by the number of bits.

In some embodiments, the fourth entity is a PRU or a terminal, and the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold.

In some embodiments, the apparatus further comprises:

The first sending module is used to send monitoring capability information to the third entity.

In some embodiments, when the data processing model is deployed on the monitoring entity, the apparatus further comprises:

A receiving module, configured to receive a fourth request sent by a third entity, wherein the fourth request indicates obtaining monitoring capability information;

The first sending module is specifically configured to send monitoring capability information to the third entity according to the fourth request.

In some embodiments, the monitoring capability information includes at least one of the following: a maximum number of samples supported by the monitoring entity.

In some embodiments, when the data processing model is deployed on a fifth entity, the apparatus further comprises:

The second sending module is used to send the monitoring result to the fifth entity after obtaining the monitoring result.

In the technical solution provided by the embodiment of the present application, the monitoring entity clusters the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model into two clusters, namely, the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, and compares the Euclidean distance between the center point of the first cluster and the center point of the second cluster with the preset monitoring threshold, so as to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring result. The difference between the input data of the data processing model obtained in actual production and the training data of the data processing model indicates that the environmental data is offset and the data processing model cannot accurately process the input data in actual production. Using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

Corresponding to the above-mentioned model monitoring method, the embodiment of the present application also provides a monitoring entity, as shown in FIG21, including a processor 211, a communication interface 212, a memory 213 and a communication bus 214, wherein the processor 211, the communication interface 212, and the memory 213 communicate with each other through the communication bus 214. Line 214 completes the communication between them;

The memory 213 is used to store computer programs;

The processor 211 is used to implement any of the above-mentioned model monitoring method steps when executing the program stored in the memory 213.

The communication bus mentioned by the above monitoring entity can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above monitoring entity and other devices.

The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processors can be general-purpose processors, including central processing units (CPU), network processors (NP), etc.; they can also be digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

Corresponding to the above-mentioned model monitoring method, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned steps of the model monitoring method is implemented.

In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the model monitoring method described in any one of the above embodiments.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, monitoring entity, storage medium, and program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims (74)

A model monitoring method, characterized in that it is applied to a monitoring entity, and the method comprises: Acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, the first entity and the second entity being located in a service area of a data processing model, and the first measurement data being input data of the data processing model; Obtaining training data for the data processing model; Determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs; The monitoring result of the data processing model is determined according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold. The method according to claim 1, characterized in that when the monitoring entity is the first entity, the step of obtaining first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period comprises: receiving a first reference signal sent by a second entity within a current monitoring period; The first reference signal is measured to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal. The method according to claim 2, characterized in that when the first entity is a terminal, the second entity is a base station, and the first reference signal is a positioning reference signal PRS, a channel state information reference signal CSI-RS, a sounding reference signal SRS, a synchronization signal block SSB, a demodulation reference signal DMRS, or a phase tracking reference signal PTRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is an SRS; The first measurement data includes a channel impulse response CIR and a power delay profile PDP. The method according to claim 1, characterized in that when the monitoring entity is the first entity, the step of obtaining first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period comprises: Sending a first reference signal to the second entity within a current monitoring period; A second measurement result sent by the second entity is received, where the second measurement result includes first measurement data of the first reference signal. The method according to claim 4, characterized in that when the first entity is a terminal, the second entity is a base station, and the first reference signal is an SRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is a PRS, a CSI-RS, an SRS, a SSB, a DMRS, or a PTRS; The first measurement data includes CIR and PDP. The method according to claim 3 or 5 is characterized in that the first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal. The method according to claim 6 is characterized in that the first configuration information is configuration information sent by the base station to the terminal according to a first request sent by a third entity, and the first request instructs the base station to send the first configuration information to the terminal. The method according to claim 1, characterized in that when the monitoring entity is the first entity, the step of obtaining first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period comprises: Sending a second request to a third entity, wherein the third entity stores first measurement data of a first reference signal between the first entity and the second entity, and the second request instructs the third entity to send the first measurement data to the monitoring entity; A third measurement result corresponding to the second request sent by the third entity is received, where the third measurement result includes the first measurement data. The method according to claim 8, characterized in that the first entity is a terminal or a base station, and the third entity is a management entity; The first measurement data includes CIR and PDP. The method according to claim 1, characterized in that when the monitoring entity is a third entity, the step of obtaining first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period comprises: First measurement data of a first reference signal between the first entity and the second entity in a current monitoring period is acquired from a first target entity, where the first target entity is an entity between the first entity and the second entity that measures the first reference signal. The method according to claim 10, characterized in that the third entity is a management entity; When the first entity is a terminal, the second entity is a base station, and the first target entity is the terminal, the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS; When the first entity is a base station, the second entity is a terminal, and the first target entity is the base station, the first reference signal is an SRS; The first measurement data includes CIR and PDP. The method according to any one of claims 1-5 and 8-11 is characterized in that the first measurement data is first measurement data of the first reference signal obtained by the monitoring entity according to second configuration information sent by a third entity. The method according to claim 12 is characterized in that the second configuration information includes at least one of the following: monitoring cycle information, measurement configuration information, a monitoring algorithm and the preset monitoring threshold. The method according to claim 13 is characterized in that the monitoring cycle information includes a cycle unit and a bit number. The method according to claim 13 is characterized in that the measurement configuration information includes a measurement cycle length, a measurement time slice length and a measurement frequency. The method according to claim 13 is characterized in that the measurement configuration information, the monitoring algorithm and the preset monitoring threshold are represented by the number of bits. The method according to claim 1, characterized in that the step of determining the first cluster to which the first measurement data belongs and the second cluster to which the training data belongs comprises: According to a preset monitoring algorithm, performing dimensionality reduction processing on the first measurement data and the training data to obtain dimensionality reduced data; The dimension-reduced data is clustered to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs. The method according to claim 17, characterized in that the step of performing dimensionality reduction processing on the first measurement data and the training data according to a preset monitoring algorithm to obtain dimensionality reduced data comprises: converting the first measurement data and the training data into intermediate data matching the data processing model; According to a preset monitoring algorithm, the intermediate data is subjected to dimensionality reduction processing to obtain dimensionality reduced data. The method according to claim 17 or 18 is characterized in that the preset monitoring algorithm is a t-distributed random neighborhood embedding algorithm. The method according to claim 1, characterized in that the step of determining the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold comprises: Calculating a ratio of a first distance to a second distance to obtain a monitoring value, wherein the first distance is a Euclidean distance between a center point of the first cluster and a center point of the second cluster, and the second distance is a sum of a radius of the first cluster and a radius of the second cluster; If the monitoring value is greater than the preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the monitoring value is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available. The method according to claim 1, characterized in that the step of determining the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold comprises: If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available. The method according to claim 1, characterized in that when the data processing model is deployed on the monitoring entity, the method further comprises: When the monitoring result indicates that the data processing model is unavailable, obtaining second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, the fourth entity being located in the service area, and a processing result of the data processing model corresponding to the fourth entity being known, and the second target entity being an entity of the first entity and the second entity that sends the second reference signal; The data processing model is updated according to the second measurement data. The method according to claim 22, characterized in that the step of obtaining second measurement data of a second reference signal between the second target entity and the fourth entity in the current monitoring period comprises: A fourth measurement result is obtained from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity. The method according to claim 23, characterized in that when the monitoring entity is the first entity, the step of obtaining the fourth measurement result from the third entity comprises: Sending a third request to a third entity, wherein the third request instructs the third entity to send the second measurement data to the monitoring entity; A fourth measurement result corresponding to the third request sent by the third entity is received. The method according to claim 24 is characterized in that the third request includes a minimum number of samples, and the number of second measurement data included in the fourth measurement result is greater than or equal to the minimum number of samples. The method according to claim 25, characterized in that the method further comprises: receiving a first response or a second response corresponding to the third request sent by the third entity, the first response indicating that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicating that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity; After receiving the first response, the step of receiving a fourth measurement result corresponding to the third request sent by the third entity is performed. The method according to claim 25 or 26 is characterized in that the minimum number of samples is expressed in quantity units and number of bits. The method according to claim 23 is characterized in that the second measurement data is measurement data sent by the fourth entity to the third entity according to third configuration information of the second reference signal sent by the third entity. The method according to claim 28 is characterized in that the third configuration information includes at least one of the following: measurement-related information of the second reference signal and an identifier of a monitoring entity. The method according to any one of claims 22-26, 28-29 is characterized in that the second measurement data includes at least one of the following: a measurement value, a true value label corresponding to the measurement value, and a data quality corresponding to the true value label. The method according to claim 30 is characterized in that the measurement value, the true value label, and the data quality are represented by the number of bits. The method according to any one of claims 22-26, 28-29 is characterized in that the fourth entity is a positioning reference unit PRU or a terminal, and the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold. The method according to claim 1, characterized in that the method further comprises: The monitoring capability information is sent to the third entity. The method according to claim 33, characterized in that when the data processing model is deployed on the monitoring entity, the method further comprises: receiving a fourth request sent by the third entity, where the fourth request indicates obtaining monitoring capability information; According to the fourth request, the step of sending monitoring capability information to the third entity is performed. The method according to claim 33 or 34 is characterized in that the monitoring capability information includes at least one of the following: the maximum number of samples supported by the monitoring entity. The method according to claim 1, characterized in that when the data processing model is deployed on the fifth entity, after obtaining the monitoring result, the method further comprises: The monitoring result is sent to the fifth entity. A model monitoring device, characterized in that it is applied to a monitoring entity, and the device comprises: A first acquisition module, configured to acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of a data processing model, and the first measurement data is input data of the data processing model; A second acquisition module, used to acquire training data of the data processing model; A first determination module, configured to determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs; The second determination module is used to determine the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold. The device according to claim 37, characterized in that when the monitoring entity is a first entity, the first acquisition module is specifically used to: receiving a first reference signal sent by a second entity within a current monitoring period; The first reference signal is measured to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal. The device according to claim 38, characterized in that when the first entity is a terminal, the second entity is a base station, and the first reference signal is a positioning reference signal PRS, a channel state information reference signal CSI-RS, a sounding reference signal SRS, a synchronization signal block SSB, a demodulation reference signal DMRS, or a phase tracking reference signal PTRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is an SRS; The first measurement data includes a channel impulse response CIR and a power delay profile PDP. The device according to claim 37, characterized in that when the monitoring entity is a first entity, the first acquisition module is specifically used to: Sending a first reference signal to the second entity within a current monitoring period; A second measurement result sent by the second entity is received, where the second measurement result includes first measurement data of the first reference signal. The apparatus according to claim 40, wherein when the first entity is a terminal, the second entity is a base station, and the first reference signal is an SRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is a PRS, a CSI-RS, an SRS, a SSB, a DMRS, or a PTRS; The first measurement data includes CIR and PDP. The device according to claim 39 or 41 is characterized in that the first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal. The device according to claim 42 is characterized in that the first configuration information is configuration information sent by the base station to the terminal according to a first request sent by a third entity, and the first request instructs the base station to send the first configuration information to the terminal. The device according to claim 37, characterized in that when the monitoring entity is a first entity, the first acquisition module is specifically used to: Sending a second request to a third entity, wherein the third entity stores first measurement data of a first reference signal between the first entity and the second entity, and the second request instructs the third entity to send the first measurement data to the monitoring entity; A third measurement result corresponding to the second request sent by the third entity is received, where the third measurement result includes the first measurement data. The apparatus according to claim 44, wherein the first entity is a terminal or a base station, and the third entity is a management entity; The first measurement data includes CIR and PDP. The device according to claim 37, characterized in that when the monitoring entity is a third entity, the first acquisition module is specifically used to: First measurement data of a first reference signal between the first entity and the second entity in a current monitoring period is acquired from a first target entity, where the first target entity is an entity between the first entity and the second entity that measures the first reference signal. The apparatus according to claim 46, wherein the third entity is a management entity; When the first entity is a terminal, the second entity is a base station, and the first target entity is the terminal, the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS; When the first entity is a base station, the second entity is a terminal, and the first target entity is the base station, the first reference signal is an SRS; The first measurement data includes CIR and PDP. The device according to any one of claims 37-41 and 44-47 is characterized in that the first measurement data is the first measurement data of the first reference signal obtained by the monitoring entity according to the second configuration information sent by the third entity. The device according to claim 48 is characterized in that the second configuration information includes at least one of the following: monitoring cycle information, measurement configuration information, a monitoring algorithm and the preset monitoring threshold. The device according to claim 49 is characterized in that the monitoring cycle information includes a cycle unit and a number of bits. The device according to claim 49 is characterized in that the measurement configuration information includes a measurement cycle length, a measurement time slice length and a measurement frequency. The device according to claim 49 is characterized in that the measurement configuration information, the monitoring algorithm and the preset monitoring threshold are represented by the number of bits. The device according to claim 37, characterized in that the first determining module is specifically configured to: According to a preset monitoring algorithm, performing dimensionality reduction processing on the first measurement data and the training data to obtain dimensionality reduced data; The dimension-reduced data is clustered to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs. The device according to claim 53, characterized in that the first determining module is specifically configured to: converting the first measurement data and the training data into intermediate data matching the data processing model; According to a preset monitoring algorithm, the intermediate data is subjected to dimensionality reduction processing to obtain dimensionality reduced data. The device according to claim 53 or 54 is characterized in that the preset monitoring algorithm is a t-distributed random neighborhood embedding algorithm. The device according to claim 37, characterized in that the second determining module is specifically configured to: Calculating a ratio of a first distance to a second distance to obtain a monitoring value, wherein the first distance is a Euclidean distance between a center point of the first cluster and a center point of the second cluster, and the second distance is a sum of a radius of the first cluster and a radius of the second cluster; If the monitoring value is greater than the preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the monitoring value is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available. The device according to claim 37, characterized in that the second determining module is specifically configured to: If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available. The device according to claim 37, characterized in that when the data processing model is deployed on the monitoring entity, the device further comprises: a third acquisition module, configured to, when the monitoring result indicates that the data processing model is unavailable, acquire second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, wherein the fourth entity is located in the service area, and a processing result of the data processing model corresponding to the fourth entity is known, and the second target entity is an entity of the first entity and the second entity that sends the second reference signal; An updating module is used to update the data processing model according to the second measurement data. The device according to claim 58, characterized in that the third acquisition module is specifically used to: A fourth measurement result is obtained from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity. The device according to claim 59, characterized in that when the monitoring entity is the first entity, the third acquisition module is specifically used to: Sending a third request to a third entity, wherein the third request instructs the third entity to send the second measurement data to the monitoring entity; A fourth measurement result corresponding to the third request sent by the third entity is received. The device according to claim 60 is characterized in that the third request includes a minimum number of samples, and the number of second measurement data included in the fourth measurement result is greater than or equal to the minimum number of samples. The device according to claim 61, characterized in that the third acquisition module is further used to: receiving a first response or a second response corresponding to the third request sent by the third entity, the first response indicating that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicating that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity; After receiving the first response, the step of receiving a fourth measurement result corresponding to the third request sent by the third entity is performed. The device according to claim 61 or 62 is characterized in that the minimum number of samples is expressed in quantitative units and number of bits. The device according to claim 59 is characterized in that the second measurement data is measurement data sent by the fourth entity to the third entity according to third configuration information of the second reference signal sent by the third entity. The device according to claim 64 is characterized in that the third configuration information includes at least one of the following: measurement-related information of the second reference signal and an identifier of the monitoring entity. The device according to any one of claims 58-62, 64-65 is characterized in that the second measurement data includes at least one of the following: a measurement value, a true value label corresponding to the measurement value, and a data quality corresponding to the true value label. The device according to claim 66 is characterized in that the measurement value, the true value label, and the data quality are represented by the number of bits. The device according to any one of claims 58-62, 64-65 is characterized in that the fourth entity is a positioning reference unit PRU or a terminal, and the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold. The device according to claim 37, characterized in that the device further comprises: The first sending module is used to send monitoring capability information to the third entity. The device according to claim 69, characterized in that when the data processing model is deployed on the monitoring entity, the device further comprises: A receiving module, configured to receive a fourth request sent by a third entity, wherein the fourth request indicates obtaining monitoring capability information; The first sending module is specifically configured to send monitoring capability information to the third entity according to the fourth request. The device according to claim 69 or 70 is characterized in that the monitoring capability information includes at least one of the following: the maximum number of samples supported by the monitoring entity. The apparatus according to claim 37, characterized in that when the data processing model is deployed on a fifth entity, the apparatus further comprises: The second sending module is used to send the monitoring result to the fifth entity after obtaining the monitoring result. A monitoring entity, characterized in that it comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor, when used to execute the program stored in the memory, implements the method steps described in any one of claims 1-36. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of claims 1-36 are implemented.