CN119396786B

CN119396786B - Scientific satellite data processing system crossing APID mixed dependency relationship

Info

Publication number: CN119396786B
Application number: CN202411408086.1A
Authority: CN
Inventors: 魏明月; 马福利; 于勤思; 佟继周; 邹自明
Original assignee: National Space Science Center of CAS
Current assignee: National Space Science Center of CAS
Priority date: 2024-10-10
Filing date: 2024-10-10
Publication date: 2025-05-13
Anticipated expiration: 2044-10-10
Also published as: CN119396786A

Abstract

The invention belongs to the field of scientific satellite data processing, and discloses a scientific satellite data processing system crossing APID mixed dependency relationship, which comprises an original data preprocessing module, an engineering parameter template and dependency matrix analysis module, a Redis cache library sharing architecture module, an APID crossing dependency parameter scheduling module and a data product formatting generation module, wherein the original data preprocessing module is used for preprocessing multi-source data and generating 0-level products, the engineering parameter template and dependency matrix analysis module is used for analyzing the 0-level products according to a scientific satellite task engineering parameter template of a parameter analysis configuration file and a parameter dependency matrix crossing APID source package, the Redis cache library sharing architecture module is used for sharing parameter values of each data product crossing the APID source package by using a Redis cache, the APID crossing dependency parameter scheduling module is used for scheduling processing sequences according to different data types and dependency conditions, and the data product formatting generation module is used for generating various types and levels of products based on product definition specifications after all parameter processing is completed and cached.

Description

Scientific satellite data processing system crossing APID mixed dependency relationship

Technical Field

The application belongs to the field of scientific satellite data processing, and particularly relates to a scientific satellite data processing system crossing APID mixed dependency relations.

Background

Along with the increasing complexity of data processing requirements of modern scientific satellites, the requirements of data fusion and analysis among multiple loads and multiple tasks of the satellites are gradually increased. In a scientific satellite engineering task, parameter resolution and production of scientific data products typically depend on parameters in multiple APID (application process identification, application identification) source packages, which may originate within the same data processing unit or across data processing units (different production modules, different processing batches, different rounds) with complex dependencies between each other, such dependency across APIDs across rounds increasing the complexity of data processing.

Conventional parameter resolution methods face many challenges in handling these cross-APID cross-turn dependencies. For example, conventional methods rely on predefined parsing rules or sequential processing schemes, are difficult to flexibly cope with dependencies between multiple data sources, and are redundant for data processing of different rounds. This limitation results in more complex and lengthy data processing flows, which increases the computational burden on the system.

Therefore, an automated algorithm capable of efficiently processing dependency relationships across APIDs across rounds is needed to ensure consistency, efficiency and accuracy of scientific data processing and product production.

Disclosure of Invention

The invention aims to overcome the defects of the prior art, provides a scientific satellite data processing system crossing APID mixed dependency relationship, and is successfully applied to a medium astronomical Satellite (SVOM) engineering task, so that the correct and efficient processing of SVOM satellite downlink data is ensured.

In order to achieve the above object, the present invention provides a scientific satellite data processing system crossing APID hybrid dependencies, comprising:

the original data preprocessing module is used for preprocessing the multi-source data and generating a 0-level product;

The engineering parameter template and dependency matrix analysis module is used for analyzing the 0-level product according to the scientific satellite task engineering parameter template of the parameter analysis configuration file and the parameter dependency matrix crossing the APID source package;

The Redis cache library sharing architecture module is used for sharing the parameter values of each data product of the cross-APID source packet by using the Redis cache;

the cross APID dependent parameter scheduling module is used for scheduling the processing sequence according to different data types and dependent conditions;

and the data product formatting generation module is used for generating products of various types and levels based on the product definition specification after all parameter processing is completed and cached.

Preferably, the pretreatment includes:

When receiving the data transmission original file of the ground station, virtual channel separation and source packet extraction of monorail data are carried out firstly, a corresponding 0C-level source packet data product is generated, splicing and weight removal of multi-rail data and statistics and fault tolerance of data jump are carried out, a 0D-level source packet data product is generated, and the 0C-level source packet data product comprises scientific data and engineering data.

Preferably, the scientific satellite task engineering parameter template is used for describing the name, unit, byte size, starting and ending positions in a data packet and a calculation formula of each telemetry parameter in an engineering product, the parameter dependency matrix crossing the APID source packet is used for describing parameter dependency relations in different data products, the rows of the matrix represent parameter codes of target products, and the columns of the matrix represent APID source packet numbers, dependent parameter codes, analysis sequences and analysis method formulas of other dependent parameters.

Preferably, the engineering parameter template and the dependency matrix analysis module process the source package where the dependency parameters are located preferentially, and traverse analysis calculation of each parameter until all the parameters are processed.

Preferably, the method for caching and sharing parameter values of each data product crossing the APID source packet comprises a caching key value and a parameter value, wherein the caching key value comprises a time code, an APID source packet identifier and a parameter code, the expiration time of caching is set based on the on-satellite solid storage capacity of a scientific satellite task to be processed, and random delay of 0 to set seconds is added for staggering the expiration time point of a large amount of cached data, so that the system is prevented from processing a large amount of cache-invalid data in the same time period, and the running stability of the system is ensured.

Preferably, the Redis cache library sharing architecture module is further configured to clear the expired parameter cache according to a set time.

Preferably, the processing procedure of the cross APID dependent parameter scheduling module includes:

Step 1) setting a waiting threshold value of engineering parameter 1-level processing, triggering a parameter processing flow of scientific data when the engineering parameter processing is smaller than the threshold value and parallel engineering parameter processing in the track data is completed, and turning to step 2) when the parameter X to be calculated depends on parameters of other APID source packets, otherwise turning to step 3);

Step 2) acquiring the data packet time TS of the current parameter, inquiring a Redis cache according to the TS and the dependent parameter codes, acquiring a time code T0 nearest to the TS, and when the TS-T0 is less than or equal to TH, wherein TH represents a set threshold value, substituting the inquired dependent parameter value into a calculation formula of the parameter X, outputting the parameter value of the parameter X, and turning to step 4);

step 3) calculating and time processing of the parameter X;

Step 4) caching the parsed parameter values.

Preferably, the data product formatting generating module is further configured to verify a generated product format to ensure integrity and accuracy of data.

Compared with the prior art, the invention has the advantages that:

1. according to the cross APID mixed dependency processing system, a grading framework of 0C, 0D and 1-level scientific data products is designed, engineering parameter configuration templates and dependency configuration templates are provided, complex parameter dependency relationships among different rounds and different source packages can be effectively processed, and the data analysis, parameter calculation and product generation processes are improved in consistency and processing efficiency through automatic processing of configuration files and a caching mechanism;

2. By using Redis cache, the time cost of repeated data reading and calculation is reduced when cross-module dependence is processed, the data processing flow is optimized, and the resolving efficiency and data consistency of cross-APID dependence parameters are ensured.

Drawings

FIG. 1 is a block diagram of a cross APID hybrid dependent parameter science data processing system of the present invention;

FIG. 2 is an engineering parameter template example;

FIG. 3 is a flow chart of data transmission source packet processing (level 0 processing) for the SVOM satellite example;

FIG. 4 is a flow chart of cross APID cross type hybrid dependent parameter processing (level 1 processing) for the example of SVOM satellites.

Detailed Description

The invention aims to provide an efficient cross APID mixed dependent parameter processing system, which solves the problem of cross-circle cross APID dependence and key engineering parameter sharing in scientific satellite downlink data processing, can efficiently manage dependent parameters under the environment of multi-source packet and multi-task processing, avoids repeated calculation, improves data processing speed and ensures consistency and accuracy of data processing.

As shown in FIG. 1, the invention provides a cross APID hybrid dependent data processing system, and the modularized design scheme is as follows:

1) The original data preprocessing module is used for preprocessing multi-source data and generating 0-level products, including virtual channel separation, source packet extraction, source packet sequencing, data verification of single-track data, splicing, sequencing and duplicate removal of the multi-track data, statistics and fault tolerance of data jump and the like

2) The engineering parameter template and dependency matrix analysis module is used for analyzing the engineering parameter template of the scientific satellite task constructed in advance and the parameter dependency matrix crossing the APID source package, and automatically identifying and classifying the dependency parameters according to the dependency matrix and analysis rules;

3) The Redis cache library sharing architecture module is used for storing key engineering parameters through the Redis cache library, supporting data sharing among different products, adopting a key value structure of time codes and parameter codes, effectively managing the life cycle of cache data, avoiding repeated calculation and improving the data access speed;

4) The cross APID dependency parameter scheduling module schedules the processing sequence of 1-level engineering and scientific data according to different data types (engineering and science) and dependency conditions (whether the data are dependent source packets or not), preferentially triggers the processing flow of 1-level engineering data, firstly reads a 0D-level engineering data product file of the source packets to be processed and carries out parameter calculation, analyzes a configuration file and a analysis dependency matrix according to engineering parameters, schedules and traverses each parameter, calculates the parameters with cross APID dependency in parameter calculation, obtains parameter values with cross APID dependency from a Redis cache, carries out calculation, directly calculates the parameters without cross APID dependency, triggers the production of 1-level scientific data products after the 1-level engineering parameter processing is completed (within a threshold range), obtains the data packet time TS of the current parameters, inquires the Redis cache according to TS and dependent parameter codes, and obtains the nearest engineering parameter value from TS and carries out calculation.

5) And the data product formatting generation module is used for generating products of various types and levels based on the product definition specifications after all parameter analyses are completed and cached.

Through the cooperative work of the modules, the invention can realize the integrated solution of cross APID cross-circle parameter dependent processing, key engineering parameter sharing and data product formatting generation. The algorithm is effectively verified in the engineering task of the astronomical Satellite (SVOM) in the middle method, and the high-efficiency and correct processing of the class 0 and class 1 products of the SVOM satellite is ensured.

The technical scheme of the invention is described in detail below with reference to the accompanying drawings and examples.

Examples

An embodiment of the present invention provides a cross APID hybrid dependent data processing system, comprising:

The following is a detailed description:

1. engineering parameter template analysis and dependency matrix construction

Engineering parameter template definition:

The engineering XML template is used for describing the structure, the position and the calculation method of each parameter in the telemetry data packet. Taking SVOM satellite engineering tasks as an example, the template contains the APID of the data packet, the name, unit, byte size of each telemetry parameter, the start and end positions in the data packet, and the calculation formula (including the dependency relationship inside the engineering product, i.e., the inter-dependency inside the same data processing unit, the same virtual channel downstream file). An engineering parameter template example is shown in fig. 2.

When XML is analyzed, firstly data packet information is extracted, then the original data of the corresponding position is obtained, and calculation is carried out according to a formula. The template defines the calculation information of each parameter, and is convenient for the subsequent processing and maintenance of engineering parameters.

Dependency matrix definition the parameter dependency matrix is used to describe the key parameter dependencies in different types of data products, mainly for different virtual channel downstream files, i.e. different data processing units. Taking SVOM satellite engineering tasks as an example, the GRM L1a scientific data product depends on a plurality of parameters in the HK1 level engineering data product, including load temperature, orbit state, etc. Through the dependency matrix, the dependency relationship between the parameters in one product and the parameters of other types of products can be intuitively and simply displayed, and input is provided for the parameter processing sequence.

The rows of the matrix represent parameters in the target product, the columns represent information such as dependent APID source packets and parameter codes, the analysis sequence columns indicate the sequence of calculation of the parameters of the target product, the data in the same analysis sequence are processed in parallel at the same time, and the analysis method columns give specific calculation formulas. The specific matrix design is shown in table 1:

TABLE 1 dependency parameter matrix schematic

In this example, GRM_TIMES and GRM_TEMP3 can be processed concurrently in order 1, while GRM_V_main depends on the calculation of GRM_TIMES, so GRM_V_main is processed in order 2.

And loading the configuration file, namely writing information of the engineering parameter template and the dependency matrix into the configuration file, and loading the configuration file and analyzing and storing the configuration file in a memory for subsequent parameter analysis when the system is started.

Redis shared cache library construction and optimization

Cache library design-engineering parameters that use the Redis cache library to store emphasis. Each cache key pair consists of a time code, an APID source packet identification, and a parameter code to ensure that data across APIDs can be effectively shared between different modules and tasks. The structural design of the Redis cache follows the principles of uniqueness and efficiency. For example, a typical key may be denoted as "202408220943000_apid0633_hk1_temp", where the time code identifies the time of data generation, the APID source packet identifies the source of data, and the parameter code indicates the specific parameter code.

TABLE 2Redis cache schematic

The buffer mechanism is optimized, namely, the situation of on-satellite on-demand data is considered, certain redundancy is considered, the buffer data is provided with a valid period of 10 days (864000 seconds) (which can be adjusted according to the design of the solid storage capacity of a specific satellite), and meanwhile, the random time delay of 0 to 1000 seconds is added, so that the time point of invalidation of a large amount of buffer data is staggered, the system is prevented from processing a large amount of data with invalidation of buffer in the same time period, and the running stability of the system is ensured.

3. Data processing and scheduling

When the system receives the data transmission original file of the ground station, virtual channel separation and source packet extraction of monorail data and sequencing and duplication of current rail data are firstly carried out to generate corresponding 0C-level source packet data products (comprising science and engineering class 2), and then splicing, sequencing, duplication, data jump statistics and fault tolerance of multi-rail data are carried out on the basis of 0C to generate 0D-level source packet data products.

Cross APID dependency parameter scheduling:

according to different data types (engineering and science) and dependency conditions (whether the data are dependent source packets) scheduling processing sequences, the processing flow of 1-level engineering data is triggered preferentially, firstly, a source packet 0D-level engineering data product file to be processed is read and parameter resolving is carried out, according to engineering parameter resolving configuration files and resolving dependency matrixes, each parameter is scheduled and traversed and calculated, and the source packet where the dependent parameter is located is processed preferentially.

After the level 1 engineering parameter processing is completed (within the threshold range), the production of the level 1 scientific data product is triggered. For parameters which are dependent across APID in the scientific product parameter calculation, inquiring the parameter values which are dependent across APID from a Redis cache according to the current parameter time and the dependent parameter codes, and then calculating, and for the parameters which are not dependent across APID, directly calculating.

Cache priority policy and schedule the system first queries the Redis cache repository for the value of the dependent parameter. If the corresponding data exists in the cache, the corresponding data is directly read and substituted into a calculation formula for analysis, and if the parameters without the dependency relationship exist, the system directly calculates according to analysis rules in the configuration file.

Time trace back query and exception handling if no dependent parameters are found in the cache, the system starts the trace back query mechanism, trace back seconds by seconds according to time, and the maximum trace back time is usually set to 8 seconds (the threshold value can be adjusted according to the characteristics of the data). If the matching value is not found finally, the system calculates by adopting a preset default value and records the abnormal condition.

4. Parameter calculation and product formatting generation

And after the parameter analysis is completed, the system writes the analysis result into a Redis cache for the subsequent task to call so as to avoid repeated calculation. For example, when the HK1 class product is generated, the analysis result of the dependent parameters is immediately stored in a slow library, so that the GRM L1a product can quickly acquire data when the dependent parameters are relied on.

Data product formatting generation and verification, namely after all parameter analyses are completed and cached, the system generates a final scientific data product file (such as a FITS file and the like) according to a predefined product format. The generated product is subjected to format verification to ensure the integrity and accuracy of the data, and is stored or distributed according to task requirements.

The specific process flow is shown in fig. 3 and 4:

When the system receives a data transmission original file of a ground station, firstly, performing virtual channel separation and source packet extraction of monorail data to generate a corresponding 0C-level source packet data product (comprising science and engineering class 2), secondly, performing splicing, sequencing and weight arrangement of the monorail data, statistics and fault tolerance of data jump on the basis of 0C to generate a 0D-level source packet data product, and firstly triggering the production flow of a 1-level engineering data product, namely, reading a to-be-processed 0D-level engineering data product file, an engineering parameter template and a dependency matrix analysis module, preferentially processing a source packet where a dependent parameter is located, and performing analysis calculation of each parameter in a traversing way until all parameters are processed:

Step 2) setting a waiting threshold value of engineering parameter 1-level processing, triggering a parameter processing flow of scientific data when the engineering parameter processing is smaller than the threshold value and parallel engineering parameter processing in the track data is completed, and turning to step 3) when the parameter X to be calculated depends on parameters of other APID source packets, otherwise turning to step 4);

Step 3) acquiring the data packet time TS of the current parameter, inquiring a Redis cache according to the TS and the dependent parameter codes, acquiring a time code T0 nearest to the TS, and when the TS-T0 is less than or equal to TH, wherein TH represents a set threshold value, substituting the inquired dependent parameter value into a calculation formula of the parameter X, outputting the parameter value of the parameter X, and turning to step 5), and when the TS-T0 is not less than or equal to TH, adopting a default value for calculation, and turning to step 5);

Step 4) calculating and time processing of the parameter X;

Step 5) caching the parsed parameter values;

Step 6) formatting the corresponding data product according to the format definition specification.

The key point of the invention is to provide a data processing method for crossing APID mixed dependency relationship, which has the core of effectively managing and scheduling dependency parameters among different APID source packets of different rounds, realizing data sharing and cache management and effectively supporting mixed dependency and fusion processing of scientific satellite multi-source data.

It should be noted that the present invention does not relate to a specific load data processing algorithm, i.e. the present invention does not propose a new calculation formula or processing method for how to parse the observed data of a specific load or how to calculate specific scientific parameters.

The cross APID mixed dependent parameter processing algorithm provided by the invention realizes efficient multi-source data mixed dependent fusion processing by designing engineering parameter templates, dependent matrixes, data scheduling flows and a caching mechanism. The concrete steps are as follows:

1) The scientific product grading definition is that different levels of product design frames are constructed according to satellite and load scientific data characteristics, a solid foundation is laid for subsequent processing flow and system development, and data compatibility and expansibility among different levels of products are ensured, so that the flexibility and adaptability of the system are improved.

2) The consistency and accuracy are guaranteed, aiming at the dependency relationship among different source packets, the system ensures that a Redis caching mechanism is adopted in parameter calculation to reduce the delay of reading data of a cross-module, maintain the consistency of the data and improve the accuracy of calculation.

3) The system utilizes Redis buffer and automatic dispatch mechanism to obviously optimize the data processing flow, reduce the resource consumption under complex dependency relationship and improve the processing efficiency.

4) The system supports the flexible processing and generating of various scientific data products by updating the dependent configuration, thereby meeting the diversified requirements of satellite engineering.

Compared with the prior art, the invention realizes breakthrough in the aspect of automation of cross APID data processing, can effectively support complex data processing tasks in satellite engineering, and provides a high-efficiency and reliable technical solution for data production and application of scientific satellites.

Finally, it should be noted that the above embodiments are only for illustrating the technical solution of the present invention and are not limiting. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the present invention, which is intended to be covered by the appended claims.

Claims

1. A scientific satellite data processing system crossing APID hybrid dependencies, comprising:

a cross APID dependent parameter scheduling module for scheduling processing sequences according to different data types and dependent conditions, and

The data product formatting generation module is used for generating products of various types and levels based on the product definition specification after all parameter processing is completed and cached;

The pretreatment comprises the following steps:

when receiving a data transmission original file of a ground station, firstly separating a virtual channel of monorail data and extracting a source packet to generate a corresponding 0C-level source packet data product, and performing splicing, weight removal, statistics of data jump and fault tolerance on the multi-rail data to generate a 0D-level source packet data product, wherein the 0C-level source packet data product comprises scientific data and engineering data;

the engineering parameter template and the dependency matrix analysis module preferentially process source packages where the dependency parameters are located, and traverse analysis calculation of each parameter until all the parameters are processed;

the processing procedure of the cross APID dependent parameter scheduling module comprises the following steps:

step 3) calculating and time processing of the parameter X;

Step 4) caching the parsed parameter values.

2. The system of claim 1, wherein the scientific satellite task engineering parameter template is used for describing the name, unit, byte size, starting and ending positions in data packets and calculation formulas of each telemetry parameter in engineering products, the parameter dependency matrix of the cross APID source packet is used for describing parameter dependency relations in different data products, the row of the matrix represents the parameter code of a target product, and the column of the matrix represents the APID source packet number, the dependent parameter code, the analysis sequence and the analysis method formulas of other one or more dependent parameters.

3. The system for processing scientific satellite data crossing APID mixed dependency relationship according to claim 1, wherein the cache sharing mode of each data product parameter value crossing APID source package comprises a cache key value and a parameter value, wherein the cache key value comprises a time code, an APID source package identifier and a parameter code, the expiration time of the cache is set based on the on-board solid-state capacity of a scientific satellite task to be processed, and random delay of 0 to set seconds is increased for staggering time points of a large amount of cache data invalidation, so that the system is prevented from processing a large amount of cache invalidation data in the same time period, and the running stability of the system is ensured.

4. The system for processing scientific satellite data across APID hybrid dependencies according to claim 1, wherein said Redis cache shared architecture module is further configured to purge out expired parameter caches according to a set time.

5. The system for processing scientific satellite data across APID hybrid dependencies according to claim 1, wherein said data product formatting generation module is further configured to verify the generated product format to ensure data integrity and accuracy.