CN120614308A - Data hierarchical transmission method, system, device, storage medium and program product - Google Patents
Data hierarchical transmission method, system, device, storage medium and program productInfo
- Publication number
- CN120614308A CN120614308A CN202510844314.8A CN202510844314A CN120614308A CN 120614308 A CN120614308 A CN 120614308A CN 202510844314 A CN202510844314 A CN 202510844314A CN 120614308 A CN120614308 A CN 120614308A
- Authority
- CN
- China
- Prior art keywords
- data
- service data
- airport
- transmission
- key value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a data hierarchical transmission method, a system, equipment, a storage medium and a program product, wherein the method comprises the steps of performing hierarchical processing on airport service data to generate service data with different levels; the method comprises the steps of receiving service data of different levels, distributing independent transmission channels for the service data of different levels by utilizing message middleware, generating message sequence key values of each service data based on flight characteristic information, determining target partitions of each service data in the transmission channels, and taking out the service data from the partitions of the transmission channels in sequence for consumption based on the message sequence key values of each service data to realize ordered transmission of the data. The invention can improve the transmission performance of airport service data and simultaneously can ensure the ordered transmission of the airport service data.
Description
Technical Field
The present invention relates to the field of data transmission technology, and in particular, to a data hierarchical transmission method, system, device, storage medium, and program product.
Background
According to the statistical analysis result of the daily passenger message quantity of the civil aviation airport, a large amount of instantaneous passenger data are usually issued from the center to the airport in the peak period, if the data sent by other external systems (such as an airport information host system, a data service system, an external aviation low-cost airline, and the like) for gathering the airport information bus are overlapped, the data quantity exceeds the maximum message processing quantity supported by a data receiving party system of the airport, and the situation of insufficient receiving performance in the data transmission process leads to data backlog, thereby influencing the departure business such as passenger security check, baggage sorting, and the like.
Therefore, a data transmission method is needed to improve the performance of airport service data transmission, and because airport passengers and baggage messages have high concurrence and time sequence requirements during transmission, the data transmitted by the center is ensured to be transmitted in sequence while solving the performance.
Disclosure of Invention
The invention aims to provide a data hierarchical transmission method, a system, equipment, a storage medium and a program product, which can improve the transmission performance of airport service data and simultaneously ensure the ordered transmission of the airport service data.
In order to achieve the above purpose, the present invention provides the following technical solutions:
in a first aspect, an embodiment of the present invention provides a data hierarchical transmission method, where the method includes:
Classifying airport service data to generate service data of different levels;
utilizing message middleware to allocate independent transmission channels for service data of different levels;
Generating a message sequence key value of each service data based on flight characteristic information, and determining a target partition of each service data in a transmission channel;
And based on the message sequence key value of each service data, sequentially taking out the service data from the subareas of the transmission channel for consumption, so as to realize the ordered transmission of the data.
In a second aspect, an embodiment of the present invention provides a hierarchical data transmission system, where the system includes a central terminal, message middleware, and an airport receiver,
The central terminal is used for carrying out grading processing on airport service data to generate service data of different levels;
The message middleware is used for distributing independent transmission channels for the business data of different levels, and generating a message sequence key value of each business data based on flight characteristic information so as to determine a target partition of each business data in the transmission channel;
The airport receiving terminal is used for sequentially taking out the service data from the subareas of the transmission channel for consumption based on the message sequence key value of each service data, so as to realize the ordered transmission of the data.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program or instructions to implement a data hierarchical transmission method as described above.
In a fourth aspect, embodiments of the present invention further provide a computer storage medium having stored therein a computer program or instructions which, when executed by a processor, implement a data hierarchical transmission method as described above.
In a fifth aspect, embodiments of the present invention further provide a computer program product comprising a computer program or instructions which, when executed by a processor, implement a data hierarchical transmission method as described above.
The invention provides a hierarchical ordered transmission technology, and utilizes the message middleware to establish independent transmission channels of different levels of data, so that different types of airport service data can be simultaneously and concurrently and orderly transmitted, thereby not only ensuring that key data in the airport data are timely processed and orderly transmitted, but also reducing waiting time and congestion phenomenon of data transmission and improving efficiency and safety of data transmission.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a first flowchart of a data hierarchical transmission method according to an embodiment of the present invention;
FIG. 2 is a second flowchart of a data hierarchical transmission method according to an embodiment of the present invention;
FIG. 3 is a flow chart of starting different threads according to an embodiment of the present invention;
Fig. 4 is a schematic structural diagram of a hierarchical data transmission system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order to solve the defects in the prior art, the embodiment of the invention discloses a data hierarchical transmission method, as shown in fig. 1 and 2, comprising the following steps:
step S1, carrying out grading processing on airport service data to generate service data of different levels;
S2, utilizing message middleware to allocate independent transmission channels for service data of different levels;
Step S3, generating a message sequence key value of each service data based on flight characteristic information, and determining a target partition of each service data in a transmission channel;
and S4, based on the message sequence key value of each service data, sequentially taking out the service data from the subareas of the transmission channel for consumption, and realizing the ordered transmission of the data.
In some specific embodiments, step S1, grading the airport service data to generate different levels of service data, including the following:
and according to the service priority, grading the airport service data issued by the central terminal according to the data type to generate service data of different levels.
The data types of the airport service data comprise passenger luggage data, flight class data (particularly comprising flight plan data and flight dynamic data), passenger transaction class data and passenger non-transaction class data. Passenger baggage data is generally used in baggage sorting systems, the real-time requirements for the data are highest, flight class data and passenger transaction class data are generally used in security check services, the real-time requirements for the data are second highest, passenger non-transaction class data are generally used in statistics class services, and the real-time requirements for the data are lowest.
The different levels of business data include first level data including passenger baggage data and second level data including flight class data, passenger transaction class data and passenger non-transaction class data.
It should be noted that the classification does not mean that the low-level service data is subjected to degradation processing, and the service data of each level is processed in parallel after the classification, and in this embodiment, the classification of the data type of the airport service data is mainly based on setting the relationship between the data type and the subject in the subsequent steps.
In some specific embodiments, step S2, the message middleware is utilized to allocate independent transmission channels for the service data of different levels, wherein the independent transmission channels comprise the following contents:
The transmission channel is a theme in the message middleware, and each theme corresponds to one or more partitions. Wherein the message middleware is similar to Kafka, the producer writes the data into a partition of the message middleware's topic, and the consumer consumes the data from the partition of the topic.
Setting a theme corresponding to each data type according to the level after the grading processing in the step S1 aiming at all the data types of the airport service data so as to establish a first corresponding relation table between the data types of the airport service data and the theme, and storing the first corresponding relation table in a database.
The first corresponding relation table comprises the corresponding relation between the data types of the airport service data and the topics, and the configurable management of the corresponding relation between the data types and the topics is realized by adding, deleting or modifying records in the first corresponding relation table, namely, when the airport service data needs to add/delete/modify one data type, the corresponding relation between the data types and the topics is realized by adding/deleting/modifying the corresponding relation between the data types and the topics in the first corresponding relation table.
In this example, as shown in table 1, the attribute in the first correspondence table includes a data type and a theme, and the correspondence between the data type and the transmission theme is stored in the table, where the passenger baggage data corresponds to the theme 1, and the passenger transaction data, the passenger flight type data, and the passenger non-transaction data correspond to the theme 2.
TABLE 1 first correspondence table
In some specific embodiments, step S3, generating a message sequence key value of each service data based on the flight characteristic information, and determining a target partition of each service data in a transmission channel, wherein the message sequence key value comprises the following contents:
the flight characteristic information comprises a flight number and a flight date, the flight number and the flight date are used as message sequence key values of each service data, namely, the message sequence key value=the flight number_flight date, and if any one of the flight number and the flight date does not exist, the flight number and the flight date are replaced by 'empty'.
And processing the message sequence key value of each service data by adopting a hash algorithm, so that each service data is mapped to an available partition in a corresponding transmission channel.
When a fault occurs in one partition, the message sequence key value of the service data in the fault partition needs to be processed by a hash algorithm again, so that the service data in the fault partition is mapped to an available partition in a corresponding transmission channel, and the automatic redistribution of the partition is realized.
In some specific embodiments, after step S3, the central side writes each service data into the partition of the transmission channel in the level order based on the first correspondence table and the message sequence key value of each service data, including:
the central terminal reads the first corresponding relation table from the database, acquires the number of data types of airport service data and the theme corresponding to each data type, starts a corresponding number of sending threads from a thread pool according to the number of the data types, and adopts an independent thread transmission mechanism for each data type of service data;
Each sending thread queries and acquires corresponding data to be sent from a database according to a database entering sequence, analyzes the flight number and the flight date of each data to be sent, sets the theme and the message sequence key value of each data to be sent, and writes the corresponding data to be sent into the subarea of the transmission channel according to the level sequence based on the message sequence key value of each service data.
In this embodiment, all the sending threads include a passenger baggage data thread, a passenger transaction data thread, a flight data thread and a passenger non-transaction data thread.
The passenger luggage data thread queries and obtains the passenger luggage data to be sent according to the data warehouse-in sequence from the database, analyzes the flight number and the flight date corresponding to the flight class data to be sent, sets the theme=theme 1 of the passenger luggage data to be sent, sets the message sequence key value=flight number_flight date of the passenger luggage data to be sent, and then the passenger luggage data thread sends the passenger luggage data to be sent;
The passenger transaction data thread inquires and obtains passenger transaction data to be transmitted from a database according to a data warehouse-in sequence, analyzes a flight number and a flight date corresponding to the flight data to be transmitted, sets a theme=theme 2 of the passenger transaction data to be transmitted, sets a message sequence key value=flight number_flight date of the passenger transaction data to be transmitted, and then the passenger transaction data thread transmits the passenger transaction data to be transmitted;
The flight class data thread queries and obtains flight class data to be sent (namely flight planning data to be sent and flight dynamic data) from a database according to a data storage sequence, analyzes a flight number and a flight date corresponding to the flight class data to be sent, sets a theme=theme 2 of the flight class data to be sent, sets a message sequence key value=flight number_flight date of the flight class data to be sent, and then sends the flight class data to be sent;
The passenger non-transaction data thread inquires and obtains passenger non-transaction data to be sent from a database according to a data warehouse-in sequence, analyzes a flight number and a flight date corresponding to the passenger non-transaction data to be sent, sets a theme=theme 2 of the passenger transaction data to be sent, sets a message sequence key value=flight number_flight date of the passenger transaction data to be sent, and then the passenger transaction data thread sends the passenger transaction data to be sent.
In some specific embodiments, step S4, based on the message sequence key value of each service data, sequentially fetches the service data from the partition of the transmission channel for consumption, so as to realize the ordered transmission of the data, as shown in fig. 3, including the following contents:
The central end writes each service data into the subarea of the transmission channel according to the level sequence based on the first corresponding relation table and the message sequence key value of each service data; and the airport receiving terminal sequentially takes out the service data for consumption according to the message sequence key value of each service data by using the consumption thread which is completed by deployment.
The airport receiving terminal uses the deployed consumption thread to sequentially take out the service data for consumption according to the message sequence key value of each service data, and specifically comprises the following steps:
the airport receiving terminal establishes a second corresponding relation table which is the same as the first corresponding relation table, and acquires the number of data types of airport service data and the theme corresponding to each data type according to the second corresponding relation table.
And starting a corresponding number of consumption threads according to the number of the data types, and adopting an independent thread consumption mechanism for the business data of each data type, wherein in the embodiment, all the consumption threads comprise a passenger luggage data thread, a passenger transaction data thread, a flight data thread and a passenger non-transaction data thread.
After each consumption thread sequentially inquires corresponding data to be consumed from the second corresponding relation table, each consumption thread parallelly takes out the data to be consumed from the corresponding transmission channel based on a message sequence key value and the transmission channel corresponding to the data to be consumed in the second corresponding relation table. The airport receiving terminal comprises a plurality of airport hosts.
Based on the same inventive concept, the embodiment of the invention discloses a data hierarchical transmission system, as shown in fig. 4, which comprises a central terminal, message middleware and an airport receiving terminal,
The central terminal is used for carrying out grading processing on airport service data to generate service data of different levels;
The message middleware is used for distributing independent transmission channels for the business data of different levels, and generating a message sequence key value of each business data based on flight characteristic information so as to determine a target partition of each business data in the transmission channel;
The airport receiving terminal is used for sequentially taking out the service data from the subareas of the transmission channel for consumption based on the message sequence key value of each service data, so as to realize the ordered transmission of the data.
The central terminal of the invention classifies the data, establishes a plurality of data transmission channels, simultaneously issues the data through the plurality of transmission channels, ensures that the data are sequentially placed in the transmission channels, and thus, the airport receiving terminal can consume the data of the plurality of transmission channels in parallel according to the priority of the production service, and avoids the situation that the data with high priority cannot be issued in time due to blockage.
The specific manner in which the respective unit modules perform the operations in the above-described embodiments has been described in detail in relation to the embodiments of the method, and will not be described in detail herein.
The invention provides a hierarchical ordered transmission technology, which utilizes message middleware to establish independent transmission channels of different levels of data, so that different types of airport service data can be simultaneously and orderly transmitted, thereby not only ensuring that key data in the airport data are timely processed and orderly transmitted, but also reducing waiting time and congestion phenomenon of the data transmission, improving the efficiency and safety of the data transmission, and supporting a real-time monitoring and early warning system of airport operation and improving the safety and stability of the airport operation by ensuring timely and accurate transmission of the key data.
Based on the same inventive concept, the embodiment of the invention also provides an electronic device, the structure of which is shown in fig. 5, including a memory, a processor and a computer program stored on the memory, where the processor executes the computer program or instructions to implement a data hierarchical transmission method as described above.
Based on the same inventive concept, the embodiment of the invention further provides a computer storage medium, wherein a computer program or an instruction is stored in the computer storage medium, and the computer program or the instruction realizes the data hierarchical transmission method when being executed by a processor.
Based on the same inventive concept, the embodiments of the present invention also provide a computer program product, including a computer program or an instruction, which when executed by a processor, implements a data hierarchical transmission method as described above.
It should be noted that the foregoing description is only a preferred embodiment of the present invention, and although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood that modifications, equivalents, improvements and modifications to the technical solution described in the foregoing embodiments may occur to those skilled in the art, and all modifications, equivalents, and improvements are intended to be included within the spirit and principle of the present invention.
Claims (10)
1.A method for hierarchical transmission of data, the method comprising:
Classifying airport service data to generate service data of different levels;
utilizing message middleware to allocate independent transmission channels for service data of different levels;
Generating a message sequence key value of each service data based on flight characteristic information, and determining a target partition of each service data in a transmission channel;
And based on the message sequence key value of each service data, sequentially taking out the service data from the subareas of the transmission channel for consumption, so as to realize the ordered transmission of the data.
2. The method of claim 1, wherein the step of hierarchically processing airport traffic data to generate different levels of traffic data comprises:
according to the service priority and the data type, grading the airport service data issued by the central terminal to generate service data of different levels;
The data types of the airport service data comprise passenger luggage data, flight class data, passenger transaction class data and passenger non-transaction class data;
The different levels of business data include first level data including passenger baggage data and second level data including flight class data, passenger transaction class data and passenger non-transaction class data.
3. A data hierarchical transmission method according to claim 1 or 2, wherein the allocation of independent transmission channels for different levels of service data using message middleware comprises:
The transmission channel is a theme in the message middleware, and each theme corresponds to one or more partitions;
Setting topics corresponding to all data types of airport service data according to the grading level, establishing a first corresponding relation table between the data types of the airport service data and the topics, and storing the first corresponding relation table in a database;
The first corresponding relation table comprises the corresponding relation between the data type of the airport service data and the theme, and the configurable management of the corresponding relation between the data type and the theme is realized by adding, deleting or modifying the record in the first corresponding relation table.
4. The data hierarchical transmission method according to claim 1, wherein generating a message sequence key value for each of the service data based on flight characteristic information and determining a target partition of each of the service data in a transmission channel, comprises:
The flight characteristic information comprises a flight number and a flight date;
Taking the flight number and the flight date as message sequence key values of each service data;
Mapping each service data to a partition in a corresponding transmission channel through a hash algorithm based on a message sequence key value of each service data;
Wherein the service data having the same message sequence key value is mapped into the same partition.
5. The method of claim 4, further comprising monitoring a partition status, automatically triggering a partition reassignment mechanism upon detection of a partition failure, comprising:
When a fault occurs in one partition, carrying out hash algorithm processing on the message sequence key value of the service data in the fault partition again, and mapping the service data in the fault partition to an available partition in a corresponding transmission channel to realize automatic partition reassignment.
6. A data hierarchical transmission method according to claim 3, wherein sequentially retrieving said service data from a partition of a transmission channel for consumption based on a message sequence key value of each said service data, comprising:
The central end writes each service data into the subarea of the transmission channel according to the level sequence based on the first corresponding relation table and the message sequence key value of each service data;
the airport receiving terminal uses the deployed consumption thread to sequentially take out the service data for consumption according to the message sequence key value of each service data;
the airport receiving terminal uses the deployed consumption thread to sequentially take out the service data for consumption according to the message sequence key value of each service data, and comprises the following steps:
the airport receiving terminal establishes a second corresponding relation table which is the same as the first corresponding relation table, and acquires the number of data types of airport service data and the theme corresponding to each data type according to the second corresponding relation table;
Starting a corresponding number of consumption threads according to the number of the data types, and adopting an independent thread consumption mechanism for the business data of each data type;
based on the message sequence key value and the transmission channel corresponding to the data to be consumed, each consumption thread parallelly takes out the data to be transmitted from the corresponding transmission channel.
7. A data hierarchical transmission system is characterized in that the system comprises a central terminal, message middleware and an airport receiving terminal,
The central terminal is used for carrying out grading processing on airport service data to generate service data of different levels;
The message middleware is used for distributing independent transmission channels for the business data of different levels, and generating a message sequence key value of each business data based on flight characteristic information so as to determine a target partition of each business data in the transmission channel;
The airport receiving terminal is used for sequentially taking out the service data from the subareas of the transmission channel for consumption based on the message sequence key value of each service data, so as to realize the ordered transmission of the data.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory, the processor executing the computer program or instructions to implement a data hierarchical transmission method according to any one of claims 1-6.
9. A computer readable storage medium, wherein a computer program or instructions is stored in the computer readable storage medium, which when executed by a processor, implements a data hierarchical transmission method according to any one of claims 1-6.
10. A computer program product comprising a computer program or instructions which, when executed by a processor, implements a data hierarchical transmission method as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510844314.8A CN120614308A (en) | 2025-06-23 | 2025-06-23 | Data hierarchical transmission method, system, device, storage medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510844314.8A CN120614308A (en) | 2025-06-23 | 2025-06-23 | Data hierarchical transmission method, system, device, storage medium and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN120614308A true CN120614308A (en) | 2025-09-09 |
Family
ID=96923886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510844314.8A Pending CN120614308A (en) | 2025-06-23 | 2025-06-23 | Data hierarchical transmission method, system, device, storage medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120614308A (en) |
-
2025
- 2025-06-23 CN CN202510844314.8A patent/CN120614308A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258737B (en) | Resource scheduling method and device and filter scheduler | |
| CN111835562B (en) | Log uploading method, log inquiring method and system | |
| US11429566B2 (en) | Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as splunk or sumo | |
| US11922059B2 (en) | Method and device for distributed data storage | |
| CN108335075A (en) | A kind of processing system and method for Logistics Oriented big data | |
| US8479219B2 (en) | Allocating space in message queue for heterogeneous messages | |
| CN103905533A (en) | Distributed type alarm monitoring method and system based on cloud storage | |
| CN106407244A (en) | Multi-database-based data query method, system and apparatus | |
| CN114756301B (en) | Log processing method, device and system | |
| CN112035516B (en) | Processing method, device, intelligent workstation and electronic device for operator service | |
| CN112559476A (en) | Log storage method for improving performance of target system and related equipment thereof | |
| CN112291214A (en) | Industry message parsing mode based on redis cache | |
| EP4109261A2 (en) | Access processing method, device, storage medium and program product | |
| CN113342551B (en) | Message subscription mechanism cross-region service message push method | |
| CN110333916A (en) | Request message processing method, device, computer system and readable storage medium storing program for executing | |
| CN116226067A (en) | Log management method, log management device, processor and log platform | |
| CN114443302A (en) | Container cluster expansion method, system, terminal and storage medium | |
| CN120614308A (en) | Data hierarchical transmission method, system, device, storage medium and program product | |
| CN108595367B (en) | Server system based on computer cluster in local area network | |
| CN117997715A (en) | Alarm processing method, device, equipment, storage medium and program product | |
| CN115022402B (en) | Agent acquisition method and system based on stack-type integration technology | |
| CN117492956A (en) | Application resource scheduling method, device and distributed system based on kubernetes cluster | |
| CN114448976B (en) | Method, device, equipment, medium and program product for assembling network message | |
| CN117435337A (en) | Resource allocation method, device, electronic equipment and storage medium | |
| CN115065685A (en) | Cloud computing resource scheduling method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |