CN116055573B

CN116055573B - Hybrid data processing method and architecture

Info

Publication number: CN116055573B
Application number: CN202310030129.6A
Authority: CN
Inventors: 陈泽华; 王小梅; 冯子滕; 张�杰; 刘泽顺; 王雷
Original assignee: Shenzhen Dongsheng Data Co ltd
Current assignee: Shenzhen Dongsheng Data Co ltd
Priority date: 2023-01-09
Filing date: 2023-01-09
Publication date: 2024-08-23
Anticipated expiration: 2043-01-09
Also published as: CN116055573A

Abstract

The application discloses a hybrid data processing method and architecture, wherein the method comprises the following steps: encapsulating various data sources of IP data and non-IP data into an Ethernet, storing key parameters of the data sources, and outputting the key parameters through an Ethernet physical interface; collecting the data output in the step S1 through an Ethernet physical interface, extracting key parameters of a data source, and putting the key parameters into a cache queue; and reading the IP data and the non-IP data in the cache queue, and distributing the IP data and the non-IP data to corresponding processing software for data processing. The technical scheme of the application solves the problems of more interfaces and high cost of the existing serial output and processing scheme.

Description

Hybrid data processing method and architecture

Technical Field

The present application relates to the field of communications technologies, and in particular, to a hybrid data processing method and architecture.

Background

In the field of communication data processing, there are data input and data processing requirements under the condition of hybrid access, that is, there are multiple types of data input and data processing conditions such as IP, non-IP and the like at the same time. The current communication data processing generally adopts a processing scheme of classified access, classified processing and convergence output, and a processing framework is shown in fig. 1, wherein a source device is used for providing input data required by the processing device, and one source device may output more than one type of data, such as several types of IP, several types of non-IP, or several types of IP plus several types of non-IP; the IP class can be divided into a plurality of data types, and the non-IP class can be divided into a plurality of data types; the processing equipment is used for respectively processing the IP type or non-IP type data of the corresponding type; the convergence output is used for converging the processing results of various processing devices; in the figure, several kinds of data of IP class and several kinds of data of non-IP class are input into correspondent processing equipment, and finally the processing results of all kinds of processing equipment are converged and outputted.

However, in the existing data processing mode, interfaces required for several types of data of the IP class may be different, and the non-IP class data further requires a special physical interface to be provided between the source device and the processing device for docking; IP class and non-IP class data in the same source equipment need to be processed in a time-sharing way; the special physical interface has the problems of slow development, high cost, difficult maintenance and the like.

Disclosure of Invention

The application provides a hybrid data processing method and architecture, which solve the problems of more interfaces and high cost of the existing serial output and processing scheme.

The embodiment of the application provides a mixed data processing method, which comprises the following steps:

s1: encapsulating various data sources of IP data and non-IP data into an Ethernet, storing key parameters of the data sources, and outputting the key parameters through an Ethernet physical interface;

S2: collecting the data output in the step S1 through an Ethernet physical interface, extracting key parameters of a data source, and putting the key parameters into a cache queue;

s3: and reading the IP data and the non-IP data in the cache queue, and distributing the IP data and the non-IP data to corresponding processing software for data processing.

In some embodiments, step S1 specifically includes:

transmitting various IP data directly through Ethernet, and storing key parameters of a data source in a source MAC or a destination MAC, wherein the key parameters comprise line numbers of the data source;

Cutting and packaging various non-IP data into Ethernet, transmitting, and storing key parameters of the data source in the source MAC or the destination MAC, wherein the key parameters comprise line numbers, cutting sequence numbers and offset of the data source.

In some embodiments, the IP data comprises ethernet, GFP, FR, PPP, HDLC.

In some embodiments, the non-IP data comprises OTN or ODU or OPU or OH, SDH or VC4 or C4 or E1 or OH, ATM.

In some embodiments, step S2 specifically includes:

Extracting key parameters of a data source from a source MAC or a destination MAC, identifying and extracting IP data, and putting the IP data into a cache queue;

And extracting key parameters of a data source from the source MAC or the destination MAC, further identifying and extracting non-IP data, putting the non-IP data into a cache queue, and sequentially combining a plurality of pieces of data to recover the original data.

In some embodiments, the ethernet physical interface comprises a 10GE physical interface or a 100GE physical interface or a 200GE physical interface or a 400GE physical interface.

In some embodiments, the ethernet physical interface is set to be a 100GE physical interface, in step S1, if the total output traffic is not greater than 100Gbps, a 100GE physical interface is adopted, and if the total output traffic is increased by 100Gbps, a 100GE physical interface is added.

The embodiment of the application also provides a hybrid data processing architecture, which comprises:

the unified output unit is used for packaging the data sources of various IP data and non-IP data output by various source devices into the Ethernet, storing key parameters of the data sources and outputting the key parameters through an Ethernet physical interface;

The unified input unit is connected with the unified output unit, acquires data output by the unified output unit through an Ethernet physical interface, extracts key parameters of a data source, and puts the key parameters into a cache queue;

and the cooperative scheduling unit is used for reading the IP data and the non-IP data in the cache queue and distributing the IP data and the non-IP data to corresponding processing software for data processing.

In some embodiments, the unified output unit is specifically configured to send various types of IP data directly through ethernet, and store key parameters of a data source in a source MAC or a destination MAC, where the key parameters include a line number of the data source;

Cutting various non-IP data, packaging the cut non-IP data into Ethernet in a slicing way, transmitting the Ethernet, and storing key parameters of a data source in a source MAC or a destination MAC, wherein the key parameters comprise line numbers, cutting sequence numbers and offset of the data source;

The unified input unit is specifically used for extracting key parameters of a data source from a source MAC or a destination MAC, identifying and extracting IP data, and placing the IP data into a cache queue;

In some embodiments, the unified output unit and the unified input unit are directly connected through a 100GE physical interface, or the unified output unit and the unified input unit are connected through a 100GE switching device.

Compared with the prior art, the application has the beneficial effects that:

1. the virtual IP bus thought is adopted, so that multi-class data exchange is supported, and the usability, expansibility and stability are good;

2. Complicated and special physical interfaces are abandoned, mature and stable Ethernet physical interfaces are adopted instead, and hardware development cost and software development cost are reduced;

3. The method supports the simultaneous output of IP data and non-IP data in the same source equipment, and upgrades the original serial output and serial processing scheme into a parallel processing scheme.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a diagram of a prior art hybrid data processing architecture;

FIG. 2 is a diagram of a hybrid data processing architecture of the present application;

FIG. 3 is a schematic diagram of an IP data encapsulation of the present application;

FIG. 4 is a schematic diagram of a non-IP data encapsulation of the present application;

FIG. 5 is a schematic diagram of IP data extraction according to the present application;

FIG. 6 is a diagram illustrating non-IP data extraction according to the present application;

FIG. 7 is a schematic diagram of the connection between the unified input unit and the unified output unit of the present application;

The achievement of the objects, functional features and advantages of the present application will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

The hybrid data processing method provided by the embodiment comprises the following steps:

referring to fig. 3 and 4, step S1 specifically includes:

Referring to fig. 5 and 6, step S2 specifically includes:

S3: reading IP data and non-IP data in the cache queue, and distributing the IP data and the non-IP data to corresponding processing software for data processing;

Specifically, IP data such as ethernet, GFP, etc. are distributed to the original ethernet, GFP processing software for data processing.

And distributing the non-IP data such as OTN, SDH and the like to the original OTN and SDH processing software for data processing.

It should be noted that, the present embodiment improves the existing method for processing mixed data, cancels the traditional mode of a special physical interface, adopts the thought of a virtual IP exchange bus to replace the special physical interface by a general physical interface, so that various IP-class and non-IP-class data can be sent and received through a unified bus, thereby enabling users to focus on data processing without paying attention to the development of the special equipment physical interface.

Further, the IP data includes ethernet, GFP, FR, PPP, HDLC. The non-IP data comprises OTN or ODU or OPU or OH, SDH or VC4 or C4 or E1 or OH, ATM.

Further, the ethernet physical interface includes a 10GE physical interface or a 100GE physical interface or a 200GE physical interface or a 400GE physical interface. The virtual IP exchange bus is realized through the physical interfaces of the Ethernet with the rates of 200GE and 400GE and even higher, the exchange capacity of the system architecture can be further improved, the better high-speed data access capacity is provided, the virtual IP exchange bus can be also established by replacing 100GE with 10GE in consideration of the condition that the output flow of all source devices is lower, and the hardware cost and the system power consumption are further reduced.

Further, considering the condition that the output flow of all source devices is limited, on the premise of improving the configuration of a CPU, a memory, a storage and the like of the processing device, the single set of processing device can directly complete the processing of various IP-type and non-IP-type data through a small number of 100GE ports.

In this embodiment, the ethernet physical interface is set to be a 100GE physical interface, in step S1, the number of 100GE physical interfaces is determined according to the total output traffic of all the data sources, so as to ensure that all the traffic can be output, where if the total output traffic is not greater than 100Gbps, one 100GE physical interface is adopted, and if the total output traffic is increased by 100Gbps, one 100GE physical interface is added.

The present embodiment further proposes a hybrid data processing architecture, referring to fig. 2, including:

Further, the unified output unit is specifically configured to send various types of IP data directly through ethernet, and store key parameters of a data source in a source MAC or a destination MAC, where the key parameters include a line number of the data source;

Further, referring to fig. 2, the unified output unit is directly connected with the unified input unit through a 100GE physical interface, or referring to fig. 7, the unified output unit is connected with the unified input unit through a 100GE switching device, so as to implement richer data exchange, for example, forwarding the same type of IP or the same type of non-IP data to the same processing device as much as possible.

The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the application, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims

1. A method of hybrid data processing comprising the steps of:

The step S1 specifically includes:

the IP data comprises Ethernet and GFP, FR, PPP, HDLC;

the non-IP data comprises OTN or ODU or OPU or OH, SDH or VC4 or C4 or E1 or OH, ATM;

The step S2 specifically comprises the following steps:

Extracting key parameters of a data source from a source MAC or a destination MAC, further identifying and extracting non-IP data, putting the non-IP data into a cache queue, and sequentially combining a plurality of pieces of data to recover the original data;

The Ethernet physical interface comprises a 10GE physical interface or a 100GE physical interface or a 200GE physical interface or a 400GE physical interface.

2. The method of claim 1, wherein the ethernet physical interface is set as a 100GE physical interface, and in step S1, if the total output traffic is not greater than 100Gbps, a 100GE physical interface is adopted, and if the total output traffic increases by 100Gbps, a 100GE physical interface is added.

3. A hybrid data processing architecture, comprising:

the collaborative scheduling unit is used for reading the IP data and the non-IP data in the cache queue and distributing the IP data and the non-IP data to the corresponding processing software for data processing;

The unified output unit is specifically configured to send various types of IP data directly through ethernet, and store key parameters of a data source in a source MAC or a destination MAC, where the key parameters include a line number of the data source;

4. The hybrid data processing architecture of claim 3, wherein the unified output unit and the unified input unit are directly connected through a 100GE physical interface or are connected through a 100GE switching device.