CN100452739C - Optical parallel transmission in usage of grid - Google Patents

Abstract

This invention discloses an optical parallel transmission method which is open to grid application. When grid users send grid operations to grid server; the server finds the operational grid resources according to grid resource list. According to this grid resource node address inquiry optical network, find the spare available optical network resource; based on the above inquiry result, send one or more optical channel-building requests to optical control layer on demand; after the optical control layer establishes the corresponding optical channel, it returns to the result and updates the list of optical network resource; grid server begin to operate the user submit grid operation; after the operation is finished, it informs optical control layer to close the optical channel links. This invention can solve the capacity demand problem that grid application large bandwidth data transmission, improve the data transmission speed and make full use of the grid network bandwidth resources.

Description

A Mesh Application Oriented Optical Parallel Transmission Method

technical field

The invention relates to a method for realizing optical parallel transmission, in particular to a grid application-oriented optical parallel transmission method, which belongs to the technical field of optical communication.

Background technique

With the rapid development of national economy, national defense construction and high technology, more and more fields need to solve large-scale scientific and engineering calculations. In order to solve people's actual demand for high-performance computing, people put forward the concept of grid computing in combination with the development of Internet technology.

The grid is a concept borrowed from the power grid, which means that computing power and computing resources can be used just like electricity, "turn on the power switch and use it", regardless of who and how to provide these services. The goal of the grid is to abstract and quantify computing resources, and to complete "quantitative" computing-related work through the network "quota" anytime and anywhere. By using grid technology to coordinate and deploy remote computing resources and storage resources, tasks can be processed quickly.

With the great improvement of bandwidth and network speed, new changes have taken place in the way of distributed computing, and the grid has become the development direction of the next generation Internet. However, the current physical transmission environment of grid applications mainly adopts the traditional Internet method, that is, grid nodes are connected through the Internet, data packets are stored and forwarded through routers, and routers are connected through fixed optical channels. In the existing Internet environment, the grid data transmission mechanism is as follows:

Step 1: grid user submits grid job to grid server;

Step 2: The server searches for idle computing resources and storage resources for executing jobs according to the distribution of computing resources and storage resources;

Step 3: Send the data to the selected computing resources and storage resources through the high-speed router;

Step 4: The high-speed router sends data to the optical transmission network through the optical port;

Step 5: The optical transmission network transmits data to the next high-speed router through a fixed optical connection;

Step 6: The next high-speed router repeats steps 4 and 5 until the data is delivered to the target grid resource;

Step 7: After the grid resource acquires the data, execute the grid job, and send the job execution result back to the server (transmission steps are the same as steps 4-6);

Step 8: The grid server returns the calculation result to the grid user.

The above-mentioned Internet-based grid physical transmission environment cannot fully meet the large-capacity and distributed business requirements brought about by large-scale grid services. For this reason, people have carried out a lot of research work, think that using optical fiber communication and dense wavelength division multiplexing DWDM facility in grid network is a promising technical solution.

However, the existing WDM technology only solves the problem of point-to-point large-capacity transmission. Due to the distributed characteristics of grid applications, it is likely that grid service applicants and providers are located on three or more dispersed nodes. In order to allow these nodes to work together, it is also necessary to provide a fast parallel connection establishment mechanism. However, there is no mature technical solution for this parallel connection mechanism at present. In recent years, due to the introduction of Automatically Switched Optical Network (ASON) technology, users can dynamically apply for bandwidth resources from the optical network through the UNI interface, but this application is still limited to point-to-point serial connections.

In addition, since the transmission capacity of a single optical fiber is much larger than the core capacity of the Internet, the processing capability of existing routers cannot match the increasing optical transmission speed at all, resulting in the waste of a large amount of optical transmission capacity. If the existing serial transmission method continues to be used, many data-intensive grid applications cannot get a quick response and cannot fully utilize the bandwidth resources of optical transmission.

Contents of the invention

The purpose of the present invention is to provide a grid application-oriented optical parallel transmission method for grid applications requiring large-capacity and distributed network transmission. This method is based on the Gridftp of GT4.0 released by the Globus organization, and adopts the GMPLS protocol group to realize the optical layer control.

For realizing above-mentioned purpose of the invention, the present invention adopts following technical scheme:

A kind of optical parallel transmission method oriented to grid application, it is characterized in that comprising the following steps:

Step 1, the grid application node submits a grid job request to the grid server;

Step 2, the grid server queries the grid resource list, finds a grid resource node that meets the job requirements, obtains the network address of the grid resource node, and proceeds to step 3; if there is no grid resource node that meets the job requirements, return As a result, the user is notified that there are currently no available optical grid resources, and the service ends;

Step 3: Query the optical network resource list, find out the free and available optical network resource information of the grid resource node, and proceed to step 4; if the grid resource node has no free and available optical network resources, inform the user that there is no available free optical network resource at present Optical network resource, end of service;

Step 4: According to the query result, the grid server initiates one or more optical channel establishment requests to the optical control layer as needed, and the optical channel is from the grid application node to the grid resource node with free and available optical network resources. channel between

Step 5: After the optical control layer establishes the corresponding optical channel, return the result;

Step 6, the grid server starts to execute the grid job request submitted by the user;

In step 7, the operation is completed, and the optical control layer is notified to close the optical channel connection.

Wherein, in the step 3, the step of querying the optical network resource list and finding the free and available optical network resource information of the grid resource node includes:

When there is a grid resource node that meets the required grid resources, judge that N optical channels can be established between the grid application node and the grid resource node according to the optical network resource list of the grid resource node , N is an integer greater than 1; and according to the size of the data to be transmitted, it is judged that at most M optical channels need to be established; if M≥N, N optical channels are established; if M<N, M optical channels are established.

If more than one grid resource node has the required grid resources, according to the optical network resource list of each grid resource node, if more than one grid resource node has available optical network resources, then in One optical channel is established between the grid application node and each grid resource node; if only one grid resource node has one optical channel resource, then one optical channel is established between the grid application node and the grid resource node If only one grid resource node has available optical network resources, and the grid resource node can establish more than one optical channel, it is judged that N optical channels can be established between the grid application node and the grid resource node Optical channel, N is an integer greater than 1; and according to the size of the data to be transmitted, it is judged that at most M optical channels need to be established; if M≥N, N optical channels are established; if M<N, M optical channels are established .

In the step 6, when a data transmission service request is initiated according to the Gridftp protocol, the Gridftp control connection establishes a TCP connection through the Internet, and the data connection is established at the optical control layer.

Each Optical Channel connection starts a Gridftp program.

The grid application-oriented optical parallel transmission method of the present invention has the following advantages and effects: it solves the capacity requirement of large-bandwidth data transmission in grid applications; improves the transmission speed of data; fully utilizes the bandwidth resources of the grid network .

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is a schematic diagram of the photonic grid network structure used in the present invention;

Fig. 2 is the photonic grid node data transmission processing flowchart embodying the method of the present invention;

FIG. 3 is a processing flow chart of the parallel optical channel establishment process during the data transmission process described in FIG. 4 .

Fig. 4 is an example of an optical parallel transmission channel implemented in the photonic grid network.

FIG. 5 is another example of optical parallel transmission channels implemented in the photonic grid network.

Detailed ways

In the embodiment shown in FIG. 1 , a schematic structural diagram of a grid network used in the present invention is given. As shown in Figure 1, there are n grid nodes, and the grid nodes communicate through the optical network. The PC of each node has 2 optical interfaces and 2 common network cards. The ordinary network card is used to establish the Gridftp control connection, and the optical interface is used as the interface device for establishing the optical channel for transmitting data.

In the present invention, the grid resources refer to traditional grid resources other than network resources, such as computing resources, storage resources, data, application programs, various electronic files, and the like. Network resources refer to the facilities that connect each grid resource node and provide data transmission and communication channels between nodes, which can be optical cross-connectors, optical switching devices, optical fibers, wavelengths, bands, general labels, optical time slots and interfaces, etc. .

A notable feature of the present invention is based on GT4.0 Gridftp issued by Globus (http://www.globus.org/), and adopts GMPLS protocol group to realize optical layer control. Traditionally standardized FTP establishes two TCP connections between Client and Server: a control connection and a data connection. The control connection is used to send commands and receive responses; the data connection is used for data transmission. In the present invention, the control connection still establishes a TCP connection through the Internet, while the data connection requests the optical control layer to establish an optical channel, and several Gridftp programs are started simultaneously if there are several optical channels.

Gridftp is a safe and reliable high-performance data transfer protocol based on FTP, a popular Internet file transfer protocol, and optimized for high-bandwidth wide area networks. The protocol can support the following functions, so that a robust transport mechanism can be implemented:

• Parallel data transfer: Using multiple TCP streams increases bandwidth over a single TCP stream. Parallel data transfer is supported by FTP command extensions and data channel extensions.

Grid Security Infrastructure (GSI) and Kerberos authentication support: user-controlled settings for various data integrity and confidentiality levels. This capability provides a robust and flexible authentication, integrity, and confidentiality mechanism for transferring files.

· Third-party control of data transmission: Supports the management of large data sets for large distributed communities. It enables third parties to control transfers between storage servers.

·Blocked data transmission: Ability to split data and place it on multiple servers, thereby increasing aggregate bandwidth. Gridftp supports chunked data transfers through extensions defined in the Grid Forum draft.

• Partial file transfers: Unlike standard FTP, which requires applications to transfer entire files, the new FTP commands support transfers of certain areas of a file.

· Reliable data delivery: The failure recovery method can handle transient network failures and server failures, and can restart failed transmissions at the same time.

·Manually control the size of TCP buffer: support to obtain the maximum TCP/IP bandwidth.

· Integrated detection (instrumentation): support for returning to restart and performance markers.

For further description of Gridftp, please refer to its white paper (URL is http://www-fp.globus.org/datagrid/deliverables/C2WPdraft3.pdf). I won't go into details one by one here.

Referring to Figure 2, when a grid user submits a grid job to the grid server, the execution steps of the job are:

Step 1: The grid user submits the grid job to the grid server.

Step 2: The grid server queries the grid resource list, finds a grid resource node that meets the job requirements, and obtains the address of the node.

When discovering the grid resources in step 2, there may be one or more grid resources found available that meet the requirements of the grid application. In addition, when searching for grid resources, in addition to obtaining the node address where the grid resources are located, some related attributes of the available resources, such as resource size, can also be known. These attribute information will be used in subsequent optical parallel transmission.

After discovering the available grid resources, query the optical network resource list according to the resource node address obtained above, and find out the optical network resources possessed by the corresponding node, as shown in Figure 3, there are two situations at this time:

1) When only one grid node has the required grid resources, according to the optical network resource list, if N (N > 1) optical channels can be established between the grid application node and the resource node, then firstly according to the needs Based on the size of the transmitted data, it is judged that a maximum of M optical channels need to be established; at this time, according to the size of M and N, the following steps are performed:

A. If M≥N, establish N optical channels;

B. If M<N, establish M optical channels.

2) There are P (P>1) grid nodes with required grid resources. According to the optical network resource list, only one optical channel is established between the grid application node and P resource nodes. If there is no available optical channel between the grid application node and P-1 resource nodes, only one of them If the resource node has more than one optical channel, follow the regulations in 1).

Step 3: Query the optical network resource list, and find out the free and available optical network resource information of the grid resource node.

Step 4: According to the query result, initiate one or more optical channel establishment requests to the optical control layer as needed.

Step 5: After the optical control layer establishes the corresponding optical channel, return the result to the grid middleware, and update the optical network resource list at the same time.

An example of the optical channel established in step 5 is shown in FIG. 4 or FIG. 5 .

Step 6: The grid server starts to execute the grid job submitted by the user.

Step 7: After the job is executed, the optical control layer is notified to close the optical channel connection, and at the same time, the optical network resource list is immediately updated.

In the present invention, once the network resource starts to be used, it is exclusive for the grid operation. This feature is obviously different from the sharing of network resources in traditional grid computing.

When a grid user submits a grid job request to the grid server, if the grid node needs to transmit data, it will be processed as follows:

Step 1: The grid node middleware queries the list of grid resources according to the grid job request, and finds the resources that meet the requirements.

Step 2: If no available grid resource meeting the requirements is found, the result will be returned, informing the user that there is currently no available grid resource, and the service ends; List of web resources;

Step 3: If the query result shows that there is no available optical network resource, then inform the user that there is currently no available optical network resource, and the service ends; if there is free available optical network resource, then establish a suitable optical channel according to the processing flow in Figure 3;

Step 4: After the optical control layer establishes the corresponding optical channel, return the result;

Step 5: The grid server starts to execute the grid job request submitted by the user, starts to transmit data, and updates the optical network resource list at the same time;

Step 6: After the job is executed, notify the optical control layer to close the optical channel connection;

Step 7: Update the optical network resource list again.

In step 3 of the above-mentioned photonic grid node data transmission process flow, when there are free and available optical network resources, the steps for establishing a suitable parallel optical channel are as follows:

Step 31: Judging whether there is a grid resource node with the required grid resources, if not, return the result; if yes, go to the next step;

Step 32: Determine the free and available optical channel of the grid resource node, if more than one grid resource node has an available optical channel or only one grid resource node has one available optical channel, then perform step 33; if there is only one grid resource node If the grid resource node has an available optical channel, and there is more than one available optical channel, perform step 34;

Step 33: Establish an optical channel to each grid resource node;

Step 34: compare the number N of optical channels that can be established between the grid application node and the resource node according to the optical network resource list, and the maximum number of optical channels that need to be established based on the size of the data to be transmitted. M; if M ≥N, establish N optical channels; if M<N, establish M optical channels.

In the present invention, the data transmission protocol uses Gridftp, and a Gridftp program is started for each optical channel connection, and the division of file data also follows Gridftp. Among them, the GridFTP control connection still establishes a TCP connection through the Internet, and the data connection requests the optical control layer to establish an optical channel. If there are several optical channels, start several Gridftp programs at the same time. Moreover, the grid middleware uses Globus toolkit4.0. Compared with the sharing of network resources in traditional grid computing, the network resources here are exclusively used for grid operations. On the other hand, in the control plane of the existing intelligent optical network, Generalized Multi-Protocol Label Switching (GMPLS) is used as a control signaling protocol to realize dynamic optical channel connection and link control functions. The present invention also adopts the control signaling protocol as the basis of optical layer control. How to use the GMPLS protocol is a conventional technique familiar to those skilled in the optical communication field, so details will not be described here.

The grid application-oriented optical parallel transmission method of the present invention has been described in detail above, but the specific implementation form of the present invention is not limited thereto. For those skilled in the art, various obvious changes made to the method of the present invention without departing from the spirit of the method and the scope of the claims are within the protection scope of the present invention.

Claims (5)

1. a kind of optical parallel transmission method facing grid application, it is characterized in that comprising the following steps: Step 1, the grid application node submits a grid job request to the grid server; Step 2, the grid server queries the grid resource list, finds a grid resource node that meets the job requirements, obtains the network address of the grid resource node, and proceeds to step 3; if there is no grid resource node that meets the job requirements, return As a result, the user is notified that there are currently no available optical grid resources, and the service ends; Step 3: Query the optical network resource list, find out the free and available optical network resource information of the grid resource node, and proceed to step 4; if the grid resource node has no free and available optical network resources, inform the user that there is no available free optical network resource at present Optical network resource, end of service; Step 4: According to the query result, the grid server initiates one or more optical channel establishment requests to the optical control layer as needed, and the optical channel is from the grid application node to the grid resource node with free and available optical network resources. channel between Step 5: After the optical control layer establishes the corresponding optical channel, return the result; Step 6, the grid server starts to execute the grid job request submitted by the user; In step 7, the operation is completed, and the optical control layer is notified to close the optical channel connection. 2. The optical parallel transmission method as claimed in claim 1, characterized in that: In the step 3, the step of querying the optical network resource list and finding the free and available optical network resource information of the grid resource node includes: When there is a grid resource node that meets the required grid resources, judge that N optical channels can be established between the grid application node and the grid resource node according to the optical network resource list of the grid resource node , N is an integer greater than 1; and according to the size of the data to be transmitted, it is judged that at most M optical channels need to be established; if M≥N, N optical channels are established; if M<N, M optical channels are established. 3. The optical parallel transmission method as claimed in claim 1, characterized in that: In the step 3, the step of querying the optical network resource list and finding the free and available optical network resource information of the grid resource node further includes: In the case that more than one grid resource node has the required grid resources, according to the optical network resource list of each grid resource node, if more than one grid resource node has available optical network resources, then in One optical channel is established between the grid application node and each grid resource node; if only one grid resource node has one optical channel resource, then a 1 optical channel is established between the grid application node and the grid resource node If only one grid resource node has available optical network resources, and the grid resource node can establish more than one optical channel, it is judged that N optical channels can be established between the grid application node and the grid resource node According to the size of the data to be transmitted, it is determined that at most M optical channels need to be established; if M≥N, N optical channels are established; if M<N, M optical channels are established. 4. The optical parallel transmission method as claimed in claim 1, characterized in that: In the step 6, when a data transmission service request is initiated according to the Gridftp protocol, the Gridftp control connection establishes a TCP connection through the Internet, and the data connection is established at the optical control layer. 5. The optical parallel transmission method according to claim 4, characterized in that: In the step 6, each optical channel connection starts a Gridftp program.

Images