CN101247309B - A system for general access to multi-grid platform - Google Patents

Abstract

The invention discloses a system for universally accessing a multi-grid platform, which includes a task manager, a storage manager, a registration manager and a security manager. The invention realizes the common task access of the grid programming environment and the public support platform of the Chinese education and scientific research grid, and fully improves the integration and utilization rate of the grid resources. The registration manager manages each target system, maintains the topology of the grid, allows grid nodes to register with the registration service, and coordinates the load balancing of each node. The storage manager provides unified grid data storage management, abstracts basic data operations such as accessing data, and provides users with a unified, simple and complete storage space service. The task manager executes the tasks submitted by the user, first obtains the input data from the user data space, then generates platform-specific execution scripts according to the submitted tasks, and returns the results to the user space after executing the scripts. The security manager performs security detection and control on the use of services.

Description

A system for general access to multi-grid platform

technical field

The invention belongs to a distributed server system, and in particular relates to a general grid access system based on a grid programming environment and a public support platform of China's education and scientific research grid.

Background technique

With the continuous development of grid computing, some grid systems have gradually matured. Many companies and research institutions have successively launched their own grid platforms to build grid systems, such as China Education and Research Grid Public Support Platform, American TeraGrid, German Internet Seoul grid programming environment, China Vega grid system, American Globus, China National Grid, etc. In the grid based on these grid system platforms, resources are shared, the utilization rate is greatly improved, and some high-performance computing projects are solved, which have a wide range of applications in medical, physical, chemical, biological, energy and other fields. However, because these grid platforms have their own characteristics, their internal structures and organizational methods are different, and the services they provide are also quite different. Therefore, resources cannot be shared among grid systems, let alone unified management. New islands of heterogeneous information are formed among the grid systems, which goes against the original intention of the grid.

Under such circumstances, in order to build a grid that fully utilizes global resource sharing, resource sharing and transparent access among existing grid systems is particularly important. The grid programming environment aims to access the resources of different heterogeneous grid systems through the clients of the grid programming environment. The basic idea is to define the common abstraction layer of the grid system. The grid system implements the interface provided by the abstraction layer, so as to realize the universal access of the multi-grid platform. For grid application developers, the underlying details such as the protocol implemented by the grid platform and programming language are transparent. Through these abstract interfaces, grid applications can easily access heterogeneous grid platform. The grid programming environment will define three layers, application layer, service layer and tool layer. Lattice applications only need to implement corresponding interfaces, manage input and output and application logic, and do not need to care about specific processes such as job submission. The service layer defines some high-level services, including dynamic resource and service registration, business process execution language, virtual workspace, etc. The tools layer defines basic atomic grid services that drive the target system interface of a grid programming environment, or the execution subsystem of a heterogeneous grid system.

China Education and Research Grid is a grid system based on China Education and Research Network and Globus′Toolkit 4. In the existing grid system, it already contains a large number of valuable resources and rich grid applications, such as online university courses, fluid Mechanics grid, biological grid, and medical image grid have been joined by many universities and research institutions. Through the China Education and Research Grid, it is convenient, fast and reliable to share and access these teaching and research resources, and improve the utilization of various computing and data resources. utilization rate. Since the public support platform of China Educational Research Grid is developed based on Globus Toolkit 4, the German national grid is completely different from Globus Toolkit 4, and transparent access cannot be achieved. At the same time, the public support platform of China Educational Research Grid provides some things that Globus Toolkit 4 does not have Services and resources, so it cannot realize direct access to the China Education and Research Grid and the grid programming environment.

Contents of the invention

The purpose of the present invention is to provide a system for universally accessing multi-grid platforms in view of the characteristics that the above-mentioned grids cannot be universally accessed and the shortcomings of the existing access methods between multi-grid platforms. You can access and use the resources of China Education and Research Grid without modification.

The system for universally accessing the multi-grid platform of the present invention is characterized in that: the system includes a task manager, a storage manager, a registration manager and a security manager;

The task manager is used to accept job tasks submitted by users, analyze task parameters, generate local objects according to the general operation service address provided by China Education and Research Grid, call general operation services through local objects to complete jobs, and notify storage before executing tasks submitted by users The manager transmits the input data from the grid programming environment to the China Education and Research Grid public support platform, and after the task is executed, notifies the storage manager to transfer the output data from the China Education and Research Grid public support platform to the grid programming environment;

The storage manager uses the virtual user space service of the China Education and Research Grid public support platform to perform data transmission for the task manager, and completes the data transmission between the grid programming environment and the China Education and Research Grid public support platform;

The registration manager invokes the information center service of the China Education and Research Grid Public Support Platform to monitor the status of grid nodes, and is used to exchange information between the grid programming environment and the China Education and Research Grid Public Support Platform; the China Education and Research Grid Public Support Platform When the general operation service of the target system is registered in the grid programming environment, the registration manager coordinates the information between the grid nodes between the grid programming environment and the public support platform of the China Education and Research Grid;

The security manager is used to implement security control on data access, task execution and target system registration, and to verify the user rights of the task manager, storage manager and registration manager.

The present invention integrates the grid programming environment with the public support platform of China's education and scientific research grid, makes full use of the unified interface of the grid programming environment and the rich grid resources of China's education and scientific research grid, and calls the China Education and Research Network through the grid programming environment The relevant services of the grid public support platform realize the transparent access of the grid programming environment and the public support platform of China's education and scientific research grid. Specifically, the present invention has the following technical effects:

(1) Unified programming and user interface

The client layer of the grid programming environment is not changed, so users can access the China Education and Research Grid through the client of the grid programming environment without modification. At the same time, the interface to grid application developers remains unchanged, so the original grid programming environment grid application components such as GridBean can also be deployed to the system without modification. In this way, the system provides a client and grid application program interface fully compatible with the grid programming environment, and both end users and developers can use the system without adjustment.

(2) Effective utilization of grid resources

This system enables the grid programming environment client and grid application programs to easily access the resources of the China Education and Research Grid, while the original China Education and Research Grid resources can only be accessed and used through the portal of the China Education and Research Grid Public Support Platform , other heterogeneous grid systems cannot be accessed conveniently. The system has uniformly packaged the services provided by the China Education and Research Grid Public Support Platform, shielding the differences between the China Education and Research Grid Public Support Platform and other grid platforms. The grid programming environment and other heterogeneous grid platforms can access China Education and Research Grid resources through a unified interface, submit homework tasks to the China Education and Research Grid public support platform, and cooperate with other grid systems, so that China Education and Research Network Lattice and other heterogeneous grid resources are more fully utilized.

(3) Heterogeneous grid system load balancing

In the system, the general operation service of the China Education and Research Grid public support platform is used as a target system. Users can submit jobs to the China Education and Research Grid public support platform through the grid programming environment, and the general operation service returns to the user after executing the job. status result. In a heterogeneous grid system, the target systems are equal in the grid programming environment. When the loads of different grid systems are unbalanced, more job tasks can be conveniently submitted to relatively idle target systems, so as to achieve heterogeneous Grid system load balancing.

Description of drawings

Fig. 1 is a schematic structural diagram of the universal access multi-grid platform system of the present invention.

Fig. 2 is a schematic diagram of the structure of the task manager.

Fig. 3 is a schematic diagram of the processing flow of the task manager.

Fig. 4 is a schematic diagram of the structure of the storage manager.

Fig. 5 is a schematic diagram of the processing flow of the storage manager.

Fig. 6 is a schematic diagram of the registration manager structure.

FIG. 7 is a schematic diagram of the registration process flow of the registration manager target system.

FIG. 8 is a schematic diagram of the process flow of the registration manager target system management.

Fig. 9 is a schematic diagram of the structure of the security manager.

Detailed ways

Below in conjunction with accompanying drawing and example the present invention is described in further detail.

The Chinese education and scientific research grid public support platform and the grid programming environment grid universal access system provided by the invention include a task manager, a storage manager, a registration manager, and a security manager. The task manager adds a new type to the target system interface of the grid programming environment—the general operating service of the China Education and Research Grid Public Support Platform. It is different from the mechanism of the grid programming environment, and it is not called through the Java language remote method To perform a task, but directly generate an object to call the service provided by the general operation service to perform a task; the storage manager enables the storage management service, file import service, and file export service of the grid programming environment to call the China Education and Research Grid public The storage management service and user space service of the supporting platform are used to access and use the data resources of the China Education and Research Grid. These services are used to transmit data when performing tasks; the registry manager management provides the ability to monitor the status of the grid, and the grid programming environment and The China Education and Research Grid public support platform can view the status information of each other and the other party through the management terminal, and monitor grid resources; the security device introduces a general security access framework, through which security detection and control can be performed when calling services.

As shown in FIG. 1 , the system of the present invention for universally accessing a multi-grid platform is implemented based on the China Education and Research Grid, and includes a task manager 1 , a storage manager 2 , a registration manager 3 and a security manager 4 .

Task manager 1 is used to accept job tasks submitted by users, analyze task parameters, generate local objects according to the general operation service address provided by China Scientific Research Grid, call general operation service services through local objects to complete jobs, and execute tasks submitted by users Notify the storage manager 2 to transmit the input data from the grid programming environment to the public support platform of the China Education and Research Grid, and notify the storage manager 2 to transmit the output data from the public support platform of the China Education and Research Grid to the grid programming environment after the task is executed .

Storage Manager 2 uses the virtual user space service of China Education and Research Grid Public Support Platform to transmit data for Task Manager 1, and the transmission of data between the grid programming environment and China Education and Research Grid Public Support Platform is managed by storage Device 2 is complete.

The registration manager 3 invokes the information center service to monitor the grid node status to the China Education and Research Grid public support platform, and is used for exchanging information between the grid programming environment and the China Education and Research Grid public support platform. When the general operation service of the China Education and Research Grid public support platform is registered to the grid programming environment as the target system, the registration manager coordinates the communication between the grid nodes between the grid programming environment and the China Education and Research Grid public support platform. information.

Security manager 4 is used to implement security control on data access, task execution, and target system registration. Storage manager 2 uses the virtual user space of the China Education and Research Grid public support platform to store and access file data. Before accessing data, the security manager 4 Verify the user and access rights; the task manager 1 invokes the general operation service of the public support platform of the China Education and Research Grid to complete the job submitted by the user. Before the job is executed, the security manager 4 verifies the user's execution rights.

Each component is described in detail below.

As shown in Figure 2, the task manager 1 includes a task analysis module 1.1, a task execution module 1.2, a task data transmission module 1.3 and a task monitoring module 1.4.

The task parsing module 1.1 receives the task from the client, and the task is described in the form of job submission description language. The task parsing module 1.1 parses the job submission description language, obtains the user name, working directory, executable command and input and output parameters of the task, and creates the task object and transfer it to the task execution module 1.2.

The task execution module 1.2 requests input data transmission from the task data transmission module 1.3 according to the input parameters in the task object. After the task data transmission module 1.3 completes the input data transmission, the task execution module 1.2 invokes the general operation service of China Education and Research Grid to execute the task. After the task object is submitted, the task object is provided to the task monitoring module 1.4.

The task monitoring module 1.4 monitors the status of the task in real time by sending a task status query request to the general operation service of the China Education and Research Grid, and notifies the task execution module 1.2 to request output data transmission from the task data transmission module 1.3 when the task status is completed.

As shown in Figure 3, the processing flow of task manager 1 is as follows:

1) Parsing parameters and creating task objects. The task submitted by the user is described in the job submission description language. The task parsing module 1.1 first parses the task to obtain the parameters of the requested task, including requested services, resources, executed command parameters, input and output parameters, and creates task objects.

2) Input data transfer. According to the input parameters in the task request parameters analyzed in 1), the input data is transmitted through the task data transmission module 1.3, and the data is transmitted from the grid programming environment to the virtual user space of the China Education and Research Grid Public Support Platform.

3) Perform tasks. When 1), 2), and 3) are completed, the parameters and input data of the task are ready, and the task execution module 1.2 invokes the general operation service of China Education and Research Grid to execute the task.

4) Monitoring tasks. After the task is submitted, the task monitoring module 1.4 monitors the status of the task in real time, and the status of the task includes submitted, ready, running, and completed.

5) Determine whether the task status is completed, if not, return to 5) continue to monitor the task status.

6) Output data transmission. When the task is completed, output data such as standard output, standard error, log, task-specific output, etc. are transmitted from the virtual user space of the China Education and Research Grid public support platform to the system data space.

As shown in Figure 4, the storage manager 2 includes a file transfer client module 2.1, a file transfer creator module 2.2 and a file transfer service module 2.3.

The file transfer client module 2.1 is used for the data transfer request, and parses the transfer request to obtain general information such as the file name, user name, and size of the file to be transferred, and sends it to the file transfer creator module 2.2.

The file transfer creator module 2.2 is responsible for mapping the general file information to the file information in the user's virtual space of the China Education and Research Grid public support platform, and sending the mapped file information to the file transfer service module 2.3.

The file transfer service module 2.3 sends a request to the storage resource service and user space service of the China Education and Research Grid according to the mapped file information, and performs actual file transfer.

The storage resource service of the China Education and Research Grid Public Support Platform is responsible for managing various heterogeneous storage resources widely distributed in the China Education and Research Grid, and the user space service of the China Education and Research Grid Public Support Platform is responsible for managing the user virtual space The metadata of the file, the address format of the file is vs://username@domain name/pathname/…/…/filename.

As shown in Figure 5, the processing flow of the storage manager 2 is as follows:

1) Application client or task management module 1 requests data transmission

2) The file transfer client module 2.1 parses the transfer request and obtains the general information of the file to be transferred.

3) The file transmission creator module 2.2 maps the file information to the file information format of the user's virtual space of the China Education and Research Grid Public Support Platform.

4) The file transfer service module 2.3 sends requests to the China Education and Research Grid storage resource service and user space service for actual file transfer.

As shown in Fig. 6, the registration manager 3 includes a target system registration module 3.1 and a target system interface module 3.2.

The target system registration module 3.1 is responsible for registering the target system to the target system library of the grid programming environment. The target system registered to the target system library can be managed uniformly through the grid programming environment, and various resources in the target system can be managed through the grid programming environment. The client seamlessly accesses, accesses and manages the registered China Education and Research Grid resources through the target system interface module 3.2.

The target system interface module 3.2 is responsible for packaging the nodes and domains of the China Education and Research Grid into the target system in the grid programming environment. The nodes and domains of the China Education and Research Grid are the management units of the public support platform of the China Education and Research Grid. System is the management unit in the grid programming environment, which includes storage system, execution system, data transmission protocol, system users, etc. Grid applications are deployed on the target system.

As shown in FIG. 7 , the target system registration processing flow of the registration manager 3 is as follows:

1) The target system registration module 3.1 processes the registration request, and obtains the addresses of general operation service, user space service and storage resource service.

2) The target system interface module 3.2 checks whether the above services are available through the China Education and Research Grid Information Center service.

3) If the above service is unavailable as a result of 2), the registration fails and an exception notification is issued.

4) If the above-mentioned services are available as a result of 2), the target system interface module 3.2 creates a target system, sets its general operation service, user space service, address of storage resource service, and other general parameters.

5) The target system registration module 3.1 adds 4) the newly created target system to the target system library of the grid programming environment, and updates the metadata of the target system library.

As shown in FIG. 8, the target system management processing flow of the registration manager 3 is as follows:

1) The target system interface module 3.2 inquires the service of the China Educational Research Information Center to obtain the status information of the target system;

2) Process status information to obtain the management type;

3) If the management type obtained in 2) is query, end;

4) If the management type obtained in 2) is to add a grid application, obtain application parameters such as application name, version, running script, etc., use these parameters to set a new grid application, and add the new application to the corresponding target system;

5) If the management type obtained in 2) is to delete the grid application, delete the specified application from the target system;

6) If the management type obtained in 2) is to add a user, obtain user parameters such as user name, user group, authority, etc., set a new user with these parameters, and add the new user to the corresponding target system;

7) If the management type obtained in 2) is to delete the user, delete the specified user from the target system;

8) If the management type obtained in 2) is to add storage space, obtain storage space parameters such as name, capacity size, storage address, etc., use these parameters to set a new storage space, and add the new storage space to the corresponding target system;

9) If the management type obtained in 2) is delete storage space, delete the specified storage space from the target system.

As shown in Fig. 9, the security manager 4 includes an agent plug-in module 4.1, an authentication plug-in module 4.2, a signaling plug-in module 4.3 and an operation format plug-in module 4.4.

The proxy plug-in module 4.1 receives the request token issued by the grid programming environment, generates an authentication proxy, calls the authentication plug-in module 4.2, and sends the token obtained after authentication to the operation format plug-in module 4.4.

The authentication plug-in module 4.2 then calls the signaling plug-in module 4.3 to confirm the identity of the user, and the signaling plug-in module 4.3 completes the actual authentication.

The operating format plug-in module 4.4 is responsible for converting the token into the format recognized by the China Education and Research Grid public support platform.

Claims (5)

1. the system of a universal accesses to multi-cell platform, it is characterized in that: this system comprises task manager (1), storage manager (2), registration manager (3) and security manager (4);

Task manager (1) is used to accept the job task that the user submits to, resolve task parameters, the general operation service address that provides according to the CERNET lattice generates native object, calling general operation service by native object fulfils assignment, notified storage manager (2) will import data before the task of carrying out user's submission and be transferred to CERNET lattice common support platform from the grid programmed environment, task notifies storage manager (2) that dateout is transferred to the grid programmed environment from Chinese education scientific research grid common support platform after carrying out;

Storage manager (2) uses the Virtual User space service of CERNET lattice common support platform, for task manager (1) carries out transfer of data, finishes the transmission of data between grid programmed environment and CERNET lattice common support platform;

Registration manager (3) calls information centre's service monitoring grid node state of CERNET lattice common support platform, is used for exchange message between grid programmed environment and the CERNET lattice common support platform; When the general operation service of CERNET lattice common support platform was registered to the grid programmed environment as goal systems, registration manager (3) was in the information of coordinating between grid programmed environment and the CERNET lattice common support platform between the mesh node;

Security manager (4) is used for data visit, task are carried out and goal systems registration enforcement security control, and the user right of task manager (1), storage manager (2) and registration manager (3) is verified.

2. the system of universal accesses to multi-cell platform according to claim 1, it is characterized in that: task manager (1) comprises task parsing module (1.1), task execution module (1.2), task data transport module (1.3) and Mission Monitor module (1.4);

Task parsing module (1.1) is used to receive the task from client, submits to descriptive language to resolve to operation, the creation task object, and send it to task execution module (1.2);

Task execution module (1.2) is imported transfer of data according to the input parameter in the task object to task data transport module (1.3) request, call the general operation service of CERNET lattice and execute the task, and provide Mission Monitor module (1.4) task object;

Task data transport module (1.3) is used for the data transfer command that task execution module (1.2) is submitted to is transmitted to storage manager (2);

Mission Monitor module (1.4) is by sending the state of the real-time monitor task of task status query requests to the general operation service of CERNET lattice, when task status notice task execution module (1.2) when finishing is transmitted to task data transport module (1.3) request dateout.

3. the system of universal accesses to multi-cell platform according to claim 1 and 2, it is characterized in that: storage manager (2) comprises file transmitting to customer end module (2.1), file transfer builder module (2.2) and file transfer services module (2.3);

File transmitting to customer end module (2.1) is used to receive data transfer request, and resolves the transmission request, sends the general information of the transfer files of wanting that obtains to file transfer builder module (2.2);

File transfer builder module (2.2) is responsible for the file general information is mapped to the fileinfo of CERNET lattice common support platform user Virtual Space, and the fileinfo after will shining upon sends file transfer services module (2.3) to;

File transfer services module (2.3) sends request according to the fileinfo after shining upon to service of CERNET lattice storage resources and user's space service, carries out actual file transmission.

4. the system of universal accesses to multi-cell platform according to claim 3, it is characterized in that: registration manager (3) comprises goal systems Registering modules (3.1) and goal systems interface module (3.2);

Goal systems Registering modules (3.1) is responsible for goal systems is registered to the target system library of grid programmed environment, by goal systems interface module (3.2) to registered CERNET lattice resource access and management;

Goal systems interface module (3.2) is used for the node of CERNET lattice and territory are packaged into the goal systems of grid programmed environment.

5. the system of universal accesses to multi-cell platform according to claim 4 is characterized in that: security manager (4) comprises proxy plug-ins module (4.1), authentication card module (4.2), signaling card module (4.3) and operational format card module (4.4);

Proxy plug-ins module (4.1) is used to receive the request token that the grid programmed environment is sent, and generates authentication proxy, and invokes authentication card module (4.2), and the token that obtains after the authentication is sent to operational format card module (4.4);

Authentication card module (4.2) calls signaling card module (4.3) again and confirms user identity, finishes actual authentication by signaling card module (4.3);

Operational format card module (4.4) is responsible for token is converted to the form of CERNET lattice common support land identification.

Images