[go: up one dir, main page]

CN106331160A - Data migration method and system - Google Patents

Data migration method and system Download PDF

Info

Publication number
CN106331160A
CN106331160A CN201610863983.0A CN201610863983A CN106331160A CN 106331160 A CN106331160 A CN 106331160A CN 201610863983 A CN201610863983 A CN 201610863983A CN 106331160 A CN106331160 A CN 106331160A
Authority
CN
China
Prior art keywords
data
cloud service
task
data migration
service node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610863983.0A
Other languages
Chinese (zh)
Inventor
吴兴义
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LeTV Holding Beijing Co Ltd
LeTV Cloud Computing Co Ltd
Original Assignee
LeTV Holding Beijing Co Ltd
LeTV Cloud Computing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by LeTV Holding Beijing Co Ltd, LeTV Cloud Computing Co Ltd filed Critical LeTV Holding Beijing Co Ltd
Priority to CN201610863983.0A priority Critical patent/CN106331160A/en
Publication of CN106331160A publication Critical patent/CN106331160A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/60Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data migration method and a system. The method comprises the following steps: receiving a request message for a data migration task including the to-be-migrated data address and the data migration destination address; determining the resources required by the data mitigation task; searching the available resources at various cloud service nodes; and according to the resources required by the data migration task and the available resources at the cloud service nodes, sending the data migration task to the cloud service nodes satisfying the preset conditions so that the cloud service nodes download locally the files corresponding to the to-be-migrated data address and uploaded them to the data migration destination address. The system of the invention is used to execute the method. According to the embodiments of the invention, a data mitigation task is sent to cloud service nodes satisfying preset conditions so that the cloud service nodes download locally the files corresponding to the to-be-migrated data address and uploaded them to the data migration destination address. In this way, the resource utilization efficiency of the cloud service nodes are increased and the time required for data mitigation is reduced.

Description

A kind of data migration method and system
Technical field
The present embodiments relate to computer communication technology field, particularly relate to a kind of data migration method and system.
Background technology
Distributed memory system is data according to the cutting of certain rule and to be broken up and be stored in many platform independent common store clothes On business device.Traditional network store system uses all data of storage server repository concentrated, storage server to become system The bottleneck of performance, is also the focus of reliability and safety, it is impossible to meet the needs of Mass storage application, and distributed storage System uses extendible system structure, utilizes multiple stage storage server to share storage load, utilizes location server location to deposit Storage information, it not only increases the reliability of system, availability and access efficiency, is also easy to extension.Storage cluster thousands of on Ten thousand station servers can be substantially redundant by data, such that it is able to significantly improve the safety of data.Existing cloud storage manufacturer one As store in a distributed manner and store system as bottom, cloud storage service is externally provided.
During realizing the embodiment of the present invention, inventor finds, user can need the need of Data Migration Asking, the mode of its Data Migration mainly has following two kinds: first, migrates from the home server of user.Second, deposit from other clouds Storage manufacturer migrates, and cloud storage service is that enterprise-class tools is selecting any cloud platform to be required for, because storage is the base of enterprise Plinth demand, based on cloud storage service, if a cloud service manufacturer is unsatisfied with and is switched to other one, then by enterprise-class tools Need to carry out mass data migration.The simple ability relying on user will consume the time and efforts that user is bigger, and user needs The data that will be stored in vendor A first download, then upload to the cloud storage service of manufacturer B.When there being more data migration task Time, task may be caused not to need to wait due to the cloud service node resource that some execution tasks are more, thus cause Data Migration needs the longer time.And other cloud service nodes also have huge surplus resources, these resources waste And can not be given full play to it and be worth.
Therefore, how to improve the resource utilization of cloud service node, reducing Data Migration required time is the most urgently The problem solved.
Summary of the invention
The problem existed for prior art, the embodiment of the present invention provides a kind of data migration method and system.
On the one hand, the embodiment of the present invention provides a kind of data migration method, including:
Receiving the request message of data migration task, the request message of described data migration task includes data to be migrated ground Location and data move target address;
Determine the resource needed for described data migration task, and inquire about the available resources of each cloud service node;
According to the resource needed for described data migration task and the available resources of described each cloud service node, by described data Migration task is sent to meet pre-conditioned described cloud service node, so that described cloud service node is by described data to be migrated This locality of the file download extremely described cloud service node that address is corresponding, and be uploaded in described Data Migration destination address.
The embodiment of the present invention provides a kind of data mover system, including:
Request receiver module, for receiving the request message of data migration task, the request of described data migration task disappears Breath includes data address to be migrated and data move target address;
Resource management module, for determining the resource needed for described data migration task, and inquires about each cloud service node Available resources;
Data Migration module, for according to the resource needed for described data migration task and described each cloud service node can By resource, it is sent to described data migration task meet pre-conditioned described cloud service node, so that described cloud service joint Point is by file download corresponding for described data address to be migrated to this locality of described cloud service node, and is uploaded to described data and moves Move in destination address.
Data migration method that the embodiment of the present invention provides and system, meet pre-conditioned cloud by task being handed down to Service node, so that cloud service node is by this locality of file download corresponding for data address to be migrated to cloud service node, and on Reach in Data Migration destination address, improve the resource utilization of cloud service node, reduce Data Migration required time.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing In having technology to describe, the required accompanying drawing used is briefly described, it should be apparent that, the accompanying drawing in describing below is this Some bright embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, it is also possible to root Other accompanying drawing is obtained according to these accompanying drawings.
A kind of data migration method schematic flow sheet that Fig. 1 provides for one embodiment of the invention;
A kind of data migration method schematic flow sheet that Fig. 2 provides for another embodiment of the present invention;
A kind of data migration method schematic flow sheet that Fig. 3 provides for further embodiment of this invention;
A kind of data migration method schematic flow sheet that Fig. 4 provides for yet another embodiment of the invention;
A kind of data mover system structural representation that Fig. 5 provides for one embodiment of the invention;
A kind of data mover system structural representation that Fig. 6 provides for another embodiment of the present invention;
A kind of data mover system structural representation that Fig. 7 provides for further embodiment of this invention;
A kind of data mover system structural representation that Fig. 8 provides for further embodiment of this invention;
A kind of data mover system entity structure schematic diagram that Fig. 9 provides for the embodiment of the present invention.
Detailed description of the invention
For making the purpose of the embodiment of the present invention, technical scheme and advantage clearer, below in conjunction with the embodiment of the present invention In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is The a part of embodiment of the present invention rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art The every other embodiment obtained under not making creative work premise, broadly falls into the scope of protection of the invention.
A kind of data migration method schematic flow sheet that Fig. 1 provides for one embodiment of the invention, as it is shown in figure 1, described side Method, including:
Step 101: receiving the request message of data migration task, the request message of described data migration task includes waiting to move Move data address and data move target address;
Specifically, system receives the solicited message of the data migration task that user is submitted to, wherein, data by electronic equipment The solicited message of migration task includes data address to be migrated and data move target address.It should be noted that, wait to move Moving data address is a public network address, and data to be migrated are stored in this public network address;Data Migration destination address is to use Family is intended the address in Data Migration to cloud service manufacturer.
Step 102: determine the resource needed for described data migration task, and inquire about the available resources of each cloud service node;
Specifically, after system receives the request message of the data migration task that user is submitted to by electronic equipment, determine Resource needed for this request message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, please Resource needed for seeking message includes required CPU, internal memory and the bandwidth consumed of this data migration task.Such as, user passes through Resource needed for the request message of the data migration task that electronic equipment is submitted to is 0.1 CPU, 100M internal memory and 10M bandwidth.Institute State cloud service node and belong to data cloud service manufacturer to be moved to and for performing data migration task.It is understood that Can also is that other resources, this is not especially limited by the present embodiment.
Step 103: according to the resource needed for described data migration task and the available resources of described each cloud service node, will Described data migration task is sent to meet pre-conditioned described cloud service node, so that described cloud service node is treated described Migrate this locality of file download corresponding to data address extremely described cloud service node, and be uploaded to described Data Migration destination address In.
Specifically, system is according to the resource needed for data migration task and each cloud service node of inquiring from data base Available resources, be sent to data migration task meet pre-conditioned cloud service node, cloud service node receives data After the request message of migration task, according to the data address to be migrated carried in the request message of data migration task, will wait to move Move the data in data address and download to this locality of cloud service node, then data to be migrated are uploaded to data from this locality and move Move in destination address.It should be noted that, this locality of cloud service node is hard for depositing the cloud of data in cloud service node In dish rather than user use physical disk.From this locality of cloud service node to Data Migration destination upload data be Intranet is carried out, now need not pass through outer net.
Such as, if user wants the mass data that will be stored in cloud service vendor A to move to cloud service manufacturer B, user Can be submitted to the solicited message of data migration task to system by electronic equipment, wherein, solicited message includes number to be migrated According to address url1 and data move target address url2.Now, system receives the data that user submitted to by electronic equipment and moves The solicited message of shifting task.System determines that this data migration task needs 0.1 CPU, 100M internal memory and 10M bandwidth, and Inquiring the available resources in each cloud service node from data base, the resource needed according to this data migration task is with each The available resources of cloud service node, are sent to data migration task meet pre-conditioned cloud service node;Cloud service node Start after receiving task to perform, according to data address url1 to be migrated, it would be desirable to the data of migration download to cloud service node This locality, then be uploaded to Data Migration destination address url2 from this locality of cloud service node.
The embodiment of the present invention meets pre-conditioned cloud service node by task being handed down to, so that cloud service node will File download corresponding to data address to be migrated is to this locality of cloud service node, and is uploaded in Data Migration destination address, carries The resource utilization of high cloud service node, reduces Data Migration required time.
On the basis of above-described embodiment, further, described pre-conditioned, including:
The available resources of described cloud service node are more than the resource needed for described data migration task.
Specifically, storing the information of the available resources of all cloud service nodes in data base, system passes through electricity according to user Resource needed for the data migration task that subset is submitted to, the available resources that data migration task is sent to cloud service node are big In data migration task resource requirement.Optionally, the available resources that data migration task can be sent to cloud service node are big That cloud service node that available resources are maximum in data migration task resource requirement.
The embodiment of the present invention by being sent to the available resources of cloud service node more than Data Migration by data migration task The cloud service node of required by task resource, reasonable distribution task, it is to avoid task waiting time, thus it is complete to decrease required by task Become duration.
A kind of data migration method schematic flow sheet that Fig. 2 provides for another embodiment of the present invention, as in figure 2 it is shown, described Method, including:
Step 201: receiving the request message of data migration task, the request message of described data migration task includes waiting to move Move data address and data move target address;
Specifically, system receives the solicited message of the data migration task that user is submitted to, wherein, number by electronic equipment Data address to be migrated and data move target address is included according to the solicited message of the task of migration.It should be noted that, treat The data address migrated is a public network address, and data to be migrated are stored in this public network address;Data Migration destination address It is that user intends the address in Data Migration to cloud service manufacturer.
Step 202: determine the resource needed for described data migration task, and inquire about the available resources of each cloud service node;
Specifically, after system receives the request message of the data migration task that user is submitted to by electronic equipment, determine Resource needed for this request message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, please Resource needed for seeking message includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user passes through electronics Resource needed for the request message of the data migration task that equipment is submitted to is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud Service node belongs to manufacturer B and for performing data migration task.It is understood that can also is that other resources, this enforcement This is not especially limited by example.
Step 203: according to the resource needed for described data migration task and the available resources of described each cloud service node, will Described data migration task is sent to meet pre-conditioned described cloud service node, so that described cloud service node is treated described Migrate this locality of file download corresponding to data address extremely described cloud service node, and be uploaded to described Data Migration destination address In;
Specifically, system is according to the resource needed for data migration task and each cloud service node of inquiring from data base Available resources, be sent to data migration task meet pre-conditioned cloud service node, cloud service node receives data After the request message of migration task, according to the data address to be migrated carried in the request message of data migration task, will wait to move Move the data in data address and download to this locality of cloud service node, then data to be migrated are uploaded to data from this locality and move Move in destination address.It should be noted that, this locality of cloud service node is hard for depositing the cloud of data in cloud service node In dish rather than user use physical disk.From this locality of cloud service node to Data Migration destination upload data be Intranet is carried out, now need not pass through outer net.
Step 204: receive the available resources that described each cloud service node reports, and store in data base.
Specifically, each cloud service node can regularly report available resources, and system will receive what each cloud service node reported Available resources store in data base.
The embodiment of the present invention is by receiving the available resources that each cloud service node reports, and stores in data base, improves System judges whether each cloud service node meets pre-conditioned accuracy.
A kind of data migration method schematic flow sheet that Fig. 3 provides for further embodiment of this invention, as it is shown on figure 3, described Method, including:
Step 301: receiving the request message of data migration task, the request message of described data migration task includes waiting to move Move data address and data move target address;
Specifically, system receives the solicited message of the data migration task that user is submitted to, wherein, number by electronic equipment Data address to be migrated and data move target address is included according to the solicited message of the task of migration.It should be noted that, treat The data address migrated is a public network address, and data to be migrated are stored in this public network address;Data Migration destination address Be user intend by Data Migration to cloud service manufacturer in address.
Step 302: determine the resource needed for described data migration task, and inquire about the available resources of each cloud service node;
Specifically, after system receives the request message of the data migration task that user is submitted to by electronic equipment, determine Resource needed for this request message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, please Resource needed for seeking message includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user passes through electronics Resource needed for the request message of the data migration task that equipment is submitted to is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud Service node belongs to manufacturer B and for performing data migration task.It is understood that can also is that other resources, this enforcement This is not especially limited by example.
Step 303: according to the resource needed for described data migration task and the available resources of described each cloud service node, will Described data migration task is sent to meet pre-conditioned described cloud service node, so that described cloud service node is treated described Migrate this locality of file download corresponding to data address extremely described cloud service node, and be uploaded to described Data Migration destination address In;
Specifically, system is according to the resource needed for data migration task and each cloud service node of inquiring from data base Available resources, be sent to data migration task meet pre-conditioned cloud service node, cloud service node receives data After the request message of migration task, according to the data address to be migrated carried in the request message of data migration task, will wait to move Move the data in data address and download to this locality of cloud service node, then data to be migrated are uploaded to data from this locality and move Move in destination address.It should be noted that, this locality of cloud service node is hard for depositing the cloud of data in cloud service node In dish rather than user use physical disk.From this locality of cloud service node to Data Migration destination upload data be Intranet is carried out, now need not pass through outer net.
Step 304: receive the available resources that described each cloud service node reports, and store in data base.
Specifically, each cloud service node can regularly report available resources, and system will receive what each cloud service node reported Available resources store in data base;
Step 305: described be sent to described data migration task meet pre-conditioned cloud service node after, by described Task status corresponding to data address to be migrated is set to first task state;Described cloud service node is by described migration data ground File download that location is corresponding after being uploaded in described Data Migration destination address, by corresponding for described data address to be migrated One task status is set to the second task status.
Specifically, after being sent to data migration task meet pre-conditioned cloud service node, arrange in data base Task status corresponding to data address to be migrated is first task state, and optionally, first task state can be " issuing ". Cloud service node starts after receiving task to perform, and after task completes, arranges data address to be migrated corresponding in data base Task status is the second task status, and optionally, the second task status can be " completing ".It is understood that it is first Business state and the second task status are not limited to " issuing " and " completing " cited by the present embodiment.
The embodiment of the present invention meets pre-conditioned cloud service node by task being handed down to, so that cloud service node will File download corresponding to data address to be migrated is to this locality of cloud service node, and is uploaded in Data Migration destination address, carries The resource utilization of high cloud service node, reduces Data Migration required time.
A kind of data migration method schematic flow sheet that Fig. 4 provides for yet another embodiment of the invention, as shown in Figure 4, described Method, including:
Step 401: receiving the request message of data migration task, the request message of described data migration task includes waiting to move Move data address and data move target address;
Specifically, system receives the solicited message of the data migration task that user is submitted to, wherein, number by electronic equipment Data address to be migrated and data move target address is included according to the solicited message of the task of migration.It should be noted that, treat The data address migrated is a public network address, and data to be migrated are stored in this public network address;Data Migration destination address It is that user intends the address in Data Migration to cloud service manufacturer B.
Step 402: determine the resource needed for described data migration task, and inquire about the available resources of each cloud service node;
Specifically, after system receives the request message of the data migration task that user is submitted to by electronic equipment, determine Resource needed for this request message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, please Resource needed for seeking message includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user passes through electronics Resource needed for the request message of the data migration task that equipment is submitted to is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud Service node belongs to manufacturer B and for performing data migration task.It is understood that can also is that other resources, this enforcement This is not especially limited by example.
Step 403: according to the resource needed for described data migration task and the available resources of described each cloud service node, will Described data migration task is sent to meet pre-conditioned described cloud service node, so that described cloud service node is treated described Migrate this locality of file download corresponding to data address extremely described cloud service node, and be uploaded to described Data Migration destination address In;
Specifically, system is according to the resource needed for data migration task and each cloud service node of inquiring from data base Available resources, be sent to data migration task meet pre-conditioned cloud service node, cloud service node receives data After the request message of migration task, according to the data address to be migrated carried in the request message of data migration task, will wait to move Move the data in data address and download to this locality of cloud service node, then data to be migrated are uploaded to data from this locality and move Move in destination address.It should be noted that, this locality of cloud service node is hard for depositing the cloud of data in cloud service node In dish rather than user use physical disk.From this locality of cloud service node to Data Migration destination upload data be Intranet is carried out, now need not pass through outer net.
Step 404: receive the available resources that described each cloud service node reports, and store in data base.
Specifically, each cloud service node can regularly report available resources, and system will receive what each cloud service node reported Available resources store in data base;
Step 405: described be sent to described data migration task meet pre-conditioned cloud service node after, by described Task status corresponding to data address to be migrated is set to first task state;Described cloud service node is by described migration data ground File download that location is corresponding after being uploaded in described Data Migration destination address, by corresponding for described data address to be migrated One task status is set to the second task status;
Specifically, after being sent to data migration task meet pre-conditioned cloud service node, arrange in data base Task status corresponding to data address to be migrated is first task state, and optionally, first task state can be " issuing ". Cloud service node starts after receiving task to perform, and after task completes, arranges data address to be migrated corresponding in data base Task status is the second task status, and optionally, the second task status can be " completing ".It is understood that it is first Business state and the second task status are not limited to " issuing " and " completing " cited by the present embodiment.
Step 406: receive Data Migration status query request, described Data Migration status query request include described in wait to move Move data address, inquire about, according to described data address to be migrated, the task status that described data address to be migrated is corresponding, and return The described task status inquired.
Specifically, the Data Migration status query request that user is submitted to, wherein Data Migration shape are received by electronic equipment State inquiry request includes data address to be migrated, and the data address to be migrated that system is submitted to by electronic equipment according to user is counting The task status corresponding according to inquiring about data address to be migrated in storehouse, and the task status inquired is returned to electronic equipment.Example As: user submits the request of Data Migration status poll to by electronic equipment to system, request includes to be inquired about to be migrated Data address url, system is inquired about in data base according to url, if the state in data base is " issuing ", then sets to electronics The standby message returning this url well afoot;If in inquiry data base, state corresponding for url is " completing ", then to electronics Equipment returns the message that task corresponding for this url is complete;If system does not inquire corresponding the appointing of url in data base Business state, then return to electronic equipment and do not have the message of task corresponding for url.
The embodiment of the present invention meets pre-conditioned cloud service node by task being handed down to, so that cloud service node will File download corresponding to data address to be migrated is to this locality of cloud service node, and is uploaded in Data Migration destination address, and Can be inquired the task of data to be migrated by the Data Migration status query request that electronic equipment is submitted to according to user State, improves the resource utilization of cloud service node, while reducing Data Migration required time, can inquire about for user and wait to move Move the task status of data, improve Consumer's Experience.
A kind of data mover system structural representation that Fig. 5 provides for one embodiment of the invention, as it is shown in figure 5, described system System, including request receiver module 501, resource management module 502 and data transferring module 503, wherein:
Request receiver module 501 is for receiving the request message of data migration task, the request of described data migration task Message includes data address to be migrated and data move target address;Resource management module 502 is used for determining described Data Migration The resource of required by task, and inquire about the available resources of each cloud service node;Data Migration module 503 is for moving according to described data Move resource and the available resources of described each cloud service node of required by task, be sent to described data migration task meet and preset The described cloud service node of condition so that described cloud service node by file download corresponding for described data address to be migrated to institute State this locality of cloud service node, and be uploaded in described Data Migration destination address.
Specifically, request receiver module 501 receives the request of the data migration task that user is submitted to by electronic equipment Information, wherein, the solicited message of data migration task includes data address to be migrated and data move target address.Should Illustrating, data address to be migrated is a public network address, and data to be migrated are stored in this public network address;Data are moved Moving destination address is that user intends the address in Data Migration to cloud service manufacturer;Resource management module 502 determines this request Resource needed for message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, request message Required resource includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user is carried by electronic equipment Resource needed for the request message of the data migration task handed over is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud service saves Point belongs to data cloud service manufacturer to be moved to and for performing data migration task.It is understood that can also is that it His resource, this is not especially limited by the present embodiment.Data Migration module 503 according to the resource needed for data migration task and from The available resources of each cloud service node inquired in data base, are sent to data migration task meet pre-conditioned cloud clothes Business node, after cloud service node receives the request message of data migration task, according in the request message of data migration task Data in data address to be migrated are downloaded to this locality of cloud service node, then will treat by the data address to be migrated carried The data migrated are uploaded to Data Migration destination address from this locality.It should be noted that, this locality of cloud service node is cloud clothes In business node for the physical disk depositing in the cloud hard disk of data rather than user uses.This locality from cloud service node Uploading data to Data Migration destination is to carry out in Intranet, now need not pass through outer net.
The embodiment of the present invention meets pre-conditioned cloud service node by task being handed down to, so that cloud service node will File download corresponding to data address to be migrated is to this locality of cloud service node, and is uploaded in Data Migration destination address, carries The resource utilization of high cloud service node, reduces Data Migration required time.
On the basis of above-described embodiment, further, described pre-conditioned, including:
The available resources of described cloud service node are more than the resource needed for described data migration task.
Specifically, data base stores the information of the available resources of all cloud service nodes, Data Migration module 503 basis Resource needed for the data migration task that user is submitted to by electronic equipment, is sent to cloud service node by data migration task Available resources are more than data migration task resource requirement.Optionally, data migration task can be sent to cloud service node That cloud service node that available resources are maximum more than available resources in data migration task resource requirement.
Data migration task is sent to the available resources of cloud service node by the embodiment of the present invention by Data Migration module More than the cloud service node of data migration task resource requirement, reasonable distribution task, it is to avoid task waiting time, thus reduce Required by task completes duration.
A kind of data mover system structural representation that Fig. 6 provides for another embodiment of the present invention, as shown in Figure 6, described System, including request receiver module 601, resource management module 602, Data Migration module 603 and resource collection module 604, its In:
Request receiver module 601 is for receiving the request message of data migration task, the request of described data migration task Message includes data address to be migrated and data move target address;Resource management module 602 is used for determining described Data Migration The resource of required by task, and inquire about the available resources of each cloud service node;Data Migration module 603 is for moving according to described data Move resource and the available resources of described each cloud service node of required by task, be sent to described data migration task meet and preset The described cloud service node of condition so that described cloud service node by file download corresponding for described data address to be migrated to institute Stating this locality of cloud service node, and be uploaded in described Data Migration destination address, resource collection module 604 is used for receiving described The available resources that each cloud service node reports, and store in data base.
Specifically, request receiver module 601 receives the request of the data migration task that user is submitted to by electronic equipment Information, wherein, the solicited message of data migration task includes data address to be migrated and data move target address.Should Illustrating, data address to be migrated is a public network address, and data to be migrated are stored in this public network address;Data are moved Moving destination address is that user intends the address in Data Migration to cloud service manufacturer;Resource management module 602 determines this request Resource needed for message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, request message Required resource includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user is carried by electronic equipment Resource needed for the request message of the data migration task handed over is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud service saves Point belongs to data cloud service manufacturer to be moved to and for performing data migration task.It is understood that can also is that it His resource, this is not especially limited by the present embodiment.Data Migration module 603 according to the resource needed for data migration task and from The available resources of each cloud service node inquired in data base, are sent to data migration task meet pre-conditioned cloud clothes Business node, after cloud service node receives the request message of data migration task, according in the request message of data migration task Data in data address to be migrated are downloaded to this locality of cloud service node, then will treat by the data address to be migrated carried The data migrated are uploaded to Data Migration destination address from this locality.It should be noted that, this locality of cloud service node is cloud clothes In business node for the physical disk depositing in the cloud hard disk of data rather than user uses.This locality from cloud service node Uploading data to Data Migration destination is to carry out in Intranet, now need not pass through outer net.Each cloud service node can timing Reporting available resources, resource collection module 604 stores receiving the available resources that each cloud service node reports in data base.
The embodiment of the present invention receives, by resource collection module, the available resources that each cloud service node reports, and stores number According in storehouse, the system of improve judges whether each cloud service node meets pre-conditioned accuracy.
A kind of data mover system structural representation that Fig. 7 provides for further embodiment of this invention, as it is shown in fig. 7, described System, including: request receiver module 701, resource management module 702, Data Migration module 703, resource collection module 704 and shape State arranges module 705, wherein:
Request receiver module 701 is for receiving the request message of data migration task, the request of described data migration task Message includes data address to be migrated and data move target address;Resource management module 702 is used for determining described Data Migration The resource of required by task, and inquire about the available resources of each cloud service node;Data Migration module 703 is for moving according to described data Move resource and the available resources of described each cloud service node of required by task, be sent to described data migration task meet and preset The described cloud service node of condition so that described cloud service node by file download corresponding for described data address to be migrated to institute Stating this locality of cloud service node, and be uploaded in described Data Migration destination address, resource collection module 704 is used for receiving described The available resources that each cloud service node reports, and store in data base;State setting module 705 is for described by described data After migration task is sent to meet pre-conditioned cloud service node, task status corresponding for described data address to be migrated is set It is set to first task state;Described cloud service node is by file download corresponding for described migration data address and is uploaded to described number After in move target address, first task state corresponding for described data address to be migrated is set to the second task status.
Specifically, request receiver module 701 receives the request of the data migration task that user is submitted to by electronic equipment Information, wherein, the solicited message of data migration task includes data address to be migrated and data move target address.Should Illustrating, data address to be migrated is a public network address, and data to be migrated are stored in this public network address;Data are moved Moving destination address is that user intends the address in Data Migration to cloud service manufacturer;Resource management module 702 determines this request Resource needed for message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, request message Required resource includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user is carried by electronic equipment Resource needed for the request message of the data migration task handed over is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud service saves Point belongs to data cloud service manufacturer to be moved to and for performing data migration task.It is understood that can also is that it His resource, this is not especially limited by the present embodiment.Data Migration module 703 according to the resource needed for data migration task and from The available resources of each cloud service node inquired in data base, are sent to data migration task meet pre-conditioned cloud clothes Business node, after cloud service node receives the request message of data migration task, according in the request message of data migration task Data in data address to be migrated are downloaded to this locality of cloud service node, then will treat by the data address to be migrated carried The data migrated are uploaded to Data Migration destination address from this locality.It should be noted that, this locality of cloud service node is cloud clothes In business node for the physical disk depositing in the cloud hard disk of data rather than user uses.This locality from cloud service node Uploading data to Data Migration destination is to carry out in Intranet, now need not pass through outer net.Each cloud service node can timing Reporting available resources, resource collection module 704 stores receiving the available resources that each cloud service node reports in data base. After being sent to data migration task meet pre-conditioned cloud service node, state setting module 705 is arranged in data base Task status corresponding to data address to be migrated is first task state, and optionally, first task state can be " issuing ". Cloud service node starts after receiving task to perform, and after task completes, state setting module 705 arranges to be migrated in data base Task status corresponding to data address is the second task status, and optionally, the second task status can be " completing ".Can manage Solving, first task state and the second task status are not limited to " issuing " cited by the present embodiment and " the completeest Become ".
The embodiment of the present invention meets pre-conditioned cloud service node by task being handed down to, so that cloud service node will File download corresponding to data address to be migrated is to this locality of cloud service node, and is uploaded in Data Migration destination address, carries The resource utilization of high cloud service node, reduces Data Migration required time.
A kind of data mover system structural representation that Fig. 8 provides for further embodiment of this invention, as shown in Figure 8, described System, including request receiver module 801, resource management module 802, Data Migration module 803, resource collection module 804, state Module 805 and state-query module 806 are set, wherein:
Request receiver module 801 is for receiving the request message of data migration task, the request of described data migration task Message includes data address to be migrated and data move target address;Resource management module 802 is used for determining described Data Migration The resource of required by task, and inquire about the available resources of each cloud service node;Data Migration module 803 is for moving according to described data Move resource and the available resources of described each cloud service node of required by task, be sent to described data migration task meet and preset The described cloud service node of condition so that described cloud service node by file download corresponding for described data address to be migrated to institute Stating this locality of cloud service node, and be uploaded in described Data Migration destination address, resource collection module 804 is used for receiving described The available resources that each cloud service node reports, and store in data base;State setting module 805 is for described by described data After migration task is sent to meet pre-conditioned cloud service node, task status corresponding for described data address to be migrated is set It is set to first task state;Described cloud service node is by file download corresponding for described migration data address and is uploaded to described number After in move target address, first task state corresponding for described data address to be migrated is set to the second task status. State-query module 806 is used for receiving Data Migration status query request, and described Data Migration status query request includes described Data address to be migrated, inquires about, according to described data address to be migrated, the task status that described data address to be migrated is corresponding, and Return the described task status inquired.
Specifically, request receiver module 801 receives the request of the data migration task that user is submitted to by electronic equipment Information, wherein, the solicited message of data migration task includes data address to be migrated and data move target address.Should Illustrating, data address to be migrated is a public network address, and data to be migrated are stored in this public network address;Data are moved Moving destination address is that user intends the address in Data Migration to cloud service manufacturer;Resource management module 802 determines this request Resource needed for message, and in data base, inquire about the available resources of each cloud service node.It should be noted that, request message Required resource includes the CPU required for this data migration task, internal memory and bandwidth.Such as, user is carried by electronic equipment Resource needed for the request message of the data migration task handed over is 0.1 CPU, 100M internal memory and 10M bandwidth.Described cloud service saves Point belongs to data cloud service manufacturer to be moved to and for performing data migration task.It is understood that can also is that it His resource, this is not especially limited by the present embodiment.Data Migration module 803 according to the resource needed for data migration task and from The available resources of each cloud service node inquired in data base, are sent to data migration task meet pre-conditioned cloud clothes Business node, after cloud service node receives the request message of data migration task, according in the request message of data migration task Data in data address to be migrated are downloaded to this locality of cloud service node, then will treat by the data address to be migrated carried The data migrated are uploaded to Data Migration destination address from this locality.It should be noted that, this locality of cloud service node is cloud clothes In business node for the physical disk depositing in the cloud hard disk of data rather than user uses.This locality from cloud service node Uploading data to Data Migration destination is to carry out in Intranet, now need not pass through outer net.Each cloud service node can timing Reporting available resources, resource collection module 804 stores receiving the available resources that each cloud service node reports in data base. After being sent to data migration task meet pre-conditioned cloud service node, state setting module 805 is arranged in data base Task status corresponding to data address to be migrated is first task state, and optionally, first task state can be " issuing ". Cloud service node starts after receiving task to perform, and after task completes, state setting module 805 arranges to be migrated in data base Task status corresponding to data address is the second task status, and optionally, the second task status can be " completing ".Can manage Solving, first task state and the second task status are not limited to " issuing " cited by the present embodiment and " the completeest Become ".When for needing the state inquiring about data to be migrated, state-query module 806 is received user and is submitted to by electronic equipment Data Migration status query request, wherein Data Migration status query request includes data address to be migrated, status poll mould It is corresponding that block 806 inquires about data address to be migrated according to user by the data address to be migrated that electronic equipment is submitted in data base Task status, and the task status inquired is returned to electronic equipment.
The embodiment of the present invention meets pre-conditioned cloud service node by task being handed down to, so that cloud service node will File download corresponding to data address to be migrated is to this locality of cloud service node, and is uploaded in Data Migration destination address, and Can be inquired the task of data to be migrated by the Data Migration status query request that electronic equipment is submitted to according to user State, improves the resource utilization of cloud service node, while reducing Data Migration required time, can inquire about for user and wait to move Move the task status of data, improve Consumer's Experience.
A kind of data mover system entity structure schematic diagram that Fig. 9 provides for the embodiment of the present invention, the most described system System may include that processor (processor) 901, communication interface (Communications Interface) 902, memorizer (memory) 903 and communication bus 904, wherein, processor 901, communication interface 902, memorizer 903 is by communication bus 904 Complete mutual communication.Communication interface 902 may be used for the information transmission between system and electronic equipment.Processor 901 can To call the logical order in memorizer 903, to perform following method: receive the request message of data migration task, described number Data address to be migrated and data move target address is included according to the request message of the task of migration;Determine described data migration task Required resource, and inquire about the available resources of each cloud service node;According to the resource needed for described data migration task and described The available resources of each cloud service node, are sent to described data migration task meet pre-conditioned described cloud service node, So that described cloud service node is by this locality of file download corresponding for described data address to be migrated to described cloud service node, and It is uploaded in described Data Migration destination address.
Additionally, the logical order in above-mentioned memorizer 903 can be realized and conduct by the form of SFU software functional unit When independent production marketing or use, can be stored in a computer read/write memory medium.Based on such understanding, this The part that contributes prior art the most in other words of technical scheme of invention or the part of this technical scheme can be with The form of software product embodies, and this computer software product is stored in a storage medium, including some instructions in order to Each is implemented to make a computer equipment (can be personal computer, server, or the network equipment etc.) perform the present invention All or part of step of method described in example.And aforesaid storage medium includes: USB flash disk, portable hard drive, read only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disc or CD etc. various The medium of program code can be stored.
Device embodiment described above is only schematically, and the wherein said unit illustrated as separating component can To be or to may not be physically separate, the parts shown as unit can be or may not be physics list Unit, i.e. may be located at a place, or can also be distributed on multiple NE.Can be selected it according to the actual needs In some or all of module realize the purpose of the present embodiment scheme.Those of ordinary skill in the art are not paying creativeness Work in the case of, be i.e. appreciated that and implement.
Through the above description of the embodiments, those skilled in the art it can be understood that to each embodiment can The mode adding required general hardware platform by software realizes, naturally it is also possible to pass through hardware.Based on such understanding, on State the part that prior art contributes by technical scheme the most in other words to embody with the form of software product, should Computer software product can store in a computer-readable storage medium, such as ROM/RAM, magnetic disc, CD etc., including some fingers Make with so that a computer equipment (can be personal computer, server, or the network equipment etc.) performs each and implements The method described in some part of example or embodiment.
Last it is noted that above example is only in order to illustrate technical scheme, it is not intended to limit;Although With reference to previous embodiment, the present invention is described in detail, it will be understood by those within the art that: it still may be used So that the technical scheme described in foregoing embodiments to be modified, or wherein portion of techniques feature is carried out equivalent; And these amendment or replace, do not make appropriate technical solution essence depart from various embodiments of the present invention technical scheme spirit and Scope.

Claims (10)

1. a data migration method, it is characterised in that including:
Receive data migration task request message, the request message of described data migration task include data address to be migrated and Data Migration destination address;
Determine the resource needed for described data migration task, and inquire about the available resources of each cloud service node;
According to the resource needed for described data migration task and the available resources of described each cloud service node, by described Data Migration Task is sent to meet pre-conditioned described cloud service node, so that described cloud service node is by described data address to be migrated This locality of corresponding file download extremely described cloud service node, and be uploaded in described Data Migration destination address.
Method the most according to claim 1, it is characterised in that described pre-conditioned, including:
The available resources of described cloud service node are more than the resource needed for described data migration task.
Method the most according to claim 1, it is characterised in that described method, also includes:
Receive the available resources that described each cloud service node reports, and store in data base.
4., according to the method described in any one of claim 1-3, it is characterised in that described method, also include:
Described be sent to described data migration task meet pre-conditioned cloud service node after, by described data to be migrated ground Task status corresponding to location is set to first task state;Described cloud service node is by file corresponding for described migration data address After downloading and be uploaded in described Data Migration destination address, first task state corresponding for described data address to be migrated is set It is set to the second task status.
Method the most according to claim 4, it is characterised in that described method, also includes:
Receiving Data Migration status query request, described Data Migration status query request includes described data address to be migrated, Inquire about task status corresponding to described data address to be migrated according to described data address to be migrated, and return inquire described Task status.
6. a data mover system, it is characterised in that including:
Request receiver module, for receiving the request message of data migration task, the request message bag of described data migration task Include data address to be migrated and data move target address;
Resource management module, for determining the resource needed for described data migration task, and inquires about the available of each cloud service node Resource;
Data Migration module, for according to the resource needed for described data migration task and the available money of described each cloud service node Source, is sent to described data migration task meet pre-conditioned described cloud service node, so that described cloud service node will This locality of the file download extremely described cloud service node that described data address to be migrated is corresponding, and it is uploaded to described Data Migration mesh Address in.
System the most according to claim 6, it is characterised in that described pre-conditioned, including:
The available resources of described cloud service node are more than the resource needed for described data migration task.
System the most according to claim 6, it is characterised in that described system, also includes:
Resource collection module, for receiving the available resources that described each cloud service node reports, and stores in data base.
9., according to the system described in any one of claim 6-8, it is characterised in that described system, also include:
State setting module, for described be sent to meet pre-conditioned cloud service node by described data migration task after, Task status corresponding for described data address to be migrated is set to first task state;Described cloud service node is by described migration File download that data address is corresponding after being uploaded in described Data Migration destination address, by described data address pair to be migrated The first task state answered is set to the second task status.
System the most according to claim 9, it is characterised in that described system, also includes:
State-query module, is used for receiving Data Migration status query request, and described Data Migration status query request includes institute State data address to be migrated, inquire about, according to described data address to be migrated, the task status that described data address to be migrated is corresponding, And return the described task status inquired.
CN201610863983.0A 2016-09-28 2016-09-28 Data migration method and system Pending CN106331160A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610863983.0A CN106331160A (en) 2016-09-28 2016-09-28 Data migration method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610863983.0A CN106331160A (en) 2016-09-28 2016-09-28 Data migration method and system

Publications (1)

Publication Number Publication Date
CN106331160A true CN106331160A (en) 2017-01-11

Family

ID=57820628

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610863983.0A Pending CN106331160A (en) 2016-09-28 2016-09-28 Data migration method and system

Country Status (1)

Country Link
CN (1) CN106331160A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109933287A (en) * 2019-03-05 2019-06-25 中国联合网络通信集团有限公司 IoT-based data migration method, device and device
CN110737717A (en) * 2018-07-03 2020-01-31 北京国双科技有限公司 database migration method and device
CN110971647A (en) * 2018-09-30 2020-04-07 南京工程学院 Node migration method of big data system
CN111930715A (en) * 2020-07-16 2020-11-13 北京金山云网络技术有限公司 Data migration method and device, computer equipment and storage medium
CN113468140A (en) * 2021-06-30 2021-10-01 上海掌门科技有限公司 Data migration processing method, electronic device and computer-readable storage medium
CN113660510A (en) * 2021-08-19 2021-11-16 杭州时趣信息技术有限公司 Video processing cloud manufacturer configuration method, device and system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102170440A (en) * 2011-03-24 2011-08-31 北京大学 Method suitable for safely migrating data between storage clouds
CN104426925A (en) * 2013-08-21 2015-03-18 腾讯科技(深圳)有限公司 Webpage resource obtaining method and apparatus
WO2015054832A1 (en) * 2013-10-16 2015-04-23 Empire Technology Development Llc Two-level cloud system migration
CN104731888A (en) * 2015-03-12 2015-06-24 北京奇虎科技有限公司 Data migration method, device and system
US9225791B2 (en) * 2011-02-28 2015-12-29 Red Hat, Inc. Staged data migration between data sources and cloud-based storage network
CN105653630A (en) * 2015-12-25 2016-06-08 北京奇虎科技有限公司 Data migration method and apparatus for distributed database

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9225791B2 (en) * 2011-02-28 2015-12-29 Red Hat, Inc. Staged data migration between data sources and cloud-based storage network
CN102170440A (en) * 2011-03-24 2011-08-31 北京大学 Method suitable for safely migrating data between storage clouds
CN104426925A (en) * 2013-08-21 2015-03-18 腾讯科技(深圳)有限公司 Webpage resource obtaining method and apparatus
WO2015054832A1 (en) * 2013-10-16 2015-04-23 Empire Technology Development Llc Two-level cloud system migration
CN104731888A (en) * 2015-03-12 2015-06-24 北京奇虎科技有限公司 Data migration method, device and system
CN105653630A (en) * 2015-12-25 2016-06-08 北京奇虎科技有限公司 Data migration method and apparatus for distributed database

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110737717A (en) * 2018-07-03 2020-01-31 北京国双科技有限公司 database migration method and device
CN110971647A (en) * 2018-09-30 2020-04-07 南京工程学院 Node migration method of big data system
CN110971647B (en) * 2018-09-30 2023-12-05 南京工程学院 A node migration method for big data systems
CN109933287A (en) * 2019-03-05 2019-06-25 中国联合网络通信集团有限公司 IoT-based data migration method, device and device
CN111930715A (en) * 2020-07-16 2020-11-13 北京金山云网络技术有限公司 Data migration method and device, computer equipment and storage medium
CN113468140A (en) * 2021-06-30 2021-10-01 上海掌门科技有限公司 Data migration processing method, electronic device and computer-readable storage medium
CN113468140B (en) * 2021-06-30 2023-04-07 上海掌门科技有限公司 Data migration processing method, electronic device and computer-readable storage medium
CN113660510A (en) * 2021-08-19 2021-11-16 杭州时趣信息技术有限公司 Video processing cloud manufacturer configuration method, device and system

Similar Documents

Publication Publication Date Title
CN106331160A (en) Data migration method and system
US9491313B2 (en) Optimizing storage between mobile devices and cloud storage providers
US9888065B2 (en) Defragmenting clusters with reserved resources
CN105025053A (en) Distributed file upload method based on cloud storage technology and system
CN107291536B (en) Application task flow scheduling method in cloud computing environment
US9774676B2 (en) Storing and moving data in a distributed storage system
CN112520284B (en) Goods shelf storage position adjusting method and system
CN104731516A (en) Method and device for accessing files and distributed storage system
CN102982116A (en) Multi-media transfer method and system based on cloud
CN110727738B (en) Global routing system based on data fragmentation, electronic equipment and storage medium
CN106302640A (en) Data request processing method and device
CN109241099A (en) A kind of data query method and terminal device
CN106960011A (en) Metadata of distributed type file system management system and method
CN110309229A (en) The data processing method and distributed system of distributed system
CN111597259B (en) Data storage system, method, device, electronic equipment and storage medium
CN102480502A (en) I/O load balancing method and I/O server
CN101483668A (en) Network storage and access method, device and system for hot spot data
CN106293509A (en) Date storage method and system
CN108667920A (en) A fog computing environment business traffic acceleration system and business traffic acceleration method
CN104092735A (en) Cloud computing data access method and system based on binary tree
CN104683480A (en) Distribution type calculation method based on applications
US9626377B1 (en) Cluster file system with metadata server for controlling movement of data between storage tiers
US12014061B2 (en) Capacity and performance optimization in non-homogeneous storage
CN112887349B (en) Method and device for distributing files
CN104219096B (en) A kind of resource reconfiguration method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20170111

WD01 Invention patent application deemed withdrawn after publication