CN111835812A - Global data acquisition method and device, electronic equipment and computer storage medium - Google Patents
Global data acquisition method and device, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN111835812A CN111835812A CN201911108099.6A CN201911108099A CN111835812A CN 111835812 A CN111835812 A CN 111835812A CN 201911108099 A CN201911108099 A CN 201911108099A CN 111835812 A CN111835812 A CN 111835812A
- Authority
- CN
- China
- Prior art keywords
- data
- overseas
- node
- domestic
- update
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a global data acquisition method and device, electronic equipment and a computer readable storage medium, and relates to the field of data processing. The method comprises the following steps: when a preset condition for acquiring data is met, a preset overseas data node acquires overseas update data; sending the overseas updating data to a preset domestic data node so that the domestic data node integrates the overseas updating data and domestic updating data to obtain target updating data; and the domestic updating data is obtained when the domestic data nodes meet the preset conditions. The application realizes the global data acquisition.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a global data acquisition method, apparatus, electronic device, and computer-readable storage medium.
Background
In the current global data collection scheme, global data is usually collected through one node, but any single node cannot realize the function of collecting global data, the network speed is slow, the packet loss number in data transmission is high, and even data intercommunication cannot be realized between part of countries.
Moreover, while data intercommunication is performed, FTP (File Transfer Protocol), BT (Bit stream), blockchain, rsync and other modes are usually adopted for deployment, but the deployment cost of FTP, BT and regional chain is too high, the deployment is too cumbersome, and meanwhile, FTP cannot ensure data security; and the global data collection effect is poor due to the fact that the rsync mode is high in delay, high in packet loss rate and poor in stability.
Disclosure of Invention
The application provides a global data acquisition method, a global data acquisition device, electronic equipment and a computer readable storage medium, which can solve the problem of a global data acquisition deployment scheme. The technical scheme is as follows:
in a first aspect, a global data acquisition method is provided, the method comprising:
when a preset condition for acquiring data is met, a preset overseas data node acquires overseas update data;
sending the overseas updating data to a preset domestic data node so that the domestic data node integrates the overseas updating data and domestic updating data to obtain target updating data; and the domestic updating data is obtained when the domestic data nodes meet the preset conditions.
Preferably, the overseas data nodes comprise a first overseas data node and a plurality of second overseas data nodes, and the first overseas data node and the plurality of second overseas data nodes are respectively provided with log information;
the step of acquiring the overseas update data by the preset overseas data node when the preset condition for acquiring the data is satisfied includes:
when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, acquiring corresponding overseas update data from the at least one data node based on the updated log information.
Preferably, the log information is provided with time information;
the step of acquiring the overseas update data by the preset overseas data node when the preset condition for acquiring the data is satisfied includes:
when detecting that the current time information exceeds a preset time information threshold value, the first oversea data node acquires each piece of updated log information in the first oversea data node and each second oversea data node; the time information of the updated log information does not exceed the preset time information threshold;
and respectively acquiring corresponding update data from the first overseas data node and each second overseas data node based on each update log information.
Preferably, the log information is generated by:
when any data node receives a data updating instruction, data updating is carried out on the data in any data node based on the data updating instruction to obtain updated data, and corresponding log information is generated; the data update includes at least one of adding data, deleting data, or modifying data.
Preferably, before the step of sending the overseas update data to the preset domestic data nodes, the method further comprises:
verifying the overseas update data;
sending the overseas update data to a preset domestic data node, comprising:
if the verification is passed, the overseas updating data is sent to the domestic data node; if the verification fails, the step that the preset overseas data nodes acquire the overseas updating data when the preset condition of acquiring the data is met is continuously executed until the verification is passed.
In a second aspect, there is provided an apparatus for global data acquisition, the apparatus comprising:
the acquisition module is used for acquiring the overseas updating data by a preset overseas data node when a preset condition for acquiring the data is met;
the sending module is used for sending the overseas updating data to a preset domestic data node so that the domestic data node integrates the overseas updating data and the domestic updating data to obtain target updating data; and the domestic updating data is obtained when the domestic data nodes meet the preset conditions.
Preferably, the overseas data nodes comprise a first overseas data node and a plurality of second overseas data nodes, and the first overseas data node and the plurality of second overseas data nodes are respectively provided with log information;
the acquisition module is specifically configured to:
when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, acquiring corresponding overseas update data from the at least one data node based on the updated log information.
Preferably, the log information is provided with time information;
the acquisition module includes:
the first oversea data node is used for acquiring the update log information of each first oversea data node and each second oversea data node; the time information of the updated log information does not exceed the preset time information threshold;
and the update data acquisition sub-module is used for respectively acquiring corresponding update data from the first oversea data node and each second oversea data node based on each update log information.
Preferably, the log information is generated by:
when any data node receives a data updating instruction, data updating is carried out on the data in any data node based on the data updating instruction to obtain updated data, and corresponding log information is generated; the data update includes at least one of adding data, deleting data, or modifying data.
Preferably, the method further comprises the following steps:
the verification module is used for verifying the overseas updating data before the step of sending the overseas updating data to a preset domestic data node;
the sending module is specifically configured to:
if the verification is passed, the overseas updating data is sent to the domestic data node; and if the verification fails, continuing to call the acquisition module and the verification module until the verification is passed.
In a third aspect, an electronic device is provided, which includes:
a processor, a memory, and a bus;
the bus is used for connecting the processor and the memory;
the memory is used for storing operation instructions;
the processor is configured to invoke the operation instruction, and the executable instruction enables the processor to perform an operation corresponding to the global data acquisition method as shown in the first aspect of the present application.
In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, implements the global data acquisition method of the first aspect of the present application
The beneficial effect that technical scheme that this application provided brought is:
in the embodiment of the invention, when the preset condition of acquiring data is met, the preset overseas data node acquires the overseas update data and then sends the overseas update data to the preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions. Thus, the overseas data node acquires all the overseas update data, the domestic data node acquires all the domestic update data, and then all the overseas update data and all the domestic update data are integrated in the domestic data node, so that global update data is obtained, and global data acquisition is realized.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments of the present application will be briefly described below.
Fig. 1 is a schematic flowchart of a global data acquisition method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a global data acquisition method according to another embodiment of the present application;
fig. 3 is a schematic structural diagram of a global data acquisition device according to another embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device for global data acquisition according to yet another embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
The present application provides a global data acquisition method, apparatus, electronic device and computer-readable storage medium, which are intended to solve the above technical problems in the prior art.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
In one embodiment, a global data acquisition method is provided, as shown in fig. 1, the method comprising:
step S101, when the preset condition of acquiring data is satisfied, a preset overseas data node acquires the overseas update data;
the application can be applied to acquiring any data around the world, such as file data, data generated by a user using an application program, financial data and the like.
Specifically, the data is acquired by data nodes, and the data nodes comprise overseas data nodes and domestic data nodes. Wherein overseas refers to all countries and regions outside the border of china (national boundary) and regions where the government in china has not been subject to administrative administration, including hong Kong and Macau Taiwan and other foreign countries and regions; domestic refers to the territory of the people's republic of china except the special administrative areas of hong Kong, Macau and Taiwan, where the people's republic of china have the master rights.
In practical application, a manager may configure preset conditions for acquiring data in the overseas data node and the domestic data node, respectively, and when the preset conditions are met, the overseas data node and the domestic data node may start to acquire updated data.
Each data node can be used for storing data, and when the data in the data node is updated, the updated data is the updated data. For example, if the user uploads data to the data node in the environment, the data information is the updated data.
Step S102, the overseas updating data is sent to a preset domestic data node, so that the domestic data node integrates the overseas updating data and the domestic updating data to obtain target updating data; the domestic updating data is obtained when domestic data nodes meet preset conditions.
Specifically, after acquiring all the overseas update data, the overseas data node transmits all the overseas update data to the domestic data node, and the domestic data node integrates the acquired domestic update data and the overseas update data, thereby acquiring target update data, that is, update data in all the data nodes.
In the embodiment of the invention, when the preset condition of acquiring data is met, the preset overseas data node acquires the overseas update data and then sends the overseas update data to the preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions. Thus, the overseas data node acquires all the overseas update data, the domestic data node acquires all the domestic update data, and then all the overseas update data and all the domestic update data are integrated in the domestic data node, so that global update data is obtained, and global data acquisition is realized.
In another embodiment, a method for global data acquisition is provided, as shown in fig. 2, the method comprising:
step S201, when the preset condition of acquiring data is satisfied, a preset overseas data node acquires the overseas update data;
the application can be applied to acquiring any data around the world, such as file data, data generated by a user using an application program, financial data and the like.
Specifically, the data is acquired by data nodes, and the data nodes comprise overseas data nodes and domestic data nodes. Wherein overseas refers to all countries and regions outside the border of china (national boundary) and regions where the government in china has not been subject to administrative administration, including hong Kong and Macau Taiwan and other foreign countries and regions; domestic refers to the territory of the people's republic of china except the special administrative areas of hong Kong, Macau and Taiwan, where the people's republic of china have the master rights.
In practical application, a manager may configure preset conditions for acquiring data in the overseas data node and the domestic data node, respectively, and when the preset conditions are met, the overseas data node and the domestic data node may start to acquire updated data.
Each data node can be used for storing data, and when the data in the data node is updated, the updated data is the updated data. For example, if a user uploads a file to a data node in the home, the file is the updated data.
Furthermore, in order to obtain global data through each data node, an internal data node and a plurality of external data nodes may be provided in the present application. Preferably, the domestic data nodes may be arranged in beijing, and the plurality of foreign data nodes may be arranged in hong kong, finland, singapore, and the united states, respectively, where the beijing data node is configured to obtain data in all the countries, the hong kong data node is configured to obtain data in the hong kong and macadamia regions, the finland data node is configured to obtain data in the european and russian regions, the singapore data node is configured to obtain data in the asia-pacific region, and the united states data node is configured to obtain data in the american, australian, and african regions, so that the global data can be obtained through each data node.
It should be noted that the above-mentioned deployment scheme of the data nodes is only an optimal scheme, and in practical application, a certain number of inbound data nodes and outbound data nodes may be set according to actual requirements, and the addresses of the data nodes may also be set according to actual requirements, which is not limited in this application.
In the present application, each data node is further provided with a rule for acquiring data, such as a proximity rule. In practical applications, each data node is configured with an IP (internet Protocol) address and a DNS (Domain Name System), so that each data node can obtain data based on the IP address and the DNS of the data. For example, data with an IP address of beijing can be obtained through domestic data nodes, and data with a DNS of hong kong can be obtained through data nodes of hong kong.
In a preferred embodiment of the present invention, the overseas data nodes include a first overseas data node and a plurality of second overseas data nodes, the first overseas data node and the plurality of second overseas data nodes are respectively provided with log information;
when a preset condition for acquiring data is met, acquiring the overseas updating data by a preset overseas data node, wherein the step comprises the following steps:
when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, corresponding overseas update data is acquired from the at least one data node based on the updated log information.
In practical application, because the locations of the respective overseas data nodes are different, if the data acquired by each overseas data node is directly transmitted to the domestic data node, situations of network information obstruction, high delay, high packet loss rate and the like may occur, so to avoid these situations, in the present application, the overseas data node having the shortest distance from the domestic data node among the plurality of overseas data nodes is set as the first overseas data node, and other overseas data nodes are set as the second overseas data nodes, and when acquiring data, the first overseas node is used to acquire update data in the first overseas node and each second overseas node.
Specifically, each data node is provided with log information, and when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, corresponding overseas update data is acquired from at least one data node based on the updated log information. For example, when detecting that the local log information is updated and the log information in the singapore data node is updated, the hong kong data node acquires the corresponding hong kong update data based on the local log information, and meanwhile acquires the corresponding singapore update data based on the log information updated in the singapore data node, and then the hong kong data node transmits the hong kong update data and the singapore update data to the beijing data node. That is, in the present application, when the log information in any data node is updated, the corresponding update data can be acquired.
In a preferred embodiment of the present invention, the log information is provided with time information;
when a preset condition for acquiring data is met, acquiring the overseas updating data by a preset overseas data node, wherein the step comprises the following steps:
when detecting that the current time information exceeds a preset time information threshold value, the first oversea data node acquires each piece of update log information in the first oversea data node and each second oversea data node; the time information of the updated log information does not exceed a preset time information threshold;
and respectively acquiring corresponding update data from the first overseas data node and each second overseas data node based on each update log information.
Specifically, the log information may be provided with time information, such as year, month, day, time, minute, second, and the like, and at the same time, a time information threshold may be further set in each data node, and when the first overseas data node detects that the current time information exceeds the preset time information threshold, each piece of update log information in the first overseas data node and each second overseas data node may be acquired, and then corresponding update data is acquired from the first overseas data node and each second overseas data node, respectively, based on each piece of update log information; and the time information of the updated log information does not exceed a preset time information threshold value.
For example, the set time information threshold is zero point of each day, and when the first overseas data node detects that the current time information exceeds zero point, the updated log information in the first overseas data node and each second overseas data node can be acquired, wherein the time information of the updated log information is that the time information threshold of this time is not exceeded after the updated data is acquired last time, for example, the current time is 31/17: 59 in 2019, and the set time information threshold is 00:00:00 in 11/1/2019, so that when the current time exceeds the time information threshold, for example, when the current time is 00:00:01 in 11/1/2019, the first overseas data node acquires each updated log information in the first overseas data node and each second overseas data node, wherein the time information of each updated log information is 31/00: 00 in 10/2019 (data acquired last time is 31/00: 00: 2019) in 2019: and 59, the first oversea data node then acquires corresponding update data from the first oversea data node and each second oversea data node respectively according to each update log information.
The log information is generated in the following way:
when any data node receives a data updating instruction, data updating is carried out on the data in any data node based on the data updating instruction to obtain updated data, and corresponding log information is generated; the data update includes at least one of adding data, deleting data, or modifying data.
Specifically, when a user updates data stored in any data node, the data node generates corresponding log information for the update, for example, if the user uploads a certain file to a certain data node, the corresponding log information may be generated based on information about the file uploaded by the user, upload time (time information of the log information), a storage address of the file, and the like.
Wherein the update to the data may include, but is not limited to, at least one of adding data, deleting data, or modifying data.
In addition, each data node is provided with a nginx and lua framework for collecting log information, storing the log information and the like. Of course, other manners for collecting log information and saving log information are also applicable to the present application, and the present application does not limit this.
Moreover, data encryption and decryption technologies are adopted for data collection and data transmission, and data are guaranteed not to be intercepted and cracked.
It should be noted that, when the preset condition for acquiring data is satisfied, besides the overseas data node acquiring the overseas update data, the domestic data node may also acquire domestic update data, and the specific acquisition method may refer to the aforementioned acquisition method, which is not described herein again.
Further, in the present application, each data node may further include an HDFS (Hadoop distributed file System), where the HDFS is used to store data, but the HDFS cannot be directly accessed, so that in order to access the data stored in the HDFS, a UNIX server needs to be mounted, and the data node on which the UNIX server is mounted is equivalent to a storage container, so that updated data can be obtained from the data node, and data verification are performed.
Furthermore, each data node is also provided with a flash + kafka for monitoring the log data of each node in real time and transmitting the updated data.
The system comprises a data acquisition system, a data transmission system, a data aggregation system and a data transmission system, wherein the data acquisition system is used for acquiring data and transmitting the data to the data acquisition system; at the same time, flash has the ability to simply process data and write to various data recipients (customizable).
Kafka is an open source streaming platform, written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site. These data are typically addressed by handling logs and log aggregations due to throughput requirements. The purpose of Kafka is to unify online and offline message processing through the parallel loading mechanism of Hadoop, and also to provide real-time messages through clustering.
Step S202, verifying the overseas updating data;
specifically, the data uploading tool is further provided, and the data uploading tool has functions of file signing, failed retransmission and the like and is used for guaranteeing accuracy of files, so that the first overseas data node can verify the overseas update data through the data uploading tool after acquiring all the overseas update data.
Step S203, the overseas updating data is sent to a preset domestic data node, so that the domestic data node integrates the overseas updating data and the domestic updating data to obtain target updating data; the domestic updating data is obtained when domestic data nodes meet preset conditions.
Specifically, after the first overseas data node acquires local update data and update data of each second overseas data node in real time and quasi-real time, all the overseas update data are sent to the domestic data node, and the domestic data node integrates the acquired domestic update data and the overseas update data, so that target update data, namely, update data in all the data nodes, is acquired.
In a preferred embodiment of the present invention, the sending the overseas update data to the preset domestic data nodes comprises:
if the verification is passed, the overseas updating data is sent to the domestic data nodes; if the verification fails, the step of acquiring the overseas updating data by the preset overseas data nodes when the preset condition of acquiring the data is met is continuously executed until the verification is passed, so that the data quality can be improved, the input of dirty data is reduced, and the later storage cost and the later calculation cost are reduced.
Because the data uploading tool has the function of failed retransmission, in practical application, when the first overseas data node acquires the updated data of each second overseas data node, if some data are failed to be verified, retransmission can be performed until the data are successfully verified. Similarly, the above method may be adopted when the overseas data node acquires all the overseas update data from the first overseas data node, which is not described herein.
In addition, during real-time data capture, the flash captures changed update data, uploads the data to the kafka and stores the data in a kafka message queue, and the message queue has a certain storage period, so that when data acquisition fails, the next acquired data can be continuously acquired from the offset of the last failure. Ensuring that data is not lost.
In the embodiment of the invention, when the preset condition of acquiring data is met, the preset overseas data node acquires the overseas update data and then sends the overseas update data to the preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions. Thus, the overseas data node acquires all the overseas update data, the domestic data node acquires all the domestic update data, and then all the overseas update data and all the domestic update data are integrated in the domestic data node, so that global update data is obtained, and global data acquisition is realized.
Furthermore, the specified foreign data nodes acquire the update data in all other foreign data nodes and the local update data, so that the situation of poor stability caused by too long transmission distance when domestic data nodes acquire the update data from each foreign data node respectively is avoided, and the stability of data transmission is improved. Meanwhile, data transmission is carried out in a hadoop-HDFS, Unix server and flash + kafka mode, and compared with the existing FTP, BT, block chain, rsync and other modes, the deployment is simpler, the cost is lower, and the labor cost and the financial cost are greatly reduced.
Fig. 3 is a schematic structural diagram of a global data acquisition device according to another embodiment of the present application, and as shown in fig. 3, the global data acquisition device of the present embodiment may include:
an obtaining module 301, configured to, when a preset condition for obtaining data is met, obtain the overseas update data by a preset overseas data node;
a sending module 302, configured to send the overseas update data to a preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions.
In a preferred embodiment of the present invention, the overseas data nodes include a first overseas data node and a plurality of second overseas data nodes, the first overseas data node and the plurality of second overseas data nodes are respectively provided with log information;
the acquisition module is specifically configured to:
when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, corresponding overseas update data is acquired from the at least one data node based on the updated log information.
In a preferred embodiment of the present invention, the log information is provided with time information;
the acquisition module comprises:
the first oversea data node is used for acquiring the update log information of each first oversea data node and each second oversea data node; the time information of the updated log information does not exceed a preset time information threshold;
and the update data acquisition sub-module is used for respectively acquiring corresponding update data from the first oversea data node and each second oversea data node based on each update log information.
In a preferred embodiment of the present invention, the log information is generated by:
when any data node receives a data updating instruction, data updating is carried out on the data in any data node based on the data updating instruction to obtain updated data, and corresponding log information is generated; the data update includes at least one of adding data, deleting data, or modifying data.
In a preferred embodiment of the present invention, the method further comprises:
the verification module is used for verifying the overseas updating data before the step of sending the overseas updating data to the preset domestic data nodes;
the sending module is specifically configured to:
if the verification is passed, the overseas updating data is sent to the domestic data nodes; and if the verification fails, continuing to call the acquisition module and the verification module until the verification is passed.
The global data acquisition device of this embodiment can execute the global data acquisition method shown in the first embodiment of this application, and the implementation principles thereof are similar, and are not described herein again.
In the embodiment of the invention, when the preset condition of acquiring data is met, the preset overseas data node acquires the overseas update data and then sends the overseas update data to the preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions. Thus, the overseas data node acquires all the overseas update data, the domestic data node acquires all the domestic update data, and then all the overseas update data and all the domestic update data are integrated in the domestic data node, so that global update data is obtained, and global data acquisition is realized.
Furthermore, the specified foreign data nodes acquire the update data in all other foreign data nodes and the local update data, so that the situation of poor stability caused by too long transmission distance when domestic data nodes acquire the update data from each foreign data node respectively is avoided, and the stability of data transmission is improved. Meanwhile, data transmission is carried out in a hadoop-HDFS, Unix server and flash + kafka mode, and compared with the existing FTP, BT, block chain, rsync and other modes, the deployment is simpler, the cost is lower, and the labor cost and the financial cost are greatly reduced.
In another embodiment of the present application, there is provided an electronic device including: a memory and a processor; at least one program stored in the memory for execution by the processor, which when executed by the processor, implements: when the preset conditions for acquiring the data are met, the preset overseas data node acquires the overseas update data and then sends the overseas update data to the preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions. Thus, the overseas data node acquires all the overseas update data, the domestic data node acquires all the domestic update data, and then all the overseas update data and all the domestic update data are integrated in the domestic data node, so that global update data is obtained, and global data acquisition is realized.
In an alternative embodiment, an electronic device is provided, as shown in fig. 4, the electronic device 4000 shown in fig. 4 comprising: a processor 4001 and a memory 4003. Processor 4001 is coupled to memory 4003, such as via bus 4002. Optionally, the electronic device 4000 may further comprise a transceiver 4004. In addition, the transceiver 4004 is not limited to one in practical applications, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.
The memory 4003 is used for storing application codes for executing the scheme of the present application, and the execution is controlled by the processor 4001. Processor 4001 is configured to execute application code stored in memory 4003 to implement what is shown in any of the foregoing method embodiments.
Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like.
Yet another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, which, when run on a computer, enables the computer to perform the corresponding content in the aforementioned method embodiments. Compared with the prior art, when the preset condition for acquiring the data is met, the preset overseas data node acquires the overseas update data and then sends the overseas update data to the preset domestic data node, so that the domestic data node integrates the overseas update data and the domestic update data to obtain target update data; the domestic updating data is obtained when domestic data nodes meet preset conditions. Thus, the overseas data node acquires all the overseas update data, the domestic data node acquires all the domestic update data, and then all the overseas update data and all the domestic update data are integrated in the domestic data node, so that global update data is obtained, and global data acquisition is realized.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A global data acquisition method, comprising:
when a preset condition for acquiring data is met, a preset overseas data node acquires overseas update data;
sending the overseas updating data to a preset domestic data node so that the domestic data node integrates the overseas updating data and domestic updating data to obtain target updating data; and the domestic updating data is obtained when the domestic data nodes meet the preset conditions.
2. The global data acquisition method according to claim 1, wherein the foreign data nodes include a first foreign data node and a plurality of second foreign data nodes, the first foreign data node and the plurality of second foreign data nodes being provided with log information, respectively;
the step of acquiring the overseas update data by the preset overseas data node when the preset condition for acquiring the data is satisfied includes:
when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, acquiring corresponding overseas update data from the at least one data node based on the updated log information.
3. The global data acquisition method according to claim 2, wherein the log information is provided with time information;
the step of acquiring the overseas update data by the preset overseas data node when the preset condition for acquiring the data is satisfied includes:
when detecting that the current time information exceeds a preset time information threshold value, the first oversea data node acquires each piece of updated log information in the first oversea data node and each second oversea data node; the time information of the updated log information does not exceed the preset time information threshold;
and respectively acquiring corresponding update data from the first overseas data node and each second overseas data node based on each update log information.
4. The global data acquisition method according to any one of claims 1 to 3, wherein the log information is generated by:
when any data node receives a data updating instruction, data updating is carried out on the data in any data node based on the data updating instruction to obtain updated data, and corresponding log information is generated; the data update includes at least one of adding data, deleting data, or modifying data.
5. The global data acquisition method according to any one of claims 1 to 3, wherein before the step of sending the overseas update data to a preset domestic data node, the global data acquisition method further comprises:
verifying the overseas update data;
sending the overseas update data to a preset domestic data node, comprising:
if the verification is passed, the overseas updating data is sent to the domestic data node; if the verification fails, the step that the preset overseas data nodes acquire the overseas updating data when the preset condition of acquiring the data is met is continuously executed until the verification is passed.
6. A global data acquisition device, comprising:
the acquisition module is used for acquiring the overseas updating data by a preset overseas data node when a preset condition for acquiring the data is met;
the sending module is used for sending the overseas updating data to a preset domestic data node so that the domestic data node integrates the overseas updating data and the domestic updating data to obtain target updating data; and the domestic updating data is obtained when the domestic data nodes meet the preset conditions.
7. The global data acquisition device of claim 6, wherein said foreign data nodes comprise a first foreign data node and a plurality of second foreign data nodes, said first foreign data node and said plurality of second foreign data nodes being provided with log information, respectively;
the acquisition module is specifically configured to:
when the first overseas data node detects that the log information of at least one data node in the first overseas data node or the plurality of second overseas data nodes is updated, acquiring corresponding overseas update data from the at least one data node based on the updated log information.
8. The global data acquisition device of claim 7, wherein said log information is provided with time information;
the acquisition module includes:
the first oversea data node is used for acquiring the update log information of each first oversea data node and each second oversea data node; the time information of the updated log information does not exceed the preset time information threshold;
and the update data acquisition sub-module is used for respectively acquiring corresponding update data from the first oversea data node and each second oversea data node based on each update log information.
9. An electronic device, comprising:
a processor, a memory, and a bus;
the bus is used for connecting the processor and the memory;
the memory is used for storing operation instructions;
the processor is used for executing the global data acquisition method of any one of the claims 1-5 by calling the operation instruction.
10. A computer-readable storage medium for storing computer instructions which, when executed on a computer, cause the computer to perform the global data acquisition method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911108099.6A CN111835812B (en) | 2019-11-13 | 2019-11-13 | Global data acquisition method and device, electronic equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911108099.6A CN111835812B (en) | 2019-11-13 | 2019-11-13 | Global data acquisition method and device, electronic equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111835812A true CN111835812A (en) | 2020-10-27 |
| CN111835812B CN111835812B (en) | 2023-01-31 |
Family
ID=72912669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911108099.6A Active CN111835812B (en) | 2019-11-13 | 2019-11-13 | Global data acquisition method and device, electronic equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111835812B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8997117B1 (en) * | 2012-10-02 | 2015-03-31 | Linkedin Corporation | System and method for creating personal connection alerts |
| US20170346914A1 (en) * | 2016-05-25 | 2017-11-30 | New Asia Technology Development Limited | Systems and methods to prioritize and schedule notifications with user behaviour and contextual data analysis |
| CN108390914A (en) * | 2018-01-23 | 2018-08-10 | 阿里巴巴集团控股有限公司 | A service update method, device, and system |
| CN109040183A (en) * | 2018-06-27 | 2018-12-18 | 郑州云海信息技术有限公司 | Node information acquisition method, device, equipment and computer readable storage medium |
| CN109379277A (en) * | 2018-12-10 | 2019-02-22 | 深圳贝尔创意科教有限公司 | A distributed IM communication method and device based on routing addressing |
| CN109670126A (en) * | 2018-11-27 | 2019-04-23 | 广东耐思智慧科技有限公司 | A kind of cross-border access accelerating method and device |
-
2019
- 2019-11-13 CN CN201911108099.6A patent/CN111835812B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8997117B1 (en) * | 2012-10-02 | 2015-03-31 | Linkedin Corporation | System and method for creating personal connection alerts |
| US20170346914A1 (en) * | 2016-05-25 | 2017-11-30 | New Asia Technology Development Limited | Systems and methods to prioritize and schedule notifications with user behaviour and contextual data analysis |
| CN108390914A (en) * | 2018-01-23 | 2018-08-10 | 阿里巴巴集团控股有限公司 | A service update method, device, and system |
| CN109040183A (en) * | 2018-06-27 | 2018-12-18 | 郑州云海信息技术有限公司 | Node information acquisition method, device, equipment and computer readable storage medium |
| CN109670126A (en) * | 2018-11-27 | 2019-04-23 | 广东耐思智慧科技有限公司 | A kind of cross-border access accelerating method and device |
| CN109379277A (en) * | 2018-12-10 | 2019-02-22 | 深圳贝尔创意科教有限公司 | A distributed IM communication method and device based on routing addressing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111835812B (en) | 2023-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104219316B (en) | A kind of call request processing method and processing device in distributed system | |
| CN106850402B (en) | Message transmission method and device | |
| CN111614761B (en) | Block chain message transmission method, device, computer and readable storage medium | |
| JP2021504847A (en) | Cross-blockchain interaction methods, devices, systems, and electronic devices | |
| CN108718347B (en) | Domain name resolution method, system, device and storage medium | |
| CN113259479B (en) | Data processing method and equipment | |
| US8903972B2 (en) | Method and apparatus for sharing contents using information of group change in content oriented network environment | |
| US20170055247A1 (en) | Data transmission method and apparatus and data receiving method and apparatus | |
| CN109154968B (en) | System and method for secure and efficient communication within an organization | |
| EP4030314A1 (en) | Blockchain-based data processing method, apparatus and device, and readable storage medium | |
| KR20160039280A (en) | Messaging api over http protocol to establish context for data exchange | |
| CN105657000A (en) | Message transmission method and device | |
| CN111163130A (en) | A network service system and data transmission method thereof | |
| CN116915827A (en) | Data transmission method, device, electronic equipment and media for Internet of Things edge gateway | |
| US20150067066A1 (en) | Provisioning Communication Services using Proxy Server in a Cloud | |
| CN117591294A (en) | Service dynamic load access method and device, internet of things platform and medium | |
| CN104468594A (en) | Data request method, device and system | |
| CN107968848A (en) | A method for obtaining an IP address, a terminal device and a storage medium | |
| JP2016537884A (en) | Client-side flow control apparatus and method in remote access environment | |
| CN103842960A (en) | Method and apparatus for inter-control communication | |
| CN113055193B (en) | A data multicast transmission method, device, equipment and storage medium | |
| CN105282112A (en) | Terminal and method for detecting security of data interaction in terminal | |
| US20120124430A1 (en) | Mechanism to Prevent Escaped Associations in Multi-Association RPC Based Protocols | |
| CN102724195A (en) | Access request tracing method and related device | |
| CN111835812B (en) | Global data acquisition method and device, electronic equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |