CN109726004A - A data processing method and device - Google Patents
A data processing method and device Download PDFInfo
- Publication number
- CN109726004A CN109726004A CN201711024565.3A CN201711024565A CN109726004A CN 109726004 A CN109726004 A CN 109726004A CN 201711024565 A CN201711024565 A CN 201711024565A CN 109726004 A CN109726004 A CN 109726004A
- Authority
- CN
- China
- Prior art keywords
- data
- computational frame
- source data
- processing
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请涉及大数据技术领域，尤其涉及一种数据处理方法及装置，用以解决现有技术中数据接入平台使用成本较高的问题；本申请实施例提供的数据处理方法包括：接收数据处理请求，所述数据处理请求中携带有源数据的地址信息和对所述源数据需执行的处理任务；根据所述源数据的地址信息获取源数据；根据所述源数据的属性信息和所述数据接入平台中各计算框架的负载状况，确定用于执行所述处理任务的计算框架；将所述处理任务提交给所述计算框架，由所述计算框架对所述源数据进行处理，这样，用户只需将需要数据接入平台处理的任务写入接入请求中，就可以对数据接入平台进行命令发布，降低了使用数据接入平台的难度，降低了数据接入平台的使用成本。
The present application relates to the technical field of big data, and in particular, to a data processing method and device, which are used to solve the problem of high usage cost of a data access platform in the prior art; the data processing method provided by the embodiments of the present application includes: receiving data processing The data processing request carries the address information of the source data and the processing task to be performed on the source data; obtains the source data according to the address information of the source data; according to the attribute information of the source data and the The load status of each computing framework in the data access platform determines the computing framework used to execute the processing task; submit the processing task to the computing framework, and the computing framework processes the source data, so that , users only need to write the tasks that need to be processed by the data access platform into the access request, and then they can issue commands to the data access platform, which reduces the difficulty of using the data access platform and reduces the cost of using the data access platform. .
Description
Technical field
This application involves big data technical field more particularly to a kind of data processing method and devices.
Background technique
Currently, the Data entries that data access platform is important as big data platform, are mainly used for realizing that data source connects
Enter and pre-processed with data buffer storage, wherein data source access mainly includes to the logs such as online message, off-line files, Binlog text
Part carries out data acquisition；Mainly the adaptation including data source format, the encrypted transmission of data, message cache for data buffer storage pretreatment
Distribute with subscribing to.
In the prior art, user must come by Flume-NG, Logstach or Scribe distributed deployment way
Data access is realized, in this way, each user requires voluntarily to carry out Technology Selection, to different acquisition tasks or preprocessing tasks
It needs using different technical solutions, and needs oneself to complete the bottom layer realization of software, therefore, development process is comparatively laborious,
Complexity causes the use cost of data access platform also relatively high.
As it can be seen that there is development processes when using data access platform is cumbersome, complicated by user in the prior art, cause to count
According to the relatively high problem of the use cost of access platform.
Summary of the invention
The embodiment of the present application provides a kind of data processing method and device, is using number to solve user in the prior art
It is cumbersome, complicated that there is development processes when according to access platform, the problem for causing the use cost of data access platform relatively high.
A kind of data processing method provided by the embodiments of the present application, it is flat applied to the data access comprising multiple Computational frames
Platform, comprising:
Data access platform receives data processing request, and the address information of active data is carried in the data processing request
With the processing task that need to be executed to the source data；
Source data is obtained according to the address information of the source data；
According to the load state of each Computational frame in the attribute information of the source data and the data access platform, determine
For executing the Computational frame of the processing task, the attribute information of the source data include at least data source, time delay size,
Total amount of data and the quality of data；
The processing task is submitted into the Computational frame, the source data is handled by the Computational frame.
A kind of data processing equipment provided by the embodiments of the present application, positioned at the data access platform comprising multiple Computational frames
In, comprising:
Receiving module, processing is requested for receiving data, and the address letter of active data is carried in the data processing request
Breath and the processing task that the source data need to be executed；
Module is obtained, for obtaining source data according to the address information of the source data；
Computational frame chooses module, based in the attribute information and the data access platform according to the source data respectively
The load state for calculating frame, determines the Computational frame for executing the processing task, the attribute information of the source data is at least
Include data source, time delay size, total amount of data and the quality of data；
Module is submitted, for the processing task to be submitted to the Computational frame, by the Computational frame to the source
Data are handled.
A kind of electronic equipment provided by the embodiments of the present application, including at least one processing unit and at least one storage
Unit, wherein the storage unit is stored with program code, when said program code is executed by the processing unit, so that
The electronic equipment executes the step of above-mentioned data processing method.
A kind of computer readable storage medium provided by the embodiments of the present application, including program code, work as said program code
When running on an electronic device, the step of making the electronic equipment execute above-mentioned data processing method.
In the embodiment of the present application, data access platform receives data processing request, carries active number in data processing request
According to address information and processing task that source data need to be executed, source data is obtained according to the address information of source data later, into
And according to the load state of each Computational frame in the attribute information of source data and data access platform, it determines and appoints for executing processing
The Computational frame of business, and processing task is submitted into the Computational frame, source data is handled by the Computational frame, in this way,
User only need to will be in the address information of source data and the processing task write-in data processing request that operates to source data, so that it may
To carry out order publication to data access platform, cumbersome Floor layer Technology is real when without being focused on again using data access platform
It is existing, the difficulty using data access platform is reduced, the use cost of data access platform is reduced.
Detailed description of the invention
Fig. 1 is data processing method flow chart provided by the embodiments of the present application；
Fig. 2 is data processing equipment structure chart provided by the embodiments of the present application；
Fig. 3 is that the hardware configuration of the electronic equipment provided by the embodiments of the present application for realizing data processing method is illustrated
Figure.
Specific embodiment
The embodiment of the present application is intended to shield user and uses the software reality for acquiring Computational frame when data access platform to bottom
It is existing, and data definition, data acquisition and data consumption interface are provided to user, simplify point that user in the prior art needs to complete
Cloth deployment and configuration work, realize the external unified management of data access platform, and can provide a part of generalization
Pretreatment potentiality (data pick-up, data desensitization, data merge etc.), it is not necessary to user carries out low level development again, and then reduces and use
Family uses the difficulty of data access platform.
The embodiment of the present application is described in further detail with reference to the accompanying drawings of the specification.
Embodiment one
As shown in Figure 1, being data processing method flow chart provided by the embodiments of the present application, it is applied to include multiple calculation blocks
The data access platform of frame, comprising the following steps:
S101: data access platform receives data processing request, and the ground of active data is carried in the data processing request
Location information and the processing task that source data need to be executed.
Wherein, data processing request is the request of XML format, parses and takes in the available data processing request of XML file
The address information of the source data of band and the processing task that source data need to be executed, here, the place for needing data access platform to execute
Reason task such as data acquisition, encrypted transmission, subscription distribution etc..
S102: source data is obtained according to the address information of source data.
S103: it according to the load state of each Computational frame in the attribute information of source data and data access platform, determines and uses
In the Computational frame for executing the processing task.
Optionally, the attribute information of source data includes at least data source, time delay size, total amount of data and the quality of data,
Wherein, data source such as File Transfer Protocol (File Transfer Protocol, FTP), Hadoop distributed file system
(Hadoop Distributed File System, HDFS), Kafka, serial data transport protocol (Serial Data
Transport Protocol, SDTP) etc.；Time delay size refers to the time that data need to wait in transmission process；Total amount of data
Refer to the data volume size for needing Computational frame to handle；The quality of data refers to the number of data that single allows to execute, if single is permitted
Perhaps the number of data executed is less, and such as 1, it is determined that the quality of data is high, if the number of data that single allows to execute is more, such as
1000, it is determined that the quality of data is low.
In the specific implementation process, it after obtaining source data, can be determined according to the attribute information of source data to source number
According to processing mode, wherein the processing mode to source data includes batch processing mode and immediate processing mode, and then according to source
The load state of each Computational frame in the processing mode and data access platform of data is determined for executing the processing task
Computational frame.
Specifically, if data source is FTP or HDFS, source data can be handled with batch processing mode, if data are come
Source is Kafka or SDTP, then can handle source data with immediate processing mode.If it is determined that the time delay of data is larger, such as larger than
10s then can handle source data with batch processing mode, otherwise, can handle source data with immediate processing mode.If it is determined that data
Total amount is larger, and if data are greater than 1GB, then can handle source data with batch processing mode otherwise can be at immediate processing mode
Manage source data.If it is determined that the quality of data is high, then otherwise can handle source data with immediate processing mode can use batch processing side
Formula handles source data.
Also, in the absence of conflict, the method for above-mentioned determining source data processing mode can be used in combination, right at this
Specific combination is without limitation.
Further, however, it is determined that source data is handled with immediate processing mode, it is determined that with real-time streams in data access platform
Processing mode handles the Computational frame of data flow, and selection load is made lower than the Computational frame of first threshold from these Computational frames
For the Computational frame for executing the processing task；If it is determined that handling source data with batch processing mode, it is determined that data access platform
In with batch processing mode processing data flow Computational frame, from these Computational frames selection load be lower than second threshold calculating
Frame is as the Computational frame for executing the processing task.
S104: the processing task is submitted into determining Computational frame, source data is handled by the Computational frame.
Optionally, also include destination address in data processing request, Computational frame after handling source data,
The data of generation can be stored in destination address.
In addition, user with customized Computational frame and can introduce the Computational frame of oneself deployment, for example, can be
Engine label is added in access interface can define the type of Computational frame.
In the above process, according to the load state of each Computational frame in the attribute information of source data and data access platform,
Before determining the Computational frame for executing the processing task, it can determine that there is no user is customized in data access platform
Computational frame, however, it is determined that there are the customized Computational frames of user in data access platform, then execute the processing determining
When the Computational frame of task, deploy label can be added to introduce the Computational frame of user oneself deployment, and no longer select number
According to the Computational frame in access platform.
In the embodiment of the present application, data access platform receives data processing request, carries active number in data processing request
According to address information and processing task that source data need to be executed, source data is obtained according to the address information of source data later, into
And according to the load state of each Computational frame in the attribute information of source data and data access platform, it determines and appoints for executing processing
The Computational frame of business, and processing task is submitted into the Computational frame, source data is handled by the Computational frame, in this way,
User only need to will be in the address information of source data and the processing task write-in data processing request that operates to source data, so that it may
To carry out order publication to data access platform, cumbersome Floor layer Technology is real when without being focused on again using data access platform
It is existing, the difficulty using data access platform is reduced, the use cost of data access platform is reduced.
Embodiment two
Data access platform in the embodiment of the present application includes multiple Computational frames, such as MR, Spark, Storm, is mentioned
A kind of real-time data imputing system generalization solution based on network web services has been supplied, has mainly been located in advance including data access, data
Reason and data storage three zones, wherein data access mainly provides off-line files (FTP/HDFS), online message (HTTP/
Kafka/SDTP it) is connect with the encryption of the data source datas such as journal file (MySQL Binlog/Syslog/log4j/logback)
Enter；Data prediction mainly provides data source definitions (field seperator, field description, data Source Description, field type), data
Source field desensitization, the subscription distribution of data source and data check；Data storage is mainly provided to be disappeared based on mainstreams such as Kafka/MetaQ
The data storage of queue mechanism is ceased, and can provide and encapsulate perfect external interface with for users to use.
In the specific implementation process, data access platform mainly includes three external interfaces: access interface,
Definition interface and consume interface, unified configuration template is externally provided using these interfaces, and user can pass through letter
These interfaces are realized in single object access protocol (Simple Object Access Protocol, SOAP) request, and then utilize this
A little interfaces, which issue data access platform, orders.
Specifically, access interface mainly provides the source of data source, the target storage position of data source, data prediction
Logical process and bound with data source definitions, the XML format of access interface is as follows:
Wherein, the Source tag definition source of data, wherein Type label is used for the type in mark data source,
Such as FTP/HDFS/Kafka/HTTP/SDTP, param label is for identifying other association attributes, such as other related categories of FTP
Property includes Server, Port, UserName, Passwd, FilePath, isGZ etc.；The data processing completion of Sink tag definition
Buffer address later, wherein Type mainly includes Kafka/Redis/Elasticsearch/FTP/HDFS etc.；
The pretreated logical process of Interceptor Tag identification data, wherein Type mainly includes that field merges (concat), just
Then extract (regularExtractor), field extracts (indexExtractor), based on types such as md5/Base64/SHA256
Field desensitization (desensitization) etc.；Definition of the dataSource label for data access process and data source is real
It now binds, user can be prompted to define data source format first the data source platform that name is not present, then carry out data and connect
Enter.
The format and relevant field information of definition interface main definitions data source, definition interface
XML format is as follows:
Wherein, outermost layer label is dataSource, and binding is realized in the definition for data access process and data source,
In, name is unique index, and for determining that unique data source identifies, fieldDelimiter label is for defining field isolation
Symbol, length label are used for defining field number, description label for defining data Source Description, fields label
It mainly include field name (name label), field index position (index label), field description in enumerating all fields
(description label) and field type (type label) etc..
Consume interface mainly provides consumption of the user to the data source data, and the XML format of consume interface is as follows:
Wherein, name label is the title of data source, and batch is primary available number of data,
BatchDurtition is the maximum time for obtaining data.
Data access platform in the embodiment of the present application supports a variety of Computational frames, has not only included real-time Computational frame but also has included
Batch processing Computational frame, wherein real-time Computational frame such as Storm, Spark Streaming；Batch processing Computational frame such as MR,
Spark。
In the specific implementation process, the XML request that data access platform submits Rest layers, can first parse XML file
The processing task that obtains the address information of source data and need to execute to source data obtains source according to the address information of source data later
Data, and then according to the attribute information of source data, such as data source, time delay size, total amount of data and the quality of data, determination pair
The processing mode of source data, and then the calculation block for executing the processing task is determined according to the processing mode to source data
Frame.
For example, perhaps the off-line datas such as HDFS can transfer to MR/Spark etc. to carry out batch processing and Kafka or SDTP to FTP
Etc. tasks the real-time Computational frame such as Storm/Spark Streaming can be selected to handle.
In the specific implementation process, the loading condition that each Computational frame in data access platform can also be obtained in real time,
When for the processing task choosing Computational frame, the loading condition for considering each Computational frame can be combined with, to realize each meter
The load balancing for calculating frame, makes the performance of data access platform get the greatest degree of optimization.
Further, after Computational frame has handled source data, the data of generation can also be stored in destination address
In, destination address is also that parsing XML file obtains.
Optionally, data access platform can also provide the customized selection Computational frame of user and introduce the meter of oneself deployment
Frame is calculated, for example, the type of Computational frame can be defined by adding engine label in access interface, is selected for subtask
When handling Computational frame, however, it is determined that it is added with deploy label in access interface, then introduces the service of user oneself deployment,
Otherwise, the Computational frame provided using data access platform.
In the specific implementation process, it to each Computational frame node in data access platform, is realized using agent mode
The monitoring of submission task and Computational frame loading condition on the node, for example, in the nimbus node of Strom cluster, flink
Agent is all disposed on the JobManager node of cluster, spark client node, task on each node is mentioned to realize
Hand over the monitoring with computing resource.
In addition, data access platform can also provide on Docker solution, convenient for users to key deployment, and
Integrated nginx load-balancing mechanism, Storm on docker, Flink on docker and Spark on docker etc. are solved
Scheme.
Data access platform provided by the embodiments of the present application can realize data access energy with intelligent selection Computational frame
Power and pretreated ability access task to the data source that user submits, its calculating are selected by SmartRouter module
Frame, while user customized can also select Computational frame and introduce the Computational frame of oneself deployment, and integrated on
Docker solution, the program compare artificial distributed deployment mode, can simplify user and dispose the numerous of related service
Complexity that is trivial, reducing configuration.In addition, the unitized processing that data access platform no longer uses data to converge, but by data
Pretreatment is divided into the smallest subtask and is distributed on each Computational frame node, and data prediction scheme unitized in this way can
To disperse the pressure of single Computational frame node, the working efficiency of data access platform is improved to the maximum extent.
Embodiment three
Based on the same inventive concept, it is additionally provided in the embodiment of the present application at one kind data corresponding with data processing method
Manage device, since the principle that the device solves the problems, such as is similar to the embodiment of the present application data processing method, the reality of the device
The implementation for the method for may refer to is applied, overlaps will not be repeated.
As shown in Fig. 2, being data processing equipment structure chart provided by the embodiments of the present application, which, which is located at, includes multiple meters
In the data access platform for calculating frame, comprising:
Receiving module 201, processing is requested for receiving data, and the address of active data is carried in the data processing request
Information and the processing task that the source data need to be executed；
Module 202 is obtained, for obtaining source data according to the address information of the source data；
Computational frame chooses module 203, in the attribute information and the data access platform according to the source data
The load state of each Computational frame determines the Computational frame for executing the processing task, the attribute information of the source data
Including at least data source, time delay size, total amount of data and the quality of data；
Module 204 is submitted, for the processing task to be submitted to the Computational frame, by the Computational frame to described
Source data is handled.
Optionally, the Computational frame is chosen module 203 and is specifically used for:
The processing mode to the source data is determined according to the attribute information of the source data, wherein to the source data
Processing mode include batch processing mode and immediate processing mode；
According to the load state of the processing mode to the source data and each Computational frame in the data access platform, really
Determine the Computational frame for executing the processing task.
Optionally, the Computational frame is chosen module 203 and is specifically used for:
If it is determined that handling the source data with immediate processing mode, it is determined that in the data access platform with real-time streams at
Reason mode handles the Computational frame of data flow, selected from the Computational frame load lower than first threshold Computational frame as
Execute the Computational frame of the processing task；
If it is determined that handling the source data with batch processing mode, it is determined that with batch processing mode in the data access platform
The Computational frame for handling data flow selects load lower than the Computational frame of second threshold as execution institute from the Computational frame
State the Computational frame of processing task.
Optionally, described device further includes determining module 205:
The determining module 205, based in the attribute information and the data access platform according to the source data respectively
The load state of frame is calculated, before determining the Computational frame for executing the processing task, determines that there is no user is customized
Computational frame.
Optionally, the submission module 204 is also used to:
If it is determined that there are the customized Computational frames of user, then it is customized the processing task to be submitted to the user
Computational frame is handled the source data by the customized Computational frame of the user.
Example IV
As shown in figure 3, the hardware configuration for the electronic equipment provided by the embodiments of the present application for realizing data processing shows
It is intended to, including at least one processing unit 301 and at least one storage unit 302, wherein storage unit is stored with program
Code, when program code is executed by the processing unit, so that the step of electronic equipment executes above-mentioned data processing method.
Embodiment five
A kind of computer readable storage medium provided by the embodiments of the present application, including program code, work as said program code
When running on an electronic device, make electronic equipment execute above-mentioned data processing method the step of.
It should be understood by those skilled in the art that, embodiments herein can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the application
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the application, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The application is process of the reference according to method, apparatus (system) and computer program product of the embodiment of the present application
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although the preferred embodiment of the application has been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the application range.
Obviously, those skilled in the art can carry out various modification and variations without departing from the essence of the application to the application
Mind and range.In this way, if these modifications and variations of the application belong to the range of the claim of this application and its equivalent technologies
Within, then the application is also intended to include these modifications and variations.
Claims (12)
1. a kind of data processing method, applied to the data access platform comprising multiple Computational frames characterized by comprising
Data access platform receives data processing request, and the address information of active data and right is carried in the data processing request
The processing task that the source data need to execute；
Source data is obtained according to the address information of the source data；
According to the load state of each Computational frame in the attribute information of the source data and the data access platform, determination is used for
The Computational frame of the processing task is executed, the attribute information of the source data includes at least data source, time delay size, data
Total amount and the quality of data；
The processing task is submitted into the Computational frame, the source data is handled by the Computational frame.
2. the method as described in claim 1, which is characterized in that according to the attribute information of the source data and the data access
The load state of each Computational frame in platform determines the Computational frame for executing the processing task, comprising:
The processing mode to the source data is determined according to the attribute information of the source data, wherein to the place of the source data
Reason mode includes batch processing mode and immediate processing mode；
According to the load state of the processing mode to the source data and each Computational frame in the data access platform, determines and use
In the Computational frame for executing the processing task.
3. method according to claim 2, which is characterized in that according to the source data processing mode and the data connect
Enter the load state of each Computational frame in platform, determine the Computational frame for executing the processing task, comprising:
If it is determined that handling the source data with immediate processing mode, it is determined that with real-time stream process side in the data access platform
Formula handles the Computational frame of data flow, selects load lower than the Computational frame of first threshold as execution from the Computational frame
The Computational frame of the processing task；
If it is determined that handling the source data with batch processing mode, it is determined that handled in the data access platform with batch processing mode
The Computational frame of data flow, from selected in the Computational frame load lower than second threshold Computational frame as execution it is described from
The Computational frame of reason task.
4. the method as described in claim 1, which is characterized in that according to the attribute information of the source data and the data access
The load state of each Computational frame in platform, before determining Computational frame for executing the processing task, further includes:
It determines and the customized Computational frame of user is not present.
5. method as claimed in claim 4, which is characterized in that further include:
If it is determined that there are the customized Computational frames of user, then the processing task is submitted into the customized calculating of the user
Frame is handled the source data by the customized Computational frame of the user.
6. a kind of data processing equipment, in the data access platform comprising multiple Computational frames characterized by comprising
Receiving module, processing request for receiving data, carry in the data processing request active data address information and
The processing task that the source data need to be executed；
Module is obtained, for obtaining source data according to the address information of the source data；
Computational frame chooses module, for each calculation block in the attribute information and the data access platform according to the source data
The load state of frame determines that the Computational frame for executing the processing task, the attribute information of the source data include at least
Data source, time delay size, total amount of data and the quality of data；
Module is submitted, for the processing task to be submitted to the Computational frame, by the Computational frame to the source data
It is handled.
7. device as claimed in claim 6, which is characterized in that the Computational frame is chosen module and is specifically used for:
The processing mode to the source data is determined according to the attribute information of the source data, wherein to the place of the source data
Reason mode includes batch processing mode and immediate processing mode；
According to the load state of the processing mode to the source data and each Computational frame in the data access platform, determines and use
In the Computational frame for executing the processing task.
8. device as claimed in claim 7, which is characterized in that the Computational frame is chosen module and is specifically used for:
If it is determined that handling the source data with immediate processing mode, it is determined that with real-time stream process side in the data access platform
Formula handles the Computational frame of data flow, selects load lower than the Computational frame of first threshold as execution from the Computational frame
The Computational frame of the processing task；
If it is determined that handling the source data with batch processing mode, it is determined that handled in the data access platform with batch processing mode
The Computational frame of data flow, from selected in the Computational frame load lower than second threshold Computational frame as execution it is described from
The Computational frame of reason task.
9. device as claimed in claim 6, which is characterized in that further include determining module:
The determining module, for each Computational frame in the attribute information and the data access platform according to the source data
Load state before determining the Computational frame for executing the processing task, determines and the customized calculation block of user is not present
Frame.
10. device as claimed in claim 9, which is characterized in that the submission module is also used to:
If it is determined that there are the customized Computational frames of user, then the processing task is submitted into the customized calculating of the user
Frame is handled the source data by the customized Computational frame of the user.
11. a kind of electronic equipment, which is characterized in that including at least one processing unit and at least one storage unit,
In, the storage unit is stored with program code, when said program code is executed by the processing unit, so that the electronics
Equipment perform claim requires the step of 1~5 any the method.
12. a kind of computer readable storage medium, which is characterized in that including program code, when said program code is set in electronics
When standby upper operation, the step of making the electronic equipment perform claim require 1~5 any the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711024565.3A CN109726004B (en) | 2017-10-27 | 2017-10-27 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711024565.3A CN109726004B (en) | 2017-10-27 | 2017-10-27 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109726004A true CN109726004A (en) | 2019-05-07 |
| CN109726004B CN109726004B (en) | 2021-12-03 |
Family
ID=66292118
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711024565.3A Active CN109726004B (en) | 2017-10-27 | 2017-10-27 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109726004B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110336986A (en) * | 2019-08-08 | 2019-10-15 | 杭州阿启视科技有限公司 | Video big data cloud platform stream media service method |
| CN111241180A (en) * | 2020-01-13 | 2020-06-05 | 中国建设银行股份有限公司 | Information processing method, device, equipment and storage medium |
| CN112307114A (en) * | 2019-07-31 | 2021-02-02 | 北京中关村科金技术有限公司 | Data exchange method, device and storage medium |
| CN113434908A (en) * | 2021-08-30 | 2021-09-24 | 四川新网银行股份有限公司 | Data desensitization method based on Logback log component |
| CN113890876A (en) * | 2021-01-27 | 2022-01-04 | 贺鑫 | Node control method of fully mechanized coal mining face data middling platform based on micro-service framework |
| CN114579656A (en) * | 2022-05-07 | 2022-06-03 | 中建电子商务有限责任公司 | Method for processing Mysql Binlog data |
| CN116264649A (en) * | 2022-12-01 | 2023-06-16 | 中移(苏州)软件技术有限公司 | Data acquisition method, data management system, data acquisition device and computer equipment |
| CN118170737A (en) * | 2024-05-11 | 2024-06-11 | 中移(苏州)软件技术有限公司 | Data processing method, device and related equipment |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102143198A (en) * | 2010-09-30 | 2011-08-03 | 华为技术有限公司 | Method, device and system for transferring messages |
| CN103530189A (en) * | 2013-09-29 | 2014-01-22 | 中国科学院信息工程研究所 | Automatic scaling and migrating method and device oriented to stream data |
| CN103812949A (en) * | 2014-03-06 | 2014-05-21 | 中国科学院信息工程研究所 | Task scheduling and resource allocation method and system for real-time cloud platform |
| CN104461740A (en) * | 2014-12-12 | 2015-03-25 | 国家电网公司 | Cross-domain colony computing resource gathering and distributing method |
| US20150271075A1 (en) * | 2014-03-20 | 2015-09-24 | Microsoft Corporation | Switch-based Load Balancer |
| US9264482B2 (en) * | 2013-08-28 | 2016-02-16 | Electronics & Telecommunications Research Institute | Batching-based content routing method and apparatus for performing batching-based content routing method |
| CN105453068A (en) * | 2013-07-31 | 2016-03-30 | 慧与发展有限责任合伙企业 | Data stream processing using a distributed cache |
| CN105574082A (en) * | 2015-12-08 | 2016-05-11 | 曙光信息产业(北京)有限公司 | Storm based stream processing method and system |
| CN105786600A (en) * | 2016-02-02 | 2016-07-20 | 北京京东尚科信息技术有限公司 | Task scheduling method and device |
| CN105991459A (en) * | 2015-02-15 | 2016-10-05 | 上海帝联信息科技股份有限公司 | Source-returning route distribution method, apparatus and system of CDN node |
| CN106021484A (en) * | 2016-05-18 | 2016-10-12 | 中国电子科技集团公司第三十二研究所 | A customizable multi-mode big data processing system based on in-memory computing |
| US20160306674A1 (en) * | 2015-04-17 | 2016-10-20 | Microsoft Technology Licensing, Llc | Handling Tenant Requests in a System that Uses Acceleration Components |
| CN106921684A (en) * | 2015-12-24 | 2017-07-04 | 北京奇虎科技有限公司 | The processing method and server of a kind of solicited message |
| CN107085579A (en) * | 2016-02-16 | 2017-08-22 | 中国移动通信集团福建有限公司 | A method and device for collecting and distributing data |
-
2017
- 2017-10-27 CN CN201711024565.3A patent/CN109726004B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102143198A (en) * | 2010-09-30 | 2011-08-03 | 华为技术有限公司 | Method, device and system for transferring messages |
| CN105453068A (en) * | 2013-07-31 | 2016-03-30 | 慧与发展有限责任合伙企业 | Data stream processing using a distributed cache |
| US9264482B2 (en) * | 2013-08-28 | 2016-02-16 | Electronics & Telecommunications Research Institute | Batching-based content routing method and apparatus for performing batching-based content routing method |
| CN103530189A (en) * | 2013-09-29 | 2014-01-22 | 中国科学院信息工程研究所 | Automatic scaling and migrating method and device oriented to stream data |
| CN103812949A (en) * | 2014-03-06 | 2014-05-21 | 中国科学院信息工程研究所 | Task scheduling and resource allocation method and system for real-time cloud platform |
| US20150271075A1 (en) * | 2014-03-20 | 2015-09-24 | Microsoft Corporation | Switch-based Load Balancer |
| CN104461740A (en) * | 2014-12-12 | 2015-03-25 | 国家电网公司 | Cross-domain colony computing resource gathering and distributing method |
| CN105991459A (en) * | 2015-02-15 | 2016-10-05 | 上海帝联信息科技股份有限公司 | Source-returning route distribution method, apparatus and system of CDN node |
| US20160306674A1 (en) * | 2015-04-17 | 2016-10-20 | Microsoft Technology Licensing, Llc | Handling Tenant Requests in a System that Uses Acceleration Components |
| CN105574082A (en) * | 2015-12-08 | 2016-05-11 | 曙光信息产业(北京)有限公司 | Storm based stream processing method and system |
| CN106921684A (en) * | 2015-12-24 | 2017-07-04 | 北京奇虎科技有限公司 | The processing method and server of a kind of solicited message |
| CN105786600A (en) * | 2016-02-02 | 2016-07-20 | 北京京东尚科信息技术有限公司 | Task scheduling method and device |
| CN107085579A (en) * | 2016-02-16 | 2017-08-22 | 中国移动通信集团福建有限公司 | A method and device for collecting and distributing data |
| CN106021484A (en) * | 2016-05-18 | 2016-10-12 | 中国电子科技集团公司第三十二研究所 | A customizable multi-mode big data processing system based on in-memory computing |
Non-Patent Citations (2)
| Title |
|---|
| IBOWEN: ""5种必知的大数据处理框架技术"", 《HTTPS://WWW.CNBLOGS.COM/OC-BOWEN/P/6109009.HTML》 * |
| 丁滟: ""开放式海量数据处理服务的计算完整性研究"", 《中国博士学位论文全文数据库 信息科技辑》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112307114A (en) * | 2019-07-31 | 2021-02-02 | 北京中关村科金技术有限公司 | Data exchange method, device and storage medium |
| CN110336986A (en) * | 2019-08-08 | 2019-10-15 | 杭州阿启视科技有限公司 | Video big data cloud platform stream media service method |
| CN111241180A (en) * | 2020-01-13 | 2020-06-05 | 中国建设银行股份有限公司 | Information processing method, device, equipment and storage medium |
| CN113890876A (en) * | 2021-01-27 | 2022-01-04 | 贺鑫 | Node control method of fully mechanized coal mining face data middling platform based on micro-service framework |
| CN113434908A (en) * | 2021-08-30 | 2021-09-24 | 四川新网银行股份有限公司 | Data desensitization method based on Logback log component |
| CN114579656A (en) * | 2022-05-07 | 2022-06-03 | 中建电子商务有限责任公司 | Method for processing Mysql Binlog data |
| CN114579656B (en) * | 2022-05-07 | 2022-07-22 | 中建电子商务有限责任公司 | Method for processing Mysql Binlog data |
| CN116264649A (en) * | 2022-12-01 | 2023-06-16 | 中移(苏州)软件技术有限公司 | Data acquisition method, data management system, data acquisition device and computer equipment |
| CN116264649B (en) * | 2022-12-01 | 2025-11-04 | 中移(苏州)软件技术有限公司 | Data acquisition methods, data management systems, data acquisition devices and computer equipment |
| CN118170737A (en) * | 2024-05-11 | 2024-06-11 | 中移(苏州)软件技术有限公司 | Data processing method, device and related equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109726004B (en) | 2021-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109726004A (en) | A data processing method and device | |
| Jindal et al. | Function delivery network: Extending serverless computing for heterogeneous platforms | |
| CN109075988B (en) | Task scheduling and resource issuing system and method | |
| US9401835B2 (en) | Data integration on retargetable engines in a networked environment | |
| CN109478973B (en) | SDN controller, system and method for task scheduling, resource provisioning and service provisioning | |
| US9992269B1 (en) | Distributed complex event processing | |
| US20130268937A1 (en) | Distributed processing system, scheduler node and scheduling method of distributed processing system, and program generation apparatus thereof | |
| US9110695B1 (en) | Request queues for interactive clients in a shared file system of a parallel computing system | |
| Medel et al. | Client-side scheduling based on application characterization on kubernetes | |
| CN107003906A (en) | The type of cloud computing technology part is to type analysis | |
| JP2016514311A (en) | Database system providing single tenant and multi-tenant environments | |
| CN103713935B (en) | Method and device for managing Hadoop cluster resources in online manner | |
| US11108871B2 (en) | Dynamic generation of network routing configuration with service requirements | |
| CN109614227A (en) | Task resource allocation method, apparatus, electronic device, and computer-readable medium | |
| US11861386B1 (en) | Application gateways in an on-demand network code execution system | |
| US20120266164A1 (en) | Determining starting values for virtual machine attributes in a networked computing environment | |
| Saravanakumar et al. | An Efficient On-Demand Virtual Machine Migration in Cloud Using Common Deployment Model. | |
| US11790103B2 (en) | Adaptive data loss prevention | |
| Gropengießer et al. | Database backend as a service: automatic generation, deployment, and management of database backends for mobile applications | |
| Anjos et al. | BIGhybrid: a simulator for MapReduce applications in hybrid distributed infrastructures validated with the Grid5000 experimental platform | |
| Yu et al. | Performance studies of a websphere application, trade, in scale-out and scale-up environments | |
| Zhao et al. | Multitask oriented GPU resource sharing and virtualization in cloud environment | |
| Xu et al. | Predicting virtual machine’s power via a rbf neural network | |
| CN102571600A (en) | A Node Resource Scheduling Method | |
| Fan et al. | Formally modeling and analyzing cost‐aware job scheduling for cloud data center |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |