CN110750563A - Multi-model data processing method, system, device, electronic equipment and storage medium - Google Patents

Abstract

The disclosure provides a multi-model data processing method, a system, a device, electronic equipment and a computer readable storage medium, and belongs to the technical field of data processing. The method comprises the following steps: obtaining raw model data from a plurality of raw models; merging the original model data to obtain first intermediate data; extracting tuples, of which the target fields meet the extraction conditions, from the first intermediate data according to the extraction conditions for the target fields of the first intermediate data to obtain second intermediate data; and performing packaging treatment on the second intermediate data to obtain standard model data. The method and the device realize the fusion and intercommunication of the multi-model data, improve the convenience of subsequent application, form the standardized processing flow of the multi-model data and have strong universality.

Description

Multi-model data processing method, system, device, electronic device and storage medium

technical field

The present disclosure relates to the technical field of data processing, and in particular, to a multi-model data processing method, system, apparatus, electronic device, and computer-readable storage medium.

Background technique

With the development of data processing technology, App (application) or website operators are more and more capable of storing and processing massive data, and can formulate accurate and effective marketing strategies through big data analysis to achieve the benefits of App or website operation. improve.

At present, the data management team of each app or website operator generally has its own data model, and has specific requirements for the structure of data storage, data format, field name, data processing rules, etc., which makes it difficult for the data of different models to communicate with each other. In the context of the development driven by the concept of "ecologicalization" and "one-stop" platform of the Internet, the integration and interoperability of data from different models is the mainstream trend in the development of apps and websites. The big data analysis can play a greater value.

Therefore, it is necessary to propose a multi-model data processing method.

It should be noted that the information disclosed in the above Background section is only for enhancement of understanding of the background of the present disclosure, and therefore may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

SUMMARY OF THE INVENTION

The present disclosure provides a multi-model data processing method, system, apparatus, electronic device, and computer-readable storage medium, thereby at least to a certain extent overcoming the problem that data of different models in the existing multi-model data processing methods are difficult to communicate with each other.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or be learned in part by practice of the present disclosure.

According to one aspect of the present disclosure, there is provided a multi-model data processing method, comprising: obtaining original model data from multiple original models; merging the original model data to obtain first intermediate data; An extraction condition of the target field of the intermediate data, extract the tuple whose target field satisfies the extraction condition from the first intermediate data to obtain the second intermediate data; perform packaging processing on the second intermediate data , to obtain standard model data.

In an exemplary embodiment of the present disclosure, the acquiring the original model data from the multiple original models includes: periodically traversing the file names of the original model data files sent by the multiple original models on the file storage platform; The file name is matched with the download history record, and when the matching is unsuccessful, the file name is parsed, and the parsed file name is matched with the preset whitelist; if the parsed file name is If the matching with the preset whitelist is successful, the original model data file is downloaded from the file storage platform; the original model data is extracted from the downloaded original model data file.

In an exemplary embodiment of the present disclosure, the combining the original model data to obtain the first intermediate data includes: in the current cycle, detecting whether all the original model data of the current cycle have been obtained; If it is detected that all the original model data of the current cycle has not been acquired, it is detected again after the first interval whether all the original model data of the current cycle have been acquired; if it is detected that all the original model data of the current cycle have been acquired data, the original model data of the current cycle is merged to obtain the first intermediate data.

In an exemplary embodiment of the present disclosure, the method further includes: if it is detected that all the original model data of the current cycle has not been obtained, judging whether the model data merging task activates the multi-batch mode; If the model data merging task does not activate the multi-batch mode, then perform the step of re-detecting whether all the original model data of the current cycle has been acquired after the first predetermined time; if it is determined that the model data merging task activates the multi-batch mode mode, in any batch of the multi-batch mode, check whether the original model corresponding to the original model data obtained in the current batch includes all the original models; if it is detected that the original models obtained in the current batch are If the original model corresponding to the data does not include all the original models, then after the second interval time, check again whether the original model corresponding to the original model data obtained in the current batch includes all the original models; if it is detected that the current batch If the original model corresponding to the original model data obtained in the second batch includes all the original models, the original model data obtained in the current batch is merged to obtain the first intermediate data.

In an exemplary embodiment of the present disclosure, the method further includes: after detecting that the original model corresponding to the original model data obtained in the current batch does not include all the original models, detecting the model data merging task Whether to activate the forced merging mode; if it is detected that the model data merging task has activated the forced merging mode, merge the acquired original model data at a predetermined time in the forced merging mode to obtain the first intermediate data; If it is detected that the forced merging mode is not activated in the model data merging task, the step of detecting again whether the original model corresponding to the original model data obtained in the current batch includes all the original models is performed after the second interval.

In an exemplary embodiment of the present disclosure, the combining the original model data to obtain the first intermediate data includes: acquiring a combining rule, where the combining rule includes a field mapping rule; according to the field mapping rule , perform column merging of the original model data of the same type of field, and assign standard fields to each column to obtain the first intermediate data.

In an exemplary embodiment of the present disclosure, the merging rule further includes at least one of a field screening rule, a model priority rule, a format conversion rule, a reset field calculation rule, and a filtering condition; when combining fields of the same type After merging the original model data of the original model data and assigning standard fields to each column, the method further includes: performing at least one of the following steps on the original model data after merging columns and assigning standard fields to each column. Perform merging processing: remove the fields specified by the field screening rules; according to the model priority rules, when there are multiple original models with the same type of field and the same primary index with different original model data, keep the one with the highest priority. The original model data of the original model; according to the format conversion rule, format conversion is performed on the original model data of each of the standard fields according to the customized format of each of the standard fields; according to the reset field calculation rule, the reset field is The original model data is converted and calculated; the original model data that meets the filtering conditions is removed.

In an exemplary embodiment of the present disclosure, the target field includes the most recent active time and an extraction state; the target field is extracted from the first intermediate data according to an extraction condition for the target field of the first intermediate data Extracting the tuple whose target field satisfies the extraction condition to obtain the second intermediate data includes: extracting the extraction condition from the application task, where the extraction condition includes extraction time, activity period and non-repetitive extraction; When the extraction time is selected, a tuple whose active time is within the active period and whose extraction state is unextracted is extracted from the first intermediate data to obtain the second intermediate data.

In an exemplary embodiment of the present disclosure, the method further includes: after obtaining the first intermediate data, slicing the first intermediate data into pieces according to multiple index fields of the first intermediate data A plurality of first intermediate data are divided into tables and stored in a plurality of containers respectively; the extracting the target field from the first intermediate data according to the extraction conditions for the target field of the first intermediate data satisfies The extraction of the tuple of conditions to obtain the second intermediate data includes: according to the extraction conditions, acquiring the target first intermediate data sub-table from the plurality of containers, and obtaining the target first intermediate data sub-table from the target first intermediate data sub-table Extracting a tuple whose target field satisfies the extraction condition to obtain the second intermediate data.

According to one aspect of the present disclosure, there is provided a multi-model data processing system, comprising: a rule customization module for generating merging rules, extraction conditions and packaging rules according to a rule configuration file; a data processing module for generating merging rules according to the merging rules Merging the original model data to obtain first intermediate data, extracting second intermediate data from the first intermediate data according to the extraction conditions, and packaging the second intermediate data according to the packaging rules to obtain standard model data; a data storage module is used to obtain and store the original model data, and store the first intermediate data, the second intermediate data and the standard model data respectively.

According to one aspect of the present disclosure, there is provided a multi-model data processing apparatus, comprising: an original acquisition unit for acquiring original model data from a plurality of original models; and a data merging unit for merging the original model data to Obtaining first intermediate data; a data extraction unit, configured to extract, from the first intermediate data, a tuple whose target field satisfies the extraction condition according to an extraction condition for a target field of the first intermediate data, to obtain second intermediate data; a data packaging unit, configured to perform packaging processing on the second intermediate data to obtain standard model data.

According to one aspect of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute via executing the executable instructions The method of any of the above.

According to one aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements any one of the methods described above.

Exemplary embodiments of the present disclosure have the following beneficial effects:

On the one hand, the original model data of each original model is converted into standard model data through merging, extraction and packaging processing, thereby realizing the fusion and intercommunication of different model data, and improving the convenience of subsequent applications. On the other hand, this embodiment forms a standardized processing flow for multi-model data, and can formulate corresponding merging, extracting and packaging processing rules according to the characteristics of the original model or the customized requirements of subsequent applications, so that this embodiment has a strong Universality.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 schematically shows a flowchart of a multi-model data processing method in this exemplary embodiment;

FIG. 2 schematically shows a flowchart of a method for acquiring original model data in this exemplary embodiment;

FIG. 3 schematically shows a flow chart of a method for merging original model data in this exemplary embodiment;

FIG. 4 schematically shows a schematic diagram of a rule customization in this exemplary embodiment;

FIG. 5 schematically shows a flowchart of another multi-model data processing method in this exemplary embodiment;

FIG. 6 schematically shows a flowchart of another multi-model data processing method in this exemplary embodiment;

FIG. 7 schematically shows a flowchart of a data extraction method in this exemplary embodiment;

FIG. 8 schematically shows a schematic diagram of an operating environment architecture of a multi-model data processing system according to this exemplary embodiment;

FIG. 9 schematically shows a structural block diagram of a multi-model data processing apparatus in this exemplary embodiment;

FIG. 10 schematically shows an electronic device for implementing the above method in this exemplary embodiment;

FIG. 11 schematically shows a computer-readable storage medium for implementing the above method in this exemplary embodiment.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Exemplary embodiments of the present disclosure provide a multi-model data processing method, which can be applied to a backend server of an App or a website. Referring to FIG. 1, the method may include the following steps S11-S14:

Step S11, acquiring original model data from multiple original models.

Step S12, combining the original model data to obtain the first intermediate data.

Step S13, according to the extraction condition for the target field of the first intermediate data, extract the tuple whose target field satisfies the extraction condition from the first intermediate data to obtain the second intermediate data.

Step S14, performing packaging processing on the second intermediate data to obtain standard model data.

This embodiment can be applied to a variety of scenarios, for example, a commodity on an e-commerce platform, its suppliers may include manufacturers, agents, supermarkets, third-party sellers, etc. The method of this embodiment is converted into standard model data, so that the data of multiple models is integrated, which is convenient for downstream e-commerce platforms to use; for example, the user data of different e-commerce modules such as clothing, fresh food, and take-out are different according to their respective It can be converted into standard model data by the method of this embodiment, which is convenient for downstream payment platforms to use; another example: there may be multiple teams such as big data department, model algorithm department, program development department and so on within the enterprise. Different apps or different sections of the app extract data such as users, orders, behavioral operations, etc., and perform user portrait construction and user behavior analysis according to different models, which can be converted into standard model data by the method of this embodiment, so that the operation department can follow the standard model. Data to formulate corresponding marketing strategies.

Based on the above description, the original model in step S11 may be a model for generating various original model data, such as the data model of each supplier, the data model of each e-commerce module, and the user data analysis model of each team. The original model data generated by each original model can be one or more relational data tables, where each column is an attribute of the data, identified by a specific field, and each row is a tuple.

The first intermediate data obtained in step S12 may also be a relational data table. When merging the original model data, you can follow certain merging rules, such as specifying the priority between the original models, specifying the data customization format of each field, specifying the valid fields in the original model data to cut out unnecessary data columns, Specify the reordering field in the original model data to remove duplicate tuples (data rows), specify filter conditions to customize the filtering of the original model data, and use the field index of the first intermediate data and the original model data to facilitate data review, etc., Corresponding merging rules may be formulated according to different application scenarios and subsequent data requirements, which are not particularly limited in this embodiment.

The first intermediate data can be used as a reserve database, which is all the combined data. Subsequent applications may not require all the data, and extraction conditions may be set, and the required data, that is, the second intermediate data, is extracted from the first intermediate data through step S13. In step S13, usually in units of tuples, after specifying the extraction conditions of the target field, the tuples that meet the conditions are extracted, for example, commodity data meeting the price range conditions, user data meeting the membership level conditions, etc. are extracted.

The packaging processing in step S14 refers to improving or optimizing the second intermediate data to meet the requirements of subsequent applications, which may specifically include sorting the second intermediate data, adding labels, and presetting the null value of the second intermediate data. Filling, etc., are not particularly limited in this embodiment.

In the above method, on the one hand, the original model data of each original model is converted into standard model data through merging, extraction and packaging processing, thereby realizing the fusion and intercommunication of different model data, and improving the convenience of subsequent applications. On the other hand, this embodiment forms a standardized processing flow for multi-model data, and can formulate corresponding merging, extracting and packaging processing rules according to the characteristics of the original model or the customized requirements of subsequent applications, so that this embodiment has a strong Universality.

In order to realize automatic processing of multi-model data, trigger conditions can be set for each step in Figure 1. In an exemplary embodiment, step S11 may include the following steps to achieve automated acquisition of original model data:

Periodically traverse the filenames of the original model data files sent by multiple original models on the file storage platform.

Match the file name with the download history, and parse the file name if the match is unsuccessful, and match the parsed file name with a preset whitelist.

If the parsed file name matches the preset whitelist successfully, download the original model data file from the file storage platform.

Extract raw model data from the downloaded raw model data file.

Among them, the file storage platform can be a database, container, etc. other than the background server. Each original model can periodically send original model data files to the file storage platform. The original model data files can usually store original model data according to time periods, and generate models containing The filename for information such as name, data type, data generation time, etc. The file name of the downloaded file is stored in the download history. If the match is unsuccessful, the file has not been downloaded. The preset whitelist refers to the pre-configured file name information matching conditions to prevent other types of files from being downloaded by mistake, and the matching conditions such as the time period and type of the files can be set in the preset whitelist to further filter out the required files. Original model data file. When extracting the original model data from the original model data file, certain filtering can also be performed, such as extracting the original model data of a specific field, extracting the original model data of a specific number, and so on.

The above process can be referred to as shown in Figure 2. The original model sends the original model data file to the file storage platform. The background server can automatically traverse the file name on the file storage platform every 30 minutes. If the default whitelist matches, the file can be downloaded. Of course, the number of primitive models and the period of 30 minutes in the figure are only exemplary. According to actual requirements, any number of primitive models and periods of any duration can be set, which are not particularly limited in this embodiment.

In an exemplary embodiment, step S12 may include the following steps to realize automatic triggering of original model data merging:

In the current cycle, check whether all the original model data of the current cycle have been acquired.

If it is detected that all the original model data of the current cycle has not been obtained, it is checked again after the first interval time whether all the original model data of the current cycle has been obtained.

If it is detected that all the original model data of the current cycle have been acquired, the original model data of the current cycle are merged to obtain the first intermediate data.

The cycle refers to the time interval during which the model data merging task is performed, which can usually be an integer multiple of the original model data sending cycle. For example, if the original model sends data once a day, the model data merging task cycle can be one day, two days, or three days. Wait. The first interval time may be regarded as an interval time for periodically starting the model data merging task. In this embodiment, when all the original model data of the current cycle are acquired, the merging of the original model data is triggered. It should be noted that, in each cycle, each original model may send original model data more than once, so it is necessary to check whether all data sent by all original models has been obtained. You can check the original model data file name by checking the original model. The number of data files, etc.

Further, in an exemplary embodiment, the multi-model data processing method may further include the following steps:

If it is detected that all the original model data of the current cycle has not been obtained, it is determined whether the model data merging task activates the multi-batch mode.

If it is determined that the multi-batch mode is not activated by the model data merging task, the step of detecting again whether all the original model data of the current cycle has been acquired after the first predetermined time is performed.

If it is determined that the model data merging task activates the multi-batch mode, in any batch of the multi-batch mode, check whether the original model corresponding to the original model data obtained in the current batch includes all the original models.

If it is detected that the original model corresponding to the original model data obtained in the current batch does not include all the original models, after the second interval time, it is detected again whether the original model corresponding to the original model data obtained in the current batch includes all the original models.

If it is detected that the original model corresponding to the original model data obtained in the current batch includes all original models, the original model data obtained in the current batch is merged to obtain the first intermediate data.

Among them, the multi-batch mode refers to merging the original model data in multiple batches within a cycle. For example, when the cycle is one day, the original model data can be merged once in the morning, afternoon, and evening, respectively, to merge the model data. Refinement to reduce the amount of data processed per task. The condition for triggering the data merging of original models within each batch is that all original models are in place within the batch, that is, each original model has sent data at least once to avoid missing data for a certain original model. The second interval time and the first interval time may be the same or different.

In addition, when the downstream application platform urgently needs the standard model data, or there may be a situation where a certain original model cannot send data normally, in order to trigger the original model data merge in time, the forced merge mode can be set in the model data merge task. . In an exemplary embodiment, the multi-model data processing method may further include the following steps:

After detecting that the original model corresponding to the original model data obtained in the current batch does not include all original models, check whether the model data merging task activates the forced merging mode;

If it is detected that the model data merging task has activated the forced merging mode, merge the acquired original model data at a predetermined time in the forced merging mode to obtain the first intermediate data;

If it is detected that the forced merging mode is not activated in the model data merging task, the step of detecting again whether the original model corresponding to the original model data obtained in the current batch includes all the original models is performed after the second interval.

The forced merging mode includes a predetermined time for performing the forced merging. The predetermined time may be set according to the data demand time of the application side, or the predetermined time may be set near the end of each cycle.

Based on the above embodiment, FIG. 3 shows a schematic flowchart of the execution of a model data merging task. The model data merging task is executed and started periodically. First, it is detected whether all the original model data has been acquired. If all the original model data has been acquired, the original data merging is triggered. , if not all are acquired, further check whether the multi-batch mode is activated. When the multi-batch mode is not activated, the above steps need to be repeated after a certain interval; when the multi-batch mode is activated, it is necessary to check whether all the original models are in place. The acquired original model data triggers the merge; if not, continue to detect whether the forced merge mode is activated. In the case of activation, continue to detect whether the predetermined time has been reached. If the predetermined time is reached, regardless of whether the acquired original model data is complete and whether all original models are in order, the original model data will be forced to merge; if the predetermined time is not reached Or if the forced merge mode is not activated, you need to repeat the above steps after a certain interval. It can be seen that the original model data can be merged in three ways: all the original model data of the current cycle has been acquired, the multi-batch mode is activated and all original models in the current batch have been completed, and the forced merge mode is activated and the predetermined time is reached. Therefore, corresponding model data merging strategies can be formulated according to different actual situations to meet the diverse needs of downstream application platforms.

In an exemplary embodiment, referring to FIG. 4 , combining the original model data to obtain the first intermediate data may include the following steps:

Step S41, acquiring a merging rule, where the merging rule includes a field mapping rule.

Step S42, according to the field mapping rule, perform column merging of the original model data of the same type of field, and assign standard fields to each column to obtain the first intermediate data.

Further, the merging rules may further include at least one of field filtering rules, model priority rules, format conversion rules, reset field calculation rules, and filtering conditions. After merging the original model data of the same type of field and assigning standard fields to each column, the original model data after column merging and assigning standard fields to each column can also be merged through at least one of the following steps. :

Step S43, remove the field specified by the field filter rule.

Step S44, according to the model priority rule, when there are multiple original models in the same type field and the original model data with the same primary index are different, the original model data of the original model with the highest priority is retained.

Step S45, according to the format conversion rule, format conversion is performed on the original model data of each standard field according to the customized format of each standard field.

Step S46, according to the reset field calculation rule, perform conversion calculation on the original model data of the reset field.

Step S47, remove the original model data that meets the filtering conditions.

Among them, the reset field calculation rule means that the calculation method of a certain field in each original model is different, and a new calculation method can be determined for the first intermediate data after synthesizing the calculation methods of each original model; the filter condition can be for a certain field Or some fields filter out tuples of specific values, for example, filter out the original model data of "user type" = "VIP", etc.

It should be noted that, according to actual application requirements, the merging rules may include all the above merging rules, or only a part of them, and more merging rule conditions may be set, or the above merging rules may be changed, etc. For example, referring to FIG. 5, a reordering field rule and a tracking field rule can be added to the merge rule. The reordering field rule means that the reordering field in the original model data can be determined to remove duplicate tuples (data rows); tracking The field rule means that when multiple original models contain numbers, the number of a specific original model can be specified as the number of the first intermediate data. This embodiment does not specifically limit the specific content of the merging rule.

Referring to Figure 6, in practical applications, merging rules can be mapped to customized data structures. For example, model priority rules {column,{modelid1,modelid2,...}} can be mapped to {String,PairIndex<Integer,Integer> }; Tracking field rules {"ext1":",|column,column,…","ext2":",|column",…} can be mapped to {BiConsumer<Integer,Map<String,Object>>,BiConsumer< Map<String,Object>,Map<String,Object>>}; reset field calculation rules {"2":"column,...,"1":"column2,column2,..."} can be mapped to {BiConsumer<Map <String, Object>, Map<String, Object>>, Consumer<Map<String, Object>>}, etc., so that various operations in the merging rules can be performed on the original model data to realize data merging.

In an exemplary embodiment, when the original model data is user data, each tuple of the merged first intermediate data is a summary of each user's basic information data, behavior data, analysis data, etc., and the target field can be Including the user's latest active time and extraction state, then step S13 may include the following steps:

Extraction conditions are extracted from application tasks, and extraction conditions include extraction time, activity period and non-repetitive extraction.

When the extraction time arrives, the tuples whose active time is within the active period and whose state is not extracted are extracted from the first intermediate data to obtain second intermediate data.

Wherein, the application task is obtained from the downstream application platform, and when the downstream application platform conducts the marketing activity, the user who is active in the activity period may be selected as the activity object. In this embodiment, referring to FIG. 5 above, the extraction condition actually includes two parts of conditions: the extraction condition of the target field and the extraction state condition. The extraction state can be a new field generated in the model data merging stage, and the initial value is "unextracted". In the data extraction stage, if a tuple is extracted, the field value of the tuple is changed to "extracted" . The extraction time can be the time when the activity starts or the time when the activity is prepared. Non-repetitive extraction means that users who have already made marketing pushes will not be extracted during the activity cycle.

The above data extraction process can be referred to as shown in Figure 7. After starting the data extraction task, since the original model data acquisition and original model data merging of the current cycle may not be completed, the first intermediate data of the previous cycle can be obtained and passed. Whether the active time is within the active period, and whether the tuple is not extracted are two judgment steps to determine which tuples are extracted, so that the second intermediate data can be obtained. In other cases, the range of extracted data is not limited to the first intermediate data of the previous period, and data may also be extracted from the first intermediate data of an earlier period or multiple historical periods, which is not particularly limited in this embodiment. Based on the data extraction task, a part of the data can be extracted from the first intermediate data with a huge amount of data according to the needs of the downstream application platform, and subsequent processing can be performed to finally obtain the standard model data required by the downstream application platform, thereby further reducing the entire process. Due to the amount of data processing, the configuration of standard model data does not need to be completely consistent with the original model data, which is more flexible.

Since the first intermediate data usually has a huge amount of data, in order to facilitate storage and invocation, in an exemplary embodiment, referring to FIG. 6 above, the multi-model data processing method may further include the following steps: after obtaining the first intermediate data Afterwards, the first intermediate data is segmented into multiple first intermediate data sub-tables according to the multiple index fields of the first intermediate data, and are respectively stored in multiple containers. For example, a distinct operation may be performed on the index field of the first intermediate data, and the first intermediate data may be fragmented by conditions such as column and group by to obtain each first intermediate data sub-table. Correspondingly, step S13 can be realized by the following steps: according to the extraction conditions, obtain the target first intermediate data sub-table from a plurality of containers, and extract the tuples whose target fields meet the extraction conditions from the target first intermediate data sub-table, to obtain the target first intermediate data sub-table. Obtain second intermediate data. Therefore, when performing data extraction, it is not necessary to call the entire first intermediate data, but only one of the sub-tables needs to be called to perform extraction and packaging processing. The packaging rules followed by the packaging process can be referred to as shown in Figure 5 above, which usually include data sorting rules, data label adding rules, null value filling rules, etc. Of course, other types of packaging rules can also be added according to actual needs. This is not particularly limited.

Based on the above-mentioned first intermediate data fragmentation and distributed storage method, the amount of data can be further reduced, and the processing efficiency can be improved. And by slicing and storing the first intermediate data in a distributed manner, it is beneficial to the expansion of the entire system, reducing the limit on the number of upstream original models, and realizing the fusion and intercommunication of massive model data.

Further, the container for sharded storage of the first intermediate data may be Docker. Docker is an application container engine. Compared with other container technologies, the main difference is that Docker can bundle application components in a single container, enabling Docker to achieve portability between different platforms and computing systems, and better parallel processing. Fragmentation rules for the first intermediate data and extraction conditions for the second intermediate data. In addition, Docker has a second-level startup rate, which can improve the efficiency of the system running multi-model data processing processes, and Docker has high resource utilization, supporting thousands of Docker containers running on one server at the same time, so it can further save system resources. .

Exemplary embodiments of the present disclosure also provide a multi-model data processing system. FIG. 8 shows a schematic diagram of the operating environment architecture of the system. Referring to FIG. 8 , the system 80 may include: a rule customization module 81 for generating merging rules, extraction conditions and packaging rules according to the rule configuration file; a data processing module 82 for merging original model data according to the merging rules , obtain the first intermediate data, extract the second intermediate data from the first intermediate data according to the extraction conditions, and perform packaging processing on the second intermediate data according to the packaging rules to obtain standard model data; the data storage module 83 is used to obtain The original model data is stored, and the first intermediate data, the second intermediate data and the standard model data are respectively stored. The upstream original model can send data to the data storage module 83, and the data storage module 83 stores the original model data in a file storage platform, which can be a built-in unit of the data storage module 83, or an external storage platform The rule configuration file can be manually configured rule text or script, and the rule customization module 81 can convert it into merge rules, extraction conditions and packaging rules suitable for system operation; the data processing module 82 can obtain the above-mentioned rules from the rule customization module , the original model data on the file storage platform is merged, extracted and packaged in turn, and finally the standard model data is obtained, which can be distributed to the downstream application platform for subsequent use. Through the multi-model data processing system 80, a standardized process for multi-model data processing can be formed, and any number of models or platforms can be set for the upstream original model or the downstream application platform, thereby facilitating the expansion of the system, which can be added according to actual needs The original model or application platform makes the system have strong versatility.

In an exemplary embodiment, the data storage module 83 may include: a file storage platform for periodically acquiring original model data files sent by a plurality of original models, and matching the file names of the original model data files with the download history records. , parse the file name when the matching is unsuccessful, match the parsed file name with the preset whitelist, and extract the original model from the original model data file whose parsed file name successfully matches the preset whitelist data.

In an exemplary embodiment, the data processing module 82 may include: a merging component for detecting in the current cycle whether all the original model data of the current cycle has been acquired, when it is detected that all the original model data of the current cycle has not been acquired; , after the first interval time, check again whether all the original model data of the current cycle has been obtained, and when it is detected that all the original model data of the current cycle have been obtained, combine the original model data of the current cycle to obtain the first intermediate data.

In an exemplary embodiment, the merging component can also be used to judge whether the model data merging task activates the multi-batch mode when it is detected that all the original model data of the current cycle has not been obtained, and when it is judged that the model data merging task has not activated the multi-batch mode. In batch mode, the step of re-detecting whether all the original model data of the current cycle has been obtained after the first predetermined time is performed. When it is determined that the model data merging task activates the multi-batch mode, any batch of In the second interval, check whether the original model corresponding to the original model data obtained in the current batch includes all original models, and when it is detected that the original model corresponding to the original model data obtained in the current batch does not include all original models After time, check again whether the original model corresponding to the original model data obtained in the current batch includes all original models, and when it is detected that the original model corresponding to the original model data obtained in the current batch includes all original models, the current batch The original model data obtained within the data is merged to obtain the first intermediate data.

In an exemplary embodiment, the merging component can also be used to detect whether the model data merging task activates the forced merging mode after detecting that the original model corresponding to the original model data obtained in the current batch does not include all original models, When the model data merging task has activated the forced merging mode, merge the acquired original model data at a predetermined time in the forced merging mode to obtain the first intermediate data, and when it is detected that the model data merging task has not activated the forced merging mode, execute The step of detecting again whether the original model corresponding to the original model data obtained in the current batch includes all original models after the second interval time.

In an exemplary embodiment, the rule customization module 81 can also be used to obtain merging rules, and the merging rules include field mapping rules; the merging component can also be used to perform column merging of the original model data of the same type of fields according to the field mapping rules, And assign standard fields to each column to get the first intermediate data.

In an exemplary embodiment, the merging rule may further include at least one of a field screening rule, a model priority rule, a format conversion rule, a reset field calculation rule, and a filter condition; At least one of the original model data after column merging and allocating standard fields to each column is merged to obtain the first intermediate data: remove the fields specified by the field filtering rules; according to the model priority rules, when there are multiple original models When the original model data of the same type of field and the same main index are different, the original model data of the original model with the highest priority is retained; according to the format conversion rules, the original model data of each standard field is formatted according to the customized format of each standard field. ;According to the reset field calculation rules, transform and calculate the original model data of the reset field; remove the original model data that meets the filter conditions.

In an exemplary embodiment, the data processing module 82 may include: an extraction component for extracting extraction conditions including extraction time, activity period and non-repetitive extraction from the application task, and when the extraction time is reached, extracting the extraction conditions from the first intermediate Extracting tuples whose active time is within the active period and whose status is not extracted from the data, to obtain second intermediate data.

In an exemplary embodiment, the data storage module 83 may also be configured to, after obtaining the first intermediate data, segment the first intermediate data into multiple first intermediate data sub-tables according to multiple index fields of the first intermediate data. , and store them in multiple containers respectively; the extraction component can also be used to obtain the target first intermediate data sub-table from multiple containers according to the extraction conditions, and extract the target field from the target first intermediate data sub-table to meet the extraction conditions tuple to obtain the second intermediate data.

Exemplary embodiments of the present disclosure also provide a multi-model data processing apparatus, as shown in FIG. 9 , the apparatus 90 may include: an original acquisition unit 91 for acquiring original model data from multiple original models; a data merging unit 92, for merging the original model data to obtain the first intermediate data; data extraction unit 93, for extracting the target field from the first intermediate data according to the extraction conditions for the target field of the first intermediate data to satisfy the extraction A tuple of conditions to obtain the second intermediate data; the data packaging unit 94 is configured to perform packaging processing on the second intermediate data to obtain the standard model data.

The original model may be a model for generating various original model data, such as the data model of each upstream supplier, the data model of each e-commerce module, the user data analysis model of each team, and the like. The data generated by the original model, that is, the original model data, is acquired from the original model by the original acquisition unit 91 . The data merging unit 92 can follow certain merging rules when merging the original model data, such as specifying the priority between the original models, specifying the data customization format of each field, and specifying the valid fields in the original model data to cut out unnecessary data. , specify the reordering field in the original model data to remove duplicate tuples (data rows), etc. Corresponding merging rules can be formulated according to different application scenarios and subsequent data requirements. The data extraction unit 93 usually uses a tuple as a unit, and can extract qualified tuples according to the extraction conditions of the target field, such as extracting commodity data that meets price range conditions, and extracting user data that meets membership level conditions. The data packaging unit 94 may perform packaging processing on the second intermediate data based on certain packaging rules, such as sorting the second intermediate data, adding tags, and pre-filling null values of the second intermediate data.

The multi-model data processing apparatus of this exemplary embodiment can realize the fusion and intercommunication of different model data, improve the convenience of subsequent data application, and form a standardized processing flow of multi-model data to meet the customization requirements of model data.

In an exemplary embodiment, the original acquisition unit 91 may include: a file name matching subunit, configured to periodically traverse the file names of the original model data files sent by a plurality of original models on the file storage platform, and match the file names with the downloaded files. The history records are matched, and when the match is unsuccessful, the file name is parsed, and the parsed file name is matched with the preset whitelist; the file download subunit is used when the parsed file name is compared with the preset whitelist. When the matching is successful, the original model data file is downloaded from the file storage platform; the data extraction subunit is used to extract the original model data from the downloaded original model data file.

In an exemplary embodiment, the data merging unit 92 may include: a full-quantity detection sub-unit for detecting in the current cycle whether all the original model data of the current cycle has been acquired, and when it is detected that all the original model data of the current cycle have not been acquired. When the model data is used, check again whether all the original model data of the current cycle has been obtained after the first interval time; the merge processing sub-unit is used to combine the original model data of the current cycle when it is detected that all the original model data of the current cycle have been obtained. The data are combined to obtain the first intermediate data.

In an exemplary embodiment, the full detection subunit can also be used to, when it is detected that all original model data of the current cycle has not been obtained, determine whether the model data merging task activates the multi-batch mode, and when it is determined that the model data merging task is to activate the multi-batch mode When the multi-batch mode is not activated, the step of detecting again whether all the original model data of the current cycle has been acquired is performed after the first predetermined time. In any batch, check whether the original model corresponding to the original model data obtained in the current batch includes all original models, and when it is detected that the original model corresponding to the original model data obtained in the current batch does not include all original models, After the second interval time, check again whether the original model corresponding to the original model data obtained in the current batch includes all original models; When all original models are available, the original model data obtained in the current batch is merged to obtain the first intermediate data.

In an exemplary embodiment, the full detection subunit may also be used to detect whether the model data merging task activates the forced merging mode after detecting that the original model corresponding to the original model data obtained in the current batch does not include all original models, And when it is detected that the model data merging task does not activate the forced merging mode, the step of detecting again whether the original model corresponding to the original model data obtained in the current batch includes all the original models after the second interval time is performed; the merging processing subunit also It can be used when the model data merging task has activated the forced merging mode, and at a predetermined time in the forced merging mode, the acquired original model data is merged to obtain the first intermediate data.

In an exemplary embodiment, the data merging unit 92 may also be configured to obtain merging rules including field mapping rules, and perform column merging of original model data of the same type of fields according to the field mapping rules, and assign standard fields to each column. , to obtain the first intermediate data.

In an exemplary embodiment, the merging rules may further include at least one of field screening rules, model priority rules, format conversion rules, reset field calculation rules, and filter conditions; the data merging unit may also be used to pass the following steps: At least one of the original model data after column merging and allocating standard fields to each column is merged to obtain the first intermediate data: remove the fields specified by the field filtering rules; according to the model priority rules, when there are multiple original model data When the original model data of the same type of field and the same primary index is different, the original model data of the original model with the highest priority is retained; according to the format conversion rules, the original model data of each standard field is formatted according to the customized format of each standard field Transform: According to the reset field calculation rules, transform and calculate the original model data of the reset field; remove the original model data that meets the filter conditions.

In an exemplary embodiment, the target field may include the most recent active time and extraction status; the data extraction unit may also be used to extract extraction conditions including extraction time, activity period and non-repetitive extraction from the application task, and the extraction time at the arrival of extraction time. When , extract the tuples whose active time is within the active period and whose status is not extracted from the first intermediate data to obtain the second intermediate data.

In an exemplary embodiment, the multi-model data processing apparatus 90 may further include: a sharding storage unit (not shown in the figure), configured to, after obtaining the first intermediate data, store data according to a plurality of index fields of the first intermediate data Divide the first intermediate data into multiple first intermediate data sub-tables and store them in multiple containers respectively; the data extraction unit can also be used to obtain the target first intermediate data sub-tables from multiple containers according to extraction conditions , and extract the tuples whose target fields satisfy the extraction conditions from the target first intermediate data sub-table to obtain the second intermediate data.

Exemplary embodiments of the present disclosure also provide an electronic device capable of implementing the above method.

As will be appreciated by one skilled in the art, various aspects of the present disclosure may be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be embodied in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which may be collectively referred to herein as implementations "circuit", "module" or "system".

An electronic device 1000 according to such an exemplary embodiment of the present disclosure is described below with reference to FIG. 10 . The electronic device 1000 shown in FIG. 10 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 10, electronic device 1000 takes the form of a general-purpose computing device. Components of the electronic device 1000 may include, but are not limited to: the above-mentioned at least one processing unit 1010 , the above-mentioned at least one storage unit 1020 , a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010 ), and a display unit 1040 .

The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Methods" section of this specification. For example, the processing unit 1010 may execute steps S11 to S14 shown in FIG. 1 , and may also execute steps S41 to S47 shown in FIG. 4 , and so on.

The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 1021 and/or a cache storage unit 1022 , and may further include a read only storage unit (ROM) 1023 .

The storage unit 1020 may also include a program/utility 1024 having a set (at least one) of program modules 1025 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, An implementation of a network environment may be included in each or some combination of these examples.

The bus 1030 may be representative of one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local area using any of a variety of bus structures bus.

The electronic device 1000 may also communicate with one or more external devices 1200 (eg, keyboards, pointing devices, Bluetooth devices, etc.), with one or more devices that enable a user to interact with the electronic device 1000, and/or with Any device (eg, router, modem, etc.) that enables the electronic device 1000 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interface 1050 . Also, the electronic device 1000 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through the network adapter 1060 . As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030 . It should be appreciated that, although not shown, other hardware and/or software modules may be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives and data backup storage systems.

From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein may be implemented by software, or may be implemented by software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to cause a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to an exemplary embodiment of the present disclosure.

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-described method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code, when the program product runs on a terminal device, the program code is used to cause the terminal device to execute the above-mentioned procedures in this specification. Steps according to various exemplary embodiments of the present disclosure are described in the "Example Methods" section.

Referring to FIG. 11 , a program product 1100 for implementing the above method is described according to an exemplary embodiment of the present disclosure, which can adopt a portable compact disk read only memory (CD-ROM) and include program codes, and can be stored in a terminal devices such as personal computers. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal in baseband or as part of a carrier wave with readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium can also be any readable medium, other than a readable storage medium, that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural programming Language - such as the "C" language or similar programming language. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (eg, using an Internet service provider business via an Internet connection).

In addition, the above-mentioned figures are merely schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, in multiple modules.

It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

Other embodiments of the present disclosure will readily suggest themselves to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (13)

1. a multi-model data processing method, is characterized in that, comprises: Obtain original model data from multiple original models; combining the original model data to obtain first intermediate data; According to the extraction condition for the target field of the first intermediate data, extract the tuple whose target field satisfies the extraction condition from the first intermediate data to obtain the second intermediate data; The second intermediate data is packaged to obtain standard model data. 2. The method according to claim 1, wherein the obtaining original model data from a plurality of original models comprises: Periodically traverse the file names of the original model data files sent by the multiple original models on the file storage platform; Matching the file name with the download history record, and parsing the file name when the matching is unsuccessful, and matching the parsed file name with a preset whitelist; If the parsed file name matches the preset whitelist successfully, downloading the original model data file from the file storage platform; The original model data is extracted from the downloaded original model data file. 3. The method according to claim 1, wherein the combining the original model data to obtain the first intermediate data comprises: In the current cycle, detecting whether all the original model data of the current cycle have been acquired; If it is detected that all the original model data of the current cycle has not been obtained, then after the first interval time, it is detected again whether all the original model data of the current cycle have been obtained; If it is detected that all the original model data of the current cycle have been acquired, the original model data of the current cycle are combined to obtain the first intermediate data. 4. The method according to claim 3, wherein the method further comprises: If it is detected that all the original model data of the current cycle has not been obtained, determine whether the model data merging task activates the multi-batch mode; If it is determined that the multi-batch mode is not activated by the model data merging task, performing the step of re-detecting whether all the original model data of the current cycle has been acquired after a first predetermined time; If it is determined that the multi-batch mode is activated by the model data merging task, in any batch of the multi-batch mode, it is detected whether the original model corresponding to the original model data obtained in the current batch includes all the original models. Model; If it is detected that the original model corresponding to the original model data obtained in the current batch does not include all the original models, then after the second interval time, it is detected again whether the original model corresponding to the original model data obtained in the current batch includes the original model. all said original models; If it is detected that the original model corresponding to the original model data obtained in the current batch includes all the original models, the original model data obtained in the current batch is merged to obtain the first intermediate data. 5. The method according to claim 4, wherein the method further comprises: After detecting that the original model corresponding to the original model data obtained in the current batch does not include all the original models, detecting whether the model data merging task activates the forced merging mode; If it is detected that the model data merging task has activated a forced merging mode, combining the acquired original model data at a predetermined time in the forced merging mode to obtain the first intermediate data; If it is detected that the forced merging mode is not activated in the model data merging task, the step of detecting again whether the original model corresponding to the original model data obtained in the current batch includes all the original models is performed after the second interval. 6. The method according to claim 1, wherein the combining the original model data to obtain the first intermediate data comprises: obtaining a merging rule, where the merging rule includes a field mapping rule; According to the field mapping rule, the original model data of the same type of field is column-merged, and standard fields are allocated to each column to obtain the first intermediate data. 7. The method according to claim 6, wherein the merging rule further comprises at least one of a field screening rule, a model priority rule, a format conversion rule, a reset field calculation rule and a filter condition; After merging the original model data of the same type of field and assigning standard fields to each column, the method further includes: The original model data after column merging and allocating standard fields to each column are merged by at least one of the following steps: remove the field specified by the field filter rule; According to the model priority rule, when there are multiple original models with different original model data in the same type field and the same primary index, the original model data of the original model with the highest priority is retained; According to the format conversion rule, format conversion is performed on the original model data of each of the standard fields according to the customized format of each of the standard fields; According to the reset field calculation rule, transform and calculate the original model data of the reset field; Remove raw model data that matches the filter criteria. 8. The method according to claim 1, wherein the target field comprises the latest active time and extraction state; The extracting, according to the extraction condition for the target field of the first intermediate data, the tuple of which the target field satisfies the extraction condition is extracted from the first intermediate data to obtain the second intermediate data including: Extracting the extraction condition from the application task, the extraction condition includes extraction time, activity period and non-repetitive extraction; When the extraction time is reached, a tuple whose active time is within the active period and whose extraction state is unextracted is extracted from the first intermediate data to obtain the second intermediate data. 9. The method of claim 1, wherein the method further comprises: After obtaining the first intermediate data, fragment the first intermediate data into multiple first intermediate data sub-tables according to multiple index fields of the first intermediate data, and store them in multiple containers respectively; The extracting, according to the extraction condition for the target field of the first intermediate data, the tuple of which the target field satisfies the extraction condition is extracted from the first intermediate data to obtain the second intermediate data including: According to the extraction condition, the target first intermediate data sub-table is obtained from the plurality of containers, and the tuple whose target field satisfies the extraction condition is extracted from the target first intermediate data sub-table, so as to obtain the second intermediate data. 10. A multi-model data processing system, comprising: The rule customization module is used to generate merge rules, extraction conditions and packaging rules according to the rule configuration file; a data processing module for merging the original model data according to the merging rules to obtain first intermediate data, extracting second intermediate data from the first intermediate data according to the extraction conditions, and extracting the second intermediate data from the first intermediate data according to the extraction conditions, and performing packaging processing on the second intermediate data to obtain standard model data; A data storage module, configured to acquire and store the original model data, and store the first intermediate data, the second intermediate data and the standard model data respectively. 11. A multi-model data processing device, comprising: an original acquisition unit for acquiring original model data from multiple original models; a data merging unit for merging the original model data to obtain first intermediate data; A data extraction unit, configured to extract, from the first intermediate data, a tuple whose target field satisfies the extraction condition according to an extraction condition for a target field of the first intermediate data, to obtain second intermediate data ; A data packaging unit, configured to perform packaging processing on the second intermediate data to obtain standard model data. 12. An electronic device, characterized in that, comprising: processor; and a memory for storing executable instructions for the processor; wherein the processor is configured to perform the method of any of claims 1-9 by executing the executable instructions. 13. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the method of any one of claims 1-9 is implemented.

Images