[go: up one dir, main page]

CN115168301A - Data processing method and device, storage medium and electronic equipment - Google Patents

Data processing method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN115168301A
CN115168301A CN202210724567.8A CN202210724567A CN115168301A CN 115168301 A CN115168301 A CN 115168301A CN 202210724567 A CN202210724567 A CN 202210724567A CN 115168301 A CN115168301 A CN 115168301A
Authority
CN
China
Prior art keywords
uploaded
files
data
file
target task
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210724567.8A
Other languages
Chinese (zh)
Inventor
孔令涛
蒋宁
王洪斌
郝征鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mashang Consumer Finance Co Ltd
Original Assignee
Mashang Consumer Finance Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mashang Consumer Finance Co Ltd filed Critical Mashang Consumer Finance Co Ltd
Priority to CN202210724567.8A priority Critical patent/CN115168301A/en
Publication of CN115168301A publication Critical patent/CN115168301A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提供了一种数据处理方法、装置、存储介质及电子设备,涉及数据处理领域。该数据处理方法包括:基于目标任务的文件依赖关系数据,确定目标任务对应的M个待上传文件以及M个待上传文件的文件依赖关系数据,M为正整数;基于M个待上传文件的文件依赖关系数据,打包上传M个待上传文件。本申请中的方案充分利用了目标任务中的文件之间的依赖关系打包待上传文件,从而为后续根据需要将每个待上传文件打包上传到不同的存储空间,进而降低存储集中、并扩展待上传文件对应的打包数据的存储方式提供前提条件。

Figure 202210724567

The present application provides a data processing method, device, storage medium and electronic device, and relates to the field of data processing. The data processing method includes: determining M files to be uploaded corresponding to the target task and file dependency data of the M files to be uploaded based on the file dependency data of the target task, where M is a positive integer; based on the files of the M files to be uploaded Dependency data, package and upload M files to be uploaded. The solution in this application makes full use of the dependencies between the files in the target task to package the files to be uploaded, so that each file to be uploaded is packaged and uploaded to different storage spaces for subsequent needs, thereby reducing the storage concentration and expanding the Prerequisites are provided for the storage method of the packaged data corresponding to the uploaded file.

Figure 202210724567

Description

Data processing method, data processing device, storage medium and electronic equipment
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, a storage medium, and an electronic device.
Background
In the application development process or the big data storage process, the problem of packing and storing related data according to a pre-constructed target task is often encountered. In the related art, all extracted data are generally mechanically packaged and uploaded to corresponding storage spaces according to a pre-constructed target task, however, the problem that data storage is too centralized and the storage method is single is caused.
Disclosure of Invention
The present application is proposed to solve the above-mentioned technical problems. The embodiment of the application provides a data processing method, a data processing device, a storage medium and electronic equipment.
In a first aspect, an embodiment of the present application provides a data processing method, including: determining M files to be uploaded corresponding to the target task and file dependency relationship data of the M files to be uploaded based on the file dependency relationship data of the target task, wherein M is a positive integer; and packaging and uploading the M files to be uploaded based on the file dependency relationship data of the M files to be uploaded.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including: the determining unit is used for determining M files to be uploaded corresponding to the target task and file dependency relationship data of the M files to be uploaded based on the file dependency relationship data of the target task, wherein M is a positive integer; and the packaging uploading unit is used for packaging and uploading the M files to be uploaded based on the file dependency relationship data of the M files to be uploaded.
In a third aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored, and the computer program is used to execute the data processing method according to the first aspect.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor; a memory for storing processor-executable instructions; the processor is configured to perform the data processing method of the first aspect.
In the embodiment of the application, the M files to be uploaded corresponding to the target task and the file dependency relationship data of the M files to be uploaded are determined, and the M files to be uploaded are packaged and uploaded according to the M file dependency relationship data. According to the scheme in the embodiment of the application, the dependency relationship among the files in the target task is fully utilized to package the files to be uploaded, so that the packaged data corresponding to each file to be uploaded is uploaded to different storage spaces according to the requirement subsequently, and the precondition is provided for reducing the storage concentration. In addition, the storage mode of the packed data corresponding to the file to be uploaded can be expanded by the packing uploading mode, for example, the packed data corresponding to the file to be uploaded can be stored in a common distributed database, and the packed data corresponding to the file to be uploaded can also be stored in a decentralized storage network.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps.
Fig. 1 is a schematic diagram of an implementation environment of a data processing method according to an embodiment of the present application.
Fig. 2 is a schematic view of an application scenario of a data processing method according to an exemplary embodiment of the present application.
Fig. 3 is a schematic flow chart of a data processing method according to an exemplary embodiment of the present application.
Fig. 4 is a schematic diagram illustrating file dependency data according to an exemplary embodiment of the present application.
Fig. 5 is a schematic flowchart illustrating a process of packaging and uploading M files to be uploaded according to an exemplary embodiment of the present application.
Fig. 6 is a schematic flow chart illustrating uploading of packaged M files to be uploaded to a decentralized storage network according to an exemplary embodiment of the present application.
Fig. 7 is a schematic flowchart illustrating a process of determining M files to be uploaded corresponding to a target task according to an exemplary embodiment of the present application.
Fig. 8 is a schematic flowchart illustrating a process of determining M files to be uploaded corresponding to a target task according to another exemplary embodiment of the present application.
Fig. 9 is a schematic structural diagram illustrating generation of M files to be uploaded according to an exemplary embodiment of the present application.
Fig. 10 is a flowchart illustrating a process of determining M files to be uploaded according to another exemplary embodiment of the present application.
Fig. 11 is a schematic flowchart illustrating a process of packaging and uploading M files to be uploaded according to another exemplary embodiment of the present application.
Fig. 12 is a schematic flowchart illustrating a data processing method according to another exemplary embodiment of the present application.
Fig. 13 is a schematic structural diagram illustrating persistent integration of a web application according to an exemplary embodiment of the present application.
FIG. 14 is a diagram illustrating file dependencies regarding a target task of a web application according to an exemplary embodiment of the present application.
Fig. 15 is a schematic structural diagram of a data processing apparatus according to an exemplary embodiment of the present application.
Fig. 16 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Decentralization is a social relationship form and a content generation form formed in the internet development process, and is a novel network content production process relative to centralization. Decentralization does not require a hub, but a hub and a discretionary hub are freely selected by nodes, and in a decentralization system, anyone is a node and can also be a hub.
Continuous integration is a DevOps software development practice. With continuous integration, developers will periodically incorporate code changes into a central repository, after which the system will automatically run build and test operations. Continuous integration generally refers to the construction or integration phase of a software release process, and requires the use of automation components and culture components. The primary goal of continued integration is to discover and resolve defects faster, improve software quality, and reduce the time required to validate and release new software updates.
In software development and application development, or in consumption data storage and financial data storage, a target task is generally constructed according to related projects, and information to be stored is packaged and uploaded to a corresponding storage space at one time according to the target task.
For example, in the application development process, continuous integration tasks are often constructed one by one, and when the continuous integration tasks are executed, developers generally submit codes to a code library, a server pulls the codes from the code library through tools such as jenkins and downloads corresponding configuration files, and then, all the codes are packaged and uploaded to a test environment or a production environment through the configuration files. For another example, for a consumption data storage problem, a consumption data storage task is often constructed, consumption data is pulled from a corresponding platform according to the consumption data storage task, and all the pulled data is packaged and uploaded to a corresponding storage space.
However, with the above method, the dependency relationship between data is not well utilized, which leads to a limitation in solving the problem. For example, the uploading of all data packaged together may result in problems such as too centralized data storage and single storage mode. Based on this, the embodiment of the application provides a data processing method, which can determine a file to be uploaded and a dependency relationship between the files to be uploaded according to file dependency relationship data of a target task, and further upload the file to be uploaded based on the file dependency relationship between the files to be uploaded. The data storage method and the data storage device solve the problem that the dependency relationship among the data cannot be utilized for packaging and uploading in the prior art, and enrich the data storage mode.
The data processing method provided by the embodiment of the application can be executed by an electronic device, and the electronic device can be a terminal, such as a smart phone, a tablet computer, a desktop computer, an intelligent voice interaction device or a vehicle-mounted terminal; or the electronic device may also be a server, such as a separate physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.
Based on the data processing method provided by the embodiment of the application, the embodiment of the application provides an environment schematic diagram of the embodiment of the data processing method, and the electronic device is assumed to be a server. Referring to fig. 1, a schematic diagram of an implementation environment of a data processing method according to an embodiment of the present application is provided. As shown in fig. 1, the present embodiment environment includes a terminal 110 and a server 120, and a communication connection between the terminal 110 and the server 120.
The terminal 110 may be a smartphone, a tablet, a desktop computer, etc. The terminal 110 is installed with an application program that can drive the server 120 to perform a data processing method, for example, the application program may be a browser application, a shopping application, a video application, and the like. The servers 120 may be physical machines or virtual machines, and the number of the servers may be one or more, and the type and the number of the servers are not limited in this embodiment of the application.
Illustratively, according to the target task, the application driver server 120 installed in the terminal 110 starts to work, and the server 120 determines M files to be uploaded corresponding to the target task and file dependency data of the M files to be uploaded based on the file dependency data of the target task, and packages and uploads the M files to be uploaded based on the file dependency data of the M files to be uploaded.
Fig. 2 is a schematic view of an application scenario of a data processing method according to an exemplary embodiment of the present application. As shown in fig. 2, the task execution server cluster includes m task execution servers, which are respectively a task execution server 1, a task execution server 2, \\8230 \8230:, and the task execution server m may select any one task execution server in the task execution server cluster to execute all the packaged uploading jobs according to the target task, or may select a plurality of task execution servers to execute all the packaged uploading jobs together.
Specifically, the developer submits the code to a code database. The user logs on the persistent integration system using the browser application to create a persistent integration task. The created content includes: the address of the code database of the application, and the account number of the test network or the formal network.
Further, the persistent integration system is operated according to the browser application, and the created persistent integration task is executed. The continuous integration system drives the task server cluster to pull corresponding codes from the code database according to the target tasks, download the dependency packages corresponding to the target tasks, determine M files to be uploaded corresponding to the target tasks according to the dependency packages, determine file dependency relationship data among the M files to be uploaded, and package and upload the M files to be uploaded to a test network or a formal network according to the file dependency relationship data.
Based on the above, refer to fig. 3, which is a schematic flow chart of a data processing method according to an exemplary embodiment of the present application. As shown in fig. 3, the data processing method provided in the embodiment of the present application includes the following steps:
step S310, determining M files to be uploaded corresponding to the target task and file dependency data of the M files to be uploaded based on the file dependency data of the target task. Wherein M is a positive integer.
The target task may be a continuous integration task with respect to a certain item, or a consumption statistics task for a quarter of a week by a consumer on a certain shopping platform, or a user transfer calculation task, etc. The type of the target task is not specifically limited in the embodiment of the application, as long as the file to be uploaded in the target task has a file dependency relationship.
The dependency relationship is a relationship between data expressed by whether values between attributes in a relationship are equal or not, and belongs to the inherent property of data. In computer science, data dependencies are states when a program structure causes data to reference previously processed data.
Fig. 4 is a schematic diagram illustrating file dependency data provided in an exemplary embodiment of the present application. As shown in fig. 4, the target task includes 5 files to be uploaded, which are file a, file B, file C, file D, and file E, respectively, and the file dependency relationship data of the target task includes: file B depends on files D and E, and file a depends on files B and C.
Further, the M files to be uploaded corresponding to the target task may be all files included in the target task, or may be partial files included in the target task.
And step S320, packaging and uploading the M files to be uploaded based on the file dependency relationship data of the M files to be uploaded.
Continuing with the example of fig. 4 in step S310, according to the file dependency relationship data among the file a, the file B, the file C, the file D, and the file E, the file D, the file E, and the file C are first packaged and uploaded, respectively; secondly, packaging and uploading the file B; and finally, packaging and uploading the file A.
In the embodiment of the application, the M files to be uploaded corresponding to the target task and the file dependency relationship data of the M files to be uploaded are determined, and the M files to be uploaded are packaged and uploaded according to the M file dependency relationship data to be uploaded. According to the scheme in the embodiment of the application, the dependency relationship among the files in the target task is fully utilized, and the files to be uploaded are packaged in batches, so that the packaged data corresponding to each file to be uploaded can be uploaded to different storage spaces according to needs in the following process, and the storage concentration is reduced. In addition, the storage mode of the packed data corresponding to the file to be uploaded can be expanded by the packing uploading mode, for example, the packed data corresponding to the file to be uploaded can be stored in a common distributed database, and the packed data corresponding to the file to be uploaded can also be stored in a decentralized storage network.
Fig. 5 is a schematic flowchart illustrating a process of packaging and uploading M files to be uploaded according to an exemplary embodiment of the present application. The embodiment shown in fig. 5 is extended based on the embodiment shown in fig. 3, and the differences between the embodiment shown in fig. 5 and the embodiment shown in fig. 3 will be emphasized below, and the descriptions of the same parts will not be repeated.
As shown in fig. 5, the file dependency relationship data includes tree file dependency relationship data, the tree file dependency relationship data includes M nodes, the M nodes are deployed at different levels, and each node corresponds to one file to be uploaded corresponding to the target task; parallel relation and dependency relation exist among the M nodes, and dependency relation exists among the files to be uploaded corresponding to the nodes with the dependency relation; the method comprises the following steps of packaging and uploading M files to be uploaded based on file dependency relationship data of the M files to be uploaded, and comprises the following steps:
step S510, based on the tree file dependency relationship data, sequentially traversing the nodes belonging to the lowest hierarchy among the M nodes as traversal start nodes according to a hierarchy sequence from low to high, packaging the files to be uploaded corresponding to the traversed nodes, and uploading the packaged M files to be uploaded to a decentralized storage network.
The nodes in the tree file dependency relationship data can be divided into n levels, and if n is greater than or equal to 3, the nodes in the tree file dependency relationship data can be divided into leaf nodes, father nodes and root nodes; if n is equal to 2, the nodes in the tree file dependency relationship data can be divided into leaf nodes and root nodes; and if n is equal to 1, the node in the tree file dependency relationship data is a root node.
When n is greater than or equal to 2, if all the nodes in the level a are leaf nodes, the level a is the lowest level, and the leaf nodes in the level a are the nodes of the lowest level. When n is equal to 1, the hierarchy where the root node is located is the lowest hierarchy, and the root node is the node in the lowest hierarchy.
Further, when n is greater than or equal to 2, the hierarchy order from all leaf nodes to the root node is from low to high. Illustratively, the hierarchies from the hierarchy which is all leaf nodes to the hierarchy where the root node is located are sequentially defined as the hierarchy 1 to the hierarchy n, firstly, the leaf nodes in the hierarchy 1 are used as traversal starting nodes according to the file dependency relationship data of M files to be uploaded, the files to be uploaded which correspond to the leaf nodes in the hierarchy 1 are packaged and uploaded to the decentralized storage network, secondly, the files to be uploaded which correspond to the nodes in the hierarchy 2 are packaged and uploaded to the decentralized storage network, and finally the files to be uploaded which correspond to the nodes in the hierarchy n are packaged and uploaded to the decentralized storage network.
Along the example in fig. 4, the node where the file D and the file E are located is a leaf node, the level where the file E is located is level 1, that is, the lowest level, the node where the file B is located is a parent node, the node where the file C is located is a leaf node, the level where the file B and the file C are located is level 2, the node where the file a is located is a root node, and the level where the file a is located is level 3, that is, the highest level. According to the file dependency relationship data of the file A, the file B, the file C, the file D and the file E, packaging and uploading the files in three steps, and respectively packaging and uploading the file D, the file E and the file C in the first step; secondly, packaging and uploading a file B based on the packaging and uploading results of the file D and the file E; and thirdly, packaging and uploading the file A according to the packaging and uploading results of the file B and the file C.
According to the scheme in the embodiment, the ordering of the M files to be uploaded in the packaging and uploading process is ensured by relying on the file dependency relationship data of the M files to be uploaded. In addition, the M files to be uploaded are uploaded to the decentralized storage network, so that the safety of the packed data corresponding to the files to be uploaded is enhanced, and the storage cost of the packed data corresponding to the files to be uploaded is reduced.
Fig. 6 is a schematic flowchart illustrating a process of uploading packaged M files to be uploaded to a decentralized storage network according to an exemplary embodiment of the present application. The embodiment shown in fig. 6 is extended based on the embodiment shown in fig. 5, and the differences between the embodiment shown in fig. 6 and the embodiment shown in fig. 5 will be emphasized below, and the descriptions of the same parts will not be repeated.
As shown in fig. 6, based on the tree-type file dependency relationship data, a node belonging to the lowest hierarchy among the M nodes is taken as a traversal start node, the traversal is sequentially performed according to a hierarchy order from bottom to top, the files to be uploaded corresponding to the traversed node are packaged, and the packaged M files to be uploaded are uploaded to a decentralized storage network, which includes the following steps:
and step S610, based on the tree file dependency relationship data, packaging and uploading the file to be uploaded corresponding to the node belonging to the lowest level to a decentralized storage network, and obtaining the evidence information of the file to be uploaded corresponding to the node of the lowest level.
Illustratively, the pass information includes token information. In addition, while the files to be uploaded corresponding to the nodes of the lowest hierarchy are packaged and uploaded, the files to be uploaded corresponding to leaf nodes contained in other non-lowest hierarchies can also be packaged and uploaded together.
Along the embodiment shown in fig. 4, the file D and the file E in the node of the lowest hierarchy are first packaged and uploaded to the decentralized storage network, and the token about the file D and the token about the file E returned by the decentralized storage network are obtained. Meanwhile, for the convenience of subsequent packaging and uploading, when the files D and E in the nodes of the lowest hierarchy are packaged and uploaded, the files C corresponding to the leaf nodes in the hierarchy 2 can also be packaged and uploaded to the decentralized storage network, and the token about the files C returned by the decentralized storage network is obtained.
Step S620, determine whether there is a next lower level node having a dependency relationship with the lowest level node among the M nodes.
Illustratively, in the actual application process, if the determination result in step S620 is no, that is, no next-lower-level node having a dependency relationship with the lowest-level node exists in the M nodes, step S630 is executed, and if the determination result in step S620 is yes, that is, a next-lower-level node having a dependency relationship with the lowest-level node exists in the M nodes, steps S640 and S650 are executed.
And step S630, finishing packaging and uploading.
And step S640, based on the certification information of the file to be uploaded corresponding to the node at the lowest level, packaging and uploading the file to be uploaded corresponding to the node at the next lower level to a decentralized storage network, and obtaining the certification information of the file to be uploaded corresponding to the node at the next lower level.
Specifically, the token of the file to be uploaded corresponding to the node of the lowest hierarchy and the file to be uploaded of the parent node in the node of the next lower hierarchy corresponding to the node of the lowest hierarchy are packaged and uploaded to the decentralized storage network together, so that the token corresponding to the file to be uploaded of each parent node in the node of the next lower hierarchy is obtained.
For example, according to the embodiment shown in fig. 4, the lowest level is level 1 where the file D and the file E are located, the next lowest level is level 2 where the file B and the file C are located, the token corresponding to the file D, the token corresponding to the file E, and the file B in the parent node corresponding to the file D and the file E are packaged together, and the packaged result is uploaded to the decentralized storage network, so that the token corresponding to the file B is obtained. The file C is used as a file on a leaf node, and when the file D and the file E are packaged and uploaded, the file C in the level 2 is also packaged and uploaded to a decentralized storage network, and the token corresponding to the file C is obtained.
And step S650, continuously traversing, packaging and uploading the files to be uploaded based on the file dependency relationship data of the M files to be uploaded and the corresponding certification information of the files to be uploaded corresponding to the nodes of the next lower level until all the files contained in the M files to be uploaded are packaged and uploaded to the decentralized storage network.
Specifically, tokens corresponding to the files to be uploaded of the nodes in the hierarchy b and files to be uploaded of the parent nodes in the hierarchy (b + 1) corresponding to the nodes in the hierarchy b are packaged and uploaded to a decentralized storage network, and tokens {2, \8230; (n-1) } corresponding to the files to be uploaded of the parent nodes in the hierarchy (b + 1) are obtained. Taking fig. 4 as an example, the token corresponding to the file B, the token corresponding to the file C, and the file a are packaged and uploaded to a decentralized storage network, and the token corresponding to the file a is obtained.
According to the technical scheme, the files to be uploaded in the nodes are packaged in a grading mode, and dependence between the files to be uploaded of the nodes in different levels is strengthened on the basis of the corresponding certification information of the files to be uploaded of the nodes.
Fig. 7 is a schematic flowchart illustrating a process of determining M files to be uploaded corresponding to a target task according to an exemplary embodiment of the present application. As shown in fig. 7, determining M files to be uploaded corresponding to a target task based on file dependency data of the target task includes the following steps:
and step S710, determining historical packaged uploading data corresponding to the target task.
The history packed upload data refers to data stored in relation to the target task, and the history packed upload data of the target task can be pulled from a storage space corresponding to a pre-specified target task.
The storage mode of the history package uploading data is also based on the file dependency data of the target task. The historical packed uploading data records files corresponding to nodes of each hierarchy, tokens of the files corresponding to the nodes, packed data of the files corresponding to the nodes, and dependency relationships among the nodes.
Step S720, based on the historical packaged uploading data and the file dependency relationship data of the target task, M files to be uploaded corresponding to the target task are determined.
The M files to be uploaded can be all files contained in the target task or partial files contained in the target task.
In the embodiment of the application, M files to be uploaded of the target task are finally determined through the historical packaged uploading data corresponding to the target task, namely, the files contained in the target task are subjected to incremental or full-volume packaged uploading according to the historical packaged uploading data, and the data storage amount is further reduced.
Fig. 8 is a schematic flowchart illustrating a process of determining M files to be uploaded corresponding to a target task according to another exemplary embodiment of the present application. On the basis of the embodiment shown in fig. 7, the embodiment shown in fig. 8 is extended, and the differences between the embodiment shown in fig. 8 and the embodiment shown in fig. 7 are mainly described below, and the descriptions of the same parts are omitted.
As shown in fig. 8, determining M files to be uploaded corresponding to a target task based on history package upload data and file dependency data of the target task includes the following steps:
step S810, determining whether the history package uploaded data is empty.
Specifically, if the history packaged upload data is empty, that is, there is no history packaged data related to the target task, that is, the packaged upload work of the file to be uploaded related to the target task has not been performed before. The historical packaged uploading data is not empty, namely the historical packaged data related to the target task exists, namely the packaged uploading work of the file to be uploaded related to the target task is performed before.
Illustratively, if the execution result of step S810 is yes, that is, the history packed data is empty, step S820 is executed, and if the execution result of step S810 is no, that is, the history packed data is not empty, step S830 is executed.
Step 820, based on the file dependency relationship data of the target task, generating M files to be uploaded.
Step S830, based on the file dependency relationship data of the target task and the data related to the target task and contained in the history packaging uploading data, M files to be uploaded are determined.
According to the scheme in the embodiment, M files to be uploaded of the target task are determined according to the historical packaged uploaded data of the target task, so that repeated storage of the files to be uploaded in the target task is avoided, and the data storage capacity is reduced.
In an exemplary embodiment of the present application, the file dependency relationship data corresponds to N initial files, and based on the file dependency relationship data of the target task, M files to be uploaded are generated, including: and generating M files to be uploaded based on the N initial files, wherein N is equal to M.
Fig. 9 is a schematic structural diagram illustrating generation of M files to be uploaded according to an exemplary embodiment of the present application. The following describes embodiments of the present application in detail, taking fig. 4 and 9 as examples.
FIG. 4 shows that the target task contains 5 initial files, file A, file B, file C, file D, file E, and the dependency relationships among the 5 initial files. If the historical packaged uploading data about the target task is empty, the 5 initial files and the dependency relationships among the 5 initial files shown in fig. 4 are respectively used as the 5 files to be uploaded of the target task and the dependency relationships among the 5 files to be uploaded, the 5 files to be uploaded are respectively packaged and uploaded according to the file dependency relationship data shown in fig. 4, and the packaged and uploaded version is the V1.0 version shown in fig. 9.
By the technical scheme in the embodiment, when the files to be uploaded of the target task are packaged and uploaded for the first time, the complete storage versions of all the files included by the target task can be obtained, and a basis is provided for subsequent incremental packaging and uploading.
Fig. 10 is a flowchart illustrating a process of determining M files to be uploaded according to another exemplary embodiment of the present application. On the basis of the embodiment shown in fig. 8, the embodiment shown in fig. 10 is extended, and the differences between the embodiment shown in fig. 10 and the embodiment shown in fig. 8 will be emphasized below, and the descriptions of the same parts will not be repeated.
As shown in fig. 10, the determining of M files to be uploaded based on the file dependency relationship data of the target task and the data related to the target task and included in the history package upload data, where the file dependency relationship data corresponds to N initial files, includes the following steps:
step S1010, based on the history packaged uploaded data, determining P initial files different from the file corresponding to the history packaged uploaded data from among the N initial files.
Taking fig. 9 as an example, the history packaged upload data includes the V1.0 version shown in fig. 9, for the target task, the target task is split into the V2.0 version shown in fig. 9, and by comparison, only the file D in the V2.0 version is changed relative to the file D in the V1.0 version, so that the file D is determined to be an initial file different from the history packaged upload data.
Step S1020, based on the file dependency relationship data, determining Q initial files having dependency relationships with the P initial files, and determining the P initial files and the Q initial files as M files to be uploaded.
Taking fig. 9 as an example, the file D and the file B have a dependency, and the file B and the file a have a dependency, so the file D, the file B, and the file a are files to be uploaded.
In this embodiment, a dependency of a history version dimension is added to a traditional file dependency relationship, that is, when a history version exists, data of a corresponding node in the history version is preferentially depended, so that the packaging and uploading cost is saved, and the amount of uploaded data is further reduced.
If the historically packaged uploading data is not empty, based on the certification information of the file to be uploaded corresponding to the node at the lowest level and the certification information of the file to be uploaded of the brother node corresponding to the node at the lowest level in the historically packaged uploading data, packaging and uploading the file to be uploaded corresponding to the node at the next lower level to the decentralized storage network, obtaining the certification information of the file to be uploaded corresponding to the node at the next lower level, and continuously traversing, packaging and uploading until the M files to be uploaded are packaged and uploaded to the decentralized storage network.
Taking fig. 9 as an example, the file D in the V2.0 version is an initial file different from the historical packed upload data (i.e., the file D in the V1.0 version), the file D in the V2.0 version is packed and uploaded to the decentralized storage network to obtain a token corresponding to the file D, the token corresponding to the file E in the V1.0 version, the token corresponding to the file D in the V2.0 version, and the file B in the V2.0 version are packed and uploaded to the decentralized storage network together to obtain a token corresponding to the file B, and finally the token corresponding to the file C in the V1.0 version, the token corresponding to the file B in the V2.0 version, and the file a in the V2.0 version are packed and uploaded to the decentralized storage network together to obtain a token corresponding to the file a.
Fig. 11 is a flowchart illustrating a flow chart of packaging and uploading M files to be uploaded according to another exemplary embodiment of the present application. The embodiment shown in fig. 11 is extended on the basis of the embodiment shown in fig. 5, and the differences between the embodiment shown in fig. 11 and the embodiment shown in fig. 5 will be mainly described below, and the descriptions of the same parts will not be repeated.
As shown in fig. 11, the method for uploading M files to be uploaded in a packaged manner based on the file dependency relationship data of the M files to be uploaded further includes the following steps:
step S1110, for each node in the M nodes, after the file to be uploaded corresponding to the node is packaged and uploaded to the centralized storage network, the file to be uploaded corresponding to the node is packaged and uploaded to the storage space corresponding to the target task.
The storage space may be a local storage space, and for the files in the leaf nodes, the files to be uploaded in the leaf nodes are packaged and uploaded to the local storage space corresponding to the target task. And for the files in the non-leaf nodes, packaging and uploading the files of the current node and the tokens of the files of the child nodes corresponding to the current node together to a local storage space so as to pull historical packaged uploading data of the target task from the local storage space, and determining M files to be uploaded according to the historical packaged uploading data.
And step S1120, packaging and uploading the file to be uploaded corresponding to the node to a database.
The database can be any database, and for the file to be uploaded in each node, the file to be uploaded is packaged and uploaded to a decentralized storage network and a storage space respectively, and then the file to be uploaded in each node is packaged and uploaded to the database. Specifically, for the file to be uploaded of each node, the file to be uploaded of the node, the token corresponding to the node, and the storage address of the file to be uploaded corresponding to the node in the storage space are packaged and uploaded to the database.
In the embodiment of the application, the file to be uploaded corresponding to the node is packaged and uploaded to the storage space corresponding to the target task, so that the difference between the file to be uploaded corresponding to the current target task and the historical packaged uploading data about the target task stored in the storage space can be compared conveniently when the target task is subsequently packaged and uploaded for a non-first time. And the files to be uploaded corresponding to the nodes are packaged and uploaded to the database, so that the files can be conveniently and quickly inquired in time when the data of the target task is used next time.
Fig. 12 is a schematic flowchart illustrating a data processing method according to another exemplary embodiment of the present application. The embodiment shown in fig. 12 is extended on the basis of the embodiment shown in fig. 3, and the differences between the embodiment shown in fig. 12 and the embodiment shown in fig. 3 will be mainly described below, and the descriptions of the same parts will not be repeated.
As shown in fig. 12, the method further includes the following steps after the file to be uploaded includes the object code and the M files to be uploaded are packaged and uploaded based on the file dependency relationship data of the M files to be uploaded.
Step 1210, testing the target code corresponding to the target task to obtain a test result.
And step S1220, deploying the target code according to the test result.
Specifically, the target tasks include code pulling, code packaging, code uploading, and the like.
Illustratively, the project codes are pulled from the code database according to the continuous integration task, corresponding configuration files are downloaded, the continuous integration task is divided into a plurality of files according to the configuration files, and M files to be uploaded corresponding to the continuous integration task and file dependency relationship data of the M files to be uploaded are determined. And then, based on the file dependency relationship data of the M files to be uploaded, target codes corresponding to the M files to be uploaded are packaged and uploaded, continuous integration test is carried out on the target codes, and the target codes are deployed according to the test result.
Continuing with the embodiment shown in fig. 4 as an example, if the history packaged upload data about the continuously integrated target task is empty, first, the dependency information corresponding to the file D and the file D in the lowest hierarchy and the dependency information corresponding to the file E and the file E are packaged and uploaded to the decentralized storage network, to obtain tokens corresponding to the file D and the file E respectively, and at the same time, the file C corresponding to the leaf node in another hierarchy and the dependency information corresponding to the file C may be packaged and uploaded to the decentralized storage network, and the token corresponding to the file C is obtained.
Further, packing and uploading the token corresponding to the file D, the token corresponding to the file E, and the dependency information corresponding to the file B and the file B to a decentralized storage network, and acquiring the token corresponding to the file B. And finally, packing and uploading the token corresponding to the file B, the token corresponding to the file C, the file A and the dependency information corresponding to the file A to a decentralized storage network to obtain the token corresponding to the file A. After the test is finished, the user operates the continuous integrated management system through the client application to deploy the target code of the file to be uploaded corresponding to the target task to the production environment.
Through the scheme in the embodiment, a circulating flow of circulating packaging, deployment and integration can be realized. In addition, the packaging and deployment process is disassembled into a plurality of times through the combing of the file dependency relationship of the target task, and the packaging, deployment and integration of the plurality of times can be automatically processed.
Furthermore, with the increasing maturity of blockchain technology, decentralized storage is the technological base for web3.0, gradually falling to the ground, which makes decentralized web applications possible. In the related art, the continuous integration flow about the web application is as follows: developers develop web applications and then submit the code to code libraries, such as github, gitlab, etc. Developers pull codes from a code library through tools such as jenkins and the like, download dependencies, pack the codes through configuration files, and deploy the codes to a test environment or a production environment. Developers continue to refine the functionality of web applications and repeat the above-described continuous integration process. The above-described continuous integration flow for web applications, however, does not solve the dependency problem well.
Such as data dependency problems. A plurality of pages in a web application are composed of a page template and a plurality of data corresponding to the page template, the data are directly stored in a database in a web2.0 scene, but the data are stored in a decentralized block chain network in a web3.0 scene to generate metadata, and then the pages generate tokens corresponding to the data through the decentralized block chain network. However, in the packing and uploading stage, data is not uploaded to the decentralized block chain network, a token used for querying the data is not generated, and a method for packing the data is not available, so that in the development process, the data needs to be manually submitted to the decentralized block chain network to obtain the corresponding token, and then the token is updated to a page.
For example, sub-page dependency problem, in the web3.0 scenario, a single page may be cast as a Non-homogeneous token (NFT) creating a unique token to identify the location of the page in the decentralized storage network. If the sub-page is not stored in the decentralized storage network, the corresponding token does not exist, and the sub-page cannot be called through the token in the main page. In the development process in the related technology, the sub-page is firstly tested, then the sub-page is manually submitted to a decentralized block chain network, a corresponding token is obtained, and then the link of the sub-page is written on the main page. Of course, the sub-pages can be linked to form a tree-shaped connection relationship. Such development process is cumbersome, not conducive to application modification and maintenance, and cumbersome to develop and test.
Therefore, how to better utilize the dependency problem in the web application to package and upload the file to be uploaded is particularly important.
Fig. 13 is a schematic structural diagram illustrating persistent integration of a web application according to an exemplary embodiment of the present application. FIG. 14 is a diagram illustrating file dependencies regarding a target task of a web application according to an exemplary embodiment of the present application. The method for continuously integrating the web application is described in detail below by taking fig. 13 and 14 as an example.
First, receiving a target task sent by a persistent integration management system, pulling a project code of a web application from a code library by a task execution server according to the target task, downloading a dependency package configured in package of the project from npm, and splitting the target task according to the dependency package, as shown in fig. 14, if history package upload data of the persistent integrated target task of the web application is empty, the target task is split into three steps.
The method comprises the steps of firstly, packaging a related dependency file corresponding to data E and data E, a related dependency file corresponding to page D and page D, and a related dependency file corresponding to page C and page C, respectively obtaining first package data corresponding to the data E, the page D and the page C, uploading the first package data corresponding to the data E, the page D and the page C to a decentralized storage network, and obtaining tokens corresponding to the data E, the page D and the page C. And uploading the first package data corresponding to the data E, the page D and the page C to a local storage space. And packaging the data E, the token corresponding to the data E and the storage address of the first packed data corresponding to the data E in the local storage space to obtain second packed data of the data E, and uploading the second packed data to the database. Correspondingly, the page D, the token corresponding to the page D and the storage address of the first packed data corresponding to the page D in the local storage space are packed together to obtain second packed data of the page D, and the second packed data are uploaded to the database. And packaging the page C, the token corresponding to the page C and the storage address of the first packed data corresponding to the page C in the local storage space to obtain second packed data of the page C, and uploading the second packed data to the database.
And secondly, packing the token corresponding to the data E, the token corresponding to the page D and the related dependency files corresponding to the page B and the page B together to obtain first package data related to the page B, uploading the first package data of the page B to a decentralized storage network, and obtaining the token corresponding to the page B. And uploading the first package data corresponding to the page B to a local storage space. And packaging the page B, the token corresponding to the page B and the storage address of the first packed data corresponding to the page B in the local storage space to obtain second packed data of the page B, and uploading the second packed data to a database.
And thirdly, packing the token corresponding to the page B, the token corresponding to the page C, the page A and the related dependency file of the page A together to obtain first package data related to the page A, uploading the first package data of the page A to a decentralized storage network, and obtaining the token corresponding to the page A. And uploading the first packed data corresponding to the page A to a local storage space. And packing the page A, the token corresponding to the page A and the storage address of the first packed data corresponding to the page A in the local storage space to obtain second packed data of the page A, and uploading the second packed data to a database.
Further, the first package data corresponding to the page a, the page B, the page C, the page D, and the data E may be uploaded through an api (application programming interface) provided by the decentralized storage network such as IPFS and AR, and uploaded to the decentralized storage network, and a token returned by the decentralized storage network is obtained. The decentralized storage network comprises IPFS and AR testing environments or formal environments.
Further, if it is the first step shown in fig. 14, it is only necessary to pack directly without replacing the dependent path inside the file. If it is the second and third steps shown in fig. 14 that are packaged, then we need to replace the opposite path in the web application with a plug-in through webpack as the path in the decentralized storage network.
Method embodiments of the present application are described in detail above in conjunction with fig. 1-14, and apparatus embodiments of the present application are described in detail below in conjunction with fig. 15. It is to be understood that the description of the method embodiments corresponds to the description of the apparatus embodiments, and therefore reference may be made to the preceding method embodiments for parts not described in detail.
Fig. 15 is a schematic structural diagram of a data processing apparatus according to an exemplary embodiment of the present application. As shown in fig. 15, the data processing apparatus provided in the embodiment of the present application includes:
the determining unit 1510 is configured to determine, based on the file dependency relationship data of the target task, M files to be uploaded corresponding to the target task and file dependency relationship data of the M files to be uploaded, where M is a positive integer;
and the packaging and uploading unit 1520 is configured to package and upload the M files to be uploaded based on the file dependency relationship data of the M files to be uploaded.
In an embodiment of the present application, the packaging and uploading unit 1520 is further configured to, based on the tree-type file dependency relationship data, take a node belonging to a lowest hierarchy among the M nodes as a traversal starting node, perform traversal once according to a hierarchy sequence from low to high, package files to be uploaded corresponding to the traversed node, and upload the packaged M files to be uploaded to the decentralized storage network.
In an embodiment of the present application, the packaging and uploading unit 1520 is further configured to package and upload the to-be-uploaded file corresponding to the node belonging to the lowest level to a decentralized storage network based on the tree-type file dependency relationship data, so as to obtain the certification information of the to-be-uploaded file corresponding to the node of the lowest level; if a sub-low level node which has a dependency relationship with the node of the lowest level exists in the M nodes, packaging and uploading the file to be uploaded corresponding to the sub-low level node to a decentralized storage network based on the certification information of the file to be uploaded corresponding to the node of the lowest level, and obtaining the certification information of the file to be uploaded corresponding to the node of the sub-low level; and continuously traversing, packaging and uploading the files to be uploaded based on the tree file dependency relationship data and the corresponding certification information of the files to be uploaded of the nodes of the next lower level until M files to be uploaded are packaged and uploaded to the decentralized storage network.
In an embodiment of the present application, the determining unit 1510 is further configured to determine historical package upload data corresponding to the target task; and determining M files to be uploaded corresponding to the target task based on the historical packed uploading data and the file dependency relationship data of the target task.
In an embodiment of the present application, the determining unit 1510 is further configured to generate M files to be uploaded based on the file dependency relationship data of the target task if the history package upload data is empty; and if the historical packaging uploading data is not empty, determining M files to be uploaded based on the file dependency relationship data of the target task and the data which is contained in the historical packaging uploading data and is related to the target task.
In an embodiment of the present application, the determining unit 1510 is further configured to generate M files to be uploaded based on the N initial files, where N is equal to M.
In an embodiment of the present application, the determining unit 1510 is further configured to determine, based on the history packaged upload data, P initial files, different from the file corresponding to the history packaged upload data, from among the N initial files; and determining Q initial files having a dependency relationship with the P initial files based on the file dependency relationship data, and determining the P initial files and the Q initial files as M files to be uploaded.
In an embodiment of the present application, the packaging and uploading unit 1520 is further configured to, for each node in the M nodes, package and upload the file to be uploaded corresponding to the node to a centralized storage network, and then package and upload the file to be uploaded corresponding to the node to a storage space corresponding to a target task; and packaging and uploading the files to be uploaded corresponding to the nodes to a database.
In an embodiment of the present application, the packaging uploading unit 1520 is further configured to test a target code corresponding to a target task to obtain a test result; and deploying the target code according to the test result.
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 16. Fig. 16 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
As shown in fig. 16, the electronic device 160 includes one or more processors 1601 and memory 1602.
The processor 1601 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 160 to perform desired functions.
Memory 1602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 1601 to implement the methods of the various embodiments of the present application described above and/or other desired functions. Various contents such as file dependency relationship data including a target task, M files to be uploaded, and packaged data corresponding to the M files to be uploaded can also be stored in the computer-readable storage medium.
In one example, the electronic device 160 may further include: an input device 1603 and an output device 1604, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input device 1603 may include, for example, a keyboard, a mouse, and the like.
The output device 1604 may output various information to the outside, including the file dependency data of the target task, the M files to be uploaded, and the packed data corresponding to the M files to be uploaded, and so on. The output devices 1604 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 160 relevant to the present application are shown in fig. 16, and components such as buses, input/output interfaces, and the like are omitted. In addition, electronic device 160 may include any other suitable components, depending on the particular application.
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods of the various embodiments of the present application mentioned above in this specification.
The computer program product may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages, for carrying out operations according to embodiments of the present application. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the methods according to the various embodiments of the present application described above in this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by one skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims (12)

1.一种数据处理方法,其特征在于,包括:1. a data processing method, is characterized in that, comprises: 基于目标任务的文件依赖关系数据,确定所述目标任务对应的M个待上传文件以及所述M个待上传文件的文件依赖关系数据,M为正整数;Based on the file dependency data of the target task, determine the M files to be uploaded corresponding to the target task and the file dependency data of the M files to be uploaded, where M is a positive integer; 基于所述M个待上传文件的文件依赖关系数据,打包上传所述M个待上传文件。Based on the file dependency relationship data of the M files to be uploaded, the M files to be uploaded are packaged and uploaded. 2.根据权利要求1所述的数据处理方法,其特征在于,所述文件依赖关系数据包括树型文件依赖关系数据,所述树型文件依赖关系数据中包括M个节点,所述M个节点部署在不同层级,每个所述节点对应所述目标任务对应的一个待上传文件;所述M个节点之间存在并列关系和从属关系,存在所述从属关系的节点对应的待上传文件之间存在依赖关系;2. The data processing method according to claim 1, wherein the file dependency relationship data comprises tree-type file dependency relationship data, wherein the tree-type file dependency relationship data includes M nodes, and the M nodes Deployed at different levels, each of the nodes corresponds to a file to be uploaded corresponding to the target task; there is a parallel relationship and a subordinate relationship between the M nodes, and the files to be uploaded corresponding to the nodes with the subordinate relationship exist there are dependencies; 所述基于所述M个待上传文件的文件依赖关系数据,打包上传所述M个待上传文件,包括:The described M files to be uploaded are packaged and uploaded based on the file dependency data of the M files to be uploaded, including: 基于所述树型文件依赖关系数据,以所述M个节点中属于最低层级的节点为遍历起始节点,按照由低到高的层级顺序依次遍历、将遍历到的所述节点对应的待上传文件进行打包,并上传打包后的M个待上传文件至去中心化存储网络。Based on the tree-type file dependency relationship data, take the node belonging to the lowest level among the M nodes as the starting node for traversal, traverse in order from low to high level, and traverse the traversed node corresponding to the to-be-uploaded node The files are packaged, and the packaged M files to be uploaded are uploaded to the decentralized storage network. 3.根据权利要求2所述的数据处理方法,其特征在于,所述基于所述树型文件依赖关系数据,以所述M个节点中属于最低层级的节点为遍历起始节点,按照由低到高的层级顺序依次遍历、将遍历到的所述节点对应的待上传文件进行打包,并上传打包后的M个待上传文件至去中心化存储网络,包括:3. The data processing method according to claim 2, characterized in that, based on the tree-type file dependency relationship data, the node belonging to the lowest level in the M nodes is used as a traversal start node, according to Traversing sequentially to the highest level, packaging the files to be uploaded corresponding to the traversed nodes, and uploading the packaged M files to be uploaded to the decentralized storage network, including: 基于所述树型文件依赖关系数据,将属于所述最低层级的节点对应的待上传文件打包上传至所述去中心化存储网络,得到所述最低层级的节点对应的待上传文件的通证信息;Based on the tree-type file dependency relationship data, package and upload the files to be uploaded corresponding to the nodes belonging to the lowest level to the decentralized storage network, and obtain the token information of the files to be uploaded corresponding to the nodes of the lowest level ; 若所述M个节点中存在、与所述最低层级的节点具有依赖关系的次低层级的节点,则基于所述最低层级的节点对应的待上传文件的通证信息,将所述次低层级的节点对应的待上传文件打包上传至所述去中心化存储网络,得到所述次低层级的节点对应的待上传文件的通证信息;If there is a next-low-level node in the M nodes that has a dependency relationship with the lowest-level node, based on the token information of the file to be uploaded corresponding to the lowest-level node, the next-low-level node The to-be-uploaded file corresponding to the node is packaged and uploaded to the decentralized storage network, and the token information of the to-be-uploaded file corresponding to the next lower-level node is obtained; 基于所述树型文件依赖关系数据、以及所述次低层级的节点对应的待上传文件的通证信息,继续遍历打包上传,直至将所述M个待上传文件打包上传至所述去中心化存储网络。Based on the tree-type file dependency data and the token information of the files to be uploaded corresponding to the nodes of the next lower level, continue to traverse the package and upload until the M files to be uploaded are packaged and uploaded to the decentralized storage network. 4.根据权利要求1至3任一项所述的数据处理方法,其特征在于,所述基于目标任务的文件依赖关系数据,确定所述目标任务对应的M个待上传文件,包括:4. The data processing method according to any one of claims 1 to 3, wherein the file dependency data based on the target task determines M files to be uploaded corresponding to the target task, comprising: 确定所述目标任务对应的历史打包上传数据;Determine the historical package upload data corresponding to the target task; 基于所述历史打包上传数据和所述目标任务的文件依赖关系数据,确定所述目标任务对应的M个待上传文件。Based on the historical package upload data and the file dependency relationship data of the target task, M files to be uploaded corresponding to the target task are determined. 5.根据权利要求4所述的数据处理方法,其特征在于,所述基于所述历史打包上传数据和所述目标任务的文件依赖关系数据,确定所述目标任务对应的M个待上传文件,包括:5. data processing method according to claim 4, is characterized in that, described based on the file dependency relation data of described historical package upload data and described target task, determine the corresponding M files to be uploaded of described target task, include: 若所述历史打包上传数据为空,则基于所述目标任务的文件依赖关系数据,生成所述M个待上传文件;If the historical package upload data is empty, the M files to be uploaded are generated based on the file dependency data of the target task; 若所述历史打包上传数据非空,则基于所述目标任务的文件依赖关系数据以及所述历史打包上传数据中包含的、与所述目标任务相关的数据,确定所述M个待上传文件。If the historical packaged and uploaded data is not empty, the M files to be uploaded are determined based on the file dependency data of the target task and the data related to the target task included in the historical packaged and uploaded data. 6.根据权利要求5所述的数据处理方法,其特征在于,所述文件依赖关系数据对应N个初始文件,所述基于所述目标任务的文件依赖关系数据,生成所述M个待上传文件,包括:6. The data processing method according to claim 5, wherein the file dependency data corresponds to N initial files, and the M files to be uploaded are generated based on the file dependency data of the target task. ,include: 基于所述N个初始文件,生成所述M个待上传文件,其中,N等于M。Based on the N initial files, the M files to be uploaded are generated, where N is equal to M. 7.根据权利要求5所述的数据处理方法,其特征在于,所述文件依赖关系数据对应N个初始文件,所述基于所述目标任务的文件依赖关系数据,以及所述历史打包上传数据中包含的、与所述目标任务相关的数据,确定所述M个待上传文件,包括:7. The data processing method according to claim 5, wherein the file dependency data corresponds to N initial files, the file dependency data based on the target task, and the historical package upload data. The included data related to the target task determines the M files to be uploaded, including: 基于所述历史打包上传数据,确定所述N个初始文件中、与所述历史打包上传数据对应的文件不同的P个初始文件;Based on the historical packaging and uploading data, determine, among the N initial files, P initial files that are different from the files corresponding to the historical packaging and uploading data; 基于所述文件依赖关系数据,确定与所述P个初始文件具有依赖关系的Q个初始文件,将所述P个初始文件和所述Q个初始文件确定为所述M个待上传文件。Based on the file dependency relationship data, Q initial files having a dependency relationship with the P initial files are determined, and the P initial files and the Q initial files are determined as the M files to be uploaded. 8.根据权利要求2或3所述的数据处理方法,其特征在于,所述基于所述M个待上传文件的文件依赖关系数据,打包上传所述M个待上传文件,所述方法还包括:8. The data processing method according to claim 2 or 3, wherein the M files to be uploaded are packaged and uploaded based on the file dependency data of the M files to be uploaded, the method further comprising: : 针对所述M个节点中的每个节点,For each of the M nodes, 在所述节点对应的待上传文件打包上传至所述中心化存储网络之后,将所述节点对应的待上传文件打包上传至所述目标任务对应的存储空间;After the file to be uploaded corresponding to the node is packaged and uploaded to the centralized storage network, the file to be uploaded corresponding to the node is packaged and uploaded to the storage space corresponding to the target task; 将所述节点对应的待上传文件打包上传至数据库。The file to be uploaded corresponding to the node is packaged and uploaded to the database. 9.根据权利要求1至3任一项所述的数据处理方法,其特征在于,所述待上传文件包括目标代码,在所述基于所述M个待上传文件的文件依赖关系数据,打包上传所述M个待上传文件之后,还包括:9. The data processing method according to any one of claims 1 to 3, wherein the files to be uploaded include target codes, and in the file dependency data based on the M files to be uploaded, packaged and uploaded After the M files to be uploaded, it also includes: 对所述目标任务对应的目标代码进行测试,得到测试结果;Test the target code corresponding to the target task to obtain a test result; 根据所述测试结果,对所述目标代码进行部署。The target code is deployed according to the test result. 10.一种数据处理装置,其特征在于,包括:10. A data processing device, comprising: 确定单元,用于基于目标任务的文件依赖关系数据,确定所述目标任务对应的M个待上传文件以及所述M个待上传文件的文件依赖关系数据,M为正整数;a determining unit, configured to determine, based on the file dependency data of the target task, the M files to be uploaded corresponding to the target task and the file dependency data of the M files to be uploaded, where M is a positive integer; 打包上传单元,用于基于所述M个待上传文件的文件依赖关系数据,打包上传所述M个待上传文件。A packaging and uploading unit, configured to package and upload the M to-be-uploaded files based on the file dependency data of the M to-be-uploaded files. 11.一种计算机可读存储介质,其特征在于,所述计算机存储介质存储有计算机程序,所述计算机程序用于执行上述权利要求1至9任一项所述的数据处理方法。11. A computer-readable storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to execute the data processing method according to any one of claims 1 to 9. 12.一种电子设备,其特征在于,所述电子设备包括:12. An electronic device, characterized in that the electronic device comprises: 处理器;processor; 用于存储所述处理器可执行指令的存储器;a memory for storing the processor-executable instructions; 所述处理器,用于执行上述权利要求1至9任一项所述的数据处理方法。The processor is configured to execute the data processing method according to any one of claims 1 to 9.
CN202210724567.8A 2022-06-24 2022-06-24 Data processing method and device, storage medium and electronic equipment Pending CN115168301A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210724567.8A CN115168301A (en) 2022-06-24 2022-06-24 Data processing method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210724567.8A CN115168301A (en) 2022-06-24 2022-06-24 Data processing method and device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN115168301A true CN115168301A (en) 2022-10-11

Family

ID=83487408

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210724567.8A Pending CN115168301A (en) 2022-06-24 2022-06-24 Data processing method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN115168301A (en)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140181791A1 (en) * 2012-12-21 2014-06-26 International Business Machines Corporation Software development work item management system
CN104462352A (en) * 2014-12-04 2015-03-25 北京国双科技有限公司 Data packing method and device
CN106548331A (en) * 2015-09-21 2017-03-29 阿里巴巴集团控股有限公司 A kind of method and device for determining distribution order
CN108958753A (en) * 2018-07-13 2018-12-07 中国银行股份有限公司 A kind of code is packaged dispositions method and device
US20190005108A1 (en) * 2017-06-29 2019-01-03 Sap Se Deployment of independent database artifact groups
CN111142903A (en) * 2019-12-23 2020-05-12 杭州安恒信息技术股份有限公司 Configuration file interactive updating method and device based on file comparison
CN111258620A (en) * 2018-11-30 2020-06-09 中兴通讯股份有限公司 Method, device and device for generating differential upgrade package
CN112947983A (en) * 2021-04-15 2021-06-11 网易(杭州)网络有限公司 Application program updating method and device, electronic equipment and storage medium
CN113032006A (en) * 2021-04-16 2021-06-25 北京奇艺世纪科技有限公司 Plug-in construction method, device, equipment and storage medium
CN113098961A (en) * 2021-03-31 2021-07-09 北京自如信息科技有限公司 Component uploading method, device and system, computer equipment and readable storage medium
CN113885935A (en) * 2021-09-29 2022-01-04 厦门雅基软件有限公司 Resource packaging method and device, electronic equipment and computer readable storage medium
CN114567647A (en) * 2022-02-28 2022-05-31 浪潮云信息技术股份公司 Distributed cloud file storage method and system based on IPFS

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140181791A1 (en) * 2012-12-21 2014-06-26 International Business Machines Corporation Software development work item management system
CN104462352A (en) * 2014-12-04 2015-03-25 北京国双科技有限公司 Data packing method and device
CN106548331A (en) * 2015-09-21 2017-03-29 阿里巴巴集团控股有限公司 A kind of method and device for determining distribution order
US20190005108A1 (en) * 2017-06-29 2019-01-03 Sap Se Deployment of independent database artifact groups
CN108958753A (en) * 2018-07-13 2018-12-07 中国银行股份有限公司 A kind of code is packaged dispositions method and device
CN111258620A (en) * 2018-11-30 2020-06-09 中兴通讯股份有限公司 Method, device and device for generating differential upgrade package
CN111142903A (en) * 2019-12-23 2020-05-12 杭州安恒信息技术股份有限公司 Configuration file interactive updating method and device based on file comparison
CN113098961A (en) * 2021-03-31 2021-07-09 北京自如信息科技有限公司 Component uploading method, device and system, computer equipment and readable storage medium
CN112947983A (en) * 2021-04-15 2021-06-11 网易(杭州)网络有限公司 Application program updating method and device, electronic equipment and storage medium
CN113032006A (en) * 2021-04-16 2021-06-25 北京奇艺世纪科技有限公司 Plug-in construction method, device, equipment and storage medium
CN113885935A (en) * 2021-09-29 2022-01-04 厦门雅基软件有限公司 Resource packaging method and device, electronic equipment and computer readable storage medium
CN114567647A (en) * 2022-02-28 2022-05-31 浪潮云信息技术股份公司 Distributed cloud file storage method and system based on IPFS

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
陈宏君;王国栋;: "组件系统接口匹配识别方案和应用", 计算机技术与发展, no. 02, 31 December 2016 (2016-12-31) *

Similar Documents

Publication Publication Date Title
CN108279932B (en) Method and device for dynamically configuring user interface of mobile terminal
US10990365B2 (en) Container image building using dependency container images
CN108628661B (en) Automatic establishment method of cloud manufacturing service and cloud manufacturing system
CN106559438B (en) A program upload method and device based on a target network platform
JP6816136B2 (en) Unified interface specification for interacting with and running models in a variety of runtime environments
US10901804B2 (en) Apparatus and method to select services for executing a user program based on a code pattern included therein
US20180101371A1 (en) Deployment manager
EP3751411B1 (en) A system for building, managing, deploying and executing reusable analytical solution modules for industry applications
US20100058331A1 (en) Automated deployment of defined topology in distributed computing environment
US20140040791A1 (en) Development platform for software as a service (saas) in a multi-tenant environment
CN114879976B (en) Version of the environment deployment methods, devices and electronic equipment
US11487595B2 (en) API adapter creation device, API adapter creation method, and API adapter creation program
TW201727481A (en) Converting desktop applications to web applications
US20210247969A1 (en) Automated source code log generation
CN110737460A (en) platform project management method and device
McCormick et al. IBM SPSS modeler cookbook
US11630647B2 (en) Method and system for configuring processes of software applications using activity fragments
US20200174757A1 (en) Application development preview tool and methods
CN112052011A (en) Method and device for combining small programs, electronic equipment and medium
CN111722873A (en) Code reconstruction method, device, equipment and medium
CN112256287A (en) Application deployment method and device
CN110704031A (en) Software application project creating method and device and electronic equipment
CN112181644B (en) Method, system and device for cross-domain Jupyter machine learning component
CN115168301A (en) Data processing method and device, storage medium and electronic equipment
CN113626510A (en) Transaction checking method, device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination