Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Decentralization is a social relationship form and a content generation form formed in the internet development process, and is a novel network content production process relative to centralization. Decentralization does not require a hub, but a hub and a discretionary hub are freely selected by nodes, and in a decentralization system, anyone is a node and can also be a hub.
Continuous integration is a DevOps software development practice. With continuous integration, developers will periodically incorporate code changes into a central repository, after which the system will automatically run build and test operations. Continuous integration generally refers to the construction or integration phase of a software release process, and requires the use of automation components and culture components. The primary goal of continued integration is to discover and resolve defects faster, improve software quality, and reduce the time required to validate and release new software updates.
In software development and application development, or in consumption data storage and financial data storage, a target task is generally constructed according to related projects, and information to be stored is packaged and uploaded to a corresponding storage space at one time according to the target task.
For example, in the application development process, continuous integration tasks are often constructed one by one, and when the continuous integration tasks are executed, developers generally submit codes to a code library, a server pulls the codes from the code library through tools such as jenkins and downloads corresponding configuration files, and then, all the codes are packaged and uploaded to a test environment or a production environment through the configuration files. For another example, for a consumption data storage problem, a consumption data storage task is often constructed, consumption data is pulled from a corresponding platform according to the consumption data storage task, and all the pulled data is packaged and uploaded to a corresponding storage space.
However, with the above method, the dependency relationship between data is not well utilized, which leads to a limitation in solving the problem. For example, the uploading of all data packaged together may result in problems such as too centralized data storage and single storage mode. Based on this, the embodiment of the application provides a data processing method, which can determine a file to be uploaded and a dependency relationship between the files to be uploaded according to file dependency relationship data of a target task, and further upload the file to be uploaded based on the file dependency relationship between the files to be uploaded. The data storage method and the data storage device solve the problem that the dependency relationship among the data cannot be utilized for packaging and uploading in the prior art, and enrich the data storage mode.
The data processing method provided by the embodiment of the application can be executed by an electronic device, and the electronic device can be a terminal, such as a smart phone, a tablet computer, a desktop computer, an intelligent voice interaction device or a vehicle-mounted terminal; or the electronic device may also be a server, such as a separate physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.
Based on the data processing method provided by the embodiment of the application, the embodiment of the application provides an environment schematic diagram of the embodiment of the data processing method, and the electronic device is assumed to be a server. Referring to fig. 1, a schematic diagram of an implementation environment of a data processing method according to an embodiment of the present application is provided. As shown in fig. 1, the present embodiment environment includes a terminal 110 and a server 120, and a communication connection between the terminal 110 and the server 120.
The terminal 110 may be a smartphone, a tablet, a desktop computer, etc. The terminal 110 is installed with an application program that can drive the server 120 to perform a data processing method, for example, the application program may be a browser application, a shopping application, a video application, and the like. The servers 120 may be physical machines or virtual machines, and the number of the servers may be one or more, and the type and the number of the servers are not limited in this embodiment of the application.
Illustratively, according to the target task, the application driver server 120 installed in the terminal 110 starts to work, and the server 120 determines M files to be uploaded corresponding to the target task and file dependency data of the M files to be uploaded based on the file dependency data of the target task, and packages and uploads the M files to be uploaded based on the file dependency data of the M files to be uploaded.
Fig. 2 is a schematic view of an application scenario of a data processing method according to an exemplary embodiment of the present application. As shown in fig. 2, the task execution server cluster includes m task execution servers, which are respectively a task execution server 1, a task execution server 2, \\8230 \8230:, and the task execution server m may select any one task execution server in the task execution server cluster to execute all the packaged uploading jobs according to the target task, or may select a plurality of task execution servers to execute all the packaged uploading jobs together.
Specifically, the developer submits the code to a code database. The user logs on the persistent integration system using the browser application to create a persistent integration task. The created content includes: the address of the code database of the application, and the account number of the test network or the formal network.
Further, the persistent integration system is operated according to the browser application, and the created persistent integration task is executed. The continuous integration system drives the task server cluster to pull corresponding codes from the code database according to the target tasks, download the dependency packages corresponding to the target tasks, determine M files to be uploaded corresponding to the target tasks according to the dependency packages, determine file dependency relationship data among the M files to be uploaded, and package and upload the M files to be uploaded to a test network or a formal network according to the file dependency relationship data.
Based on the above, refer to fig. 3, which is a schematic flow chart of a data processing method according to an exemplary embodiment of the present application. As shown in fig. 3, the data processing method provided in the embodiment of the present application includes the following steps:
step S310, determining M files to be uploaded corresponding to the target task and file dependency data of the M files to be uploaded based on the file dependency data of the target task. Wherein M is a positive integer.
The target task may be a continuous integration task with respect to a certain item, or a consumption statistics task for a quarter of a week by a consumer on a certain shopping platform, or a user transfer calculation task, etc. The type of the target task is not specifically limited in the embodiment of the application, as long as the file to be uploaded in the target task has a file dependency relationship.
The dependency relationship is a relationship between data expressed by whether values between attributes in a relationship are equal or not, and belongs to the inherent property of data. In computer science, data dependencies are states when a program structure causes data to reference previously processed data.
Fig. 4 is a schematic diagram illustrating file dependency data provided in an exemplary embodiment of the present application. As shown in fig. 4, the target task includes 5 files to be uploaded, which are file a, file B, file C, file D, and file E, respectively, and the file dependency relationship data of the target task includes: file B depends on files D and E, and file a depends on files B and C.
Further, the M files to be uploaded corresponding to the target task may be all files included in the target task, or may be partial files included in the target task.
And step S320, packaging and uploading the M files to be uploaded based on the file dependency relationship data of the M files to be uploaded.
Continuing with the example of fig. 4 in step S310, according to the file dependency relationship data among the file a, the file B, the file C, the file D, and the file E, the file D, the file E, and the file C are first packaged and uploaded, respectively; secondly, packaging and uploading the file B; and finally, packaging and uploading the file A.
In the embodiment of the application, the M files to be uploaded corresponding to the target task and the file dependency relationship data of the M files to be uploaded are determined, and the M files to be uploaded are packaged and uploaded according to the M file dependency relationship data to be uploaded. According to the scheme in the embodiment of the application, the dependency relationship among the files in the target task is fully utilized, and the files to be uploaded are packaged in batches, so that the packaged data corresponding to each file to be uploaded can be uploaded to different storage spaces according to needs in the following process, and the storage concentration is reduced. In addition, the storage mode of the packed data corresponding to the file to be uploaded can be expanded by the packing uploading mode, for example, the packed data corresponding to the file to be uploaded can be stored in a common distributed database, and the packed data corresponding to the file to be uploaded can also be stored in a decentralized storage network.
Fig. 5 is a schematic flowchart illustrating a process of packaging and uploading M files to be uploaded according to an exemplary embodiment of the present application. The embodiment shown in fig. 5 is extended based on the embodiment shown in fig. 3, and the differences between the embodiment shown in fig. 5 and the embodiment shown in fig. 3 will be emphasized below, and the descriptions of the same parts will not be repeated.
As shown in fig. 5, the file dependency relationship data includes tree file dependency relationship data, the tree file dependency relationship data includes M nodes, the M nodes are deployed at different levels, and each node corresponds to one file to be uploaded corresponding to the target task; parallel relation and dependency relation exist among the M nodes, and dependency relation exists among the files to be uploaded corresponding to the nodes with the dependency relation; the method comprises the following steps of packaging and uploading M files to be uploaded based on file dependency relationship data of the M files to be uploaded, and comprises the following steps:
step S510, based on the tree file dependency relationship data, sequentially traversing the nodes belonging to the lowest hierarchy among the M nodes as traversal start nodes according to a hierarchy sequence from low to high, packaging the files to be uploaded corresponding to the traversed nodes, and uploading the packaged M files to be uploaded to a decentralized storage network.
The nodes in the tree file dependency relationship data can be divided into n levels, and if n is greater than or equal to 3, the nodes in the tree file dependency relationship data can be divided into leaf nodes, father nodes and root nodes; if n is equal to 2, the nodes in the tree file dependency relationship data can be divided into leaf nodes and root nodes; and if n is equal to 1, the node in the tree file dependency relationship data is a root node.
When n is greater than or equal to 2, if all the nodes in the level a are leaf nodes, the level a is the lowest level, and the leaf nodes in the level a are the nodes of the lowest level. When n is equal to 1, the hierarchy where the root node is located is the lowest hierarchy, and the root node is the node in the lowest hierarchy.
Further, when n is greater than or equal to 2, the hierarchy order from all leaf nodes to the root node is from low to high. Illustratively, the hierarchies from the hierarchy which is all leaf nodes to the hierarchy where the root node is located are sequentially defined as the hierarchy 1 to the hierarchy n, firstly, the leaf nodes in the hierarchy 1 are used as traversal starting nodes according to the file dependency relationship data of M files to be uploaded, the files to be uploaded which correspond to the leaf nodes in the hierarchy 1 are packaged and uploaded to the decentralized storage network, secondly, the files to be uploaded which correspond to the nodes in the hierarchy 2 are packaged and uploaded to the decentralized storage network, and finally the files to be uploaded which correspond to the nodes in the hierarchy n are packaged and uploaded to the decentralized storage network.
Along the example in fig. 4, the node where the file D and the file E are located is a leaf node, the level where the file E is located is level 1, that is, the lowest level, the node where the file B is located is a parent node, the node where the file C is located is a leaf node, the level where the file B and the file C are located is level 2, the node where the file a is located is a root node, and the level where the file a is located is level 3, that is, the highest level. According to the file dependency relationship data of the file A, the file B, the file C, the file D and the file E, packaging and uploading the files in three steps, and respectively packaging and uploading the file D, the file E and the file C in the first step; secondly, packaging and uploading a file B based on the packaging and uploading results of the file D and the file E; and thirdly, packaging and uploading the file A according to the packaging and uploading results of the file B and the file C.
According to the scheme in the embodiment, the ordering of the M files to be uploaded in the packaging and uploading process is ensured by relying on the file dependency relationship data of the M files to be uploaded. In addition, the M files to be uploaded are uploaded to the decentralized storage network, so that the safety of the packed data corresponding to the files to be uploaded is enhanced, and the storage cost of the packed data corresponding to the files to be uploaded is reduced.
Fig. 6 is a schematic flowchart illustrating a process of uploading packaged M files to be uploaded to a decentralized storage network according to an exemplary embodiment of the present application. The embodiment shown in fig. 6 is extended based on the embodiment shown in fig. 5, and the differences between the embodiment shown in fig. 6 and the embodiment shown in fig. 5 will be emphasized below, and the descriptions of the same parts will not be repeated.
As shown in fig. 6, based on the tree-type file dependency relationship data, a node belonging to the lowest hierarchy among the M nodes is taken as a traversal start node, the traversal is sequentially performed according to a hierarchy order from bottom to top, the files to be uploaded corresponding to the traversed node are packaged, and the packaged M files to be uploaded are uploaded to a decentralized storage network, which includes the following steps:
and step S610, based on the tree file dependency relationship data, packaging and uploading the file to be uploaded corresponding to the node belonging to the lowest level to a decentralized storage network, and obtaining the evidence information of the file to be uploaded corresponding to the node of the lowest level.
Illustratively, the pass information includes token information. In addition, while the files to be uploaded corresponding to the nodes of the lowest hierarchy are packaged and uploaded, the files to be uploaded corresponding to leaf nodes contained in other non-lowest hierarchies can also be packaged and uploaded together.
Along the embodiment shown in fig. 4, the file D and the file E in the node of the lowest hierarchy are first packaged and uploaded to the decentralized storage network, and the token about the file D and the token about the file E returned by the decentralized storage network are obtained. Meanwhile, for the convenience of subsequent packaging and uploading, when the files D and E in the nodes of the lowest hierarchy are packaged and uploaded, the files C corresponding to the leaf nodes in the hierarchy 2 can also be packaged and uploaded to the decentralized storage network, and the token about the files C returned by the decentralized storage network is obtained.
Step S620, determine whether there is a next lower level node having a dependency relationship with the lowest level node among the M nodes.
Illustratively, in the actual application process, if the determination result in step S620 is no, that is, no next-lower-level node having a dependency relationship with the lowest-level node exists in the M nodes, step S630 is executed, and if the determination result in step S620 is yes, that is, a next-lower-level node having a dependency relationship with the lowest-level node exists in the M nodes, steps S640 and S650 are executed.
And step S630, finishing packaging and uploading.
And step S640, based on the certification information of the file to be uploaded corresponding to the node at the lowest level, packaging and uploading the file to be uploaded corresponding to the node at the next lower level to a decentralized storage network, and obtaining the certification information of the file to be uploaded corresponding to the node at the next lower level.
Specifically, the token of the file to be uploaded corresponding to the node of the lowest hierarchy and the file to be uploaded of the parent node in the node of the next lower hierarchy corresponding to the node of the lowest hierarchy are packaged and uploaded to the decentralized storage network together, so that the token corresponding to the file to be uploaded of each parent node in the node of the next lower hierarchy is obtained.
For example, according to the embodiment shown in fig. 4, the lowest level is level 1 where the file D and the file E are located, the next lowest level is level 2 where the file B and the file C are located, the token corresponding to the file D, the token corresponding to the file E, and the file B in the parent node corresponding to the file D and the file E are packaged together, and the packaged result is uploaded to the decentralized storage network, so that the token corresponding to the file B is obtained. The file C is used as a file on a leaf node, and when the file D and the file E are packaged and uploaded, the file C in the level 2 is also packaged and uploaded to a decentralized storage network, and the token corresponding to the file C is obtained.
And step S650, continuously traversing, packaging and uploading the files to be uploaded based on the file dependency relationship data of the M files to be uploaded and the corresponding certification information of the files to be uploaded corresponding to the nodes of the next lower level until all the files contained in the M files to be uploaded are packaged and uploaded to the decentralized storage network.
Specifically, tokens corresponding to the files to be uploaded of the nodes in the hierarchy b and files to be uploaded of the parent nodes in the hierarchy (b + 1) corresponding to the nodes in the hierarchy b are packaged and uploaded to a decentralized storage network, and tokens {2, \8230; (n-1) } corresponding to the files to be uploaded of the parent nodes in the hierarchy (b + 1) are obtained. Taking fig. 4 as an example, the token corresponding to the file B, the token corresponding to the file C, and the file a are packaged and uploaded to a decentralized storage network, and the token corresponding to the file a is obtained.
According to the technical scheme, the files to be uploaded in the nodes are packaged in a grading mode, and dependence between the files to be uploaded of the nodes in different levels is strengthened on the basis of the corresponding certification information of the files to be uploaded of the nodes.
Fig. 7 is a schematic flowchart illustrating a process of determining M files to be uploaded corresponding to a target task according to an exemplary embodiment of the present application. As shown in fig. 7, determining M files to be uploaded corresponding to a target task based on file dependency data of the target task includes the following steps:
and step S710, determining historical packaged uploading data corresponding to the target task.
The history packed upload data refers to data stored in relation to the target task, and the history packed upload data of the target task can be pulled from a storage space corresponding to a pre-specified target task.
The storage mode of the history package uploading data is also based on the file dependency data of the target task. The historical packed uploading data records files corresponding to nodes of each hierarchy, tokens of the files corresponding to the nodes, packed data of the files corresponding to the nodes, and dependency relationships among the nodes.
Step S720, based on the historical packaged uploading data and the file dependency relationship data of the target task, M files to be uploaded corresponding to the target task are determined.
The M files to be uploaded can be all files contained in the target task or partial files contained in the target task.
In the embodiment of the application, M files to be uploaded of the target task are finally determined through the historical packaged uploading data corresponding to the target task, namely, the files contained in the target task are subjected to incremental or full-volume packaged uploading according to the historical packaged uploading data, and the data storage amount is further reduced.
Fig. 8 is a schematic flowchart illustrating a process of determining M files to be uploaded corresponding to a target task according to another exemplary embodiment of the present application. On the basis of the embodiment shown in fig. 7, the embodiment shown in fig. 8 is extended, and the differences between the embodiment shown in fig. 8 and the embodiment shown in fig. 7 are mainly described below, and the descriptions of the same parts are omitted.
As shown in fig. 8, determining M files to be uploaded corresponding to a target task based on history package upload data and file dependency data of the target task includes the following steps:
step S810, determining whether the history package uploaded data is empty.
Specifically, if the history packaged upload data is empty, that is, there is no history packaged data related to the target task, that is, the packaged upload work of the file to be uploaded related to the target task has not been performed before. The historical packaged uploading data is not empty, namely the historical packaged data related to the target task exists, namely the packaged uploading work of the file to be uploaded related to the target task is performed before.
Illustratively, if the execution result of step S810 is yes, that is, the history packed data is empty, step S820 is executed, and if the execution result of step S810 is no, that is, the history packed data is not empty, step S830 is executed.
Step 820, based on the file dependency relationship data of the target task, generating M files to be uploaded.
Step S830, based on the file dependency relationship data of the target task and the data related to the target task and contained in the history packaging uploading data, M files to be uploaded are determined.
According to the scheme in the embodiment, M files to be uploaded of the target task are determined according to the historical packaged uploaded data of the target task, so that repeated storage of the files to be uploaded in the target task is avoided, and the data storage capacity is reduced.
In an exemplary embodiment of the present application, the file dependency relationship data corresponds to N initial files, and based on the file dependency relationship data of the target task, M files to be uploaded are generated, including: and generating M files to be uploaded based on the N initial files, wherein N is equal to M.
Fig. 9 is a schematic structural diagram illustrating generation of M files to be uploaded according to an exemplary embodiment of the present application. The following describes embodiments of the present application in detail, taking fig. 4 and 9 as examples.
FIG. 4 shows that the target task contains 5 initial files, file A, file B, file C, file D, file E, and the dependency relationships among the 5 initial files. If the historical packaged uploading data about the target task is empty, the 5 initial files and the dependency relationships among the 5 initial files shown in fig. 4 are respectively used as the 5 files to be uploaded of the target task and the dependency relationships among the 5 files to be uploaded, the 5 files to be uploaded are respectively packaged and uploaded according to the file dependency relationship data shown in fig. 4, and the packaged and uploaded version is the V1.0 version shown in fig. 9.
By the technical scheme in the embodiment, when the files to be uploaded of the target task are packaged and uploaded for the first time, the complete storage versions of all the files included by the target task can be obtained, and a basis is provided for subsequent incremental packaging and uploading.
Fig. 10 is a flowchart illustrating a process of determining M files to be uploaded according to another exemplary embodiment of the present application. On the basis of the embodiment shown in fig. 8, the embodiment shown in fig. 10 is extended, and the differences between the embodiment shown in fig. 10 and the embodiment shown in fig. 8 will be emphasized below, and the descriptions of the same parts will not be repeated.
As shown in fig. 10, the determining of M files to be uploaded based on the file dependency relationship data of the target task and the data related to the target task and included in the history package upload data, where the file dependency relationship data corresponds to N initial files, includes the following steps:
step S1010, based on the history packaged uploaded data, determining P initial files different from the file corresponding to the history packaged uploaded data from among the N initial files.
Taking fig. 9 as an example, the history packaged upload data includes the V1.0 version shown in fig. 9, for the target task, the target task is split into the V2.0 version shown in fig. 9, and by comparison, only the file D in the V2.0 version is changed relative to the file D in the V1.0 version, so that the file D is determined to be an initial file different from the history packaged upload data.
Step S1020, based on the file dependency relationship data, determining Q initial files having dependency relationships with the P initial files, and determining the P initial files and the Q initial files as M files to be uploaded.
Taking fig. 9 as an example, the file D and the file B have a dependency, and the file B and the file a have a dependency, so the file D, the file B, and the file a are files to be uploaded.
In this embodiment, a dependency of a history version dimension is added to a traditional file dependency relationship, that is, when a history version exists, data of a corresponding node in the history version is preferentially depended, so that the packaging and uploading cost is saved, and the amount of uploaded data is further reduced.
If the historically packaged uploading data is not empty, based on the certification information of the file to be uploaded corresponding to the node at the lowest level and the certification information of the file to be uploaded of the brother node corresponding to the node at the lowest level in the historically packaged uploading data, packaging and uploading the file to be uploaded corresponding to the node at the next lower level to the decentralized storage network, obtaining the certification information of the file to be uploaded corresponding to the node at the next lower level, and continuously traversing, packaging and uploading until the M files to be uploaded are packaged and uploaded to the decentralized storage network.
Taking fig. 9 as an example, the file D in the V2.0 version is an initial file different from the historical packed upload data (i.e., the file D in the V1.0 version), the file D in the V2.0 version is packed and uploaded to the decentralized storage network to obtain a token corresponding to the file D, the token corresponding to the file E in the V1.0 version, the token corresponding to the file D in the V2.0 version, and the file B in the V2.0 version are packed and uploaded to the decentralized storage network together to obtain a token corresponding to the file B, and finally the token corresponding to the file C in the V1.0 version, the token corresponding to the file B in the V2.0 version, and the file a in the V2.0 version are packed and uploaded to the decentralized storage network together to obtain a token corresponding to the file a.
Fig. 11 is a flowchart illustrating a flow chart of packaging and uploading M files to be uploaded according to another exemplary embodiment of the present application. The embodiment shown in fig. 11 is extended on the basis of the embodiment shown in fig. 5, and the differences between the embodiment shown in fig. 11 and the embodiment shown in fig. 5 will be mainly described below, and the descriptions of the same parts will not be repeated.
As shown in fig. 11, the method for uploading M files to be uploaded in a packaged manner based on the file dependency relationship data of the M files to be uploaded further includes the following steps:
step S1110, for each node in the M nodes, after the file to be uploaded corresponding to the node is packaged and uploaded to the centralized storage network, the file to be uploaded corresponding to the node is packaged and uploaded to the storage space corresponding to the target task.
The storage space may be a local storage space, and for the files in the leaf nodes, the files to be uploaded in the leaf nodes are packaged and uploaded to the local storage space corresponding to the target task. And for the files in the non-leaf nodes, packaging and uploading the files of the current node and the tokens of the files of the child nodes corresponding to the current node together to a local storage space so as to pull historical packaged uploading data of the target task from the local storage space, and determining M files to be uploaded according to the historical packaged uploading data.
And step S1120, packaging and uploading the file to be uploaded corresponding to the node to a database.
The database can be any database, and for the file to be uploaded in each node, the file to be uploaded is packaged and uploaded to a decentralized storage network and a storage space respectively, and then the file to be uploaded in each node is packaged and uploaded to the database. Specifically, for the file to be uploaded of each node, the file to be uploaded of the node, the token corresponding to the node, and the storage address of the file to be uploaded corresponding to the node in the storage space are packaged and uploaded to the database.
In the embodiment of the application, the file to be uploaded corresponding to the node is packaged and uploaded to the storage space corresponding to the target task, so that the difference between the file to be uploaded corresponding to the current target task and the historical packaged uploading data about the target task stored in the storage space can be compared conveniently when the target task is subsequently packaged and uploaded for a non-first time. And the files to be uploaded corresponding to the nodes are packaged and uploaded to the database, so that the files can be conveniently and quickly inquired in time when the data of the target task is used next time.
Fig. 12 is a schematic flowchart illustrating a data processing method according to another exemplary embodiment of the present application. The embodiment shown in fig. 12 is extended on the basis of the embodiment shown in fig. 3, and the differences between the embodiment shown in fig. 12 and the embodiment shown in fig. 3 will be mainly described below, and the descriptions of the same parts will not be repeated.
As shown in fig. 12, the method further includes the following steps after the file to be uploaded includes the object code and the M files to be uploaded are packaged and uploaded based on the file dependency relationship data of the M files to be uploaded.
Step 1210, testing the target code corresponding to the target task to obtain a test result.
And step S1220, deploying the target code according to the test result.
Specifically, the target tasks include code pulling, code packaging, code uploading, and the like.
Illustratively, the project codes are pulled from the code database according to the continuous integration task, corresponding configuration files are downloaded, the continuous integration task is divided into a plurality of files according to the configuration files, and M files to be uploaded corresponding to the continuous integration task and file dependency relationship data of the M files to be uploaded are determined. And then, based on the file dependency relationship data of the M files to be uploaded, target codes corresponding to the M files to be uploaded are packaged and uploaded, continuous integration test is carried out on the target codes, and the target codes are deployed according to the test result.
Continuing with the embodiment shown in fig. 4 as an example, if the history packaged upload data about the continuously integrated target task is empty, first, the dependency information corresponding to the file D and the file D in the lowest hierarchy and the dependency information corresponding to the file E and the file E are packaged and uploaded to the decentralized storage network, to obtain tokens corresponding to the file D and the file E respectively, and at the same time, the file C corresponding to the leaf node in another hierarchy and the dependency information corresponding to the file C may be packaged and uploaded to the decentralized storage network, and the token corresponding to the file C is obtained.
Further, packing and uploading the token corresponding to the file D, the token corresponding to the file E, and the dependency information corresponding to the file B and the file B to a decentralized storage network, and acquiring the token corresponding to the file B. And finally, packing and uploading the token corresponding to the file B, the token corresponding to the file C, the file A and the dependency information corresponding to the file A to a decentralized storage network to obtain the token corresponding to the file A. After the test is finished, the user operates the continuous integrated management system through the client application to deploy the target code of the file to be uploaded corresponding to the target task to the production environment.
Through the scheme in the embodiment, a circulating flow of circulating packaging, deployment and integration can be realized. In addition, the packaging and deployment process is disassembled into a plurality of times through the combing of the file dependency relationship of the target task, and the packaging, deployment and integration of the plurality of times can be automatically processed.
Furthermore, with the increasing maturity of blockchain technology, decentralized storage is the technological base for web3.0, gradually falling to the ground, which makes decentralized web applications possible. In the related art, the continuous integration flow about the web application is as follows: developers develop web applications and then submit the code to code libraries, such as github, gitlab, etc. Developers pull codes from a code library through tools such as jenkins and the like, download dependencies, pack the codes through configuration files, and deploy the codes to a test environment or a production environment. Developers continue to refine the functionality of web applications and repeat the above-described continuous integration process. The above-described continuous integration flow for web applications, however, does not solve the dependency problem well.
Such as data dependency problems. A plurality of pages in a web application are composed of a page template and a plurality of data corresponding to the page template, the data are directly stored in a database in a web2.0 scene, but the data are stored in a decentralized block chain network in a web3.0 scene to generate metadata, and then the pages generate tokens corresponding to the data through the decentralized block chain network. However, in the packing and uploading stage, data is not uploaded to the decentralized block chain network, a token used for querying the data is not generated, and a method for packing the data is not available, so that in the development process, the data needs to be manually submitted to the decentralized block chain network to obtain the corresponding token, and then the token is updated to a page.
For example, sub-page dependency problem, in the web3.0 scenario, a single page may be cast as a Non-homogeneous token (NFT) creating a unique token to identify the location of the page in the decentralized storage network. If the sub-page is not stored in the decentralized storage network, the corresponding token does not exist, and the sub-page cannot be called through the token in the main page. In the development process in the related technology, the sub-page is firstly tested, then the sub-page is manually submitted to a decentralized block chain network, a corresponding token is obtained, and then the link of the sub-page is written on the main page. Of course, the sub-pages can be linked to form a tree-shaped connection relationship. Such development process is cumbersome, not conducive to application modification and maintenance, and cumbersome to develop and test.
Therefore, how to better utilize the dependency problem in the web application to package and upload the file to be uploaded is particularly important.
Fig. 13 is a schematic structural diagram illustrating persistent integration of a web application according to an exemplary embodiment of the present application. FIG. 14 is a diagram illustrating file dependencies regarding a target task of a web application according to an exemplary embodiment of the present application. The method for continuously integrating the web application is described in detail below by taking fig. 13 and 14 as an example.
First, receiving a target task sent by a persistent integration management system, pulling a project code of a web application from a code library by a task execution server according to the target task, downloading a dependency package configured in package of the project from npm, and splitting the target task according to the dependency package, as shown in fig. 14, if history package upload data of the persistent integrated target task of the web application is empty, the target task is split into three steps.
The method comprises the steps of firstly, packaging a related dependency file corresponding to data E and data E, a related dependency file corresponding to page D and page D, and a related dependency file corresponding to page C and page C, respectively obtaining first package data corresponding to the data E, the page D and the page C, uploading the first package data corresponding to the data E, the page D and the page C to a decentralized storage network, and obtaining tokens corresponding to the data E, the page D and the page C. And uploading the first package data corresponding to the data E, the page D and the page C to a local storage space. And packaging the data E, the token corresponding to the data E and the storage address of the first packed data corresponding to the data E in the local storage space to obtain second packed data of the data E, and uploading the second packed data to the database. Correspondingly, the page D, the token corresponding to the page D and the storage address of the first packed data corresponding to the page D in the local storage space are packed together to obtain second packed data of the page D, and the second packed data are uploaded to the database. And packaging the page C, the token corresponding to the page C and the storage address of the first packed data corresponding to the page C in the local storage space to obtain second packed data of the page C, and uploading the second packed data to the database.
And secondly, packing the token corresponding to the data E, the token corresponding to the page D and the related dependency files corresponding to the page B and the page B together to obtain first package data related to the page B, uploading the first package data of the page B to a decentralized storage network, and obtaining the token corresponding to the page B. And uploading the first package data corresponding to the page B to a local storage space. And packaging the page B, the token corresponding to the page B and the storage address of the first packed data corresponding to the page B in the local storage space to obtain second packed data of the page B, and uploading the second packed data to a database.
And thirdly, packing the token corresponding to the page B, the token corresponding to the page C, the page A and the related dependency file of the page A together to obtain first package data related to the page A, uploading the first package data of the page A to a decentralized storage network, and obtaining the token corresponding to the page A. And uploading the first packed data corresponding to the page A to a local storage space. And packing the page A, the token corresponding to the page A and the storage address of the first packed data corresponding to the page A in the local storage space to obtain second packed data of the page A, and uploading the second packed data to a database.
Further, the first package data corresponding to the page a, the page B, the page C, the page D, and the data E may be uploaded through an api (application programming interface) provided by the decentralized storage network such as IPFS and AR, and uploaded to the decentralized storage network, and a token returned by the decentralized storage network is obtained. The decentralized storage network comprises IPFS and AR testing environments or formal environments.
Further, if it is the first step shown in fig. 14, it is only necessary to pack directly without replacing the dependent path inside the file. If it is the second and third steps shown in fig. 14 that are packaged, then we need to replace the opposite path in the web application with a plug-in through webpack as the path in the decentralized storage network.
Method embodiments of the present application are described in detail above in conjunction with fig. 1-14, and apparatus embodiments of the present application are described in detail below in conjunction with fig. 15. It is to be understood that the description of the method embodiments corresponds to the description of the apparatus embodiments, and therefore reference may be made to the preceding method embodiments for parts not described in detail.
Fig. 15 is a schematic structural diagram of a data processing apparatus according to an exemplary embodiment of the present application. As shown in fig. 15, the data processing apparatus provided in the embodiment of the present application includes:
the determining unit 1510 is configured to determine, based on the file dependency relationship data of the target task, M files to be uploaded corresponding to the target task and file dependency relationship data of the M files to be uploaded, where M is a positive integer;
and the packaging and uploading unit 1520 is configured to package and upload the M files to be uploaded based on the file dependency relationship data of the M files to be uploaded.
In an embodiment of the present application, the packaging and uploading unit 1520 is further configured to, based on the tree-type file dependency relationship data, take a node belonging to a lowest hierarchy among the M nodes as a traversal starting node, perform traversal once according to a hierarchy sequence from low to high, package files to be uploaded corresponding to the traversed node, and upload the packaged M files to be uploaded to the decentralized storage network.
In an embodiment of the present application, the packaging and uploading unit 1520 is further configured to package and upload the to-be-uploaded file corresponding to the node belonging to the lowest level to a decentralized storage network based on the tree-type file dependency relationship data, so as to obtain the certification information of the to-be-uploaded file corresponding to the node of the lowest level; if a sub-low level node which has a dependency relationship with the node of the lowest level exists in the M nodes, packaging and uploading the file to be uploaded corresponding to the sub-low level node to a decentralized storage network based on the certification information of the file to be uploaded corresponding to the node of the lowest level, and obtaining the certification information of the file to be uploaded corresponding to the node of the sub-low level; and continuously traversing, packaging and uploading the files to be uploaded based on the tree file dependency relationship data and the corresponding certification information of the files to be uploaded of the nodes of the next lower level until M files to be uploaded are packaged and uploaded to the decentralized storage network.
In an embodiment of the present application, the determining unit 1510 is further configured to determine historical package upload data corresponding to the target task; and determining M files to be uploaded corresponding to the target task based on the historical packed uploading data and the file dependency relationship data of the target task.
In an embodiment of the present application, the determining unit 1510 is further configured to generate M files to be uploaded based on the file dependency relationship data of the target task if the history package upload data is empty; and if the historical packaging uploading data is not empty, determining M files to be uploaded based on the file dependency relationship data of the target task and the data which is contained in the historical packaging uploading data and is related to the target task.
In an embodiment of the present application, the determining unit 1510 is further configured to generate M files to be uploaded based on the N initial files, where N is equal to M.
In an embodiment of the present application, the determining unit 1510 is further configured to determine, based on the history packaged upload data, P initial files, different from the file corresponding to the history packaged upload data, from among the N initial files; and determining Q initial files having a dependency relationship with the P initial files based on the file dependency relationship data, and determining the P initial files and the Q initial files as M files to be uploaded.
In an embodiment of the present application, the packaging and uploading unit 1520 is further configured to, for each node in the M nodes, package and upload the file to be uploaded corresponding to the node to a centralized storage network, and then package and upload the file to be uploaded corresponding to the node to a storage space corresponding to a target task; and packaging and uploading the files to be uploaded corresponding to the nodes to a database.
In an embodiment of the present application, the packaging uploading unit 1520 is further configured to test a target code corresponding to a target task to obtain a test result; and deploying the target code according to the test result.
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 16. Fig. 16 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
As shown in fig. 16, the electronic device 160 includes one or more processors 1601 and memory 1602.
The processor 1601 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 160 to perform desired functions.
Memory 1602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 1601 to implement the methods of the various embodiments of the present application described above and/or other desired functions. Various contents such as file dependency relationship data including a target task, M files to be uploaded, and packaged data corresponding to the M files to be uploaded can also be stored in the computer-readable storage medium.
In one example, the electronic device 160 may further include: an input device 1603 and an output device 1604, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input device 1603 may include, for example, a keyboard, a mouse, and the like.
The output device 1604 may output various information to the outside, including the file dependency data of the target task, the M files to be uploaded, and the packed data corresponding to the M files to be uploaded, and so on. The output devices 1604 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 160 relevant to the present application are shown in fig. 16, and components such as buses, input/output interfaces, and the like are omitted. In addition, electronic device 160 may include any other suitable components, depending on the particular application.
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods of the various embodiments of the present application mentioned above in this specification.
The computer program product may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages, for carrying out operations according to embodiments of the present application. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the methods according to the various embodiments of the present application described above in this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by one skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.