CN111104102A - Method for constructing multi-service scene automatic assembly line - Google Patents
Method for constructing multi-service scene automatic assembly line Download PDFInfo
- Publication number
- CN111104102A CN111104102A CN202010127053.5A CN202010127053A CN111104102A CN 111104102 A CN111104102 A CN 111104102A CN 202010127053 A CN202010127053 A CN 202010127053A CN 111104102 A CN111104102 A CN 111104102A
- Authority
- CN
- China
- Prior art keywords
- pipeline
- action
- building
- description file
- automation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a method for constructing a multi-service scene automatic assembly line, which comprises an integral framework, a process engine, an expanded market and an assembly line description file, wherein the integral framework comprises the following steps: the system comprises an interaction layer capable of submitting a pipeline description file and optional source information to an API Server, the API Server responsible for receiving and processing requests sent by the interaction layer, a Crond used for managing and triggering a timing pipeline, a flow engine used for executing a specific pipeline, dependent middleware and architecture advantages. The pipeline scheme of the invention simultaneously supports scenes such as CI/CD, big data, Ops automation operation and maintenance, QA code quality analysis, API-Test and the like; the pipeline description file and grammar proposed in the design are sufficiently simplified and complete in function, so that the user has almost no cost for starting; meanwhile, the market design is expanded, and the possibility of infinite expansion is provided for a production line; the user can develop or use the extended Action developed by other developers, and the use scene is greatly enriched.
Description
Technical Field
The invention belongs to the field of PaaS platforms, and particularly relates to a method for constructing an automatic assembly line of a multi-service scene.
Background
At present, production line products with large use amount on the market mainly comprise Jenkins, Circle CI, Travis CI and the like, and the open-source software items only support CI/CD scenes, and are applied to continuous integration and continuous delivery, so that the single scene is supported; the assembly line description file is difficult to manually compile, and the use difficulty of a user is high; the pipeline task is not abstracted enough or is not abstracted at all, and a user needs to describe concrete steps of each task from zero, so that the multiplexing degree is low. In the traditional mode, a user often has a large amount of repeated logic in the pipeline description, and often only very individual parameters are different. The same content is copied between different pipeline description files, errors are easy to occur, if problems occur in the repeated logic and need to be repaired, all the description files need to be modified in the same way, the cost is extremely high, and omission is easy. The user wants to perform quality analysis on the code, does not know how to do so, only can search on a search engine how to perform code quality analysis on the specified language, then defines a script by himself, installs a pile of analysis software in the script to analyze the code, and is too high in cost, and the user cannot share experience.
The method for constructing the multi-service scene automatic assembly line can simultaneously support scenes such as CI/CD, big data calculation, Ops automation operation and maintenance, QA code quality analysis, API-Test interface Test and the like through one assembly line technology.
Disclosure of Invention
In order to solve the problems, the invention provides a method for constructing an automatic assembly line of multiple service scenes, which can overcome the problems of single supporting scene and low multiplexing degree of the traditional open source software project, realize that one assembly line technology supports multiple scenes and improve the assembly line service degree.
The technical scheme of the invention is as follows: the invention relates to a method for constructing a multi-service scene automatic pipeline, which is characterized by comprising an overall architecture, a process engine, an expanded market and a pipeline description file, wherein the overall architecture comprises: the system comprises an interaction layer capable of submitting a pipeline description file and optional source information to an API Server, the API Server responsible for receiving and processing requests sent by the interaction layer, a Crond used for managing and triggering a timing pipeline, a flow engine used for executing a specific pipeline, dependent middleware and architecture advantages.
Typical interaction means include, but are not limited to: UI interface, RESTful API, CLI (command line tool).
The process engine is responsible for executing a specific pipeline, and the pipeline is divided into execution units and is performed according to the sequence specified in the description file. Preferably, the flow engine includes an Action executer that executes each Action and a recirculer that uses the ability of the Action executer in the pipeline dimension to implement pipeline advancement.
The API Server includes, but is not limited to, creation of a pipeline, starting, executing progress queries, stopping, and querying the pipeline according to conditions. The API Server analyzes the description file and inserts the assembly line data into the database; if the assembly line has timing configuration, the assembly line is submitted to the Crond unified management after being analyzed; the pipeline ID is submitted to the pipeline engine (write etcd), waiting for execution.
Preferably, the dependent middleware includes etcd for distributed locks, event snooping, and partial data storage throughout the architecture and MySQL for persisting specific execution cases of the storage pipeline and meta-information.
Preferably, the Action execution acts as an interface mode at the code level.
Reconciler performs the following:
s1, monitoring a newly added pipeline task;
s2, calculating the overall execution condition of the production line according to the execution conditions of all actions, and if the overall execution condition is a final state, executing subsequent cleaning work without propulsion; otherwise, go to S3;
s3, calculating a schedulable Action list according to the DAG and all Action execution conditions;
s4, calling a specific Action execution or to start execution for each Action in the executable list;
s5, executing each Action to a final state of success or failure, and repeating S2, S3, S4 and S5;
and S6, finally advancing the whole assembly line to a final state, and ending Reconcle at this time.
The advent of the expanded market offers unlimited possibilities for pipelining, and as an option, the expanded market can be continually expanded and perfected by developing actions that are shared between different projects and users.
The development of actions includes: the delivery mode of the Action is Docker mirror image, the user needs to provide an executable file of/opt/Action/run, and the file contains specific processing logic; run executables may be developed using any programming language, including shell.
The pipeline description file comprises: the method comprises the steps of global file structure, Action configuration items, references among actions and Secret configuration. Preferably, the global file structure includes: version representing the version number of the pipeline description file; envs representing a global environment variable; cron as a timing configuration; stages as the core field of the pipeline description file; cron is designed to support two formats: minute scale (Linux crontab standard format); second class (Java schedule format).
Compared with the prior art, the invention has the beneficial effects that:
the pipeline solution of the invention simultaneously supports scenes such as CI/CD, big data, Ops automation operation and maintenance, QA code quality analysis, API-Test and the like; the pipeline description file and grammar proposed in the design are sufficiently simplified and complete in function, so that the user has almost no cost for starting; meanwhile, the market design is expanded, and the possibility of infinite expansion is provided for a production line; the user can develop or use the extended Action developed by other developers, and the use scene is greatly enriched.
Drawings
FIG. 1 is an overall architecture of the automated pipeline of the present invention.
FIG. 2 is an expanded market for use with the automated pipeline of the present invention.
Detailed Description
The invention relates to a method for constructing a multi-service scene automatic pipeline, which is characterized by comprising an overall architecture, a process engine, an expanded market and a pipeline description file, wherein the overall architecture comprises: the system comprises an interaction layer capable of submitting a pipeline description file and optional source information to an API Server, the API Server responsible for receiving and processing requests sent by the interaction layer, a Crond used for managing and triggering a timing pipeline, a flow engine used for executing a specific pipeline, dependent middleware and architecture advantages.
Typical interaction means include, but are not limited to: UI interface, RESTful API, CLI (command line tool).
The process engine is responsible for executing a specific pipeline, and the pipeline is divided into execution units and is performed according to the sequence specified in the description file. Preferably, the flow engine includes an Action executer that executes each Action and a recirculer that uses the ability of the Action executer in the pipeline dimension to implement pipeline advancement.
The API Server includes, but is not limited to, creation of a pipeline, starting, executing progress queries, stopping, and querying the pipeline according to conditions. The API Server analyzes the description file and inserts the assembly line data into the database; if the assembly line has timing configuration, the assembly line is submitted to the Crond unified management after being analyzed; the pipeline ID is submitted to the pipeline engine (write etcd), waiting for execution.
Preferably, the dependent middleware includes etcd for distributed locks, event snooping, and partial data storage throughout the architecture and MySQL for persisting specific execution cases of the storage pipeline and meta-information.
Preferably, the Action execution acts as an interface mode at the code level.
Reconciler performs the following:
s1, monitoring a newly added pipeline task;
s2, calculating the overall execution condition of the production line according to the execution conditions of all actions, and if the overall execution condition is a final state, executing subsequent cleaning work without propulsion; otherwise, go to S3;
s3, calculating a schedulable Action list according to the DAG and all Action execution conditions;
s4, calling a specific Action execution or to start execution for each Action in the executable list;
s5, executing each Action to a final state of success or failure, and repeating S2, S3, S4 and S5;
and S6, finally advancing the whole assembly line to a final state, and ending Reconcle at this time.
The advent of the expanded market offers unlimited possibilities for pipelining, and as an option, the expanded market can be continually expanded and perfected by developing actions that are shared between different projects and users.
The development of actions includes: the delivery mode of the Action is Docker mirror image, the user needs to provide an executable file of/opt/Action/run, and the file contains specific processing logic; run executables may be developed using any programming language, including shell.
The pipeline description file comprises: the method comprises the steps of global file structure, Action configuration items, references among actions and Secret configuration. Preferably, the global file structure includes: version representing the version number of the pipeline description file; envs representing a global environment variable; cron as a timing configuration; stages as the core field of the pipeline description file; cron is designed to support two formats: minute scale (Linux crontab standard format); second class (Java schedule format).
Claims (10)
1. The invention relates to a method for constructing a multi-service scene automatic pipeline, which is characterized by comprising an overall architecture, a process engine, an expanded market and a pipeline description file, wherein the overall architecture comprises: the system comprises an interaction layer capable of submitting a pipeline description file and optional source information to an API Server, the API Server responsible for receiving and processing requests sent by the interaction layer, a Crond used for managing and triggering a timing pipeline, a flow engine used for executing a specific pipeline, dependent middleware and architecture advantages.
2. The method for building a multi-service scenario automation pipeline of claim 1, where the flow engine includes an Action executer that executes each Action and a Reconciler that uses the ability of the Action executer in the pipeline dimension for pipeline advancement.
3. The method for building a multi-service scenario automation pipeline according to claim 1, characterized in that the API Server includes creation of the pipeline, starting, executing progress queries, stopping and querying the pipeline according to conditions.
4. The method for building a multi-business scenario automation pipeline of claim 1, where the dependent middleware comprises etcd for distributed locks, event snooping and partial data storage in the whole architecture and MySQL for persisting specific execution cases and meta-information of the storage pipeline.
5. The method for building a multi-service scenario automation pipeline of claim 2, where the Action Executor is interface mode at code level.
6. A method for building a multi-service scenario automation pipeline according to claim 2, characterized in that the Reconciler performs the following procedure:
s1, monitoring a newly added pipeline task;
s2, calculating the overall execution condition of the production line according to the execution conditions of all actions, and if the overall execution condition is a final state, executing subsequent cleaning work without propulsion; otherwise, go to S3;
s3, calculating a schedulable Action list according to the DAG and all Action execution conditions;
s4, calling a specific Action execution or to start execution for each Action in the executable list;
s5, executing each Action to a final state of success or failure, and repeating S2, S3, S4 and S5;
and S6, finally advancing the whole assembly line to a final state, and ending Reconcle at this time.
7. The method for building a multi-business scenario automation pipeline of claim 1, where the expanded market can be continuously augmented and refined by developing actions that are shared between different projects and users.
8. The method for building a multi-service scenario automation pipeline according to claim 1, characterized in that the pipeline description file comprises: the method comprises the steps of global file structure, Action configuration items, references among actions and Secret configuration.
9. The method for building a multi-service scenario automation pipeline according to claim 8, characterized in that the global file structure comprises: version representing the version number of the pipeline description file; envs representing a global environment variable; cron as a timing configuration; stages as the core field of the pipeline description file.
10. The method for building a multi-service scenario automation pipeline according to claim 9, characterized in that the cron is designed to support two formats: minute and second.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911140174 | 2019-11-20 | ||
| CN2019111401747 | 2019-11-20 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111104102A true CN111104102A (en) | 2020-05-05 |
Family
ID=70427746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010127053.5A Pending CN111104102A (en) | 2019-11-20 | 2020-02-28 | Method for constructing multi-service scene automatic assembly line |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111104102A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112099768A (en) * | 2020-08-04 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Business process processing method and device and computer readable storage medium |
| CN116933323A (en) * | 2023-09-11 | 2023-10-24 | 北京格尔国信科技有限公司 | Code auditing method, system and computer equipment based on privacy protection |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216906A (en) * | 2007-12-29 | 2008-07-09 | 中国建设银行股份有限公司 | A flow control method and a flow engine |
| WO2011053891A2 (en) * | 2009-10-31 | 2011-05-05 | Rutgers, The State University Of New Jersey | Virtual flow pipelining processing architecture |
| JP2017059273A (en) * | 2017-01-05 | 2017-03-23 | 富士通株式会社 | Arithmetic processing unit |
| CN106980530A (en) * | 2016-01-18 | 2017-07-25 | 中兴通讯股份有限公司 | A kind of distributed system workflow processing method and workflow engine system |
| CN108037919A (en) * | 2017-12-01 | 2018-05-15 | 北京博宇通达科技有限公司 | A kind of visualization big data workflow configuration method and system based on WEB |
| US20180203727A1 (en) * | 2017-01-13 | 2018-07-19 | International Business Machines Corporation | Optimizing pipeline execution scheduling based on commit activity trends, priority information, and attributes |
| CN109324943A (en) * | 2018-08-15 | 2019-02-12 | 刘鹏 | DOCLOUD flow engine system |
| CN110058932A (en) * | 2019-04-19 | 2019-07-26 | 中国科学院深圳先进技术研究院 | A kind of storage method and storage system calculated for data flow driven |
-
2020
- 2020-02-28 CN CN202010127053.5A patent/CN111104102A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216906A (en) * | 2007-12-29 | 2008-07-09 | 中国建设银行股份有限公司 | A flow control method and a flow engine |
| WO2011053891A2 (en) * | 2009-10-31 | 2011-05-05 | Rutgers, The State University Of New Jersey | Virtual flow pipelining processing architecture |
| CN106980530A (en) * | 2016-01-18 | 2017-07-25 | 中兴通讯股份有限公司 | A kind of distributed system workflow processing method and workflow engine system |
| JP2017059273A (en) * | 2017-01-05 | 2017-03-23 | 富士通株式会社 | Arithmetic processing unit |
| US20180203727A1 (en) * | 2017-01-13 | 2018-07-19 | International Business Machines Corporation | Optimizing pipeline execution scheduling based on commit activity trends, priority information, and attributes |
| CN108037919A (en) * | 2017-12-01 | 2018-05-15 | 北京博宇通达科技有限公司 | A kind of visualization big data workflow configuration method and system based on WEB |
| CN109324943A (en) * | 2018-08-15 | 2019-02-12 | 刘鹏 | DOCLOUD flow engine system |
| CN110058932A (en) * | 2019-04-19 | 2019-07-26 | 中国科学院深圳先进技术研究院 | A kind of storage method and storage system calculated for data flow driven |
Non-Patent Citations (1)
| Title |
|---|
| 潘宗奎: "云资源管理流程平台系统设计与实现" * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112099768A (en) * | 2020-08-04 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Business process processing method and device and computer readable storage medium |
| CN112099768B (en) * | 2020-08-04 | 2024-02-23 | 北京奇艺世纪科技有限公司 | Business process processing method and device and computer readable storage medium |
| CN116933323A (en) * | 2023-09-11 | 2023-10-24 | 北京格尔国信科技有限公司 | Code auditing method, system and computer equipment based on privacy protection |
| CN116933323B (en) * | 2023-09-11 | 2023-12-08 | 北京格尔国信科技有限公司 | Code auditing method, system and computer equipment based on privacy protection |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9405577B2 (en) | Realizing jumps in an executing process instance | |
| Openja et al. | Analysis of modern release engineering topics:–a large-scale study using stackoverflow– | |
| US9569204B2 (en) | End-to-end continuous integration and verification of software | |
| US10466981B1 (en) | System and method for generative programming in an integrated development environment (IDE) | |
| Gao et al. | An empirical study on low gpu utilization of deep learning jobs | |
| Leszko | Continuous Delivery with Docker and Jenkins: Create secure applications by building complete CI/CD pipelines | |
| CN116339749A (en) | Containerized DevOps method, system and equipment capable of performing task scheduling | |
| CN101317153A (en) | Object model on workflow | |
| CN116324836A (en) | Map interactive user interface elements to an RPA object repository for RPA development | |
| CN111104102A (en) | Method for constructing multi-service scene automatic assembly line | |
| Hu et al. | Repo2run: Automated building executable environment for code repository at scale | |
| Verona | Practical DevOps: Implement DevOps in your organization by effectively building, deploying, testing, and monitoring code | |
| US11809897B2 (en) | Chained triggering of builds in continuous integration environments | |
| CN113485726B (en) | Application environment delivery method and device | |
| CN119166280A (en) | A system and method for bioinformatics analysis process | |
| Baxter | Mastering Akka | |
| CN106357735B (en) | For operating the method and apparatus of the infrastructure layer of cloud computing framework | |
| CN118227296A (en) | Task backtracking method, device, computing equipment and storage medium | |
| Minella | The Definitive Guide to Spring Batch | |
| Gammelgaard | Microservices in. NET | |
| US12293151B2 (en) | AI-based language testing process for CI/CD | |
| CN112486631B (en) | Method for constructing virtual machine mirror image | |
| CN113220592A (en) | Processing method and device for automated testing resources, server and storage medium | |
| Thomas et al. | Simulation factory: Taming application configuration and workflow on high-end resources | |
| CN119718388B (en) | Artificial intelligence platform and expansion and updating method, device, and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200505 |