CN114780109A - Python project third-party library dependence automatic analysis and installation method - Google Patents

Abstract

A third-party library dependency of a Python project is an automatic analysis and installation method, which relates to the technical field of software reliability detection. First, collect the data of all third libraries on the Python package management warehouse PyPI to form a local knowledge base; enter a Python project, parse its configuration file, and obtain all direct dependencies of the project; build the project according to the local knowledge base, including all direct dependencies and Indirect dependencies and the complete dependency tree of all versions that meet the constraint range, and then solve all versions that meet the constraint range; output the final installation script according to the third-party libraries installed in the local environment. This method parses and recommends libraries that can reuse locally installed third-party libraries as much as possible while meeting project constraints. Implement automatic parsing, installation and reuse of third-party libraries for Python projects. Save users download time and installation time required to install projects and reduce hard disk space usage.

Description

Python project third-party library dependency automatic analysis and installation method

technical field

The invention relates to the technical field of software reliability detection, in particular to a third-party library dependency automatic analysis and installation method of a Python project.

Background technique

In the contemporary software development process, the reuse of third-party libraries brings great convenience to developers, making some software developers focus more on the development of their own business functions. PyPI is a successful example in the Python ecosystem, which provides a central repository to store a large number of third-party Python libraries. As of March 2022, it has indexed nearly 3 million releases of libraries, each of which is described by metadata (e.g., library name, version information, and dependencies on other libraries). In order to import a library in PyPI, developers need to declare version constraints (that is, the constraints that the version of the library must satisfy) to restrict whether the imported library in the configuration file is compatible. Then, subject to version constraints, the Python library installer pip will recursively install the imported library and other libraries that the imported library depends on. pip frees developers from the heavy burden of managing library dependencies. However, due to complex library dependencies, the automatic installation process will come with the risk of dependency conflict issues. If there are multiple version constraints on the same library in a project, dependency conflicts may occur.

Dependency conflicts in Python projects are mainly divided into two categories. The first is the project conflict caused by the update of other libraries in the Python ecological community. This is because pip's installation policy defaults to downloading and installing the latest version within the constraints of the third-party library. However, pip cannot guarantee that the downloaded third-party library is fully compatible. The second is the incompatibility between multiple Python projects in the local environment. When developers install multiple Python projects, the version constraints for the same third-party library may be inconsistent, which leads to incompatibility conflicts between Python projects and the local environment during the installation process. Therefore, there is a need for a way to automatically resolve all versions of third-party libraries required by a Python project, and install all third-party libraries without compatibility issues such as dependency conflicts.

SUMMARY OF THE INVENTION

The purpose of the present invention is to address the above deficiencies in the prior art, and the present application provides a method for automatic analysis and installation of third-party library dependencies of Python software projects. Based on all available versions of the third library in the entire dependency tree of the project, it is expected to find the most suitable version of the third library that meets the needs of the current project in the shortest time. First collect the data of all third libraries on PyPI, the Python package management warehouse, to form a local knowledge base; then enter a Python project, parse its configuration file, and obtain all the direct dependencies of the project; then build the project according to the local knowledge base. Dependencies and indirect dependencies, as well as all their complete dependency trees that meet the constraint range, and then solve all versions that meet the constraint range; finally, output the final installation script according to the third-party library installed in the local environment, so that this method can parse the recommended Libraries can reuse locally installed third-party libraries as much as possible while meeting project constraints. This method realizes automatic parsing, installation and reuse of third-party libraries of Python projects.

The present invention comprises the following steps:

Step 1: Obtain the detailed information of all third-party libraries in the Python ecological community, including the library name, version and all its required direct dependencies, and build a local knowledge base;

Step 2: Obtain the dependency configuration file (setup.py or requirements.txt) of the Python project;

Step 3: Build the complete dependency tree of the project, generate the SMT expression and solve the constraint range;

Step 4: Optimize the generated SMT expression according to the third-party library installed in the local environment;

Step 5: After the optimization operation in Step 4, obtain the final SMT expression, and then solve it to obtain all the third-party libraries required by the Python project, whose version is the latest version within the range of the version constraints, and output the installation script;

Step 6: For the installation script in Step 5, install all the third-party dependencies in the local environment where the project is located in turn. If there is a reusable third-party library, use a soft link to link to the environment where the project is located. middle.

In step 1, the step of building a local knowledge base may be:

Step 1.1: Obtain all third-party libraries of the Python ecological community;

Access the API provided by PyPI to obtain the name and version list of each third-party library in the Python ecological community, and then, for each version, download it to the local environment through the command "pip install package_name–no-deps";

Step 1.2: Parse the dependency information of each library version;

After downloading a third-party library through the command in step 1.1, parse its setup.py file to obtain the full name and version number of the library and the full name and version number information of all other third-party libraries directly dependent on the library. The class information is stored in the database table package_req respectively, and the 4 columns are package_name, package_version, package_req_name, package_req_version;

Step 1.3: Renumber the data to form a local knowledge base;

Since the SMT constraint solver only supports numerical range solving and does not support the version number format (such as 0.0.1), it is necessary to renumber the data items in the database. Each version of each library corresponds to a unique id, and the id number is based on the version. The numbers are arranged in order from small to large; finally, a local knowledge base file is formed, and the storage format is a Json file.

In step 3, the specific steps of constructing the complete dependency tree of the project, generating the SMT expression and solving the constraint range may be:

Step 3.1: Parse the project's direct dependency library;

Parse the dependency configuration file obtained in step 2. After obtaining the configuration file, use the method in step 1.2 to parse the setup.py file or directly read the requirements.txt to obtain all the required information about the directly dependent libraries of the Python project. ;According to pip's default breadth-first parsing strategy, parse each directly dependent library of the project in turn to obtain the required version constraint range;

Step 3.2: Obtain the satisfiable version list of the directly dependent library in the knowledge base;

After obtaining the version constraint range of all directly dependent libraries of the project in step 3.1, convert it into an id list and retrieve all versions of the current library that meet the version constraint range obtained in the local knowledge base;

Step 3.3: Analyze each version in Step 3.2 to obtain information about its sub-dependent libraries, including name and version constraint range;

Step 3.4: Use the depth-first traversal algorithm to repeat the above steps 3.2 to 3.3 until the entire complete dependency tree of the Python project is constructed, and the complete dependency tree contains all the required libraries and all feasible versions of the current project;

Step 3.5: Generate SMT expressions according to the complete dependency tree in step 3.4;

Convert each dependent library and its version in the complete dependency tree in step 3.4 into the corresponding unique id in the local knowledge base, and also convert the version constraint range of the library into the id range to form an SMT expression. The main reason is that SMT expressions are only supported for numeric types for comparison. There is an OR relationship between multiple versions of each library, and an AND relationship between multiple libraries, and finally an SMT expression based on a complete dependency tree is formed.

Step 3.6: Use the constraint solving tool Z3 to solve the SMT expression, and obtain the satisfiable version constraint range of each dependent library.

Definition: SMT is an extension of SAT (Boolean Expression Satisfaction Theory), which represents the satisfiability determination problem of a class of formulas. The SMT expression here is used to determine whether the version constraint range of the third-party library in the dependency tree is not Satisfied.

In step 4, the generated SMT expression is optimized according to the installed third-party library in the local environment, and the specific steps are as follows:

Step 4.1: Get the list of installed third-party libraries package_list in the local environment;

Step 4.2: For each version of each third-party library in the package_list, add the SMT expression in step 3 in turn from the higher version to the lower version, and use the constraint solver to determine whether it is satisfied;

Step 4.3: If it is satisfied, it proves that the current version complies with the SMT constraints and can be reused, so keep it;

Step 4.4: If it is not satisfied, roll back the state of the SMT expression, and then continue to add the next version, and stop until all the packages that appear in the SMT expression have been traversed once.

In step 6, the specific steps are as follows:

For each record of the installation script in step 5, first index whether the version of the library exists in the local warehouse, if so, create a soft link directly, use ln-s in Linux, use mklink in Windows, and finally all packages Index into the virtual environment where the current project is located. If the version does not exist in the local repository, use the pip installation command pip install package==version-no-deps to install a specific version of the library into the local repository, and then create a soft link to the current virtual environment.

The beneficial effects of the present invention mainly include two parts:

1. This method allows users to automatically resolve all dependent third-party libraries and their versions required by the project when installing a Python project. This method can also effectively solve the problem of dependency conflicts existing in the project, and make up for the problems of too large search space and too long search time to solve the dependency conflict problem of the default Python installation tool pip, which leads to installation failure and waste of bandwidth. In the process of using, users do not need to understand the specific process of Python dependency parsing, nor do they need to deal with potential dependency conflicts in the project, and can use this method directly.

2. When using this method to parse and install all the third-party libraries that the Python project depends on, you can obtain a list of the locally installed third-party libraries. In the list of dependency libraries to be installed recommended to the developer by this method, the locally installed dependency libraries are preferentially selected, so that local data can be reused as much as possible when the project is installed.

Description of drawings

Fig. 1 is the flow chart of Python software project third-party library dependency automatic analysis and installation method in the embodiment of the present invention;

Fig. 2 is the flow chart of constructing local knowledge base in the embodiment of the present invention;

3 is a flowchart of constructing a complete dependency tree and analyzing the version range of a project dependency library in an embodiment of the present invention;

FIG. 4 is a flowchart of optimizing and multiplexing a native library in an embodiment of the present invention.

Detailed ways

The invention provides an automatic analysis and installation method for third-party library dependencies of Python software projects, detects the configuration file of the project, automatically analyzes all third-party libraries required by the project and the most suitable version range, and finally according to the locally installed third-party libraries It can automatically solve and give all the third-party libraries and their versions required by the current project, and automatically install all versions, and reuse the locally installed third-party libraries as much as possible, so as to save the user's need for installing the project. Download time and installation time and reduce hard disk space usage.

The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. The following examples are intended to illustrate the present invention, but not to limit the scope of the present invention.

The implementation process of this method is shown in Figure 1, and the process is as follows:

Step 1: The specific content of the local knowledge base construction is shown in Figure 2, including:

Step 1.1: Obtain all third-party libraries of the Python ecological community;

Access the API provided by PyPI to get the name and version list of every third-party library in the Python ecosystem. Then, for each version, download it to the local environment with the command "pip install package_name --no-deps".

Step 1.2: Parse the dependency information of each library version;

After downloading a third-party library through the command in step 1.1, parse its setup.py file to obtain the full name and version number of the library and the full name and version number information of all other third-party libraries that the library directly depends on. The above four types of information are stored in the database table package_req respectively, and the four columns are package_name, package_version, package_req_name, package_req_version.

Step 1.3: Renumber the data to form a local knowledge base;

Since the SMT constraint solver only supports numerical range solving and does not support the version number format (such as 0.0.1), it is necessary to renumber the data items in the database. Each version of each library corresponds to a unique id, and the id number is based on the version. Numbers are arranged in order from smallest to largest. Finally, a local knowledge base file is formed, and the storage format is Json file.

Step 2: Obtain the configuration file of the Python project and parse its direct dependent library information;

Every Python project has at least one configuration file (such as setup.py or requirements.txt).

Step 3: Build a complete dependency tree, generate SMT expressions and solve the constraint range, as shown in Figure 3, including:

Step 3.1: Parse the project's direct dependency library: After obtaining the configuration file, use the method in Step 1.2 to parse the setup.py file or directly read the requirements.txt to obtain the information of all the directly dependent libraries of the Python project. According to pip's default breadth-first resolution strategy, each directly dependent library of the project is parsed in turn to obtain the required version constraint range.

Step 3.2: Obtain the satisfiable version list of the directly dependent library in the knowledge base: After obtaining the version constraint range of all the directly dependent libraries of the project in step 3.1, convert it into an id list and retrieve the corresponding version in the local knowledge base.

Step 3.3: Analyze each version in Step 3.2 to obtain information about its sub-dependent libraries, including name and version constraint range.

Step 3.4: Repeat steps 3.2 to 3.3 above using a depth-first traversal algorithm until a complete dependency tree is built that includes all the dependency libraries required by the current Python project and all their versions.

Step 3.5: Generate SMT expressions according to the complete dependency tree in step 3.4: Convert each dependency library and its version in the complete dependency tree in step 3.4 into the corresponding unique id in the local knowledge base, and also convert the version of the library Constraint ranges are converted to ranges of ids, forming SMT expressions. The main reason is that SMT expressions are only supported for numeric types for comparison. There is an OR relationship between multiple versions of each library, and an AND relationship between multiple libraries, and finally an SMT expression based on a complete dependency tree is formed.

Step 3.6: Use the constraint solving tool Z3 to solve the SMT expression, and obtain the satisfiable version constraint range of each dependent library.

Step 4: Optimize the generated SMT expression according to the installed third-party library in the local environment. The specific content is shown in Figure 4, including:

Step 4.1: Get the list of installed third-party libraries package_list in the local environment;

With this method, when the Python project is installed for the first time, a local repository will be created to store the desired third-party libraries of the Python project. When the developer uses this method to install the project again, it will first obtain the package_list of all third-party libraries installed in the local warehouse at this time.

Step 4.2: For each version of each third-party library in the package_list, use the constraint solver to determine whether it is satisfied;

For each library in package_list, if the library exists in the SMT expression, get all its downloaded versions, traverse each version in turn from the high version number to the low version number, and convert it into the local knowledge base through the transformation function transform The unique id is added to the SMT expression in step 3 in the form of package_name==id, and finally the constraint solver is used to determine whether the expression can be satisfied at this time.

Step 4.3: If it can be satisfied, it proves that the version of the current library complies with the SMT constraints and can be reused, so it is reserved;

If package_name==id is added to the SMT expression, and the solution result is in the sat state, it means that it can be satisfied, and the current constraint is retained, and then jumps to the next library for traversal search.

Step 4.4: If it is not satisfied, that is, the solution result of SMT is unsat, then roll back the state of the SMT expression, and then continue to add the next version, until all the packages that appear in the SMT expression have been traversed once, then stop;

Step 5: After the optimization operation in Step 4, obtain the final SMT expression, and then solve it to obtain all the third-party libraries required by the Python project, whose version is the latest version within the range of the version constraints, and output the installation script;

Solve the final SMT expression using the Z3 solver to obtain a list of library versions, each of the form package_name==id. Then map the id to the local knowledge base, obtain the corresponding version number, convert it into the form of package_name==version, and output it to the installation script.

Step 6: For the installation script in Step 5, install all the third-party dependencies in the local environment where the project is located in turn. If there is a reusable third-party library described in Step 4, use the soft connection method, Link to the environment where the project is located.

For each record of the installation script in step 5, first index whether the version of the library exists in the local warehouse, if so, create a soft link directly, use ln-s in Linux, use mklink in Windows, and finally all packages Index into the virtual environment where the current project is located. If the version does not exist in the local repository, use the pip installation command pip install package==version-no-deps to install a specific version of the library into the local repository, and then create a soft link to the current virtual environment.

Step 7: The installation is successful, and the installation result is output.

The invention provides an automatic analysis and installation method for third-party library dependencies of Python software projects, detects the configuration file of the project, automatically analyzes all third-party libraries required by the project and the most suitable version range, and finally according to the locally installed third-party libraries It can automatically solve and give all the third-party libraries and their versions required by the current project, and automatically install all versions, and reuse the locally installed third-party libraries as much as possible, so as to save the user's need for installing the project. Download time and installation time and reduce hard disk space usage.

Claims (5)

1. The third-party library dependency automatic analysis and installation method of the Python project is characterized in that it comprises the following steps: Step 1: Obtain the detailed information of all third-party libraries in the Python ecological community, including the library name, version and all its required direct dependencies, and build a local knowledge base; Step 2: Get the dependency configuration file of the Python project, setup.py or requirements.txt; Step 3: Build the complete dependency tree of the project, generate the SMT expression and solve the constraint range; Step 4: Optimize the generated SMT expression according to the third-party library installed in the local environment; Step 5: After the optimization operation in Step 4, obtain the final SMT expression, and then solve it to obtain all the third-party libraries required by the Python project, whose version is the latest version within the range of the version constraints, and output the installation script; Step 6: For the installation script in Step 5, install all the third-party dependencies in the local environment where the project is located in turn. If there is a reusable third-party library, use a soft link to link to the environment where the project is located. middle. 2. the third-party library of Python project as claimed in claim 1 relies on automatic analysis and installation method, it is characterized in that in step 1, the described step of building local knowledge base is: Step 1.1: Obtain all third-party libraries of the Python ecological community; Access the API provided by PyPI to obtain the name and version list of each third-party library in the Python ecological community, and then, for each version, download it to the local environment through the command "pip install package_name-no-deps"; Step 1.2: Parse the dependency information of each library version; After downloading a third-party library through the command in step 1.1, parse its setup.py file to obtain the full name and version number of the library and the full name and version number information of all other third-party libraries directly dependent on the library. The class information is stored in the database table package_req respectively, and the 4 columns are package_name, package_version, package_req_name, package_req_version; Step 1.3: Renumber the data to form a local knowledge base; Since the SMT constraint solver only supports numerical range solving and does not support the version number format (such as 0.0.1), it is necessary to renumber the data items in the database. Each version of each library corresponds to a unique id, and the id number is based on the version. The numbers are arranged in order from small to large; finally, a local knowledge base file is formed, and the storage format is a Json file. 3. the third-party library of Python project as claimed in claim 1 relies on automatic analysis and installation method, it is characterized in that in step 3, the complete dependency tree of described construction project, the concrete steps that generate SMT expression and solve constraint scope are: Step 3.1: Parse the project's direct dependency library; Parse the dependency configuration file obtained in step 2. After obtaining the configuration file, use the method in step 1.2 to parse the setup.py file or directly read the requirements.txt to obtain all the required information about the directly dependent libraries of the Python project. ;According to pip's default breadth-first parsing strategy, parse each directly dependent library of the project in turn to obtain the required version constraint range; Step 3.2: Obtain the satisfiable version list of the directly dependent library in the knowledge base; After obtaining the version constraint range of all directly dependent libraries of the project in step 3.1, convert it into an id list and retrieve all versions of the current library that conform to the version constraint range obtained in the local knowledge base; Step 3.3: Analyze each version in Step 3.2 to obtain information about its sub-dependent libraries, including name and version constraint range; Step 3.4: Use the depth-first traversal algorithm to repeat the above steps 3.2 to 3.3 until the entire complete dependency tree of the Python project is constructed, and the complete dependency tree contains all the required libraries and all feasible versions of the current project; Step 3.5: Generate SMT expressions according to the complete dependency tree in step 3.4; Convert each dependency library and its version in the complete dependency tree in step 3.4 into the corresponding unique id in the local knowledge base, and also convert the version constraint range of the library into the id range to form an SMT expression; SMT expression It only supports numeric types for comparison; the OR relationship between multiple versions of each library, and the AND relationship between multiple libraries, finally forms an SMT expression based on a complete dependency tree; Step 3.6: Use the constraint solving tool Z3 to solve the SMT expression, and obtain the satisfiable version constraint range of each dependent library. 4. the third-party library of Python project as claimed in claim 1 relies on automatic analysis and installation method, it is characterized in that in step 4, described according to the third-party library installed in the local environment to optimize the generated SMT expression, concrete Proceed as follows: Step 4.1: Get the list of installed third-party libraries package_list in the local environment; Step 4.2: For each version of each third-party library in the package_list, add the SMT expression in step 3 in turn from the higher version to the lower version, and use the constraint solver to determine whether it is satisfied; Step 4.3: If it is satisfied, it proves that the current version complies with the SMT constraints and can be reused, so keep it; Step 4.4: If it is not satisfied, roll back the state of the SMT expression, and then continue to add the next version, and stop until all the packages that appear in the SMT expression have been traversed once. 5. the third-party library of Python project as claimed in claim 1 relies on automatic analysis and installation method, it is characterized in that in step 6, concrete steps are as follows: For each record of the installation script in step 5, first index whether the version of the library exists in the local warehouse, and if so, create a soft link directly, use ln-s in Linux, use mklink in Windows, and finally all packages Index into the virtual environment where the current project is located; if the version does not exist in the local warehouse, use the pip installation command pipinstall package==version-no-deps to install a specific version of the library into the local warehouse, and then create a soft link to in the current virtual environment.

Images