DE102024206830A1 - Method and apparatus for testing software changes and/or for detecting software errors in image recognition software - Google Patents

Abstract

Method for testing software changes and/or detecting software errors of an image recognition software, in particular an error detection software, which can be implemented on an industrial machine (210), the method comprising the steps:
- Providing (S1) a software code containing the software changes to a version control system (202);
- Determining (S2) the software changes in the provided software code by the version control system (202);
- Executing (S3) at least part of the software code on a virtual machine through which a test framework is provided and through which the behavior of the industrial machine (210) in response to the software code can be simulated;
- Executing (S4) at least part of the software code on a vision engine (208) based on labeled image test data that includes labeled images for testing the software code; and
- Providing (S5) a test result based on at least part of the software code executed on the Vision Engine (208).

Description

The invention relates to a method and a device for testing software changes and/or for detecting software errors of an image recognition software, in particular an error detection software, which can be implemented on an industrial machine.

State of the art

In today's manufacturing landscape, industrial image processing systems play a central role in ensuring product quality. These systems are essential for inspecting and verifying the standards of products as they move along the production line. Despite their widespread use, there is a gap in the availability of automated test frameworks designed to evaluate the code quality of these image processing systems. Traditional software development methods are often unable to adapt to the unique and demanding environments of industrial applications, which can lead to inefficiencies and potential errors.

The lack of robust software quality standards within measurement systems is a problem in these production environments. This deficiency can lead to increased production costs, longer downtimes, and a higher number of defective parts. Consequently, there is a need for an automated test framework that evaluates the code quality of industrial image processing systems.

In the age of Industry 4.0, the reliance on complex software systems in industrial processes, particularly in image processing, continues to grow. This increasing complexity underscores the importance of ensuring the robustness and reliability of these systems. A critical challenge that arises is the quality assurance of software implementations, which is essential to prevent costly disruptions in the manufacturing process. Conventional testing methods often fail to meet the specific needs and complexities of industrial image processing systems, leading to potential software defects.

While manual testing is an alternative, it has its drawbacks. It is time-consuming, resource-intensive, and prone to human error, making consistent and automated testing after code changes difficult. The goal of a dedicated test framework is to optimize processes sustainably and reduce costs. This is particularly important in vision software, where adjustments to the optical properties of materials can disrupt the original setup and lead to errors. These challenges are compounded by rapid development cycles and the need to manage multiple products and processes within a single station. Addressing these setup conflicts is one of the biggest challenges for process engineers, and errors can lead to significant risks.

If reworking a part is permitted, the downtime costs caused by the production stoppage can be significant. Additionally, there are ongoing costs for engineers who need to inspect various parts for sustainability. Furthermore, there are costs for setting up the part at the station and for the reworking itself. If reworking is not allowed, for example, if the output material of the parts has expired, the downtime and inspection costs remain the same as in the previous scenario. Additional costs arise for scrapped parts and recycling.

The third and, from a reliability perspective, most critical point is the possibility that NOK (Not OK) parts might be mistakenly identified as OK parts. This misclassification can lead to a reduction in the part's lifespan. Furthermore, there is a risk that customers will identify the part as defective upon receipt, potentially resulting in returns. In the worst-case scenario, if this error is not detected in time, it could lead to a large-scale recall.

To provide a comprehensive understanding of the potential cost savings from integrating the patented software test framework for optical inspection software, a cost estimate was conducted at a single station in a power electronics production line. The analysis revealed that software errors accounted for more than 30% of the station's total failures and resulted in a significant number of false positives. These false positives frequently lead to unnecessary rejections of components that are actually of acceptable quality, incurring additional costs for manual intervention and reassessment by failure analysis teams.

Further investigation revealed that only about 40% of the parts identified as faulty by the software were actually defective, indicating a high rate of false positives. Consequently, most of the parts identified as defective by the software undergo manual inspection and analysis, leading to increased operating costs. The patented invention aims to solve this problem by automating the test framework, thereby reducing the The number of false-positive results is reduced and the need for manual intervention is minimized.

Against this background, it is an object of the invention to provide an improved method and/or an improved device.

The problem is solved by a method according to the features of claim 1. The problem is solved by a device according to the features of claim 10.

Disclosure of the invention

According to a first aspect, a method for testing software changes and/or detecting software errors of an image recognition software, in particular an error detection software, which can be implemented on an industrial machine, is proposed.

• - Bereitstellen eines Softwarecodes, der die Softwareänderungen aufweist, an ein Versions-Kontrollsystem;
• - Ermitteln der Softwareänderungen in dem bereitgestellten Softwarecode durch das Versions-Kontrollsystem;
• - Ausführen zumindest eines Teils des Softwarecodes auf einer virtuellen Maschine, durch die ein Testframework bereitgestellt wird, und durch die ein Verhalten der industriellen Maschine in Reaktion auf den Softwarecode simulierbar ist;
• - Ausführen zumindest eines Teils des Softwarecodes auf einer Vision Engine auf Basis von gelabelten Bildtestdaten, die gelabelte Bilder aufweisen zum Testen des Softwarecodes; und
• - Bereitstellen eines Testergebnisses auf Basis des auf der Vision Engine ausgeführten zumindest einen Teils des Softwarecodes.

The procedure includes the following steps:

• - Providing software code containing the software changes to a version control system;
• - Determining software changes in the provided software code using the version control system;
• - Executing at least part of the software code on a virtual machine, which provides a test framework and allows the simulation of the industrial machine's behavior in response to the software code;
• - Executing at least part of the software code on a vision engine based on labeled image test data that includes labeled images for testing the software code; and
• - Providing a test result based on at least part of the software code executed on the Vision Engine.

It is understood that the steps according to the invention, as well as further optional steps, do not necessarily have to be carried out in the sequence shown, but can also be carried out in a different sequence. Furthermore, additional intermediate steps may be provided. The individual steps may also comprise one or more sub-steps without thereby departing from the scope of the method according to the invention.

• - Bereitstellen eines Softwarecodes, der die Softwareänderungen aufweist, an ein Versions-Kontrollsystem;
• - Ermitteln der Softwareänderungen in dem bereitgestellten Softwarecode durch das Versions-Kontrollsystem;
• - Ausführen zumindest eines Teils des Softwarecodes auf einer virtuellen Maschine, durch die ein Testframework bereitgestellt wird, und durch die ein Verhalten der industriellen Maschine in Reaktion auf den Softwarecode simulierbar ist;
• - Ausführen zumindest eines Teils des Softwarecodes auf einer Vision Engine auf Basis von gelabelten Bildtestdaten, die gelabelte Bilder aufweisen zum Testen des Softwarecodes; und
• - Bereitstellen eines Testergebnisses auf Basis des auf der Vision Engine ausgeführten zumindest einen Teils des Softwarecodes.

According to a second aspect, a device for testing software changes and/or detecting software errors in image recognition software, in particular error detection software, which can be implemented on an industrial machine, is proposed. The device includes an evaluation and computing unit configured to perform the following steps:

• - Providing software code containing the software changes to a version control system;
• - Determining software changes in the provided software code using the version control system;
• - Executing at least part of the software code on a virtual machine, which provides a test framework and allows the simulation of the industrial machine's behavior in response to the software code;
• - Executing at least part of the software code on a vision engine based on labeled image test data that includes labeled images for testing the software code; and
• - Providing a test result based on at least part of the software code executed on the Vision Engine.

The statements made regarding the procedure apply accordingly to the device. It is understood that linguistic modifications of procedurally formulated features can be reformulated for the device according to common linguistic practice, without such formulations needing to be explicitly listed here.

The present method and apparatus provide an automated test framework specifically designed for industrial image processing systems. This framework not only improves the efficiency and effectiveness of quality assurance processes but also ensures the reliability and robustness of these critical industrial systems. The framework was developed to address the specific challenges associated with ensuring software quality and reliability in production environments where image processing plays a crucial role in quality control and inspection processes. This software test framework can be used in various manufacturing sectors to contribute to improved product quality, reduced costs, and increased operational efficiency, and in particular to detect, prevent, or at least minimize software errors.

This framework offers standardized and automated testing procedures that can be seamlessly integrated into existing software development workflows. By using the principles of By applying the principles of continuous integration and software testing, the framework strengthens the reliability and stability of software changes in industrial environments. A key aspect of the invention is its versatility, which is geared towards various image processing systems commonly used in industrial image processing. The proposed software testing framework offers several advantages over existing methods. First, it provides a comprehensive and automated testing environment that improves the efficiency of quality assurance processes. Second, by enabling continuous testing throughout the entire software development lifecycle, the framework allows for the early detection and correction of software errors, thereby reducing the risk of production downtime and the associated costs.

Furthermore, the framework's adaptability to various image processing systems ensures versatility and applicability in a wide range of industrial environments. This leads to improved software quality, as it enables comprehensive software testing of image processing algorithms, allowing software errors to be detected and corrected before they impact production processes. This results in higher software quality and reliability, reducing the risk of product defects and associated costs. Additionally, the framework leads to improved operational efficiency through the automation of software testing processes. It streamlines quality assurance procedures and reduces the time and resources required for testing.

This optimization enables improved operational efficiency, minimizes production line downtime, and maximizes productivity. Furthermore, the framework provides standardization and adaptability. For example, it offers a standardized approach to regression testing across diverse manufacturing environments. It can be adapted to various image processing systems commonly used in industry, thus facilitating seamless integration and consistent testing methods.

Furthermore, the framework contributes to cost reduction by enabling the early detection and correction of software errors. This avoids production downtime and rework. By minimizing the impact of software errors, it contributes to overall cost reduction and increased profitability.

In production, the framework reduces defective parts due to misalignment, rework, production downtime, and production costs, as the design phase addresses these issues. Furthermore, during optical inspection, it ensures that image processing algorithms are thoroughly tested, leading to a reduction in defective parts due to misalignment or errors in the inspection process. This results in less rework and fewer production downtimes, enabling on-time delivery of high-quality parts.

The software testing framework can be seamlessly implemented into existing camera platforms, such as Halcon, Keyence, and Cognex. Using modern technologies like Python, Jenkins, and Git, the invention ensures comprehensive test coverage and seamless integration into existing development processes.

Test cases are carefully defined by this framework to cover various functions of image processing algorithms and enable a thorough evaluation of software changes. The framework's architecture allows for seamless integration with existing development tools and enables continuous testing and real-time feedback throughout the entire software development lifecycle.

By taking specific requirements into account and integrating with existing infrastructures such as Jenkins Master, an efficient and robust test environment is created that meets the challenges of industrial applications. Furthermore, the invention provides a mechanism for validating its effectiveness through comparative testing with competitor products, thereby demonstrating its superiority in terms of software quality and performance.

Efficiency and effectiveness are two factors that should be considered when conducting tests in an industrial environment. Efficiency primarily refers to a test framework's ability to execute test cases quickly and with minimal resource consumption.

This is important because industrial environments often use large and complex software systems, and testing can be time-consuming and resource-intensive. An efficient regression testing framework can help reduce the time and resources spent on testing, thereby increasing overall productivity and reducing costs.

Effectiveness primarily refers to a test framework's ability to detect errors and defects in the software under test. This is crucial in an industrial environment, as faulty software can lead to product defects, production delays, and other problems that can impact the final product.

An effective software testing framework can help identify and resolve problems early in the software development process, saving time and costs later on. An efficient and effective framework is therefore preferable for ensuring the reliability and quality of software in industrial environments. This requires, among other things, continuous monitoring of the software lifecycle to respond to changing conditions in the respective test scenario.

Test design involves selecting and creating test cases to ensure that changes made do not affect existing functionality. This process helps understand the scope of test coverage and the potential reuse of test cases, which in turn allows conclusions to be drawn about the effectiveness of the test strategy. The focus is preferably on identifying test cases that are critical for detecting errors, with the goal of creating an effective test suite. The selection and prioritization of tests, as well as the selection and prioritization of test cases, play a crucial role in enabling comprehensive test coverage. During test selection, suitable test cases are preferably chosen from a pool of test cases to cover all areas. The test cases are preferably ranked according to their importance to ensure that critical areas are tested first. In this phase, it is preferable to define the goals and requirements of the regression test. Each test case should be linked to the requirements it is intended to validate to ensure that all requirements are covered by the tests.

In the development environment, the vision programs are hosted on a virtual server to ensure accessibility for all developers. Every change to the vision programs is preferably tracked using version control (Git). These vision programs preferably comprise various procedures that implement the industrial processes on the system. The procedures preferably process and transform the image data collected during the process and deliver the process measurements as a result.

In the test environment, the results of these procedures are indirectly verified as a test for the correctness of the procedures and thus of the entire program. This approach allows potential errors or inconsistencies in the procedures to be detected and corrected early on, ensuring the reliability and performance of the image processing programs. In the defined test environment, source code is preferably used to call and initialize the updated vision program and to evaluate the changes made.

When the modified program is run, the necessary initialization steps are preferably executed, and the various functions are run to ensure that the intended changes have been implemented correctly. Subsequently, the changes are preferably evaluated to ensure that performance has been improved, no undesirable side effects occur, and/or the program functions as expected. This process allows developers to verify the impact of their changes and ensure that the updated vision program continues to meet requirements.

In another aspect, it is proposed that if a positive test result is provided indicating that the software changes do not contain a software error, the software code incorporating the software changes should be implemented on the industrial machine, the industrial machine preferably being configured to perform an optical inspection of a component based on the software code, in particular the error detection software.

This industrial machine is preferably designed to perform an optical inspection of a component based on the implemented software code. Specifically, the software code is fault detection software that enables the industrial machine to identify potential defects in the components during inspection.

Automated optical inspection allows for the classification of components into OK (OK) and non-OK (reject) parts. This classification is preferably based on image data captured by an optical sensor. The data is then evaluated using a defect detection algorithm.

Another aspect proposes that, before executing at least part of the software code on the virtual machine, the identified software changes should be processed in the provided software code by an automated framework for triggering a test platform with a variety of test scenarios, wherein the automated framework, in particular Jenkins, preferably has a master and multiple agents.

The processing of the identified software changes occurs before at least a portion of the software code is executed on the virtual machine. The identified software changes in the deployed software code are processed by an automated framework. This automated framework is designed to trigger a test platform with a variety of test scenarios. The test platform comprises a variety of test scenarios that are triggered by the automated framework. The framework preferably includes Jenkins, a widely used tool for continuous integration and continuous deployment. The framework, specifically Jenkins, preferably consists of a master and multiple agents that execute the tests in parallel and efficiently.

In another aspect, it is proposed that providing the test result involves providing a test result from the Vision Engine to the test framework and processing the test result by the test framework.

Another aspect is proposed to involve providing the test result to the automated framework.

Another aspect is proposed: that the test result should also be manually reviewed by an expert.

In another aspect, a control unit is also claimed, which is included in a robotics system and/or an industrial machine, and on which the present method can be carried out in one of its aspects. The robotics system and/or the industrial machine is preferably configured to automatically optically inspect components and thus classify them as OK and not OK components.

The classification algorithm for detecting defects in components can be analytically structured and/or at least include part of a machine learning model.

In another aspect, a computer program is claimed to contain program code capable of executing at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, a computer program (product) is claimed to comprise instructions that, when executed by a computer, cause it to execute the method(s) in one of its aspects.

In a further aspect, a computer-readable data carrier containing the program code of a computer program is proposed to execute at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, the invention relates to a computer-readable (storage) medium comprising instructions which, when executed by a computer, cause it to execute the method/steps of the method in one of its aspects.

The described configurations and training programs can be combined in any way desired.

Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or subsequently with regard to the exemplary embodiments that are not explicitly mentioned.

Brief description of the drawings

The accompanying drawings are intended to provide a further understanding of the embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain the principles and concepts of the invention.

• 1 zeigt ein schematisches Flussdiagramm eines Ausführungsbeispiels des vorliegenden Verfahrens.
• 2 zeigt ein schematisches Blockdiagramm eines Ausführungsbeispiels des vorliegenden Verfahrens.
• 3 zeigt ein spezifisches Ablaufdiagramm eines spezifischen Ausführungsbeispiels des vorliegenden Verfahrens.

Other embodiments and many of the aforementioned advantages become apparent with reference to the drawings. The elements depicted in the drawings are not necessarily shown to scale.

• 1 shows a schematic flowchart of an exemplary embodiment of the present method.
• 2 shows a schematic block diagram of an embodiment of the present method.
• 3 shows a specific flowchart of a specific embodiment of the present method.

In the figures of the drawings, identical reference symbols denote identical or functionally equivalent elements, parts or components, unless otherwise stated.

1 shows a schematic flowchart of a procedure for testing software changes and/or detecting software errors of an image recognition software, in particular an error detection software, which can be implemented on an industrial machine.

The method can be carried out in any embodiment, at least partially, by a device 100, which may comprise several components not shown in detail, for example, one or more provisioning units and/or at least one evaluation and computing unit. It is understood that the provisioning unit may be designed together with the evaluation and computing unit, or it may be different from it. Furthermore, the device 100, which may be part of a system, may comprise a storage unit and/or an output unit and/or a display unit and/or an input unit.

• In einem Schritt S1 erfolgt ein Bereitstellen eines Softwarecodes, der die Softwareänderungen aufweist, an ein Versions-Kontrollsystem.
• In einem Schritt S2 erfolgt ein Ermitteln der Softwareänderungen in dem bereitgestellten Softwarecode durch das Versions-Kontrollsystem.
• In einem Schritt S3 erfolgt ein Ausführen zumindest eines Teils des Softwarecodes auf einer virtuellen Maschine, durch die ein Testframework bereitgestellt wird, und durch die ein Verhalten der industriellen Maschine in Reaktion auf den Softwarecode simulierbar ist.
• In einem Schritt S4 erfolgt ein Ausführen zumindest eines Teils des Softwarecodes auf einer Vision Engine auf Basis von gelabelten Bildtestdaten, die gelabelte Bilder aufweisen zum Testen des Softwarecodes.
• In einem Schritt S5 erfolgt ein Bereitstellen eines Testergebnisses auf Basis des auf der Vision Engine ausgeführten zumindest einen Teils des Softwarecodes.

The computer-implemented procedure includes at least the following steps:

• In step S1, a software code containing the software changes is provided to a version control system.
• In step S2, the version control system identifies the software changes in the provided software code.
• In step S3, at least part of the software code is executed on a virtual machine, which provides a test framework and allows the behavior of the industrial machine in response to the software code to be simulated.
• In step S4, at least part of the software code is executed on a vision engine based on labeled image test data, which contains labeled images for testing the software code.
• In step S5, a test result is provided based on at least part of the software code executed on the Vision Engine.

2 Figure 200 shows a hyper-test framework 200 underlying this invention. This framework 200 creates a comprehensive test environment that utilizes a version control system (VCS) 202, a test framework 204 (implemented, for example, using Python), an automated framework 206 for triggering a test platform (e.g., Jenkins), and optical inspection software or a vision engine 208 to automate the test process. It provides a structured basis for the development and execution of test series.

The automated framework 206 can be seamlessly integrated with the version control system 202, preferably using webhooks and GitHub Actions. The automated framework 206 manages parameter passing and execution control, while dedicated classes handle test configuration, execution, and evaluation.

The test cases are preferably defined to cover various functions of the image processing algorithms and enable a thorough evaluation of the software changes. Test framework 204 preferably loads the corresponding image processing software dynamically based on the selected test cases and compares the measurement results with reference JSON files. This image processing software preferably solves the respective container or module containing the optical inspection algorithm. Subsequently, the consistency, efficiency, and effectiveness of the implemented changes are evaluated.

The architecture of Framework 200 enables seamless integration into existing development tools and allows continuous testing throughout the entire lifecycle of the machine software.

The provision of the test result preferably comprises the provision of a test result from the Vision Engine 208 to the Test Framework 204 and the processing of the test result by the Test Framework 206 (step S6). The processed test result can be provided to the automated framework 206 in step S7. The automated framework 206 can return the test result to the VSC 202 in step S8. The test result can be verified in step S9. If a positive test result is provided, indicating that the software changes do not contain a software defect, the software code incorporating the software changes can be implemented (step S10) on the industrial machine 210, wherein the industrial machine 210 is preferably configured to perform an optical inspection of a component based on the software code, in particular the defect detection software.

3 This illustrates the workflow of the software test framework 200 for optical inspection systems. First, the requirements 300 for the test series are defined. Then, depending on the test case, a decision is made as to whether Jenkins is used for automated testing or whether manual test execution is performed. Subsequently, test cases 302 are developed and implemented, with an optional integration step with the optical inspection software.

If integration is performed, the tests are run on the inspection software (308); otherwise, they are run manually. The Jenkins component (304) triggers test execution, and upon completion, the framework evaluates whether the tests passed. If successful, test reports are generated; otherwise, the developers are notified of any errors. The Python Testing Framework (306) automates test execution and uses Python scripts to execute test cases and collect the results for analysis. Similarly, the optical inspection software provides test data, executes test scenarios, and collects inspection results for analysis.

The diagram provides a clear overview of the testing process, including automation and integration points, to enable comprehensive quality control in optical inspection systems.

Claims (10)

A method for testing software changes and/or detecting software errors in image recognition software, in particular error detection software, which can be implemented on an industrial machine (210), comprising the steps of: - Providing (S1) software code containing the software changes to a version control system (202); - Determining (S2) the software changes in the provided software code by the version control system (202); - Executing (S3) at least part of the software code on a virtual machine that provides a test framework and that simulates the behavior of the industrial machine (210) in response to the software code; - Executing (S4) at least part of the software code on a vision engine (208) based on labeled image test data containing labeled images for testing the software code; and - Providing (S5) a test result based on at least part of the software code executed on the Vision Engine (208). Procedure according to Claim 1 , wherein, if a positive test result is provided indicating that the software changes do not have a software defect, an implementation (S10) of the software code incorporating the software changes is carried out on the industrial machine (210), wherein the industrial machine (210) is preferably configured to perform an optical inspection of a component based on the software code, in particular the defect detection software. Procedure according to Claim 1 or 2 , wherein, prior to executing at least part of the software code on the virtual machine, the identified software changes in the provided software code are processed by an automated framework (206) to trigger a test platform with a multitude of test scenarios, wherein the automated framework (206), in particular Jenkins, preferably comprises a master and multiple agents. Method according to one of the preceding claims, wherein the provision of the test result comprises a provision (S6) of a test result from the vision engine to the test framework (204) and a processing (S7) of the test result by the test framework (204). Procedure according to Claim 3 and 4 , wherein providing the test result involves providing the prepared test result to the automated framework (206). Method according to one of the preceding claims, wherein furthermore a review (S9) of the test result is carried out by an expert. Control unit of an industrial machine on which a method according to one of the preceding claims is implemented. Computer program with program code to perform at least parts of a procedure according to one of the Claims 1 until 7 to be executed when the computer program is run on a computer. Computer-readable data carrier containing the program code of a computer program, in order to execute at least parts of a procedure according to one of the Claims 1 until 7 to be executed when the computer program is run on a computer. Device (100) for testing software changes and/or detecting software errors of image recognition software, in particular error detection software, which can be implemented on an industrial machine (210), wherein the device (100) has an evaluation and computing unit configured to perform the following steps: - providing software code containing the software changes to a version control system (202); - identifying the software changes in the provided software code by the version control system (202); - executing at least part of the software code on a virtual machine through which a test framework (204) is provided and through which a behavior of the industrial machine (210) is displayed. in response to the software code is simulated; - Executing at least part of the software code on a Vision Engine (208); and - Providing a test result based on at least part of the software code executed on the Vision Engine (208).