TWI904063B - Method for keyword-based testing and system therefor - Google Patents

Abstract

There are provided a method for keyword-based testing and a system therefor. The method may include obtaining information on a plurality of keywords used to create a test case of target software, the obtained information including code routines for the plurality of keywords, calculating complexity for each of the plurality of keywords by analyzing the code routines, calculating a score for each of the plurality of keywords based on the complexity, and determining ranking of the plurality of keywords based on the score. According to such a method, ranking information according to usability of keywords may be provided to testers of the target software, and as a result, difficulty in creating the test case may be reduced and convenience of the testers may be improved.

Description

Keyword-based testing methods and systems

This invention relates to a keyword-based testing method and system.

To address the difficulties and efficiency issues in test case creation, various testing methodologies have been proposed, among which keyword-based testing is the most noteworthy.

Keyword-based testing is a testing methodology that aims to define test components using keywords based on various actions (e.g., login, click, etc.) and create test cases based on combinations of keywords. Therefore, in a keyword-based testing environment, testers do not need to directly create test code based on specific actions; they can quickly create test cases by appropriately selecting keywords corresponding to the actions (i.e., predefined keywords).

However, in keyword-based testing environments, there are often a large number of keywords. For example, an action may have multiple keywords. Therefore, unless you are an experienced tester, it is not easy to determine which keyword is more useful (or suitable for the test case) by looking at the keyword description.

Existing technical literature

Patent documents

Korean Patent Publication No. 10-1728170 (Published on April 19, 2017)

Some embodiments of the present invention aim to solve the technical problem of providing a keyword-based testing method and system that can reduce the difficulty of creating test cases and improve the convenience for testers.

Another technical problem to be solved by some embodiments of the present invention is to provide a method and system for ranking various keywords used to create test cases according to their usefulness.

Another technical problem that some embodiments of the present invention aim to solve is to provide a variety of metrics that can accurately measure the usefulness of keywords.

The technical problems of this invention are not limited to those described above. Those skilled in the art will clearly understand other technical problems not mentioned in the following description.

A keyword-based testing method according to some embodiments of the present invention for solving the aforementioned technical problem is a method performed by at least one computing device and may include the following steps: acquiring information on a plurality of keywords for creating test cases of target software, the acquired information including code routines of the plurality of keywords; analyzing the code routines to calculate the complexity of each of the plurality of keywords; calculating a score for each of the plurality of keywords based on the complexity; and determining a ranking of the plurality of keywords based on the scores.

In some embodiments, the steps of calculating the complexity may include: identifying valid code lines from the code routines of a particular keyword; and calculating the complexity of the particular keyword based on the number of valid code lines.

In some embodiments, the acquired information may include multiple test cases, and the steps of calculating the score of each of the multiple keywords may include the following steps: calculating the frequency of the multiple keywords used in the multiple test cases; and further calculating the score of each of the multiple keywords based on the calculated frequency.

In some embodiments, the steps of identifying the valid code lines may include identifying the valid code lines by excluding code lines associated with declarations, comments, and spaces.

In some embodiments, the acquired information may include multiple test cases, and the steps for calculating the score of each of the multiple keywords may include the following steps: monitoring the execution status of the code routines of the multiple keywords called when the multiple test cases are running; determining the number of times errors occur in the code routines of each of the multiple keywords called based on the monitoring results, and calculating the reliability of each of the multiple keywords called based on the determined number; and further calculating the score of each of the multiple keywords based on the calculated reliability.

In some embodiments, the steps of calculating the confidence level may include determining whether the error has occurred by using monitoring code inserted in the code routine of each of the plurality of keywords.

In some embodiments, the steps of calculating the score of each of the plurality of keywords may include the following steps: analyzing the acquired information to calculate the values of the complexity and other measures; reducing only the complexity; and calculating the score of each of the plurality of keywords based on the adjusted values of the complexity and other measures.

In some embodiments, the adjustment range of the complexity of a particular keyword among the plurality of keywords is determined based on the frequency and extent of modification of the code routine of the particular keyword.

In some embodiments, the information obtained is first information, and the method may further include the following steps: obtaining second information of the plurality of keywords, the second information may include information that is newer than the first information; analyzing the second information to update the score and ranking of each of the plurality of keywords.

In some embodiments, the method may also include the step of providing ranking information for each of the plurality of keywords through a page accessible to testers of the target software.

In some embodiments, the method may further include the following steps: receiving an action to configure new test cases from a tester of the target software via a test case creation interface; responding to the input of the action by filtering at least one keyword from the plurality of keywords that corresponds to the input action; determining recommended keywords from the filtered keywords based on the ranking of the filtered keywords; and providing the recommended keywords via the test case creation interface.

A keyword-based testing system according to some embodiments of the present invention for solving the aforementioned technical problem may include: one or more processors; and memory for storing a computer program executed by the one or more processors, the computer program including multiple instructions for performing the following operations: acquiring information on multiple keywords for creating test cases of target software, the acquired information including code routines of the multiple keywords; analyzing the code routines to calculate the complexity of each of the multiple keywords; calculating a score for each of the multiple keywords based on the complexity; and determining a ranking of the multiple keywords based on the scores.

A computer program according to some embodiments of the present invention, used to solve the aforementioned technical problem, can be stored in a computer-readable recording medium and, in conjunction with a computing device, perform the following steps: acquiring information on multiple keywords for creating test cases of target software, the acquired information including code routines of the multiple keywords; analyzing the code routines to calculate the complexity of each of the multiple keywords; calculating a score for each of the multiple keywords based on the complexity; and determining a ranking of the multiple keywords based on the scores.

According to some embodiments of this invention, keyword scores are calculated by comprehensively considering frequency, complexity, and reliability, thereby accurately reflecting the usefulness of the corresponding keywords. Furthermore, keyword rankings are determined based on the scores, enabling the ranking of various keywords according to their usefulness.

Furthermore, the complexity of keywords is calculated by the number of valid lines of code based on code routines, thus accurately reflecting the actual complexity of code routines in terms of keyword complexity.

Furthermore, considering the possibility of frequent modifications to the code routines, the complexity can be reduced when calculating the keyword score. Therefore, the impact of a complexity value with relatively low accuracy (reliability) on the keyword score can be reduced, thereby improving the accuracy (reliability) of the keyword score.

Furthermore, by providing testers of the target software with ranking information based on keyword usefulness, the difficulty of creating test cases can be reduced and the convenience for testers can be improved. This, in turn, can increase the speed of test case creation.

Furthermore, by providing testers with recommended keywords based on keyword ranking information when creating test cases, the difficulty of testers creating test cases can be further reduced and the convenience for testers can be further improved.

The effects of the technical concept of this invention are not limited to those described above, and those skilled in the art will clearly understand from the following description other effects not mentioned.

The various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The advantages and features of the present invention, as well as the methods for implementing them, will become clear through the embodiments described in detail below in conjunction with the accompanying drawings. However, the technical concept of the present invention is not limited to the following embodiments and can be implemented in various different forms. The following embodiments are provided only to improve the technical concept of the present invention and to enable those skilled in the art to fully understand the scope of the present invention, which is limited only by the scope of the patent application.

In describing various embodiments of the invention, detailed descriptions of known structures or functions will be omitted if they would obscure the essence of the invention. Some embodiments of the invention are described below with reference to the accompanying figures.

Unless otherwise defined, the terms (including technical and scientific terms) used in the following embodiments may be used in accordance with their meanings as commonly understood by one of ordinary skill in the art to which this invention pertains, but may also vary depending on the intent, precedent, and emergence of new technologies by one of ordinary skill in the relevant art. The terms used in this invention are for illustrative purposes and are not intended to limit the scope of this invention.

Unless explicitly specified as singular in the context, the singular expressions used in the following examples include plural concepts. Furthermore, unless explicitly specified as plural in the context, plural expressions include singular concepts.

Furthermore, terms such as first, second, A, B, (a) and (b) used in the following embodiments are used only to distinguish certain components from other components, and these terms are not intended to restrict the nature or order of the corresponding components.

Various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Figure 1 is a schematic diagram illustrating the operation of a keyword-based testing system 10 according to some embodiments of the present invention. In Figure 1, the keyword-based testing system 10 is shown as a 'testing system', and in the following description, the keyword-based testing system 10 will be simply referred to as a 'testing system'.

As shown in Figure 1, the test system 10 according to various embodiments is a computing device/system that supports keyword-based testing of target software. For example, the test system 10 may support (provide) functions such as keyword definition, creation of test cases based on keyword combinations, execution of test cases, keyword scoring, keyword ranking, and keyword recommendation.

Specifically, the testing system 10 can acquire information about multiple keywords 11 (e.g., keyword code routines, test cases, etc.) and analyze the relevant information to calculate a score (see 12) for the usefulness of each of the multiple keywords 11. Furthermore, the testing system 10 can provide ranking information based on the scores to the testers 13 of the target software. For example, the testing system 10 can provide ranking information for the multiple keywords 11 (e.g., scores, rankings, detailed scores for usage frequency/complexity/reliability, etc.) through pages accessible to the testers 13. Thus, the testers 13 can easily select useful keywords to quickly create test cases. That is, it can reduce the difficulty of creating test cases and improve the convenience for testers.

The testing system 10 can calculate scores for multiple keywords 11 based on various metrics, which will be explained in detail with reference to Figure 4 and subsequent figures.

In some implementations, the testing system 10 can automatically recommend keywords to the tester 13 based on the ranking information of multiple keywords 11. This can further reduce the difficulty of creating test cases, as will be explained in detail with reference to Figures 4 and 9.

The test system 10 described above can be implemented as at least one computing device. For example, all functions of the test system 10 may be performed by one computing device, or a first function of the test system 10 may be performed by a first computing device and a second function by a second computing device. Alternatively, a specific function of the test system 10 may be performed by multiple computing devices.

The computing device may include any device with computing function, and an example of such a device can be found in Figure 10.

For reference, a computing device is a collection of interacting components (e.g., memory, processor, etc.), and therefore can be called a 'computing system' in some cases. Furthermore, a computing system can refer to a collection of multiple interacting computing devices.

The operation of the test system 10 according to some embodiments of the present invention has been briefly illustrated above with reference to Figure 1. In the following text, for ease of understanding, schematic diagrams illustrating the concepts of the keywords mentioned in various embodiments of the present invention are illustrated with reference to Figures 2 and 3.

The keywords mentioned in the various embodiments of this invention refer to keywords in a software testing methodology known as a 'keyword-based testing methodology'. That is, a keyword means an action (e.g., logging in, clicking, etc.) or a combination of actions related to the target software (e.g., a web application (or website)), or a test component based on these actions (e.g., an abstraction that provides test components that provide code routines based on specific actions).

For reference, the 'keyword-based testing method' is also known as the 'keyword-driven testing method' or the 'action word-based testing method'.

For example, as shown in Figure 2, suppose the target software is a web application that includes a login page 20. In this case, testing the login page 20 requires login actions (or detailed actions such as ID input, PW input, login button clicks, etc.), so keywords 21 corresponding to the login actions can be exported and defined. Furthermore, in this way, various keywords used for testing (verifying) the target software can be exported and defined.

On the other hand, the description of a test case consists of a series of actions used to test (verify) the target software. Therefore, test cases can be created by combining keywords corresponding to the actions in sequence.

For example, as shown in Figure 3, suppose test case 31 for verifying the target software is created in the order of login, a specific action ('X'), and logout. In this case, the tester can easily create test case 31 by sequentially combining multiple keywords 32 to 34 corresponding to each action (of course, if there is no keyword corresponding to a specific action, the keyword needs to be defined). That is, in a keyword-based testing environment, even if the tester does not have code-level knowledge, they can easily create test cases (e.g., 31). Multiple code routines 35 to 37 for multiple keywords 32 to 34 can be pre-created by a test engineer with development knowledge (e.g., a TQA with code-level knowledge (Technical QA)), and the corresponding multiple code routines 35 to 37 can be called sequentially when running test case 31.

The concepts of keywords mentioned in various embodiments of the present invention have been explained above with reference to Figures 2 and 3. The structure and operation of the test system 10 are further explained in detail below with reference to Figure 4.

Figure 4 is a schematic block diagram showing a test system 10 according to some embodiments of the present invention.

As shown in Figure 4, the test system 10 according to various embodiments can be configured to include a keyword information acquisition unit 41, a keyword analysis unit 42, a keyword ranking unit 43, and a keyword recommendation unit 44. However, Figure 4 only shows components relevant to embodiments of the present invention. Therefore, those skilled in the art should understand that other general-purpose components (e.g., processors, memory, etc.) may be included in addition to those shown in Figure 4. Furthermore, the components of the test system 10 shown in Figure 4 represent functionally distinct functional units 41-47. In a real physical environment, multiple components can be implemented in an integrated form, or specific components can be separated into multiple sub-functional units. The components 41-47 of the test system 10 are described in detail below.

The keyword information acquisition unit 41 is a module used to acquire various information about various keywords used to create test cases for the target software. For example, the keyword information acquisition unit 41 can collect (acquire) descriptions of multiple pre-registered keywords, code routines for these keywords, and multiple pre-created test cases. However, the scope of this invention is not limited to this. The keyword information acquisition unit 41 can collect information (e.g., descriptions, code routines, etc.) about multiple pre-registered keywords from a keyword database (DB), but the scope of this invention is not limited to this. Test engineers and others can define corresponding keywords by creating code routines and descriptions for the keywords, and then register the defined keywords in the keyword database.

In some cases, a keyword database can be named 'keyword library', etc. Similarly, in some cases, code routines can be named 'code block', 'code snippet', 'code', 'source code', 'script', etc.

Furthermore, the keyword analysis unit 42 is a module that analyzes the usefulness of multiple keywords based on various metrics (e.g., usage frequency, complexity, reliability, etc.). For example, the keyword analysis unit 42 can analyze the information obtained by the keyword information acquisition unit 41 in terms of usage frequency, complexity, and reliability.

As shown in the figure, the keyword analysis unit 42 can be configured to include a frequency calculation unit 45, a complexity calculation unit 46, and a reliability calculation unit 47.

The frequency calculation unit 45 is a module used to calculate the usage frequency of keywords. The frequency calculation unit 45 can calculate (statistically) the frequency (i.e., number of times) of multiple keywords used in multiple test cases. To avoid redundant explanation, this will be explained later with reference to Figure 5 and subsequent figures.

For reference, in some cases, frequency can be named 'frequency score', 'frequency index', etc.

Furthermore, the complexity calculation unit 46 is a module used to calculate the complexity of keywords. For example, the complexity calculation unit 46 can analyze the code routines of keywords to calculate the complexity of the corresponding keywords, which will be explained later with reference to Figure 5 and subsequent figures.

For reference, in some cases, complexity can be named 'complexity score', 'complexity index', etc.

Furthermore, the reliability calculation unit 47 is a module used to analyze the reliability of keywords. The reliability calculation unit 47 can calculate the reliability of the corresponding keywords by monitoring the execution results (or execution status) of the code routines of the keywords, which will be explained later with reference to Figure 5 and subsequent figures.

For reference, in some cases, reliability can be named 'reliability score', 'reliability index', etc.

Furthermore, Keyword Ranking Unit 43 is a module that aggregates the analysis results of Keyword Analysis Unit 42 and calculates scores for each keyword, determining the keyword ranking based on these scores. For example, Keyword Ranking Unit 43 can comprehensively consider keyword usage frequency, complexity, and reliability to calculate the corresponding keyword score, which will be explained later with reference to Figure 5 and subsequent appendices.

For reference, in some cases, the score can be named 'usability score', 'ranking score', 'comprehensive/overall score', etc.

In some cases, the keyword ranking unit 43 can provide various keyword ranking information through pages accessible to testers (e.g., general testers, test engineers, etc.).

Furthermore, Keyword Recommendation Unit 44 is a module that recommends keywords based on keyword ranking information (e.g., ranking, score). For example, Keyword Recommendation Unit 44 can recommend one or more keywords with a higher benchmark ranking (or benchmark score) to testers (e.g., general testers with insufficient development knowledge). For further explanation of Keyword Recommendation Unit 44, please refer to Figure 9.

On the other hand, although not shown in Figure 4, the test system 10 can be configured to also include a test case generation unit (not shown) and a keyword improvement unit (not shown).

The test case generation unit (not shown) is a module used to automatically generate test cases. For example, the test case generation unit (not shown) can receive information about a series of actions (i.e., a sequence of actions) that constitute a test case and combine multiple keywords corresponding to the input actions to create a test case. In this case, the test case generation unit (not shown) can generate test cases using keywords that rank higher than or equal to a benchmark value among the multiple keywords corresponding to each action (e.g., the highest-ranking keyword).

Furthermore, the Keyword Improvement Unit (not shown) is a module used to request test engineers (e.g., TQA, engineers with code-level development knowledge) to improve keywords. For example, the Keyword Improvement Unit (not shown) can send information about multiple keywords whose ranking (or score) is lower than or equal to the benchmark value to the test engineer to request improvement of the corresponding multiple keywords.

In some cases, the keyword improvement unit (not shown) may request the test engineer to improve keywords whose usage frequency (or reliability) is lower than or equal to the baseline value and/or keywords whose usage frequency (or reliability) is on a downward trend.

The structure and operation of a test system 10 according to some embodiments of the present invention are illustrated above with reference to Figure 4. In the following, various methods (i.e., detailed operations) that can be performed on the test system 10 described above are illustrated with reference to Figure 5 and the following figures.

In the following text, for ease of understanding, it will be assumed that all steps/operations of the method described below are executed within the aforementioned test system 10. Therefore, when the execution entity of a specific step/operation is omitted, it can be understood as being executed by the test system 10. However, in a real-world environment, specific steps/operations of the method described below may also be executed on other computing devices.

Figure 5 is a schematic flowchart showing some embodiments of the keyword-based testing method according to the present invention. However, this is only a preferred embodiment for achieving the purpose of the present invention, and some steps can be added or deleted as needed. Figure 5 mainly shows the method for determining keyword ranking.

As shown in Figure 5, a keyword-based testing method according to multiple embodiments can begin with step S51, which involves acquiring information about multiple keywords used to create test cases for the target software. For example, test system 10 can acquire information such as descriptions of multiple keywords, code routines for these keywords, and test cases. This information can be acquired in any way. For example, test system 10 can collect information (e.g., descriptions, code routines, etc.) of pre-registered keywords from a keyword database.

In step S52, the acquired information can be analyzed to calculate the usage frequency, complexity, and reliability of each of the multiple keywords. The methods used to calculate (analyze) the usage frequency, complexity, and reliability of keywords are explained in detail below.

First, the method used to calculate (analyze) the frequency of keyword usage will be explained.

The testing system 10 can filter valid test cases from multiple test cases and can calculate (statistically) the frequency of each of the multiple keywords used in valid test cases. Valid test cases can be, for example, test cases whose status information (fields) are set to valid, test cases whose execution (usage) count is greater than or equal to a baseline value, test cases that have a running history within a pre-set recent period, or combinations thereof, but are not limited to these.

In some cases, the testing system 10 can assign or deduct points based on the type of test case creator (or requester) (i.e., adjust the score for usage frequency by assigning or deducting points). For example, when the creator of a particular test case is a general tester (i.e., a tester with insufficient professional (code-level) development knowledge), the testing system 10 can assign points to each of the multiple keywords used in the corresponding test case to calculate the final score for usage frequency (e.g., the less experience a general tester has, the higher/more points they can be assigned). As another example, when the creator of a particular test case is a test engineer (i.e., a tester with professional (code-level) development knowledge), the test system 10 can deduct points from each of the multiple keywords used in the corresponding test case to calculate a final score for the frequency of use (e.g., the more experienced a tester is, the higher/more points they can be given).

Furthermore, in certain situations, the testing system 10 can assign (or deduct) points based on the number of times a test case is run, the run interval, etc. For example, when the number of times a specific test case is run is greater than or equal to a baseline value, the testing system 10 can assign points to each of the multiple keywords used in the corresponding test case to calculate the final score for usage frequency (e.g., more runs result in higher scores, and vice versa). As another example, when the rate of increase in the number of times a test case is run is greater than or equal to a baseline value, the testing system 10 can assign points to each of the multiple keywords used in the corresponding test case (e.g., a steeper rate of increase results in higher scores, and vice versa). As another example, when the execution interval of a particular test case is less than the baseline value, the test system 10 can award points to each of the multiple keywords used in the corresponding test case (e.g., the shorter the execution interval, the higher the award, and vice versa).

Furthermore, the method for calculating (analyzing) keyword complexity is explained below.

Test system 10 can analyze the code routines of keywords to calculate the complexity of the corresponding keywords. However, the specific method may vary depending on the implementation.

In some embodiments, the test system 10 can identify valid lines of code from the code routines of a keyword and calculate the complexity of the corresponding keyword based on the number of valid lines of code. A valid line of code can mean the line of code that is actually executed. For example, as shown in Figure 6, the test system 10 can identify the remaining lines of code (e.g., 66, 68) from the code routine 62 of keyword 61, excluding lines of code related to declarations (e.g., declarations for variables, functions, classes, libraries, etc.), spaces, comments, etc. (63 to 65, 67, hereinafter referred to as 'invalid lines of code'). Then, the test system 10 can calculate the complexity of the corresponding keyword using a value proportional to the number of valid code lines (e.g., complexity = number of valid code lines). Thus, the actual complexity of the code routine can be accurately reflected in the complexity of the keyword.

In another embodiment, the test system 10 can further consider invalid code lines when calculating keyword complexity. For example, the test system 10 can calculate keyword complexity based on a weighted sum of the number of valid code lines and the number of invalid code lines. In this case, valid code lines can be assigned a higher weight than invalid code lines. The reason for considering invalid code lines is that the more complex the code, the more likely it is to have more comments and declarations. That is, it can be understood that since invalid code lines are also related to the complexity of the code routine, the keyword complexity is further considered when calculating invalid code lines.

In another embodiment, the test system 10 can further consider the number of variables, functions, etc., appearing in the code routine to calculate the keyword complexity. For example, the test system 10 can calculate complexity based on the number of valid lines of code and assign bonuses based on the number of variables and/or functions appearing in the code routine (e.g., the more variables or functions, the higher the bonus). Thus, the final complexity of the corresponding keyword can be calculated (which can also be understood as adjusting the keyword complexity based on the number of variables, etc.). In some cases, functions can be named 'method', 'subroutine', etc.

In another embodiment, the test system 10 can assign weights to each line of code and further calculate the keyword complexity based on the weight of each line. For example, the test system 10 can calculate the keyword complexity based on the number of lines in the code routine (or the number of valid lines of code). In this case, the weight of each line can be determined in various ways. For example, the weight of a valid line of code can be determined to be higher than that of a non-valid line of code. As another example, the weight of a blank line can be determined to be the lowest, and the weight of a declaration line can be determined to be higher than that of a comment line. As yet another example, it can be determined that the more words a line of code has, the higher the weight of that line of code. As yet another example, it can be determined that the more variables appearing in a line of code, the higher the weight of that line of code. As yet another example, the weight of a line of code containing a function call statement can be determined to be higher than that of other lines of code. As yet another example, the weight of each line of code can be determined based on various combinations of the embodiments described above.

In yet another embodiment, the complexity of the keyword can be calculated based on various combinations of the multiple embodiments described above.

Furthermore, the method used to calculate (analyze) the reliability of keywords is explained below.

The test system 10 can monitor the execution status of the code routines of multiple keywords called when running test cases to calculate the reliability of the keywords. That is, the reliability of a keyword can be understood as the reliability of the code routine of the corresponding keyword.

Specifically, the test system 10 can monitor the execution status of the code routines of each of the multiple keywords called each time a test case is run. For example, as shown in Figure 7, the test system 10 determines the number of times an error occurs in the code routine 72 of a specific keyword 71. Furthermore, the test system 10 can calculate the confidence level of the corresponding keyword 71 using a value inversely proportional to the number of errors. For example, the test system 10 can decrease the confidence level value (e.g., decrease it by '1' each time) whenever an error occurs in the code routine 72, and increase the confidence level value (e.g., increase it by '1' each time) whenever the code routine 72 completes execution without errors. Therefore, the reliability of keyword 71 can be accurately calculated.

The test system 10 can monitor the running status of the code routine (e.g., 72) by means of monitoring code (e.g., code created to record/output/send information such as whether an error occurred, whether the execution was successful, etc.) inserted into the code routine (e.g., 72) of the keyword (e.g., 71). However, the scope of the invention is not limited thereto.

For reference, if the code routines (e.g., 72) for the keyword (e.g., 71) are created in Python, monitoring code can be implemented using decorators that allow for easy addition of functionality to functions. However, the scope of this invention is not limited thereto.

Figure 8 shows the results of calculating the usage frequency, complexity, and reliability of multiple keywords. Figure 8 assumes the target software is a shopping-related web application and categorizes keywords based on the target software's functions (e.g., common functions, order functions, delivery functions, etc.). However, the scope of this invention is not limited to this.

Please refer to Figure 5 for further explanation.

In step S53, the score for each of the multiple keywords can be calculated by comprehensively considering frequency, complexity, and reliability (however, in some cases, the keyword score can also be calculated using only a subset of the three metrics). However, the calculation method can vary depending on the implementation.

In some embodiments, the testing system 10 can calculate the keyword score based on a weighted sum of usage frequency, complexity, and reliability. In this case, complexity can be assigned the lowest weight among the three measures, but the scope of the invention is not limited thereto.

In another embodiment, test system 10 may simply reduce the complexity and calculate the keyword score based on the adjusted complexity and other metrics (i.e., using frequency and reliability) (this can also be understood as a concept similar to assigning the lowest weight to complexity). It is understood that reducing only the complexity reflects its lower accuracy (reliability) compared to other metrics. More specifically, code routines are characterized by frequent modifications and significant changes in the complexity value each time a modification is made; therefore, the complexity value may vary depending on the computation time. Considering this, complexity can be understood as having lower accuracy (reliability) than other metrics. Furthermore, this can be understood as reducing the impact of complexity with this characteristic on keyword scores by lowering the complexity. For example, the testing system 10 can calculate the keyword complexity using the following mathematical formula 1. The following mathematical formula 1 expresses the weight (i.e., the weight used to lower the complexity) for only the complexity among the three measures.

【Mathematical Expression 1】

In the preceding embodiments, the adjustment magnitude of keyword complexity (e.g., the weight in Formula 1) can be determined based on the modification frequency, modification degree, and interval before modification in the corresponding keyword's code routine (hereinafter referred to as the 'modification interval'). For example, a higher modification frequency, a greater modification degree, and a shorter modification interval can result in a larger adjustment magnitude (and vice versa). As another example, when the modification frequency (or modification degree) is on an increasing trend or the modification interval is on a decreasing trend, the adjustment magnitude can be determined to be a larger value (e.g., a steeper slope for increasing modification frequency results in a larger value, a steeper slope for decreasing modification interval results in a larger value, and vice versa).

In another embodiment, the testing system 10 can calculate the keyword score in a manner that is proportional to the frequency of use and reliability and inversely proportional to the complexity. That is, the score is calculated in a way that makes the keyword with a concise code routine score higher.

On the other hand, in some cases, the testing system 10 can adjust the score calculated in step S53 based on preset rules, additional metrics, etc. For example, the testing system 10 can calculate the final score of a corresponding keyword by assigning bonus points to the most recently registered keywords (e.g., the closer the registration time is to the present, the higher the bonus points are assigned). Alternatively, the testing system 10 can assign (perform) bonus points/deduction points to corresponding keywords when the frequency of keyword usage is on an increasing/decreasing trend (e.g., the higher the increase/decrease slope, the higher the bonus/deduction points are assigned). Alternatively, the testing system 10 can assign bonus points to keywords based on the development experience of the code routine creator (e.g., more experience, higher bonus points). Alternatively, the testing system 10 can assign bonus points to corresponding keywords based on the number of times test cases using the keyword are run, the increasing trend of the number of runs, etc. (e.g., bonus points are assigned when the number of runs is greater than or equal to a baseline value, and higher bonus points are assigned for more runs). Alternatively, the testing system 10 can adjust the keyword scores based on various combinations of the above embodiments.

In step S54, the ranking of multiple keywords can be determined based on scores. That is, the testing system 10 can determine the ranking based on the scores of multiple keywords. In some cases, the testing system 10 can classify multiple keywords according to a preset criterion and determine the ranking of the keywords according to the category. In this case, the preset criterion may be, for example, an action, a function of the target software, or a combination thereof, but the scope of the invention is not limited thereto.

For reference, step S51 can be performed by the keyword information acquisition unit 41, step S52 can be performed by the keyword analysis unit 42, and steps S53 and S54 can be performed by the keyword ranking unit 43.

On the other hand, according to some embodiments of the present invention, the testing system 10 can periodically or irregularly update the scores and rankings of multiple keywords. For example, the testing system 10 can obtain second information that includes information newer than the first information about the multiple keywords previously obtained, and analyze the second information to update the scores and rankings of each of the multiple keywords. For example, the testing system 10 can recalculate the scores of the multiple keywords according to a set cycle (e.g., monthly, etc.) (e.g., updating scores using information from the most recent month), and update the rankings of the multiple keywords with the corresponding scores. As another example, the testing system 10 can recalculate and reflect the complexity whenever the code routine of a keyword is modified, to immediately update the scores of the corresponding keywords (e.g., updating scores in real time). As yet another example, testing system 10 can also update and reflect the frequency of keyword usage whenever a new test case is created, thereby immediately updating the corresponding keyword score. As yet another example, testing system 10 can also update and reflect the reliability of keywords whenever a test case is run, thereby immediately updating the corresponding keyword score. As yet another example, testing system 10 can also update the scores and rankings of multiple keywords based on various combinations of the above embodiments.

Furthermore, in some embodiments, the testing system 10 can provide ranking information for each of multiple keywords (e.g., ranking, such as score, detailed scores for usage frequency/complexity/reliability, etc.) through a page accessible to testers of the target software. This reduces the perceived difficulty for testers in creating test cases, and consequently increases the speed of test case creation.

Furthermore, in some embodiments, the testing system 10 can automatically generate test cases based on ranking information of multiple keywords. For example, when the testing system 10 receives information about a series of actions (i.e., a sequence of actions) that constitute a test case, it combines multiple keywords corresponding to the input actions to create test cases. In this case, the testing system 10 can create test cases using keywords that rank higher than or equal to a benchmark value among the multiple keywords corresponding to each action (e.g., the highest-ranking keyword). If there is no keyword corresponding to a specific action, the testing system 10 can refer to similar test cases (e.g., test cases created to verify the same or similar functionality as the target software) to change the corresponding action to another action or subdivide it into multiple sub-actions. Furthermore, the test system 10 can generate test cases using keywords corresponding to the changed actions or sub-actions.

Furthermore, in some embodiments, the testing system 10 can recommend keywords to testers based on ranking information of multiple keywords (e.g., ranking, score, detailed scores for usage frequency/complexity/reliability, etc.). For example, the testing system 10 can automatically recommend keywords for specific actions through the test case creation interface. This embodiment will be explained in more detail below with reference to Figure 9.

As shown in Figure 9, suppose tester 91 can create a new test case 93 through the test case creation interface (92, hereinafter referred to as the 'creation interface'). The new test case 93 consists of a series of actions 94-1 to 94-3. Tester 91 sets (selects/inputs) the keyword 95-1 of the first action 94-1, and then sets (selects/inputs) the keyword 95-2 of the second action 94-2.

For reference, the creation interface 92 is an interface provided by the test system 10 and can be implemented in any way. The creation interface 92 may include, for example, an interface for supporting input action information, providing recommended keywords (or a list of recommended keywords), selecting keywords, etc., but the scope of the invention is not limited thereto.

In this scenario, the testing system 10 can receive information from the tester 91 via the creation interface 92 regarding a second action 94-2 ('X') to constitute a new test case 93, and respond accordingly by filtering from a pool of keywords (i.e., pre-registered keywords) that correspond to the input action 94-2. Furthermore, the testing system 10 can determine recommended keywords (e.g., keywords ranked higher than or equal to a benchmark value) based on the ranking of the filtered keywords, and provide the tester 91 with a list 96 of recommended keywords (e.g., which may include keyword names, rankings, scores, etc.) via the creation interface 92. Thus, tester 91 can confirm the recommended keyword list 96 through creation interface 92 and select the appropriate keyword 95-2 ('A'). Test system 10 sets the selected keyword 95-2 as the keyword corresponding to the second action 94-2. This process can also be repeated for other actions (e.g., 94-3).

The keyword-based testing methods according to some embodiments of the present invention are illustrated above with reference to Figures 5 to 9. As stated above, by comprehensively considering frequency, complexity, and reliability to calculate keyword scores, the keyword scores can accurately reflect the usefulness of the corresponding keywords. Furthermore, the ranking of keywords is determined based on the corresponding scores, thereby enabling the ranking of various keywords according to their usefulness.

Furthermore, by providing testers of the target software with ranking information based on keyword usefulness, the difficulty of creating test cases can be reduced and the convenience for testers can be improved. This, in turn, can increase the speed of test case creation.

Furthermore, by providing testers with recommended keywords based on keyword ranking information when creating test cases, the difficulty of testers creating test cases can be further reduced and the convenience for testers can be further improved.

In the following description, with reference to FIG10, an illustrative computing device 100 is shown that can implement a test system 10 according to some embodiments of the present invention.

Figure 10 is a schematic hardware structure diagram of the display computing device 100.

As shown in FIG10, the computing device 100 may include: one or more processors 101, a bus 103, a communication interface 104, memory 102 for loading computer programs executed by the processors 101, and auxiliary memory 105 for storing computer programs 106. However, FIG10 only shows components relevant to embodiments of the present invention. Therefore, those skilled in the art will understand that other general components may be included in addition to those shown in FIG10. That is, the computing device 100 may include various components besides those shown in FIG10. Furthermore, in some cases, the computing device 100 may be configured to omit some components shown in FIG10. The components of the computing device 100 will be described below.

Processor 101 can control the overall operation of the components of computing device 100. Processor 101 may include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), NPU (Neural Processing Unit), or any processor known in the art. Furthermore, processor 101 can execute calculations for performing at least one application or program to carry out operations/methods according to various embodiments of the present invention. Computing device 100 may include one or more processors.

Furthermore, memory 102 can store various data, commands, and/or information. Memory 102 can load computer program 106 from auxiliary memory 105 to execute operations/methods according to various embodiments of the present invention. Memory 102 can be implemented as volatile memory such as RAM, but the scope of the present invention is not limited thereto.

Furthermore, bus 103 can provide communication functions between components of computing device 100. Bus 103 can be implemented as various types of buses, such as address bus, data bus, and control bus.

Furthermore, the communication interface 104 can support wired/wireless Internet communication of the computing device 100. In addition, the communication interface 104 can support various communication methods other than Internet communication. Therefore, the communication interface 104 may include communication modules known in the art.

Furthermore, the auxiliary memory 105 can store one or more computer programs 106 non-temporarily. The auxiliary memory 105 may include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, hard disk, removable magnetic disk, or any type of computer-readable recording medium known in the art.

Furthermore, the computer program 106 may include one or more instructions that, when loaded into the memory 102, cause the processor 101 to perform operations/methods according to various embodiments of the present invention. That is, the processor 101 can perform operations/methods according to various embodiments of the present invention by executing one or more instructions.

For example, computer program 106 may include one or more instructions for performing the following operations: obtaining information on multiple keywords used to create test cases for the target software, the obtained information including code routines for the multiple keywords; analyzing the code routines to calculate the complexity of each of the multiple keywords; calculating a score for each of the multiple keywords based on the complexity; and determining a ranking of the multiple keywords based on the scores.

As another example, computer program 106 may also include multiple instructions for performing at least a portion of the multiple operations/methods described with reference to Figures 1 to 9.

As shown in the figure, a test system 10 according to some embodiments of the present invention can be implemented by a computing device 100.

On the other hand, in some embodiments, the computing device 100 shown in FIG10 can mean a virtual machine implemented based on cloud technology. For example, the computing device 100 is a virtual machine running on one or more physical servers included in a server farm. In this case, at least some of the processor 101, memory 102, and auxiliary memory 105 shown in FIG10 can be virtual hardware, and the communication interface 104 can also be implemented as a virtualized network element such as a virtual switch.

The schematic computing device 100 of the test system 10, which can implement some embodiments of the present invention, is illustrated above with reference to FIG10.

Various embodiments of the present invention and their effects have been illustrated above with reference to Figures 1 to 10. The effects of the technical concept of the present invention are not limited to those described above, and those skilled in the art will clearly understand from the following description other effects not mentioned.

Furthermore, although the above embodiments illustrate that multiple components are combined into one or more operations, the technical concept of the present invention is not necessarily limited to these embodiments. That is, within the scope of the technical concept of the present invention, all components can be selectively combined into one or more for operation.

The technical concept of the present invention described so far can be implemented as computer-readable code on a computer-readable medium. The computer program recorded on the computer-readable recording medium can be sent to another computing device via a network such as the Internet, installed in the other computing device, and used in the other computing device.

Although the operations are shown in a specific order in the accompanying figures, it should not be understood that they must be performed in the specific or sequential order shown, or that all shown operations must be performed to obtain the desired result. In certain cases, multitasking and parallel processing may be advantageous. While various embodiments of the invention have been described with reference to the accompanying figures, it should be understood by those skilled in the art that the technical concept of the invention can be implemented in different specific forms without altering the technical concept or essential characteristics of the invention. Therefore, the above embodiments should be understood as illustrative rather than restrictive in all respects. The scope of protection of the invention should be interpreted as defined by the appended claims, and all technical ideas within the equivalent scope should be interpreted as included within the scope of the technical ideas defined herein.

10: Keyword-based testing system 11: Multiple keywords 12: Keyword score 13: Tester 20: Login page 21, 32, 33, 34: Keywords 31: Test cases 35, 36, 37: Code examples 41: Keyword information retrieval unit 42: Keyword analysis unit 43: Keyword ranking unit 44: Keyword recommendation unit 45: Usage frequency calculation unit 46: Complexity calculation unit 47: Reliability calculation unit S51, S52, S53 S54: Steps 61, 71: Keywords 62, 72: Code Routine 63, 64, 65, 66, 67, 68: Code Lines 91: Tester 92: Test Case Creation Interface 93: New Test Case 94-1, 94-2, 94-3: Actions 95-1, 95-2: Keywords 96: Recommended Keyword List 100: Computing Device 101: Processor 102: Memory 103: Bus 104: Communication Interface 105: Auxiliary Memory 106: Computer Programs

Figure 1 is a schematic diagram for simply illustrating the operation of a keyword-based testing system according to some embodiments of the present invention.

Figures 2 and 3 are schematic diagrams used to illustrate the concept of keywords mentioned in various embodiments of the present invention.

Figure 4 is a schematic block diagram showing a keyword-based testing system according to some embodiments of the present invention.

Figure 5 is a schematic flowchart showing some embodiments of a keyword-based testing method according to the present invention.

Figure 6 is a schematic diagram illustrating the complexity calculation method according to some embodiments of the present invention.

Figure 7 is a schematic diagram illustrating the reliability analysis method used to explain some embodiments of the present invention.

Figure 8 shows the results of calculating the usage frequency, complexity, and reliability of multiple keywords according to some embodiments of the present invention.

Figure 9 is a schematic diagram illustrating the keyword recommendation method according to some embodiments of the present invention.

Figure 10 shows a schematic computing device that can implement a keyword-based testing system according to some embodiments of the present invention.

S51, S52, S53, S54: Steps

Claims (5)

A keyword-based testing method, executed by at least one computing device, is characterized by comprising the following steps: acquiring information on a plurality of keywords used to create test cases for target software; calculating the complexity of each of the plurality of keywords; and calculating a score for each of the plurality of keywords based on the complexity, wherein the step of calculating the score for each of the plurality of keywords includes... The steps are as follows: calculate the values of the complexity and other metrics; adjust the value of the complexity; and calculate the score of each of the plurality of keywords based on the adjusted values of the complexity and other metrics, wherein the adjustment range of the complexity of a particular keyword among the plurality of keywords is determined based on the interval between the modification frequency, modification degree, and modification time of the code routine of the particular keyword. The keyword-based testing method of claim 1 is characterized in that the adjustment range of complexity is determined to be a larger value when the modification frequency is higher, the modification degree is greater, the interval between modification moments is shorter, the modification frequency or the modification degree is on an increasing trend, or the interval between modification moments is on a decreasing trend. The keyword-based testing method of claim 1 is characterized in that the step of calculating the score of each of the plurality of keywords includes the following steps: when the frequency of use of the specific keyword is on an increasing trend or the number of times the test cases using the specific keyword are run is on an increasing trend, the specific keyword is given a bonus. A keyword-based testing system includes: one or more processors; and memory for storing a computer program executed by the one or more processors, the computer program including a plurality of instructions for performing the following operations: acquiring information on a plurality of keywords for creating test cases of target software; calculating the complexity of each of the plurality of keywords; and calculating a score for each of the plurality of keywords based on the complexity, and calculating... The steps for scoring each of the plurality of keywords include the following: calculating the values of the complexity and other metrics; adjusting the value of the complexity; and calculating the score of each of the plurality of keywords based on the adjusted values of the complexity and other metrics, wherein the adjustment range of the complexity of a particular keyword among the plurality of keywords is determined based on the interval between the modification frequency, modification degree, and modification time of the code routine of the particular keyword. A computer program, stored in a computer-readable storage medium, in conjunction with a computing device, performs the following steps: acquiring information on a plurality of keywords used to create test cases for target software; calculating the complexity of each of the plurality of keywords; and calculating a score for each of the plurality of keywords based on the complexity. The steps of calculating the score for each of the plurality of keywords include the following: Steps: Calculate the complexity and other metrics; adjust the complexity; and based on the adjusted complexity and other metrics, calculate the score for each of the plurality of keywords, wherein the adjustment range of the complexity of a particular keyword among the plurality of keywords is determined based on the interval between the modification frequency, modification degree, and modification time of the code routine of the particular keyword.