CN118656059B - Edge device accidental touch control detection method, device and storage medium - Google Patents

Abstract

The invention provides a method, a device and a storage medium for detecting false touch control of side equipment, which relate to the technical field of software engineering and comprise the following steps: detecting accessibility of each page based on a page traversal algorithm; judging whether each new page needs to be tested according to the page similarity; screening the controls in the current page to determine the controls to be detected; performing simulated clicking on a center point of the control to be detected to generate a center clicking result; performing simulated clicking on the edge of the control at a certain distance from the center point of the control to be detected, and generating an edge clicking result; and comparing the edge clicking result with the center clicking result to determine whether the control to be detected has an accessibility problem. The invention combines static detection and dynamic detection, and judges whether the control to be detected has accessibility problem by simulating the resource change and the page change after the user operation. To provide more reliable accessibility assessment and improvement advice to developers.

Description

Edge device accidental touch control detection method, device and storage medium

Technical Field

The present invention relates to the field of software engineering technology, and in particular to a method, device and storage medium for detecting and controlling false touches of edge devices.

Background Art

In the current field of Android application development, users have higher and higher expectations for the ease of operation and stability of applications. Despite this, developers usually tend to focus on the traditional core attributes of the application, such as functionality and performance, and often ignore the importance of user experience in real-world usage scenarios. In this case, design defects in the application interface may lead to user errors during operation, which in turn leads to unexpected consequences. Therefore, for Android applications, while ensuring functionality and performance, it is also necessary to pay attention to the optimization of user experience, especially in terms of improving accessibility, to avoid possible operational errors and related problems. Traditional accessibility detection methods usually determine whether page elements may cause user misoperation behaviors and thus affect the user experience by scanning the page element layout of the element. The traditional approach is to detect the accessibility of the control and potential problems that may arise by making regular judgments on the static properties of the control (size, color, font size, etc.). However, although traditional methods can capture the control properties and page layout of the current page of the application, they cannot dynamically traverse all pages to achieve automatic all-round detection. On the other hand, due to the limitations of static detection, traditional methods often judge all small controls as having poor accessibility. For some small controls, due to the rationality of the layout, accessibility issues do not exist. In the field of Android application development, users have growing expectations for the convenience and stability of operations, while developers often focus on the functionality and performance of the application and ignore the importance of user experience. This kind of application interface design flaw in reality may lead to user operation errors and unexpected consequences.

In the process of implementing the present invention, the applicant discovered that the prior art performs accessibility detection on a page-by-page basis, without combining page traversal with page detection. This means that the prior art cannot achieve large-scale automated detection. The prior art cannot automatically identify similar pages, and only relies on testers to manually determine whether a page needs to be detected. It is very likely that similar pages will be repeatedly detected, resulting in unnecessary waste of resources. The prior art is too vague in determining the accessibility conditions for small-sized controls. Since static detection cannot detect controls more accurately, this means that the prior art will generate a large number of false alarms.

Therefore, how to improve the detection efficiency and accuracy of Android operation page accessibility detection has become a technical problem that needs to be solved urgently.

Summary of the invention

The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a method, device and storage medium for detecting false touches on edge devices. It optimizes the user experience of Android applications by dynamically simulating common user operations, and solves the limitations of traditional accessibility detection methods.

The first aspect of the present invention discloses a method for detecting and controlling false touches of edge devices, including: traversing all pages to determine the page to be detected: determining whether each new page needs to be tested based on page similarity: if the current page is not similar to the page that has completed the test, recording the current page and the operation events performed on the current page, and adding them to the queue of pages to be detected; screening the controls in the current page to determine the controls to be detected; simulating a click on the center point of the control to be detected to generate a center click result; simulating a click on the edge of the control at a certain distance from the center point of the control to be detected to generate an edge click result; if it is detected that the current page has jumped, tracing back to the current page through a predetermined fallback strategy; comparing the edge click result with the center click result: if the center point is If the simulated click on the center point generates a page jump, and the simulated click on the edge of the control also generates a page jump, the page similarity and traffic consumption of the new pages of the two jumps are further compared. When the page similarity exceeds the first threshold and the difference in traffic consumption is less than the second threshold, it is determined that the control to be detected does not have an accessibility problem; if the simulated click on the center point generates a page jump, and the simulated click on the edge of the control does not generate a page jump, it is further determined whether the simulated click at the edge of the control generates traffic consumption. If the traffic consumption is less than the third threshold, it is determined that the control to be detected does not have an accessibility problem; if the simulated click on the center point does not generate a page jump, and the simulated click on the edge of the control also does not generate a page jump, it is determined that the control to be detected has an accessibility problem.

According to the edge device false touch control detection method disclosed in the present invention, preferably, the fallback strategy specifically includes: simulating a fallback button to fall back to the previous page; or simulating exiting the application first, and then cold starting the current page to be detected; or tracing back to the current page to be detected according to the page path recorder.

According to the edge device false touch control detection method disclosed by the present invention, preferably, the cold start is implemented based on the start_activity function in the appium package of python.

According to the edge device false touch control detection method disclosed in the present invention, preferably, the page similarity detection method specifically includes: extracting the controls in the XML layout files of the two pages, comparing the control set W1 and the control set W2, and if the overlap between W1 and W2 is greater than a predetermined value, the pages are determined to be similar.

According to the edge device false touch control detection method disclosed in the present invention, preferably, an adb test tool is used to extract the xml file of the page, two pages are represented as p1 and p2, and Jaccord similarity is used to represent the similarity of the two control sets. If the similarity is greater than 0.8, the two pages are judged to be similar, and the expression is as follows: Jaccord(p1,p2) = #(W1∩W2)/ #(W1∪W2).

According to the edge device accidental touch control and detection method disclosed in the present invention, preferably, the step of screening the controls in the current page specifically includes: extracting the xml file of the current page; determining the controls whose size is smaller than a predetermined value as controls to be detected; checking the clickable attribute of each control, and determining the clickable controls as controls to be detected; if the number of clickable controls on the current page is less than the predetermined number, determining all controls on the current page as controls to be detected.

According to the edge device false touch control detection method disclosed in the present invention, preferably, a control whose length or height is less than 48dp is defined as a control to be detected; if the number of clickable controls is less than 5, all controls are defined as controls to be detected.

According to the edge device false touch control detection method disclosed in the present invention, preferably, it also includes:

Mark controls with accessibility issues and take a screenshot of the current page.

The second aspect of the present invention discloses an edge device false touch control and detection device, including: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement an edge device false touch control and detection method such as any of the above-mentioned technical solutions.

The third aspect of the present invention discloses a computer-readable storage medium, which stores a program code, and the program code is used to implement a method for detecting and controlling false touches of an edge device as described in any of the above-mentioned technical solutions.

The beneficial effects of the present invention include at least: optimizing the user experience of Android applications by dynamically simulating common user operations, taking dynamic simulation of common user operations as the core idea, restoring real user operation error scenarios by simulating user operations on the edge areas of small-sized controls, and judging whether the control really affects the user experience based on the changes in the page state and resource consumption of the application before and after the operation, thereby solving the limitations of traditional accessibility detection methods. The present invention simulates user operations to accurately reproduce real operation error scenarios to judge whether the control really affects the user experience. Through this method, the automated detection capability of the system can be greatly improved, so that application developers can achieve high-coverage automated testing without manual testing, which greatly facilitates developers and improves the overall quality and reliability of the application. At the same time, the present invention reduces the need for repeated detection and improves detection efficiency by treating pages with similar layouts as being in the same state.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a flow chart of a method for detecting false touches on edge devices according to an embodiment of the present invention.

FIG. 2 shows a schematic diagram of a system framework according to an embodiment of the present invention.

FIG3 shows a schematic block diagram of an edge device false touch control and detection apparatus according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

As shown in FIG1 , according to an embodiment of the present invention, a method for detecting and controlling false touches of edge devices is disclosed, which specifically includes:

Step S101, traverse the pages: traverse all pages to determine the page to be tested: determine whether each new page needs to be tested based on the page similarity: if the current page is not similar to the page that has been tested, then record the current page and the operation events performed on the current page and add them to the queue of pages to be tested;

Step S102, control screening: screening the controls in the current page to determine the controls to be detected;

Step S103, simulate click: simulate click on the center point of the control to be detected to generate a center click result; simulate click on the edge of the control at a certain distance from the center point of the control to be detected to generate an edge click result; during this period, if it is detected that the current page has jumped, then backtrack to the current page through a predetermined fallback strategy;

Step S104, determine accessibility issues: compare the edge click result with the center click result. If the simulated click on the center point generates a page jump, and the simulated click at the edge of the control also generates a page jump, then further compare the page similarity and traffic consumption of the new pages of the two jumps. When the page similarity exceeds the first threshold and the difference in traffic consumption is less than the second threshold, it is determined that there is no accessibility issue with the control to be detected; if the simulated click on the center point generates a page jump, and the simulated click at the edge of the control does not generate a page jump, then further determine whether the simulated click at the edge of the control generates traffic consumption. If the traffic consumption is less than the third threshold, it is determined that there is no accessibility issue with the control to be detected; if the simulated click on the center point does not generate a page jump, and the simulated click at the edge of the control also does not generate a page jump, then it is determined that there is an accessibility issue with the control to be detected.

According to the above embodiment, preferably, the fallback strategy specifically includes: simulating a back button to fall back to the previous page; or simulating exiting the application first, and then cold starting the current page to be detected; or tracing back to the current page to be detected according to the page path recorder. Among them, the cold start is implemented based on the start_activity function in the appium package of python.

According to the above embodiment, preferably, the method for detecting page similarity specifically includes: extracting controls in the xml layout files of the two pages, comparing the control set W1 and the control set W2, and if the overlap between W1 and W2 is greater than a predetermined value, then the pages are determined to be similar. Specifically: using the adb test tool to extract the xml file of the page, the two pages are represented as p1 and p2, and the Jaccord similarity is used to represent the similarity of the two control sets. If the similarity is greater than 0.8, then the two pages are determined to be similar, and the expression is as follows: Jaccord(p1,p2) = #(W1∩W2)/ #(W1∪W2).

According to the above embodiment, preferably, the step of screening the controls in the current page specifically includes: extracting the xml file of the current page; determining the controls whose control size is smaller than a predetermined value as controls to be detected; checking the clickable attribute of each control, and determining the clickable controls as controls to be detected; if the number of clickable controls on the current page is less than the predetermined number, determining all controls on the current page as controls to be detected.

According to the above embodiment, preferably, the method further includes: marking controls with accessibility issues, and taking a screenshot of the current page.

According to the above embodiment, preferably, the appium server is deployed to simulate events such as user clicks, back operations, and application activity startup; adb is used to obtain the static properties of the program's current page XML file and controls, and to monitor network transceiver traffic resources; the cv2 library is used to take a screenshot of the current page of the Android application in order to mark the problematic controls for users to view the controls.

As shown in Figure 2, according to another embodiment of the present invention, a system framework is disclosed after the edge device false touch control detection method of the present invention is encapsulated as an accessibility testing tool, including: a two-stage navigation unit, a page path recorder unit, an abstract page state confirmation unit, a page similarity detection unit, a user operation simulation unit, an accessibility detection algorithm unit and a resource monitor unit.

The specific implementation of the two-stage navigation includes:

There are too many Android application pages, and human labor alone cannot detect too many pages. Therefore, a page traversal algorithm is needed to automatically detect pages. The present invention uses two-stage navigation to improve the coverage of page states. Page navigation is used to transfer the current state to the target page, and control navigation is used to transfer the current state to the target control. Specifically:

The first stage of two-stage navigation: page navigation. When detecting a certain page p, if the page jumps, in order to return to page p or a page with a similar layout to p, this tool will adopt three strategies in sequence to try to return to the page to be detected p: 1) Simulate the back button to simulate the user's back operation, because the new page is caused by the jump from the old page p. In the design of most apps, the back button can help users return to the previous page; 2) Cold start this activity. When strategy 1) fails, this tool first simulates exiting the application, and then tries to directly cold start the activity of this page. Specifically, using the start_activity function in the appium package can help achieve this. 3) Backtracking the page according to the recorder: When both current strategies fail, start strategy 3).

In this embodiment, the tool will determine whether each new page encountered needs to be detected based on the page similarity. If the current page is not similar to the previously tested page, the recorder records the current page and the operation events performed when running to the current page, and adds it to the queue of pages to be detected. Otherwise, the current page does not need to be recorded. Therefore, each page to be detected has recorded the operation events from the main page to the current page. Starting from the main page, these event operations can be re-simulated to reach the page to be detected. Among them, in Android, the simulation of these user events is implemented through the python appium package. Appium is an open source tool for automated mobile application testing, which supports iOS, Android and Windows platforms. Appium is cross-platform and can use multiple programming languages to write test scripts, such as Java, Python, JavaScript, etc., so it is very flexible and easy to use. The working principle of Appium is to communicate with the device or simulator by sending commands of the WebDriver protocol and perform various operations, such as starting the application, entering text, clicking buttons, etc. It uses the native automation framework of various mobile platforms to interact with the application to realize the function of automated testing.

The second stage of two-stage navigation: control navigation. In a page to be detected, there may be a large number of controls. It is necessary to screen the controls to find the controls that are likely to be accidentally touched. Specifically, this tool screens in the following three aspects. When the program runs to the page to be detected, this tool will first extract the xml file of the page, which is used to describe the layout and structure of the application interface. In Android development, layout files written in XML format are stored in the res/layout directory of the application, and the static properties of the controls (control position, control size, whether the control supports click operations, etc.) can be obtained.

1) Is the size of the control smaller than 48dp? If the length or height of a control is smaller than 48dp, the control may be too small. When the user clicks this control, they may accidentally touch other controls, thus triggering undesirable events.

2) Whether the control has a clickable attribute. If the control has a clickable attribute and this attribute is set to true, then the control is considered to be operable.

3) If the number of clickable controls on a page is less than 5, it can be determined that this page is very likely to hide the clickable properties of the controls in a custom way. For such pages, all controls should be tested.

In this function, control navigation is mainly achieved by appium simulating user operations and adb extracting static properties of page controls.

The specific methods of abstracting pages with similar page layouts into page states include:

Since Android applications have a large number of duplicate pages, it is inefficient to detect duplicate pages of these pages. At the same time, even if it is the same page, due to the presence of dynamic controls and videos, it is impossible to determine whether two pages are the same page only by taking screenshots. Therefore, the present invention abstracts pages with similar page layouts into page states s, and each page state s is a collection of pages p1, p2... with similar page layouts. Only one of the pages needs to be detected to representatively obtain the accessibility issues of all pages. Among them, the page similarity comparison algorithm includes: when page p1 and page p2 are compared, (using adb test tool) the controls in the xml layout files of the two pages are extracted, and these control sets W1 and W2 are compared. If the control sets of the two pages have a high degree of overlap, then the two pages are similar. Use Jaccord similarity to represent the similarity of the control sets of two pages. If it is greater than 0.8, the two pages are considered to be similar:

Jaccord(p1,p2) = #（W1∩W2）/ #（W1∪W2）.

The specific implementation of the accessibility detection algorithm includes:

When this tool detects a certain control w to be detected, it will use a dynamic monitoring method. Specifically, this tool will simulate user operations and simulate click operations on the four edge coordinate points 24dp away from the center coordinate of the control, and compare them with the results of clicking the control. The specific comparison method is as follows:

1. If the simulated click on the center point generates a page jump, and the simulated click on the edge coordinates also generates a page jump, then compare the XML files of the two new pages. If the jaccord similarity of the control set of the XML file is greater than 0.8, and the difference in traffic consumption caused by the two times monitored by the resource manager is less than 2048KB (receiving traffic) or 1024KB (sending traffic), it is considered that the two operations will not cause accidental touches and there is no accessibility problem. Otherwise, there is an accessibility problem.

2. If the simulated click on the center point generates a page jump, while the simulated click on the edge coordinates does not generate a page jump and does not consume more than 1024KB of traffic, it is considered that the two operations will not cause accidental touches and there is no accessibility problem. Otherwise, there is an accessibility problem. The statistics of network resources are implemented using the adb test tool.

3. If the simulated click on the center point does not result in a page jump, but the simulated click on the edge coordinates results in a page jump, it is considered that there is an accidental touch problem in both operations, which has an accessibility problem.

As shown in Figure 3, according to another embodiment of the present invention, a device 300 for detecting and controlling mistaken touches of an edge device is disclosed, including: a memory 301 for storing program instructions; a processor 302 for calling the program instructions stored in the memory to implement a method for detecting and controlling mistaken touches of an edge device as in the above-mentioned embodiment.

According to another embodiment of the present invention, a computer-readable storage medium is disclosed, which stores program code, and the program code is used to implement the edge device false touch control and detection method as described in the above embodiment.

According to the above-mentioned embodiment of the present invention, the defects of static detection are overcome by simulating a dynamic detection scheme of user operation, thereby improving the accessibility of controls. By dynamically simulating user operations, the present invention aims to achieve accurate identification of controls that are easily accidentally touched in Android applications, thereby improving user experience and reducing the occurrence of misoperations. By simulating the actual operation scenarios of users, the present invention can more effectively detect those controls that are misjudged in static detection, thereby providing application developers with more reliable accessibility assessments and improvement suggestions. The present invention proposes a two-stage navigation for traversing pages, which realizes the switching and traversal of Android application page states. On the one hand, it can improve the degree of automation of accessibility, and on the other hand, it can access more pages and improve detection accuracy (in manual testing, due to the bias of testers and efficiency constraints, some pages cannot be detected).

Since traditional accessibility detection technology does not care about the page traversal technology of Android applications, or the traversal technology cannot achieve a high page coverage, it is impossible to achieve large-scale automated detection of applications. The present invention proposes an effective traversal algorithm, which consists of two stages. Page navigation is used to transfer the current state to the target page, and control navigation is used to transfer the current state to the target control. This solution can realize automatic page traversal, thus eliminating the need for manual testing by application developers. At the same time, since different pages of Android applications may have similar layouts, such as opening two videos in Douyin, the page layouts of these two video pages are basically the same. Therefore, the present invention uses abstract pages to treat pages with similar page layout files and pages reached through the same path as the same page state, thereby only detecting the page state without the need to repeatedly detect each page. Among them, the page state refers to a method similar to a finite state machine, which treats similar pages as one page for testing, reducing the number of pages tested.

All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. The utility model provides a detection method is managed in wrong touch of side equipment for detect the page of android application, its characterized in that includes:

Traversing all pages to determine a page to be detected: judging whether each new page needs to be tested according to the page similarity: if the current page is dissimilar to the page which has completed the test, recording the current page and an operation event which is carried out by running to the current page, and adding the current page and the page to be detected into a page queue;

Screening the controls in the current page to determine the controls to be detected;

performing simulated clicking on the center point of the control to be detected to generate a center clicking result;

performing simulated clicking on the edge of the control at a certain distance from the center point of the control to be detected, and generating an edge clicking result;

When the current page is detected to jump, backtracking to the current page through a preset rollback strategy;

Comparing the edge click result with the center click result:

if the simulated click on the center point generates page jump and the simulated click on the edge of the control also generates page jump, further comparing the page similarity and the flow consumption of the new page of the two jumps, and judging that the control to be detected has no accessibility problem when the page similarity exceeds a first threshold and the difference value of the flow consumption is smaller than a second threshold;

If the simulated click on the center point generates the page jump, the simulated click at the control edge does not generate the page jump, further judging whether the simulated click at the control edge generates flow consumption, and judging that the control to be detected has no accessibility problem under the condition that the flow consumption is smaller than a third threshold value;

If the simulated click on the center point does not generate page jump, and the simulated click at the edge of the control does not generate page jump, judging that the control to be detected has the accessibility problem.

2. The method for detecting false touch control of a side device according to claim 1, wherein the rollback policy specifically includes:

simulating a rollback button to roll back to the last page; or alternatively

Firstly simulating exiting the application, and then starting the current page to be detected; or alternatively

And backtracking to the current page to be detected according to a page path recorder.

3. The method for detecting false touch control of a side device according to claim 2, wherein the cold start is based on a start_activity function in a appium packet of python.

4. The method for detecting false touch control of side equipment according to claim 1, wherein the method for detecting page similarity specifically comprises the following steps:

and extracting the controls in the xml layout file of the two pages, comparing the control set W1 with the control set W2, and judging that the pages are similar if the overlap ratio of the W1 and the W2 is larger than a preset value.

5. The method for detecting false touch control of a side device according to claim 4, wherein an adb test tool is used to extract xml files of pages, two pages are denoted as p1 and p2, jaccord similarity is used to represent similarity of two control sets, if the similarity is greater than 0.8, the two pages are judged to be similar, and the expression is as follows:

Jaccord(p1,p2) = #（W1∩W2）/ #（W1∪W2）。

6. The method for detecting false touch control of a side device according to claim 1, wherein the step of screening the controls in the current page specifically includes:

Extracting an xml file of a current page;

Determining a control with the control size smaller than a preset value as a control to be detected;

Checking clickable attributes of each control, and determining the clickable control as a control to be detected;

If clickable controls in the current page are smaller than the preset number, all the controls in the current page are determined to be the controls to be detected.

7. The method for detecting false touch control of the side equipment according to claim 6, wherein the length or the height of the control is less than 48dp and is defined as the control to be detected; clickable the number of controls is less than 5, then all controls are defined as controls to be detected.

8. The method for detecting false touch control of an edge device according to any one of claims 1 to 7, further comprising: and marking the control with the accessibility problem, and capturing the screenshot of the current page.

9. The utility model provides a limit equipment mistake touch-control detection device which characterized in that includes:

a memory for storing program instructions;

a processor for invoking the program instructions stored in the memory to implement the edge device false touch detection method as recited in any one of claims 1-8.

10. A computer-readable storage medium storing a program code for implementing the edge device false touch detection method according to any one of claims 1 to 8.