TWI605378B - Method of recording operations and method of automatically re-executing operations - Google Patents

Description

Control recording method and automatic execution control method

The disclosure relates to a manipulation recording method and an automatic execution manipulation method.

The combination of smart phones or tablets and wireless networks has been widely used. Many service providers have launched applications related to mobile platforms for consumers to use. In the financial industry, for example, each bank launches an online banking application for the mobile platform. Users set their own account and password for each bank's online banking, and enter the identity number, account number and password to log in when using. After that, you can start the operation and obtain the service. For users with multiple bank accounts, users must log in at the same time when checking the information of each bank or confirming the successful transfer of funds between different banks. In addition to the online banking application of different mobile platforms, switching is also required, so the login and operation process becomes more cumbersome and time consuming for the user.

The present disclosure provides a manipulation recording method and an automatic execution manipulation method.

In an exemplary embodiment, a method of controlling recording according to the present disclosure is for an electronic device. The electronic device has an auxiliary function interface configured to perform an Accessibility Service, and a display configured to display an operation screen having a plurality of user interfaces (User) Interface, UI) components, the UI elements form a tree-like hierarchical relationship, and the auxiliary function interface sends a manipulation information according to the components operated by the user. The manipulation recording method includes: capturing the manipulation information; determining, according to the manipulation information and the tree hierarchy relationship, a manipulation event manipulated by the user, the UI component corresponding to the manipulation event in the UI elements, and assigning a node The Node Identifier corresponds to the manipulation event; and the manipulation event and the node position character are written in a manipulation log file.

In an exemplary embodiment, a method of automatically performing a method of operation is provided for an electronic device including an auxiliary function interface configured to perform an Accessibility Service, and configured to display A display screen of an operation screen, the operation screen comprising a plurality of user interface (UI) elements configured to be manipulated, the UI elements forming a tree-like hierarchical relationship. The automatic execution control method includes: reading a manipulation log file, the manipulation log file including at least one manipulation event, at least one node position corresponding to the at least one manipulation event, and at least one component corresponding to the at least one manipulation event Layout record; capturing a component layout information transmitted by the auxiliary function interface, the component layout information corresponding to the operation screen; determining, according to the at least one component layout record and the component layout information, whether the component layout record corresponds to the operation screen And transmitting, in the case of the operation screen corresponding to the at least one component layout record, a manipulation execution instruction to the auxiliary function interface according to the at least one manipulation event and the node position, and causing the auxiliary function interface to execute the Manipulate execution instructions.

In another exemplary embodiment, an automated method of controlling a method for an electronic device including an auxiliary function interface configured to perform an Accessibility Service, and configured to A display displaying a first operational screen having at least one first user interface (UI) component configured for manipulation. The automatic execution control method includes: in response to executing the first UI element, the electronic device displays a second operation screen, and the second operation screen has a plurality of second UI elements, and the second UI elements form a tree a hierarchical relationship; reading a manipulation log file including at least one manipulation event, at least one node position corresponding to the at least one manipulation event, and at least one component layout record corresponding to the at least one manipulation event; Taking a component layout information transmitted by the auxiliary function interface, the component layout information corresponding to the second operation screen; determining, according to the at least one component layout record and the component layout information, whether the at least one component layout record corresponds to the second Operating the screen; and transmitting a manipulation execution command to the auxiliary function interface according to the at least one manipulation event and the node position, only if the second operation screen corresponds to the at least one component layout record, to enable the auxiliary function The interface executes the manipulation execution instruction.

Based on the above, the manipulation recording method and the automatic execution manipulation method proposed by the embodiment of the present disclosure can record and control the operation on the electronic device as the manipulation record file, and further can be read by the electronic device, and automatically executed according to the manipulation record file. Manipulation, thus assisting in the execution of complex controls to provide convenient and time-saving manipulation.

The above described features and advantages of the present invention will be more apparent from the following description.

The term "coupled" as used throughout the specification (including the scope of the patent application) may be used in any direct or indirect connection. For example, if the processor is described as being coupled to a memory, it should be interpreted that the processor can be directly connected to the memory, or the processor can be indirectly connected to the device through other means or a connection means. Storage. In addition, wherever possible, the elements and/ Elements/components/steps that use the same reference numbers or use the same terms in different embodiments may refer to the related description.

1 and 2 are block diagrams showing an electronic device capable of performing a manipulation recording method and a display screen thereof, respectively, according to an embodiment of the present disclosure. The control recording method is applicable to the electronic device 1 , and the electronic device 1 includes a display 11 , a storage device 12 , an antenna 14 capable of transmitting and receiving wireless signals, and the touch display 11 , the storage device 12 and the An operation processor 13 of the antenna 14 has an auxiliary function interface 131 that can execute an accessibility service and drives the touch display 11 to display an operation screen 20. In the present embodiment, the display 11 is a manipulation display.

Referring again to FIG. 3, it is a node tree diagram of a tree hierarchy relationship of the manipulation recording method according to an embodiment of the present disclosure. The operation screen 20 has a plurality of user interface (UI) elements 23 configured for user manipulation, and the UI elements 23 have the tree hierarchy relationship as shown in FIG. Element 23 is each node in the tree hierarchy relationship, and the auxiliary function interface 131 is configured to send manipulation information in accordance with UI elements manipulated by the user. In this embodiment, the electronic device 1 is installed with an Android Operation System, and the auxiliary function interface 131 is an Android Accessibility Service Interface.

In the Android operating system, a frame layout is constructed by various display objects, which may be an image (Imageview) or a text (Textview), and collectively form a hierarchical relationship; the UI of the embodiment Element 23 is the display object of the Android operating system.

Further referring to FIG. 4, it is a flowchart of a method for controlling recording according to an embodiment of the present disclosure. The steps included in the manipulation recording method are as follows: Step S31 is executed, and the user controls one of the UI elements 23. In this embodiment, the user clicks on the UI component that displays "BBB" (indicated by 23' in the figure) However, the user's control is not limited to clicks. In addition to clicking, it can be other manipulation actions allowed by the system, or it can be the manipulation of typing data.

In the embodiment, step S32 is performed after step S31, and the auxiliary function interface 131 sends the manipulation information according to the user's manipulation and clicking on the UI element 23', wherein the manipulation information includes a component layout corresponding to the operation screen 20. A frame-layout, a manipulation event corresponding to the user's manipulation, and a hash code corresponding to the UI element 23 executed by the manipulation event.

The component layout information is information describing the operation screen 20, that is, information of all the UI elements 23 in the operation screen 20; the component layout information includes the component type of each UI element 23 in the operation screen 20, and the operation screen. Each of the UI elements 23 in the 20 has a number of child nodes in the tree hierarchy relationship, and a plurality of hash codes corresponding to each UI element 23 in the operation screen 20, respectively. Different operation screens can include different component layout information.

In the Android operating system, when the auxiliary function interface transmits the component layout information, it is transmitted according to the node position of each UI element, so the tree hierarchical relationship can be obtained according to the number of child nodes and the order of receiving. The component type described in this embodiment is a FrameLayout, a LinearLayout, a TextView, or an ImageView defined in the Android operating system, wherein the FrameLayout is a root node, and the LinearLayout is a node having a child node, and the left node under the root node in FIG. 3 is taken as an example, and the corresponding The component type is LinearLayout, and in the tree hierarchy relationship, it has 5 child nodes inherited in its hierarchy, so the corresponding number of child nodes is 5, which is marked as "Child=5" in FIG.

In the manipulation information, the manipulation event is information of a user operation action. When operating on the Android operating system, the manipulation event in this embodiment may be a click (TYPE_VIEW_CLICKED), a focus (TYPE_VIEW_FOCUSED), or a long click (TYPE_VIEW_LONG_CLICKED). And the disclosure is not limited to this. For example, when the user performs the click on the UI element 23 ′ displayed as “BBB”, the manipulation event sent by the auxiliary function interface 131 is a click (TYPE_VIEW_CLICKED), and the UI element corresponding to the user click is sent. The hash code for 23' is "562471".

After the operation information is sent by the auxiliary function interface 131, step S33 is executed to retrieve the manipulation information sent by the auxiliary function interface 131, and temporarily store the component layout information of the manipulation information, the manipulation event, and corresponding to the manipulation. The Node Identifier of the event.

After the component layout information sent by the auxiliary function interface 131 is retrieved, step S34 is performed to determine whether the manipulation event is a click (TYPE_VIEW_CLICKED), a focus (TYPE_VIEW_FOCUSED), or a long click (TYPE_VIEW_LONG_CLICKED).

In the case that it is determined that the manipulation event is a click (TYPE_VIEW_CLICKED), a focus (TYPE_VIEW_FOCUSED), or a long click (TYPE_VIEW_LONG_CLICKED), step S35 is performed, and according to the manipulation information and the tree hierarchy relationship, it is determined that the UI elements 23 correspond to each other. The UI element 23' of the event is manipulated and a Node Identifier is assigned to correspond to the event.

With further reference to FIG. 5, the detailed steps of step S35 of this embodiment are as follows: Step S351 is executed to perform a position function algorithm according to the tree hierarchy relationship and the component type of each UI element 23 of the component layout information, respectively. A node position symbol corresponding to each UI element 23 of the component layout information is assigned. Then, in step S352, the hash code corresponding to the manipulation event and the component are compared according to the hash code corresponding to each UI element 23 of the component layout information and the node position symbol corresponding to each UI component 23 of the component layout information. Corresponding to the hash code corresponding to each UI element 23 of the layout information, determining the UI element 23' of the UI elements 23' corresponding to the manipulation event, and assigning the node position of the UI element 23' corresponding to the manipulation event to be The node position of the event should be manipulated.

Here, the positional algorithm described in step S351 will be described. Referring to FIG. 6, it is a flowchart of the positional algorithm. The positional algorithm is used to calculate the node position corresponding to each UI element 23. The steps in an embodiment are as follows: Step S41 is executed to record the UI elements 23 in the tree hierarchy. The hierarchical parent node, next to step S42, records the location of the UI element 23 in the hierarchy, and performs step S43, according to the hierarchical parent node to which the UI element 23 belongs and the location assignment of the UI element 23 in the hierarchy. The node position character. The node location symbol is a set of locations of the hierarchical parent nodes and the UI elements 23 in their respective levels.

To explain the position algorithm more clearly, the UI element 23' is taken as an example here: the UI element 23' is located at the second position of its hierarchy, that is, the second position under the LinearLayout, and is assigned "1". In addition, the parent node to which the UI element 23' belongs is a FrameLayout and a LinearLayout, wherein the parent node FrameLayout of the UI element 23' assigns an empty set to the root level, and the other parent node LinearLayout is an inheritance. The first position below the root level is assigned "0" as the location code. The node position symbol assigned to the UI element 23' is a set of the position code "0" of the parent node and the node position code "1" of the UI element 23', and the node position symbol is indicated as [0, 1] in a collective form.

The code indicating the position of the node is encoded by "0", so the second position is represented by "1" and the first position is represented by "0". The elements in different positions in the node position symbol represent different levels, and the last element represents the position of the class to which it belongs, and sequentially advances to the position of the parent node of the previous level. The coding sequence or method is not limited to this. The starting point can be adjusted according to the system setting or size, the number of code bits can be increased, or the symbols other than numbers can be used. For example, the alphabetical order is used to represent the hierarchy represented by each position in the collection. It can also be adjusted according to the system settings, such as changing the position of the parent node by the element at the back position in the set.

For example, when the user clicks on the UI element 23 ′ that displays “BBB”, the present embodiment determines that the control event is a click (TYPE_VIEW_CLICKED), and after receiving the hash code “562471” corresponding to the manipulation event, The matching search is performed in the hash codes of the component layout information. In the hash code of the component layout information, the same as "562471" is the UI component 23', so that the UI component 23' is determined to correspond to the manipulation event. That is, the user's manipulation is to click on the UI element 23', and the node position symbol [0, 1] of the UI element 23' is assigned to be the node position symbol corresponding to the manipulation event.

Referring to FIGS. 1, 3, and 4, after step S35 is executed, it is determined whether a record stop command is received. If the record stop command is received, step S36 is executed, in response to the record stop command, in a manipulated log file. Writing the manipulation event, the node position symbol corresponding to the manipulation event, the node position symbol corresponding to each UI element 23 of the component layout information, and the number of child nodes belonging to each UI element in the tree hierarchy relationship The component type of each UI element 23, and the hash code corresponding to each UI element 23 of the component layout information. The form of the manipulation record file of this embodiment is script. After the execution of step S35, if the recording stop command is not received, step S33 is performed.

On the other hand, in step S34, if it is determined that the manipulation event is not a click (TYPE_VIEW_CLICKED), a focus (TYPE_VIEW_FOCUSED), or a long click (TYPE_VIEW_LONG_CLICKED), it is determined whether the record stop instruction is received, and the record is not received. In the case of stopping the command, step S33 is performed. When the record stop command is received, step S36 is executed.

An example of the manipulation log file of one embodiment of the present disclosure is as follows: { "event_type": "TYPE_VIEW_CLICKED", "check_node": [ { "class_name": "android.widget.FrameLayout", "get_node":[], "child_count" :2, }, { "class_name":"android.widget.LinearLayout", "get_node":[0], "child_count":5, }, { "class_name":"android.widget.ImageView", "get_node" :[0,0], "child_count":0, }, { "class_name":"android.widget.ImageView", "get_node":[0,1], "child_count":0, }, { "class_name":"android.widget.ImageView", "get_node":[0,2], "child_count":0, }, { "class_name":"android.widget.ImageView", "get_node":[0,3], "child_count":0, }, { "class_name":"android.widget.ImageView", "get_node":[0, 4], "child_count": 0, }, { "timeout":5000 } ], "perform_action": [ { "action":"ACTION_CLICK", "get_node":[0,1], } ] },

In the above control log file, the component layout information is recorded by the item check_node. In the item check_node, the information of each UI element 23 is recorded by the item class_name, the node position of the item get_node, and the number of child nodes recorded by the item child_count. And record the hash code with the project hash_code. In addition, the user manipulation is recorded with the item perform_action, in which the event is manipulated by the action record, and the node position character of the UI element 23 corresponding to the manipulation event is recorded with the item get_node.

Referring to Figures 1 and 7, an embodiment of the automatic execution control method of the present disclosure is applicable to the electronic device 1 shown in Figure 1. The electronic device 1 has a display 11 , a storage 12 , an antenna 14 capable of transmitting and receiving wireless signals, and an operation processor 13 coupled to the display 11 , the storage 12 and the antenna 14 . The device 13 includes an auxiliary function interface 131 configured to perform an Accessibility Service and drives the touch display 11 to display an operation screen 20. In the embodiment, the display 11 is a touch display.

Referring to FIG. 8 again, it is a node tree diagram of a tree hierarchy relationship of the automatic execution control method according to the embodiment of the disclosure. The operation screen 20 includes a plurality of user interface (UI) elements 23 configured for user manipulation, and the UI elements 23 constitute the tree hierarchy relationship. The electronic device 1 of the present embodiment is installed with an Android Operation System (Android Operation System), and the auxiliary function interface 131 is an Android Accessibility Service Interface.

9A and 9B are flowcharts of an automatic execution control method according to an embodiment of the present disclosure. The automatic execution control method automatically performs the manipulation according to a manipulation log file generated by the manipulation recording method shown in FIGS. 4, 5, and 6 of the previous embodiment. The steps of the automatic execution control method of an embodiment of the present disclosure are explained below.

Step S51 is executed to read the manipulation log file. The manipulation log file includes at least one component layout record, at least one manipulation event, at least one node position corresponding to the at least one manipulation event, and at least one component layout record corresponding to the at least one manipulation event. The content of the component layout record of the manipulation log file is information of each UI element 23 in the recording screen, and the component layout record of the manipulation log file includes components corresponding to each UI element 23 in the recording screen. Type, number of children, node position, and hash code. Each UI element 23 in the recorded picture constitutes a record tree hierarchical relationship, and the node position symbol is a set of hierarchical parent node information and node position information.

In the manipulation record file of the embodiment, the at least one manipulation event includes a first manipulation event, and a second manipulation event, that is, two manipulation actions are recorded in the manipulation record file of the embodiment. The at least one node position symbol includes a first node position symbol corresponding to the first manipulation event, and a second node position symbol corresponding to the second manipulation event. The at least one component layout record includes a first component layout record corresponding to the first manipulation event, and a second component layout record corresponding to the second manipulation event. The manipulation log file is generated by the manipulation recording method of the previous embodiment, and the information of the manipulation log file has been described in the previous embodiment, and therefore will not be further described in detail.

Step S53 is performed to retrieve a component layout information transmitted by the auxiliary function interface 131, and the component layout information is information describing the operation screen 20, that is, information of all UI elements 23 in the operation screen 20; The layout information includes the component type of each UI element 23 in the operation screen 20, the number of child nodes each UI element 23 in the operation screen 20 has in the tree hierarchy relationship, and each of the operation screens 20 respectively. A number of hash codes of the UI element 23. In this embodiment, the component type is a FrameLayout, a LinearLayout, a TextView, or an ImageView defined in the Android operating system, wherein the FrameLayout is a root node, and the LinearLayout is a node having a child node.

Step S52 is executed to start timing and calculate a judgment time, and step S54 is executed to determine the at least one component layout record according to the at least one component layout record of the manipulation log file and the component layout information transmitted by the auxiliary function interface 131. Whether it corresponds to the operation screen 20. Determining, according to the relationship between each node position record of the at least one component layout record and the tree hierarchy, determining a node position indicated by each node position of the at least one component layout record in the tree hierarchy relationship; Determining a node position corresponding to each node position symbol of a component layout record, determining the UI elements 23 corresponding to the component layout information of each node position symbol of the at least one component layout record, and then arranging the at least one component In the record, the number of child nodes of the UI element 23 corresponding to each node position symbol of the component layout record is obtained; then, the number of child nodes corresponding to each node position symbol of the at least one component layout record is compared with the component layout Whether the number of child nodes of the corresponding UI element 23 in the information is the same.

When the number of each child node of the at least one component layout record is the same as the number of child nodes of the UI component 23 corresponding to the component layout information, it is determined that the at least one component layout record corresponds to the operation screen 20, and the timing is stopped and Return to zero. On the other hand, when the number of each child node of the at least one component layout record is not all the same as the number of child nodes of the UI component 23 corresponding to the component layout information, step S57 is performed to determine whether the determination time is greater than a time limit. Time. When the determination time is greater than a time limit, step S58 is executed to end the automatic execution manipulation method.

It should be noted that the manipulation record file of the embodiment records two manipulation actions, that is, a first manipulation event and a second manipulation event, and the phase is reading and processing the first corresponding to the first manipulation event. The component layout record, so the at least one component layout record mentioned in the above step S54 is the first component layout record.

When the number of the child nodes of the at least one component layout record is different from the number of the child nodes of the UI component 23 corresponding to the component layout information, and the determination time is not greater than the time limit, step S59 is performed to perform a movement. The screen instruction (for example, the instruction ACTION_SCROLL_BACKWARD) moves the operation screen 20, and performs step S53 to retrieve the component layout information transmitted by the auxiliary function interface 131, and step S54 records the component layout information according to the at least one component layout. It is judged whether or not the component layout record corresponds to the operation screen 20. After the execution of the moving screen command, the operation screen 20 changes accordingly. For example, after the range of the operation screen 20 is moved, the UI element 23 that originally exceeds the display screen may appear. The auxiliary function interface 131 is based on the changed operation screen. 20 sends the corresponding component layout information, that is, the component layout information may thus include information of different UI elements 23.

Step S55 is executed, and only when the operation screen 20 corresponds to the component layout record, a manipulation execution instruction is transmitted to the auxiliary function interface 131 according to the manipulation event and the node position symbol, and the auxiliary function interface 131 performs the manipulation. Execute the instruction.

After determining that the operation screen 20 matches the component layout record, step S55 is executed to stop timing, and the at least one manipulation event of the manipulation log file and the at least one node position symbol are first read, and then according to the component layout information. The at least one node position symbol is converted to a corresponding UI element 23. The control record file of this embodiment records two manipulation actions, that is, the first manipulation event and the second manipulation event, and the stage is to read and process the first node position symbol corresponding to the first manipulation event. Therefore, the at least one node position symbol mentioned in the above S55 is the first node position symbol.

Taking the first node position symbol labeled [0, 1] as an example, the UI element 23 indicated by the first node position symbol [0, 1] is inherited from the first node of the second level. Two nodes, followed by the component layout information, obtain the UI element 23 corresponding to the first manipulation event.

Then, a first manipulation execution instruction is generated and transmitted to the auxiliary function interface 131 according to the first manipulation event and the UI element 23 corresponding to the first node position, and the auxiliary function interface 131 further executes the first manipulation execution instruction. .

After the first manipulation execution instruction corresponding to the first manipulation event is executed, proceeding to step S56, determining whether the manipulation log file has a second manipulation event that is not read, and determining that the first is not read. When the event is controlled, the second manipulation event, the second node position symbol, and the second component layout record are read. And converting the second node position symbol to the corresponding UI element 23 according to the component layout information, and generating and transmitting a second manipulation execution according to the second manipulation event and the UI element 23 corresponding to the second node position symbol. The instruction is to the auxiliary function interface 131. The auxiliary function interface 131 further executes the second manipulation execution instruction. The manipulation record file of the embodiment records two manipulation actions, that is, the first manipulation event and the second manipulation event, and the phase is reading and processing the second node position symbol corresponding to the second manipulation event. .

The operation of judging the UI element 23 corresponding to the second node position is the same as the method of the first node position corresponding to the UI element 23, and therefore the description will not be repeated. Since a plurality of manipulation events may be recorded in the manipulation log, step S56 is for reading and executing each manipulation event one by one, and confirming whether the next manipulation event needs to be executed.

On the other hand, when it is judged that there is no second manipulation event that has not been read, step S58 is executed to end the automatic execution manipulation method.

The example of the manipulation log file of this embodiment is as follows. Since the code is verbose, only the one associated with the execution of this embodiment is retained: { "event_type": "TYPE_VIEW_CLICKED", "check_node": [ { "class_name":"android.widget. FrameLayout", "get_node":[], "child_count":2, }, { "class_name":"android.widget.LinearLayout", "get_node":[0], "child_count":5, }, ... .. { "class_name":"android.widget.ImageView", "get_node":[0,1], "child_count":0, }, ..... { "timeout":5000 } ], "perform_action": [ { "action":"ACTION_CLICK", "get_node":[0,2], } ] }, { "event_type":"TYPE_VIEW_FOCUSED", "check_node": [ { "class_name":"android.widget.FrameLayout", "get_node":[], "child_count":2, }, { "class_name":"android.widget.LinearLayout", "get_node":[0], "child_count": 5, }, ... { "class_name":"android.widget.ImageView", "get_node":[0,2], "child_count":0, }, ...... { "timeout":5000 } ], "perform_action": [ { "action":"ACTION_FOCUSED", "get_node":[0,2], } ] }

In the above control log file, the two "check_node" paragraphs are the first component layout record and the second component layout record respectively, and in the two "perform_action" paragraphs, the "action" record manipulation event and the "get_node" record node are recorded. indicator.

Referring to Figures 1 and 10, an embodiment of the automatic execution control method of the present disclosure is applicable to the electronic device 1 shown in Figure 1. The electronic device 1 has a display 11 , a storage 12 , an antenna 14 capable of transmitting and receiving wireless signals, and an operation processor 13 coupled to the touch display 11 , the storage 12 , and the antenna 14 . The display 11 of this embodiment is a touch display.

The operation processor 13 includes an auxiliary function interface 131 configured to execute an Accessibility Service and drives the touch display 11 to display a first operation screen 21. The first operational screen 21 includes a plurality of first user interface (UI) elements 231 that are configured for manipulation by a user. The electronic device 1 of the present embodiment is operated by installing an Android operation system (Android Operation System), and the auxiliary function interface 131 is an Android Accessibility Service Interface.

FIG. 11 is a schematic diagram showing the display of a second operation screen 22 by the electronic device 1 according to an embodiment of the disclosure. The automatic execution control method of the present embodiment is similar to the previous embodiment shown in FIGS. 10A and 10B, and the control is automatically performed according to a control record file. The main difference between this embodiment and the previous embodiment is that the present embodiment The operation of the example is to call the second operation screen 22 shown in FIG. 13 via the first operation screen 21, and then perform the steps in the previous embodiment as shown in FIGS. 9A and 9B, as in the previous embodiment. The manipulation log file of this embodiment is also produced by the manipulation recording method shown in Figs. 4, 5, and 6.

12, 13A, and 13B are flowcharts of an embodiment of an automatic execution control method according to the present disclosure, and a node tree diagram of a tree hierarchy relationship. The steps of the automatic execution control method of the embodiment are as follows: Step S50 is executed, in response to the user manipulating the first UI element 231 ′, the electronic device 1 displays the second operation screen 22, wherein the second operation screen 22 has A plurality of second UI elements 232, and the second UI elements 232 form the tree hierarchy relationship.

Step S51 is executed to read the manipulation log file, where the manipulation log file includes at least one component layout record, at least one manipulation event, at least one node position symbol corresponding to the at least one manipulation event, and corresponding to the at least one manipulation event. At least one component layout record. The content of the component layout record of the manipulation log file is information of each UI element 23 in the recording screen, and the component layout record of the manipulation log file includes components corresponding to each UI element 23 in the recording screen. Type, number of children, node position, and hash code. Each UI element 23 in the recorded picture constitutes a record tree hierarchical relationship, and the node position symbol is a set of hierarchical parent node information and node position information.

In the manipulation record file of the embodiment, the at least one manipulation event includes a first manipulation event, and a second manipulation event, that is, two manipulation actions are recorded in the manipulation record file of the embodiment. The at least one node position symbol includes a first node position symbol corresponding to the first manipulation event, and a second node position symbol corresponding to the second manipulation event. The at least one component layout record includes a first component layout record corresponding to the first manipulation event, and a second component layout record corresponding to the second manipulation event. The manipulation log file is generated by the manipulation recording method of the embodiment corresponding to FIG. 4, and the information of the manipulation log file has been explained in the embodiment of the manipulation recording method, and therefore will not be described in detail.

Step S53 is executed to retrieve a component layout information transmitted by the auxiliary function interface 131. The component layout information transmitted by the auxiliary function interface 131 corresponds to the second operation screen 22, and the component layout information is information describing the second operation screen 22, that is, all the second UI components in the second operation screen 22. The information of the component layout information includes the component type of each of the second UI elements 232 in the second operation screen 22, and the child of each UI component 23 in the second operation screen 22 in the tree hierarchy relationship. The number of nodes, and a number of hash codes (Hash-codes) respectively belonging to each of the second UI elements 232 in the operation screen 20. In this embodiment, the component type may be a FrameLayout, a LinearLayout, a TextView, or an ImageView defined in the Android operating system, wherein the FrameLayout is a root node, and the LinearLayout is a node having a child node.

Then, step S52 is executed to start the timing and calculate a judgment time, and execute the step of determining the at least one component layout record according to the at least one component layout record of the manipulation log file and the component layout information transmitted by the auxiliary function interface 131. Whether it corresponds to the second operation screen 22. First determining, according to the relationship between each node position record of the at least one component layout and the tree hierarchy, the corresponding at least one second UI element 232 and the corresponding number of child nodes; and then, comparing each of the component layout records The number of child nodes corresponding to the node position symbol is the same as the number of child nodes of the second UI element 232 corresponding to the component layout information.

Determining that the at least one component layout record corresponds to the second operation screen 22 when the number of each child node of the at least one component layout record is the same as the number of child nodes of the second UI component 232 corresponding to the component layout information. And stop timing and return to zero. On the other hand, when the number of each child node of the at least one component layout record is not all the same as the number of child nodes of the UI component 23 corresponding to the component layout information, step S57 is performed to determine whether the determination time is greater than a time limit. . When the determination time is greater than a time limit, step S58 is executed to end the automatic execution manipulation method.

It should be noted that the manipulation record file of the embodiment records two manipulation actions, that is, a first manipulation event and a second manipulation event, and the phase is reading and processing the first corresponding to the first manipulation event. The component layout record, so the at least one component layout record mentioned in the above step S54 is the first component layout record.

When the number of the child nodes of the at least one component layout record is different from the number of the child nodes of the second UI component 232 corresponding to the component layout information, and the determination time is not greater than the time limit, step S59 is executed to execute A moving picture instruction (for example, the instruction ACTION_SCROLL_BACKWARD) moves the second operation screen 22, and performs step S53 and step S54 again. In this embodiment, the corresponding second UI element 232 and the number of child nodes are obtained in the component office information, wherein the conversion, the operation and the comparison, and the process of determining whether the second operation screen 22 corresponds to the second operation screen 22 are The previous embodiment shown in Figs. 9A and 9B is the same, and the detailed description will not be repeated.

Step S55 is executed, and only when the second operation screen 22 corresponds to the component layout record, a manipulation execution instruction is transmitted to the auxiliary function interface 131 according to the manipulation event and the node position, and the auxiliary function interface 131 is executed. This manipulation executes the instruction.

After determining that the second operation screen 22 matches the component layout record, step S55 is executed to stop timing, and the at least one manipulation event of the manipulation log file and the at least one node position symbol are first read, and then according to the component layout. The information converts the at least one node position symbol into a corresponding second UI element 232. The control record file of this embodiment records two manipulation actions, that is, the first manipulation event and the second manipulation event, and the stage is to read and process the first node position symbol corresponding to the first manipulation event. Therefore, the at least one node position symbol mentioned in the above S55 is the first node position symbol. The process of converting the node position symbol in the manipulation log to the second UI element 232 in the component layout information and the correlation operation are the same as in the previous embodiment, and therefore the description will not be repeated.

Then, a first manipulation execution instruction is generated and transmitted to the auxiliary function interface 131 according to the first manipulation event and the UI element 23 corresponding to the first node position, and the auxiliary function interface 131 further executes the first manipulation execution instruction. .

After the first manipulation execution instruction corresponding to the first manipulation event is executed, proceeding to step S56, determining whether the manipulation log file has the second manipulation event that is not read, and determining that there is an unread During the second manipulation event, the second manipulation event, the second node position character, and the second component layout record are read. And converting the second node position symbol to the corresponding second UI element 232 according to the component layout information, and generating and transmitting a second control element according to the second manipulation event and the second UI element 232 corresponding to the second node position symbol. The second execution instruction is sent to the auxiliary function interface 131, and the auxiliary function interface 131 further executes the second manipulation execution instruction. The manipulation record file of the embodiment records two manipulation actions, that is, the first manipulation event and the second manipulation event, and the phase is reading and processing the second node position symbol corresponding to the second manipulation event. . On the other hand, when it is judged that there is no second manipulation event that has not been read, the automatic execution manipulation method is ended.

It is to be noted that the first operation screen 21 and the second operation screen 22 of the embodiment may be different applications, that is, the automatic execution of the operation may be initiated through different applications, thereby assisting the user to automatically control different applications. Programs make switching and manipulation between applications more time-saving.

The present disclosure has been disclosed in the above embodiments, but it is not intended to limit the disclosure, and any person skilled in the art can make some changes and refinements without departing from the spirit and scope of the disclosure. The scope of the invention is defined by the scope of the appended claims.

1‧‧‧Electronic device
11‧‧‧ Display
12‧‧‧Storage
13‧‧‧Operation processor
131‧‧‧Accessory interface
14‧‧‧Antenna
20‧‧‧ operation screen
21‧‧‧First operation screen
22‧‧‧Second operation screen
23, 23'‧‧ User interface components
231‧‧‧First user interface component
232‧‧‧Second user interface component
S31~S36‧‧‧Steps
S351, S352‧‧‧ steps
S41~S43‧‧‧Steps
S50~S59‧‧‧Steps

1 is a block diagram of an electronic device in accordance with an embodiment of the present disclosure. FIG. 2 is a schematic diagram of an operation screen displayed by the electronic device according to the embodiment of FIG. 1. FIG. FIG. 3 is a schematic diagram of a node tree of a tree hierarchy relationship according to an embodiment of the present invention. 4 is a flow chart of a method of manipulating recording according to an embodiment of the present disclosure. FIG. 5 is a flow chart according to step S35 of FIG. 4. FIG. 6 is a flow chart of a positional algorithm for controlling a recording method according to an embodiment of the present disclosure. FIG. 7 is a schematic diagram showing an operation screen of the electronic device of FIG. 1. FIG. FIG. 8 is a schematic diagram of a node tree of a tree hierarchy relationship according to an automatic execution control method according to an embodiment of the disclosure. 9A and 9B are flowcharts of an automatic execution control method according to an embodiment of the present disclosure. FIG. 10 is a schematic diagram showing the first operation screen displayed by the electronic device of FIG. 1. FIG. FIG. 11 is a schematic diagram showing a second operation screen of the electronic device of FIG. 1. FIG. FIG. 12 is a schematic diagram of a node tree of a tree hierarchy relationship according to an automatic execution control method according to an embodiment of the disclosure. 13A and 13B are flowcharts of an automatic execution control method according to an embodiment of the present disclosure.

S52~S59‧‧‧Steps

Claims (23)

A control recording method for an electronic device having an auxiliary function interface configured to perform an Accessibility Service, and a display configured to display an operation screen, the operation screen having a use A plurality of user interface (UI) components are controlled, and the UI elements form a tree-like hierarchical relationship, and the auxiliary function interface sends a manipulation information according to the user-controlled UI component, and the manipulation recording method includes: Obtaining the manipulation information; determining, according to the manipulation information and the tree hierarchy relationship, a manipulation event controlled by the user, the UI component corresponding to the manipulation event in the UI elements, and assigning a node position symbol to the manipulation event; And writing the manipulation event and the node position character in a manipulation log file. The manipulation recording method according to claim 1, wherein the manipulation information includes a component layout information corresponding to the operation screen, the manipulation event, and a hash code corresponding to the manipulation event, the component layout information includes a plurality of hash codes corresponding to the UI elements of the operation screen, and a plurality of component types corresponding to the UI elements of the operation screen, and wherein assigning the node position symbol to the manipulation event further includes: The tree hierarchy relationship, the element type of each UI element of the component layout information respectively assigns a node position symbol corresponding to each UI element of the component layout information; and a hash corresponding to each UI element according to the component layout information a code, a node position corresponding to each UI element of the component layout information, and a hash code corresponding to each UI element of the component layout information corresponding to the hash code of the manipulation event, and determining corresponding to the UI components Manipulating the UI element of the event and assigning the node position of the UI element corresponding to the manipulation event to the node corresponding to the manipulation event Character set. The control recording method of claim 2, wherein only when the control event is clicked, focused, or long clicked, the executing step determines the user's manipulation event and the UI elements according to the manipulation information. The UI element corresponding to the event is manipulated, and the node position is assigned according to the UI element corresponding to the manipulation event. The control recording method according to claim 2, wherein the node position corresponding to each UI element of the component layout information is respectively included: recording the belonging of each UI element in the tree hierarchy relationship a parent node of each level; recording a position of each UI element in a hierarchy; and assigning each UI element according to each hierarchical parent node to which each UI element belongs and a position of each UI element in a hierarchy a node position character, the node position symbol being a set of locations of the hierarchical parent nodes and the respective UI elements in the hierarchy. The control recording method of claim 2, further comprising: temporarily storing the manipulation event, the node position symbol, the component layout information, and a node position symbol corresponding to each UI element of the component layout information; The node position symbol corresponding to each UI element in which the component layout information is written in the manipulation log file. The manipulation recording method according to claim 5, wherein the number of child nodes belonging to each UI element is also written in the manipulation record file. The control recording method described in claim 1, wherein the auxiliary function interface is an Android Accessibility Service Interface. An automatic execution control method for an electronic device comprising an auxiliary function interface configured to perform an auxiliary function service, and a display configured to display an operation screen having a number of manipulated a user interface component, the UI elements form a tree hierarchy relationship, the method comprising: reading a manipulation log file, the manipulation log file including at least one manipulation event, at least one node position corresponding to the at least one manipulation event And a component layout record corresponding to the at least one manipulation event; capturing a component layout information transmitted by the auxiliary function interface, the component layout information corresponding to the operation screen; according to the at least one component layout record a component layout information, determining whether the component layout record corresponds to the operation screen; and transmitting, in the case of the operation screen corresponding to the at least one component layout record, a manipulation execution instruction according to the at least one manipulation event and the node position symbol to The accessibility interface to enable the accessibility interface to perform the manipulation Line instruction. The automatic execution control method of claim 8, wherein the at least one component layout record includes at least one node position symbol, and at least one child node number corresponding to at least one node position symbol, and wherein the at least one The component layout record and the component layout information determine whether the at least one component layout record corresponds to the operation screen, and further includes: determining, according to the position identifier of each node of the at least one component layout record, corresponding to each of the component layout information Nodes of the UI elements of the node position, and obtaining the number of child nodes of each of the UI elements; whether the number of each child node recorded by the at least one component layout is the same as the number of child nodes of the corresponding UI element; When the number of each child node of the at least one component layout record is the same as the number of child nodes of the corresponding UI element, it is determined that the at least one component layout record corresponds to the operation screen. The automatic execution control method of claim 9, further comprising performing timing and calculating a judgment time, and determining, according to the at least one component layout record and the component layout information, whether the at least one component layout record corresponds to the operation. The screen further includes: ending the automatic execution manipulation method when the number of each child node of the at least one component layout record is different from the number of child nodes of the corresponding UI component, and the determination time is greater than a time limit; When the number of each child node of a component layout record is different from the number of child nodes of the corresponding UI component, and the determination time is not greater than the time limit value, a moving picture instruction is executed to move the operation screen, and then the step is performed. The component layout information transmitted by the auxiliary function interface, and the step determining whether the at least one component layout record corresponds to the operation screen according to the at least one component layout record and the component layout information. The automatic execution control method of claim 8, wherein the step of transmitting the manipulation execution instruction to the auxiliary function interface according to the at least one manipulation event only if the operation screen corresponds to the at least one component layout record When the auxiliary function interface executes the manipulation execution instruction, the method further includes: converting the at least one node position symbol into a corresponding UI component according to the component layout information and the hierarchical tree relationship, and according to the at least one manipulation event and The UI element corresponding to the at least one node locator transmits the manipulation execution instruction to the auxiliary function interface; and the auxiliary function interface executes the manipulation execution instruction. The automatic execution control method of claim 8, wherein the at least one manipulation event of the manipulation log file comprises a first manipulation event, and at least a second manipulation event, the at least one of the manipulation log files The node position symbol includes a first node position symbol corresponding to the first manipulation event, and at least one second node position symbol corresponding to the at least one second manipulation event, the at least one component layout record of the manipulation log file includes a corresponding a first component layout record of the first manipulation event, and at least one second component layout record corresponding to the at least one second manipulation event, wherein reading a manipulation log file includes reading the first manipulation event, the a first node locator, and the first component layout record, the automatic execution control method further comprising: reading the at least one second manipulation event, the at least one second node locator, and the at least one second component layout information Determining whether the read at least one second component layout record is correct according to the read at least one second component layout record and the component layout information The operation screen; and in the case that the operation screen corresponds to the at least one second component layout record being read, according to the at least one second manipulation event being read and the at least one second being read The node position symbol transmits a second manipulation execution instruction to the auxiliary function interface, so that the auxiliary function interface executes the second manipulation execution instruction. The automatic execution control method of claim 12, further comprising: determining whether the manipulation record file has the at least one second manipulation event that is not read; determining that the at least one is not read Reading, by the second control event, the at least one second manipulation event, the at least one second node position, and the at least one second component layout information; and determining that there is no at least one second that is not read This automatic execution control method ends when the event is manipulated. The automatic execution control method according to claim 9, wherein the node position symbol is a set of hierarchical parent node information and hierarchical position information, wherein each of the node position information is determined in the component layout information. When the UI component obtains the number of child nodes of each of the UI components, the method further includes: determining, according to the relationship between the node location and the tree hierarchy, the location identifier of each node in the tree hierarchy relationship The node location, and the corresponding UI component is obtained according to the indicated node location. The automatic execution control method described in claim 8 is wherein the auxiliary function interface is an Android auxiliary function interface. An automatic execution control method for an electronic device, the electronic device comprising an auxiliary function interface configured to perform an Accessibility Service, and a display configured to display a first operation screen, the first The operation screen includes at least one first user interface (UI) component configured to be manipulated, the method comprising: in response to executing the first UI element, the electronic device displays a second operation screen, and the The second operation screen has a plurality of second UI elements, and the second UI elements form a tree-like hierarchical relationship; reading a manipulation log file, the manipulation log file including at least one manipulation event, corresponding to at least one manipulation event a node locator, and at least one component layout record corresponding to the at least one manipulation event; capturing a component layout information transmitted by the auxiliary function interface, the component layout information corresponding to the second operation screen; The component layout record and the component layout information determine whether the at least one component layout record corresponds to the second operation And transmitting, in the case of the second operation screen corresponding to the at least one component layout record, a manipulation execution instruction to the auxiliary function interface according to the at least one manipulation event and the node position to enable the auxiliary function interface Execute the manipulation execution instruction. The automatic execution control method of claim 16, wherein the at least one component layout record comprises at least one node position symbol, and at least one child node corresponding to at least one node position symbol, wherein the at least one component is Layout record and the component layout information, determining whether the at least one component layout record corresponds to the second operation screen, further comprising: determining, according to the position identifier of each node of the at least one component layout record, correspondingly corresponding to the component layout information Each of the second UI elements of each node location identifier, and obtaining the number of child nodes of each of the second UI elements; comparing the number of each child node recorded by the at least one component layout with the corresponding second UI component Whether the number of child nodes is the same; and when the number of each child node of the at least one component layout record is the same as the number of child nodes of the corresponding second UI element, determining that the at least one component layout record corresponds to the second operation screen. The automatic execution control method of claim 17, further comprising: performing timing and calculating a judgment time, wherein determining whether the at least one component layout record corresponds to the component layout information according to the at least one component layout record The second operation screen further includes: when the number of each child node of the at least one component layout record is different from the number of child nodes of the corresponding second UI element, and the determination time is greater than a time limit value, ending the automatic execution a manipulation method; and when the number of each child node of the at least one component layout record is different from the number of child nodes of the corresponding second UI element, and the determination time is not greater than the time limit value, performing a moving screen instruction to move the Performing, by the second operation screen, the component layout information transmitted by the auxiliary function interface, and determining, according to the at least one component layout record and the component layout information, whether the at least one component layout record corresponds to the second operation screen . The automatic execution control method of claim 16, wherein the manipulation execution instruction is transmitted to the auxiliary function interface according to the at least one manipulation event, so that the auxiliary function interface executes the manipulation execution instruction, further comprising: The component layout information and the hierarchical tree relationship convert the at least one node position symbol into a corresponding second UI element, and transmit the according to the at least one manipulation event and the second UI element corresponding to the at least one node position symbol Manipulating execution instructions to the auxiliary function interface; and the auxiliary function interface executing the manipulation execution instruction. The automatic execution control method of claim 17, wherein the at least one manipulation event of the manipulation log file comprises a first manipulation event, and at least a second manipulation event, the at least one of the manipulation log files The node position symbol includes a first node position symbol corresponding to the first manipulation event, and at least one second node position symbol corresponding to the at least one second manipulation event, the at least one component layout record of the manipulation log file includes a corresponding a first component layout record of the first manipulation event, and at least one second component layout record corresponding to the at least one second manipulation event, wherein reading a manipulation log file includes reading the first manipulation event, the a first node locator, and the first component layout record, the automatic execution control method further comprising: reading the at least one second manipulation event, the at least one second node locator, and the at least one second component layout information Determining whether the read at least one second component layout record is correct according to the read at least one second component layout record and the component layout information Should be the second operation screen; and only if the second operation picture corresponds to the read at least one second component layout record, according to the read at least one second manipulation event and the read The at least one second node position assigns a second manipulation execution instruction to the auxiliary function interface, so that the auxiliary function interface executes the second manipulation execution instruction. The automatic execution control method of claim 20, further comprising: determining whether the control record file has the at least one second manipulation event that is not read; determining that the at least one is not read During the second manipulation event, the executing step reads the at least one second manipulation event, the at least one second node position symbol, and the at least one second component layout information; and determines that there is no at least one unread The automatic execution of the control method ends when the event is manipulated. The automatic execution control method according to claim 20, wherein the node position symbol is a set of hierarchical parent node information and hierarchical position information, and wherein the component corresponding to each node position symbol is determined in the component layout information. When the second UI component obtains the number of child nodes of each of the second UI components, the method further includes: determining, according to the relationship between the node location and the tree hierarchy, that each node location character is in the tree shape The node position indicated in the hierarchical relationship, and the corresponding second UI element is obtained according to the indicated node position. The automatic execution control method described in claim 17, wherein the auxiliary function interface is an Android auxiliary function interface.