CN118860550A - An automatic element extraction method based on understanding of business processes - Google Patents

Abstract

The present invention provides an automatic element extraction method based on the understanding of business processes, 1) establishing communication between the server end of the recorder C-end program and the recorder plug-in; recording the specific attributes of the event, generating a data packet of Xpath and iframe hierarchical information from the captured elements, and transmitting it to the server end of the recorder C-end program; element extraction and analysis, dynamically synthesizing the iframe path and the basic Xpath into a complete, directly accessible Xpath, realizing two-way data transmission and collaborative work; performing element verification and feedback. The method of the present invention can extract elements in depth, automatically mine the iframe hierarchy, and realize dynamic splicing, and can realize highlighting and secondary verification of the extracted elements.

Description

An automatic element extraction method based on understanding of business processes

Technical Field

The present invention belongs to the technical field of software development, and in particular relates to an automatic element extraction method based on understanding of business processes.

Background Art

As demand grows, labor costs continue to rise, and companies increasingly need to use automation technology to free workers from tedious, repetitive and low-productivity tasks and engage in decision-making work, such as abstract thinking, building connections, dealing with ambiguity, innovation, etc. Business process automation can drive business value growth and improve employee engagement.

The emerging workforce is being trained in increasingly advanced technology skills and automation. These people enter the workforce with higher expectations for job impact, satisfaction, and efficiency, and see software applications as a driver to achieve these expectations.

Within the enterprise, one of the biggest problems faced in the implementation of automated processes stems from the accurate selection of elements. Due to the dynamic variability of elements and the offset of various relative positions of elements after web page version iterations, the automated process fails. We have to re-do the complex process of element selection and analysis, which greatly consumes manpower, material and financial resources. Therefore, the automatic extractor of web page elements came into being to help us solve this problem.

Therefore, there is an urgent need for an auxiliary tool based on the Chrome browser in the web-side automation process, which can automatically extract the Xpath and FullXpath of the target element by selecting and storing them in the tool list for subsequent automation. The Path uniquely points to the target element.

Summary of the invention

In order to overcome the problems in the prior art, the present invention provides an automatic element extraction method based on the understanding of business processes to achieve deep element extraction.

The present invention provides the following technical solutions:

An automatic element extraction method based on understanding of business processes, characterized in that it includes the following steps: 1) a business person logs in to a recorder C-end program, creates a new recording task, and establishes communication between a server end of the recorder C-end program and a recorder plug-in located in a Chrome browser;

2) The recorder plug-in captures all user operations and interactions in real time, records the specific attributes of the events, generates data packets of Xpath and iframe level information from the captured elements, encapsulates them into data packets and transmits them to the server of the recorder C-end program;

3) The client of the recorder C-end program extracts and parses the elements according to the instructions of the service team, dynamically combines the iframe path and the basic Xpath into a complete and directly accessible Xpath, and feeds back the extraction results to the server;

4) Perform element verification and feedback.

Furthermore, in step 1), the recorder C-end program sends a WebSocket upgrade request to the recorder plug-in, and the recorder plug-in responds to the handshake, confirms the WebSocket connection, and establishes a TCP connection for two-way transmission, using TLS for encrypted transmission.

Furthermore, in step 2), the recorder plug-in captures the user's operation interaction events in real time through the DOM event listener, and the operations include but are not limited to: click, input, scroll, drag, hover, select, and right-click operations; the specific attributes of the recorded events include timestamp, CSS selector of the target element, input value or operation type, and the captured events and event attributes are encapsulated in JSON format, and the URL and timestamp of the current page are attached.

Furthermore, in the step of generating a data packet in step 2), the recorder plug-in uses DOM parsing technology to perform the following sub-steps:

a) Element location and data capture: The recorder plug-in monitors the user's interactive behavior in real time and captures the DOM elements related to the operation. The plug-in reads all available attributes of these elements, including ID, class, style, and their nested structure in the DOM tree;

b) Xpath and FullXpath generation: For each target element of an operation, the plug-in automatically generates two types of Xpath using DOM parsing technology. Simple Xpath provides a direct path from the nearest uniquely identified parent element to the target element; FullXpath provides a complete path starting from the root element, ensuring that the path accurately points to the target element no matter how other content on the page changes;

c) Position calculation: The plug-in calculates the absolute and relative position of the element in the DOM tree;

d) Data encapsulation and storage: The generated Xpath and location information are encapsulated into a data packet in JSON format, with necessary meta information attached, including timestamp and page URL.

Furthermore, in step 2), an iframe detection step is also included, and the specific judgment and processing flow is as follows:

a) Target element detection: When user interaction generates data, the system first checks whether the target element belongs to an iframe. If the target element is not in any iframe, it is directly processed according to the normal process;

b) Determine iframe nesting relationships: If the target element is located in one or more iframes, a recursive analysis is performed, starting from the iframe where the target element is located, tracing upward to the top-level document, and identifying each iframe layer by layer;

c) Path parsing and splicing: Basic path identification: Generate a basic Xpath for each found iframe element; Complete path construction: Dynamically splice the Xpath of the target element with the Xpath of each upper iframe;

d) Judgment and feedback: Data packet encapsulation: After constructing the complete iframe hierarchical path, the information is encapsulated into a data packet, which includes the complete hierarchical Xpath and related metadata; Send to the server: The data packet is sent to the server of the recorder C-end program, which performs further analysis and storage processing;

e) Exception handling: If an inaccessible element or path error is encountered at any step, the system will record the error information and may prompt the user to redo the operation or automatically try to fix the problem.

Furthermore, in step 3), a path resolution algorithm is executed, and the specific processing flow is as follows:

a) Path normalization: Normalize all received XPaths to ensure that the path format is unified for subsequent processing;

b) Iframe path fusion: The basic Xpath is merged with the corresponding iframe level path. If the element is directly in the top-level document, the final Xpath is the basic Xpath; if the element is in one or more iframes, starting from the innermost iframe, the path of each iframe is merged with the basic Xpath of its internal elements layer by layer.

c) Path optimization: Utilize the stability analysis of DOM structure to remove overly complex or redundant parts and simplify the path expression;

d) Result output: The final XPath generated is provided to the server as the unique identifier of the element.

Furthermore, after the synthesis into a complete and directly accessible Xpath in step 3), a step of highlighting secondary verification is also included, which is characterized in that:

a) If the client of the recorder C-side program detects that the final Xpath is valid and locates a single target element, it will automatically visually highlight the element on the user interface;

b) If the highlighted element is consistent with the user's expectations, the user can confirm the operation, and the system will record the verification as successful and store or pass the data to the next workflow;

c) If the highlighted element is inconsistent with the user's expectations or multiple elements are located, provide feedback options on the user interface to allow the user to refuse confirmation and request to re-record or modify the path;

d) If the verification passes, the client will record the verification status and timestamp of the element as a basis for subsequent processes; if the verification fails, the client will record the user's feedback and problem description.

Furthermore, in step 4), a deep learning-driven intelligent element is introduced, and the steps are as follows:

a) Training data preparation and preprocessing: If intelligent element recognition is required, before step 1), the client of the recorder C-end program collects the labeled web page element data, which will be used as the training set; the system preprocesses this data, including normalization, denoising and feature extraction;

b) Model training: Use a deep learning framework to build and train a model that can learn and identify different types of web page elements and their attributes. Once the model training is complete, the system performs a validation test to determine the optimal parameters and structure of the model.

c) Integration and implementation: If the model verification results after training are satisfactory, the model will be integrated into the element automatic extraction method. The model will receive input data from the user interface in real time, including screenshots of DOM elements and HTML codes; the model will predict the category and attributes of each element and generate the corresponding Xpath or identifier;

d) Intelligent element recognition: If the user interacts with an element in the browser, the system uses the trained deep learning model to analyze the element in real time, identify key attributes and paths, and automatically fill in or suggest the correct Xpath; the client of the recorder C-end program will provide a feedback mechanism;

e) Continuous learning and optimization: If the client of the recorder C-end program discovers new element types or encounters prediction errors, data on these situations will be automatically collected and used to retrain the model, thereby achieving continuous updating and optimization of the model.

By adopting the above technical solution, the present invention has the following beneficial effects:

1. The element automatic extraction method of the present invention is simple to operate, does not need to pay attention to the underlying principles, and can be used by business personnel.

2. The deep extraction of elements in the present invention can automatically mine the iframe hierarchy and realize dynamic splicing.

3. The present invention can realize highlighting of the extracted elements and secondary verification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic flow diagram of the extraction method of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the structural diagrams and specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

Example 1

The present invention provides an automatic element extraction method based on understanding of business processes, which is characterized by: 1) a business person logs in to a recorder C-end program, creates a new recording task, and establishes communication between a server end of the recorder C-end program and a recorder plug-in located in a Chrome browser;

2) The recorder plug-in captures all user operations and interactions in real time, records the specific attributes of the events, generates data packets of Xpath and iframe level information from the captured elements, encapsulates them into data packets and transmits them to the server of the recorder C-end program;

3) The client of the recorder C-end program extracts and parses the elements according to the instructions of the service team, dynamically combines the iframe path and the basic Xpath into a complete and directly accessible Xpath, and feeds back the extraction results to the server, thus realizing two-way data transmission and collaborative work;

4) Perform element verification and feedback.

In step 1), the recorder C-end program sends a WebSocket upgrade request to the recorder plug-in, and the recorder plug-in responds to the handshake, confirms the WebSocket connection, and establishes a TCP connection for two-way transmission, using TLS for encrypted transmission.

Example 2

Preferably, in step 2), the recorder plug-in captures the user's operation interaction events in real time through the DOM event listener, and the operations include but are not limited to: click, input, scroll, drag, hover, select, right-click operation; the specific attributes of the recorded event include timestamp, CSS selector of the target element, input value or operation type, so that the attribute record is more comprehensive, allowing the user's behavior to be reproduced or analyzed more accurately in subsequent operations. The "operation type" here refers to the specific action performed by the user on the element, such as click, double-click, drag, etc. The captured events and event attributes are encapsulated in JSON format, and the URL and timestamp of the current page are attached.

In the step of generating a data packet in step 2), the recorder plug-in uses DOM parsing technology to perform the following sub-steps:

a) Element positioning and data capture: The recorder plug-in monitors the user's interactive behavior in real time and captures the DOM elements related to the operation. The plug-in reads all available attributes of these elements, including ID, class, style, and their nested structure in the DOM tree.

b) Xpath and FullXpath generation: For each target element of an operation, the plug-in automatically generates two types of Xpath using DOM parsing technology. Simple Xpath provides a direct path from the nearest uniquely identified parent element to the target element; FullXpath provides a complete path starting from the root element (such as an HTML tag), ensuring that the path accurately points to the target element no matter how other content on the page changes.

c) Position calculation: The plugin calculates the absolute and relative positions of an element in the DOM tree. The absolute position refers to the pixel position of the element from the top of the page, while the relative position is the position of the element relative to its nearest positioned ancestor element.

d) Data packaging and storage: The generated Xpath and location information are packaged into a JSON format data packet, with necessary meta information attached, including timestamp and page URL. The data packet is then sent to the server of the recorder C-end program, which stores the data in the database to provide support for subsequent automatic element extraction and other business logic.

In step 2), an iframe detection step is also included, and the specific judgment and processing flow is as follows:

a) Target element detection: When user interaction generates data, the system first checks whether the target element belongs to an iframe. If the target element is not in any iframe, it is directly processed according to the normal process.

b) Determine iframe nesting relationships: If the target element is located in one or more iframes, a recursive analysis is performed, starting from the iframe where the target element is located, tracing upward to the top-level document, and identifying each iframe layer by layer.

c) Path parsing and splicing: Basic path identification: Generate a basic Xpath for each found iframe element; Complete path construction: Dynamically splice the Xpath of the target element with the Xpath of each upper iframe. This splicing process starts from the innermost iframe and gradually splices to the outer layers until the top-level document is reached.

d) Judgment and feedback:

Packet encapsulation: After constructing the complete iframe hierarchical path, encapsulate the information into a packet, which includes the complete hierarchical Xpath and related metadata; Send to the server: The packet is sent to the server of the recorder C-end program, which performs further analysis and storage processing;

e) Exception handling: If an inaccessible element or path error is encountered at any step, the system will record the error information and may prompt the user to redo the operation or automatically try to fix the problem.

In step 3), the path resolution algorithm is executed. The main function of the path resolution algorithm is to convert the dynamically generated Xpath and iframe hierarchical information into a stable and unique identifier, namely the final Xpath. The key issues that the algorithm needs to solve include multi-layer nested iframe resolution, dynamic content processing, and adaptation to page structure changes.

The specific processing flow is as follows:

a) Path normalization: Normalize all received XPaths to ensure that the path format is unified for subsequent processing.

b) Iframe path fusion: The basic Xpath is fused with the corresponding iframe level path through an algorithm: if the element is directly in the top-level document, the final Xpath is the basic Xpath; if the element is in one or more iframes, starting from the innermost iframe, the path of each iframe is fused with the basic Xpath of its internal elements layer by layer.

Path fusion can be simplified by the following formula:

\[X_{final}＝f(X_{n},I_{n-1},I_{n-2},...,I_1)\]

Among them, \(X_{final}\) is the final Xpath, \(X_n\) is the basic Xpath of the target element, and \(I_{n-1},I_{n-2},...,I_1\) represent the iframe paths from the first iframe of the target element to the outer layer.

c) Path optimization: Utilize the stability analysis of the DOM structure to remove overly complex or redundant parts and simplify the path expression.

d) Result output: The final XPath generated is provided to the server as the unique identifier of the element and can be used in subsequent automated operations or test scripts.

Example 3

After synthesizing into a complete and directly accessible Xpath in step 3), a highlight secondary verification step is also included. Highlight secondary verification is a method of verifying the accuracy of the element path. By highlighting the resolved elements on the user interface, it is ensured that the path correctly reflects the user's intention.

After the final Xpath of the element has been successfully generated, the system automatically triggers the highlighted secondary verification step.

a) If the client of the recorder C-end program detects that the final Xpath is valid and locates a single target element, the element will be automatically visually highlighted on the user interface. The highlighting can be a visual prompt such as a border, background color change or flashing.

b) If the highlighted element is consistent with the user's expectations, the user can confirm the operation, and the system will record the verification as successful and store or pass the data to the next workflow.

c) If the highlighted element is inconsistent with the user's expectation or multiple elements are located, provide a feedback option on the user interface to allow the user to refuse confirmation and request to re-record or modify the path.

d) If the verification passes, the client will record the verification status and timestamp of the element as a basis for subsequent processes; if the verification fails, the client will record the user's feedback and problem description for the technical team to analyze and optimize the path generation algorithm.

For example, on an online e-commerce platform, business personnel need to automatically extract the element information of the "Add to Cart" button on the product details page for use in subsequent automated testing or business process automation. The following is a detailed description of the process.

Step 1: Business personnel interaction starts

-Business personnel log in to the recorder C-end program, create a new recording task, and start communicating with the recorder plug-in in the Chrome browser.

Step 2: Real-time operation capture and recording

-Business personnel operate on the product details page of the e-commerce platform, move the mouse and click the "Add to Cart" button.

-The recorder plugin captures and records this action interaction in real time, including the specific properties of the click event such as timestamp, CSS selector, action type (click) and input value (none).

-The plugin also generates Xpath and FullXpath from the captured elements through DOM parsing technology, and calculates the absolute and relative positions of the elements in the DOM tree.

-If the button element is in an iframe, the plug-in will also detect and recursively parse the paths of all parent iframes, and dynamically concatenate the base path with the iframe-level path to form a complete iframe-level path.

Step 3: Path information analysis and optimization

-The client of the recorder C-end program receives this data and executes the path resolution algorithm to dynamically synthesize a directly accessible, optimized complete Xpath.

- Combined with the highlighted secondary verification function, the system will highlight the "Add to Cart" button on the user interface to ensure the accuracy of the path.

Step 4: Result verification and adjustment

-Business personnel check whether the highlighted elements are correct and verify the results of the operation.

-If the highlighted elements are correct and the business person confirms completion, the system marks the verification as successful.

-If a problem is found, business personnel can choose to re-record or use the delete function to adjust the incorrect element path.

Step 5: Data Storage and Reporting

-Once confirmed, the server side of the recorder C-side program will store the extraction results and related data in the database and generate a report for subsequent analysis.

Example 4

After step 3), the steps of re-recording, deleting and highlighting verification are as follows:

a) Re-recording function:

If the user is not satisfied with the currently captured element or path parsing result, the user can choose the re-recording function; the client of the recorder C-end program will allow the user to restart the recording process to capture new user interactions and element operations; re-recording can be a complete restart or start from the last detection point.

b) Delete function:

If the user determines that a captured element or generated XPath is no longer needed or is incorrect, the user can choose to delete the record of the element. The client of the recorder C-end program will provide an interface option to allow the user to delete a specific element or its associated path information to ensure the accuracy and validity of the data.

c) Highlight verification:

If the element path has been generated and needs to be verified for correctness, the client of the recorder C-end program automatically performs the highlighting process, as described in item 7 above. The system will highlight the target element located by the final Xpath on the user interface to verify its accuracy.

If the user confirms that the highlighted element is as expected, the client of the recorder C-end program marks the verification result as successful and continues the subsequent process.

If the user finds that the highlighted elements are not as expected, they can re-record or use the delete function to remove the incorrect data and need to re-capture and verify the elements.

d) User interaction and feedback:

The client of the recorder C-end program will provide an intuitive user interface for users to perform the above operations and update data and status in real time based on user input.

The client of the recorder C-end program should record all user operations and system responses to support problem diagnosis and process optimization.

Example 5

In step 4), deep learning-driven intelligent element recognition is introduced. The specific steps are as follows:

a) Training data preparation and preprocessing: If intelligent element recognition is required, before step 1), the client of the recorder C-end program collects the labeled web page element data, which will be used as the training set; the system preprocesses the data, including normalization, denoising and feature extraction, to ensure that the data quality meets the training requirements;

b) Model training: Use a deep learning framework to build and train a model that can learn and identify different types of web page elements and their attributes. Once the model training is completed, the system performs a validation test to evaluate the accuracy and efficiency of the model and determine the optimal parameters and structure of the model through cross-validation and other techniques;

c) Integration and implementation: If the model verification results after training are satisfactory, the model will be integrated into the element automatic extraction method. The model will receive input data from the user interface in real time, including screenshots of DOM elements and HTML codes; the model will predict the category and attributes of each element and generate the corresponding Xpath or identifier;

d) Intelligent element recognition: If the user interacts with elements in the browser, the system uses the trained deep learning model to analyze these elements in real time, identify key attributes and paths, and automatically fill in or suggest the correct Xpath; the client of the recorder C-end program will provide a feedback mechanism so that users can verify and correct the output of the model to further improve the accuracy of the model;

e) Continuous learning and optimization: If the client of the recorder C-end program discovers new element types or encounters prediction errors, the relevant data of these situations will be automatically collected and used for retraining the model to achieve continuous updating and optimization of the model.

The above-mentioned embodiments only express the implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims (8)

1. An element automatic extraction method based on understanding of a business process is characterized by comprising the following steps: 1) Business personnel log in a recorder C-end program to create a new recording task, and a communication is created between a server end of the recorder C-end program and a recorder plug-in located in a Chrome browser;

2) The recorder plug-in captures all operation interactions of a user in real time, records specific attributes of an event, generates data packets of Xpath and iframe level information from the captured elements, encapsulates the data packets into data packets, and transmits the data packets to a server of a recorder C-end program;

3) The client of the recorder C-end program performs element extraction and analysis according to the service team instruction, dynamically synthesizes the iframe path and the basic Xpath into a complete and directly accessible Xpath, and feeds back the extraction result to the service end;

4) Element verification and feedback are performed.

2. The method according to claim 1, wherein in step 1), the recorder C-side program sends a WebSocket upgrade request to the recorder plug-in, and the recorder plug-in responds to the handshake, confirms the WebSocket connection, establishes a TCP connection bidirectional transmission, and encrypts the transmission using TLS.

3. The method for automatically extracting elements according to claim 1, wherein in step 2), the recorder plug-in captures the operation interaction event of the user in real time through the DOM event listener, and the operations include, but are not limited to: clicking, inputting, scrolling, dragging, hovering, selecting and right-hand key operation; specific attributes of the recorded event include a timestamp, a CSS selector of the target element, an input value or an operation type, the captured event and event attributes are encapsulated into a JSON format, and a URL and a timestamp of the current page are appended.

4. The method of automatic element extraction according to claim 1, wherein in the step of generating a data packet in step 2), the recorder plug-in performs the following sub-steps using DOM parsing technique:

a) Element positioning and data capturing: the recorder plug-in monitors interaction behavior of a user in real time, captures DOM elements related to operation, and reads all available attributes of the elements, including IDs, classes, styles and nested structures thereof in a DOM tree;

b) Xpath and FullXpath generation: for each operational target element, the plug-in automatically generates two types of Xpath using DOM parsing techniques, the simple Xpath providing a direct path from the nearest parent element with a unique identification to the target element; fullXpath provides a complete path from the root element, ensuring that the path can accurately point to the target element no matter how other contents on the page change;

c) Position calculation: the plug-in calculates the absolute position and the relative position of the element in the DOM tree;

d) Data packaging and storage: the generated Xpath and location information are encapsulated into a JSON format packet, and necessary meta information including a time stamp and a page URL is attached.

5. The method for automatically extracting elements according to claim 1, wherein in step 2), the method further comprises an iframe detection step, and the specific judgment and processing flow is as follows:

a) Target element detection: when user interaction generates data, the system firstly checks whether the target element belongs to a certain iframe, and if the target element is not in any iframe, the target element is directly processed according to a conventional flow;

b) Determining iframe nesting relation: if the target element is positioned in one or more iframes, performing recursion analysis, and tracing upwards from the iframes in which the target element is positioned to the top-level document, and identifying each iframe layer by layer;

c) Path analysis and splicing: and (3) basic path identification: generating a basic Xpath for each discovered iframe element; and (3) complete path construction: dynamically splicing the Xpath of the target element with the Xpath of each upper-layer iframe;

d) Judging and feeding back:

And (3) packaging data packets: after a complete iframe hierarchical path is constructed, information is packaged into a data packet, wherein the data packet comprises a complete hierarchical Xpath and related metadata; transmitting to a server side: the data packet is sent to a server of a recorder C-end program, and the server performs further analysis and storage processing;

e) Exception handling: if an element is encountered in any step that is not accessible or has a path error, the system will record an error message and may prompt the user to re-operate or automatically attempt to fix the problem.

6. The method for automatically extracting elements according to claim 1, wherein in step 3), a path analysis algorithm is executed, and the specific process flow is as follows:

a) Path normalization: normalizing all the received Xpaths to ensure the uniform format of the paths, and facilitating the subsequent processing;

b) iframe path fusion: fusing the basic Xpath with the corresponding iframe hierarchical path through an algorithm, wherein if the element is directly positioned in the top-level document, the final Xpath is the basic Xpath; if the element is located in one or more iframes, the path of each iframe and the basic Xpath of the element inside are fused outwards layer by layer from the innermost iframe.

C) Path optimization: removing the too complex or redundant part by utilizing stability analysis of the DOM structure, and simplifying the path expression;

d) And (3) outputting results: the generated final Xpath is provided as a unique identifier of the element to the server.

7. The method for automatic element extraction according to claim 1, further comprising the step of highlighting the secondary authentication after the step 3) of synthesizing into a complete, directly accessible Xpath,

A) If the client of the recorder C-end program detects that the final Xpath is valid and a single target element is positioned, the element is automatically visually highlighted on a user interface;

b) If the highlighted element is consistent with the user expectation, the user can confirm the operation, the system records the verification as successful, and stores or transmits the data to the next workflow;

c) If the highlighted element is inconsistent with the user's expectation or a plurality of elements are positioned, providing feedback options on the user interface, allowing the user to reject the confirmation and to request re-recording or modifying the path;

d) If the verification is passed, the client uses the verification state and the time stamp of the record element as the basis of the subsequent flow; if the verification fails, the client records the feedback and the problem description of the user.

8. The automatic element extraction method according to claim 1, wherein the deep learning driven intelligent element is introduced in step 4), comprising the steps of:

a) Training data preparation and preprocessing: if intelligent element identification is needed, before step 1), collecting marked webpage element data by a client of a recorder C-end program, wherein the webpage element data is used as a training set; the system preprocesses the data, including normalization, denoising and feature extraction;

b) Model training: constructing and training a model by using a deep learning framework, wherein the model can learn and identify different types of webpage elements and attributes thereof, and if the model training is completed, the system performs a verification test to determine optimal parameters and structures of the model;

c) Integration and implementation: if the trained model verification result is satisfactory, integrating the model into an element automatic extraction method, wherein the model receives input data from a user interface in real time, wherein the input data comprises screenshot of DOM elements and HTML codes; the model predicts the class and attribute of each element and generates a corresponding Xpath or identifier;

d) Intelligent element identification: if the user interacts with the elements in the browser, the system uses the trained deep learning model to analyze the elements in real time, identify key attributes and paths, and automatically fill or suggest correct Xpath; the client of the recorder C-side program will provide a feedback mechanism;

e) Continuous learning and optimization: if the client of the recorder C-end program finds a new element type or encounters a prediction error, the data of the situations are automatically collected and used for retraining the model, so that continuous updating and optimization of the model are realized.