CN115883909A - Page operation process backtracking method and device, computer equipment and storage medium - Google Patents

Abstract

The application relates to a page operation process backtracking method and device, computer equipment, storage medium and program product. The method comprises the following steps: responding to user page operation, and acquiring a screen recording object; recording the screen recording object; acquiring sampling buffer data in the process of recording a screen recording object; creating pixel buffer area reference according to the sampling buffer data, generating an image information object according to the pixel buffer area reference which is successfully created, and caching the image information object to a preset pixel buffer area, wherein the image information object comprises information required for generating an image; and when the user page operation is confirmed to be successful, acquiring the image information object from the preset pixel buffer area, and coding the image information object to obtain the backtracking video. By adopting the method, the accurate positioning of the page operation problem can be realized.

Description

Page operation process backtracking method, device, computer equipment and storage medium

technical field

The present application relates to the field of computer technology, in particular to a page operation process backtracking method, device, computer equipment and storage medium.

Background technique

With the development of computer technology, a page operation process backtracking technology appears, and the page operation process backtracking technology is used to backtrack the page operation process. For example, the page operation process may specifically refer to a process in which a user performs an operation on a terminal page.

In the traditional technology, the backtracking method mainly adopted is to collect the static page text data corresponding to each current page during the page operation process, and add a time stamp to each static page text data according to the time of the collected static page text data. Add the text data of each static page with a time stamp for storage, restore the corresponding pages according to the time stamp order, and realize the visual log backtracking of the page operation process.

However, in the traditional method, the static page text data corresponds to the log of each operation (click, check, long press, etc.). When the page operation is to fill in the information on the page, only the corresponding graphic data can be obtained. If the page operation When a program error is incorrectly generated, it is difficult to locate the page operation problem.

Contents of the invention

Based on this, it is necessary to provide a page operation process retrospective method, device, computer equipment, storage medium and program product capable of accurately locating page operation problems for the above technical problems.

A page operation process backtracking method, the method comprising:

Get the screen recording object in response to the user page operation;

Record the screen recording object;

Obtain the sample buffer data during the recording process of the screen recording object;

Create a pixel buffer reference based on the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to the preset pixel buffer, and the image information object includes the information required to generate the image;

When it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, and the image information object is encoded to obtain the backtracking video.

A page operation process backtracking device, said device comprising:

The response module is used to respond to user page operations and obtain screen recording objects;

The recording module is used to record the screen recording object;

An acquisition module, configured to acquire sampling buffer data in the process of recording a screen recording object;

The processing module is used to create a pixel buffer reference according to the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to a preset pixel buffer, and the image information object includes the information needed to generate the image information;

The encoding module is used to obtain the image information object from the preset pixel buffer when it is confirmed that the user page operation is successful, and encode the image information object to obtain the retrospective video.

A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Get the screen recording object in response to the user page operation;

Record the screen recording object;

Obtain the sample buffer data during the recording process of the screen recording object;

Create a pixel buffer reference based on the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to the preset pixel buffer, and the image information object includes the information required to generate the image;

When it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, and the image information object is encoded to obtain the backtracking video.

A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Get the screen recording object in response to the user page operation;

Record the screen recording object;

Obtain the sample buffer data during the recording process of the screen recording object;

Create a pixel buffer reference based on the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to the preset pixel buffer, and the image information object includes the information required to generate the image;

When it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, and the image information object is encoded to obtain the backtracking video.

A computer program product comprising a computer program which, when executed by a processor, implements the following steps:

Get the screen recording object in response to the user page operation;

Record the screen recording object;

Obtain the sample buffer data during the recording process of the screen recording object;

Create a pixel buffer reference based on the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to the preset pixel buffer, and the image information object includes the information required to generate the image;

When it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, and the image information object is encoded to obtain the backtracking video.

The method, device, computer equipment, storage medium, and program product for the above-mentioned page operation process obtain the screen recording object by responding to the user page operation, record the screen recording object, and obtain sampling buffer data during the recording process of the screen recording object, It can realize real-time response to user page operations, obtain the sampling buffer data corresponding to user page operations, create pixel buffer references based on sampling buffer data, generate image information objects according to successfully created pixel buffer references, and store image information The object is cached to the preset pixel buffer, and the sampling buffer data can be managed by creating a pixel buffer reference, and then the image information object corresponding to the user page operation can be obtained by using the successfully created pixel buffer reference, so that When it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, and the backtracking video is obtained by encoding the image information object. By backtracking the video, the backtracking of the user page operation can be realized, and the user's status on the page can be restored. The operation process when filling in information can realize accurate positioning of page operation problems.

Description of drawings

Fig. 1 is an application environment diagram of a page operation process backtracking method in an embodiment;

Fig. 2 is a schematic flow chart of a page operation process backtracking method in an embodiment;

Fig. 3 is a schematic diagram of terminal environment configuration of a page operation process backtracking method in an embodiment;

Fig. 4 is an application scenario diagram of a page operation process backtracking method in an embodiment;

FIG. 5 is an application scenario diagram of a page operation process backtracking method in another embodiment;

FIG. 6 is a schematic flowchart of a page operation process backtracking method in another embodiment;

FIG. 7 is a structural block diagram of a page operation process backtracking device in an embodiment;

Figure 8 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

The page operation process backtracking method provided in this application can be applied to the application environment shown in FIG. 1 . Wherein, the terminal 102 communicates with the server 104 through the network. Terminal 102 responds to the user page operation, obtains the screen recording object, records the screen recording object, obtains the sampling buffer data during the recording process of the screen recording object, creates a pixel buffer reference according to the sampling buffer data, and creates a pixel buffer reference according to the successfully created pixel buffer The area references to generate an image information object, and caches the image information object to the preset pixel buffer. The image information object includes the information required to generate the image. When it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer. The image information object is encoded to obtain a retrospective video, and the retroactive video is uploaded to the server 104 . Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be realized by an independent server or a server cluster composed of multiple servers, or can be nodes in the blockchain.

In one embodiment, as shown in FIG. 2 , a page operation process backtracking method is provided. The method is applied to the terminal in FIG. 1 as an example for illustration, including the following steps:

Step 202, in response to the user's page operation, acquire a screen recording object.

Wherein, the user page operation refers to the user's operation on the page displayed on the terminal. For example, the user page operation may specifically refer to operation behaviors such as sliding and clicking. The screen recording object is a singleton object used to record the screen. For example, the screen recording object may specifically be a packaged RPScreenRecorder (a shared recorder object that provides the ability to record audio and video of an application).

Specifically, when the user's operation on the displayed page is detected, the terminal will respond to the user's page operation to obtain a screen recording object. For example, when the screen recording object is a packaged RPScreenRecorder, the method of obtaining the screen recording object can be obtained through the sharedRecorder function, and the sharedRecorder function is used to return the shared screen recorder instance of the application.

Step 204, recording the screen recording object.

Specifically, after obtaining the screen recording object, the terminal will call the start recording method to record the screen recording object. Among them, the start recording method is used to start screen recording. For example, the method for starting recording may specifically be a startRecording function, and the startRecording function is used to start recording an application display.

Step 206, acquiring sample buffer data during the recording process of the screen recording object.

Wherein, the sampling buffer data refers to the original data of the recorded picture. For example, the sampling buffer data may specifically refer to sampleBuffer data, and the sampleBuffer data is a CMSampleBuffer object. In this embodiment, the sampleBuffer data includes video data.

Specifically, when recording the screen recording object, the terminal will call the method of obtaining the recorded image, and obtain the sampling buffer data in the callback. Wherein, the callback here may specifically refer to a block callback, and the callback function may be pre-added in the screen recording object. Among them, the method of acquiring recorded images is used to acquire sampling buffer data. For example, the method for obtaining a recorded image may specifically refer to the startCaptureWithHandler method, which is used to start screen capture.

Furthermore, the terminal will start an asynchronous task, call the method of obtaining the recorded image, and obtain the sampling buffer data in the block callback. What needs to be explained here is that the block callback can be continuously executed within the preset time interval of the system. When the preset time interval is 60FPS (Frames Per Second, the number of frames per second), the terminal will call 60 times, sequentially execute the steps of calling the method of obtaining recorded images, obtaining sampling buffer data in the callback, and generating image information objects based on the sampling buffer data, that is, 60 image information objects will be generated per second.

Step 208, create a pixel buffer reference according to the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to a preset pixel buffer, and the image information object includes information required to generate an image.

Wherein, the image information object refers to an object including all information required to generate an image. For example, an image information object can specifically refer to a CIImage object, which refers to the representation of an image processed or generated by a core image filter, but although it has image data associated with the image, it is not an image, and the CIImage object can be used as an image The "recipe", a CIImage object has all the information needed to generate an image. The preset pixel buffer refers to a preset pixel buffer for caching image information objects.

Specifically, after obtaining the sampling buffer data, the terminal will call the capture sampling buffer to image buffer method, create a pixel buffer reference according to the sampling buffer data, and then generate an image information object according to the successfully created pixel buffer reference. After obtaining the image information object, the terminal will cache the image information object in the preset pixel buffer for standby. Among them, the pixel buffer reference is a type of pixel picture, which is a collection of pixel points of the image frame, and contains the information of the pixel points. For example, the information of the pixel points can specifically refer to the starting coordinates of each pixel point wait. For example, the pixel buffer reference can specifically refer to CVPixelBufferRef. Since it starts with CV, it belongs to the CoreVideo module, which includes all compressed image information and video frame formats.

Step 210, when it is confirmed that the user page operation is successful, obtain the image information object from the preset pixel buffer, encode the image information object, and obtain the retroactive video.

Wherein, the successful operation of the user page means that the user has successfully submitted information. For example, when purchasing insurance, the successful operation of the user page may specifically mean that the user has successfully ordered the insurance.

Specifically, when confirming that the user page operation is successful, the terminal will obtain the image information object from the preset pixel buffer, encode the image information object, and obtain the retroactive video. Further, after obtaining the backtracking video, the terminal will also upload the backtracking video to the server, so that the server can backtrack the page operation process according to the backtracking video. It should be noted that since the image information objects are stored in the preset pixel buffer according to time sequence, it is sufficient to directly encode the image information objects in the order in which they enter the preset pixel buffer. If the user page operation is unsuccessful, the terminal will directly remove the bitmap data in the pixel buffer.

For example, in this embodiment, H264 encoding may be used specifically to generate a video stream. H264 is developed by ITU-T (ITU-Tfor ITU Telecommunication Standardization Sector, International Telecommunication Union Telecommunication Standardization Branch) Video Coding Experts Group (VCEG) and ISO/IEC (International Organization for Standardization, International Organization for Standardization) Motion Picture Experts Group (MPEG) A highly compressed digital video codec standard proposed by the joint Joint Video Team (JVT, Joint Video Team).

The above method of backtracking the page operation process, by responding to the user page operation, obtaining the screen recording object, recording the screen recording object, and obtaining the sampling buffer data during the recording process of the screen recording object, can realize real-time response to the user page operation, obtain To the sampling buffer data corresponding to the user page operation, create a pixel buffer reference according to the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, and cache the image information object to the preset pixel buffer, which can be passed Create a pixel buffer reference to manage the sampling buffer data, and then use the successfully created pixel buffer reference to obtain the image information object corresponding to the user page operation, so that when it is confirmed that the user page operation is successful, from the preset Assuming that the image information object is obtained in the pixel buffer, by encoding the image information object, a backtracking video can be obtained. Through the backtracking video, the backtracking of the user's page operation can be realized, and the operation process when the user fills in the information on the page can be restored, and the page can be realized. Accurate location of operational issues.

In one embodiment, creating a pixel buffer reference according to the sampling buffer data, and generating an image information object according to the successfully created pixel buffer reference includes:

Call the capture sample buffer to image buffer method, and create a pixel buffer reference based on the sample buffer data;

When the pixel buffer reference is successfully created, an image information object is generated according to the pixel buffer reference and preset attribute information of the preset pixel buffer.

Among them, the capture sample buffer to image buffer method is used to create a pixel buffer reference according to the sample buffer data. For example, the method of capturing the sampling buffer to the image buffer may specifically refer to the CMSampleBufferGetImageBuffer method, and the CMSampleBufferGetImageBuffer method is used to return the CVImageBuffer media data (ie, the pixel buffer reference) according to the CMSampleBuffer (ie, the sampling buffer data). It should be noted that if CMSampleBuffer does not contain CVImageBuffer, CMSampleBuffer contains CMBlockBuffer, or there are other errors, the return value is empty (ie NULL), the pixel buffer reference cannot be created, so when calling the capture sample buffer to image buffer method, According to the sampling buffer data, when creating a pixel buffer reference, it is necessary to determine whether the pixel buffer reference is empty. When the pixel buffer reference is not empty, the next step can be processed.

The preset attribute information refers to the attributes of the pre-configured preset pixel buffer, including the type of the pixel buffer, whether it has the ability to generate a bitmap, the width of the pixel, the height of the pixel, and the like. For example, the type of pixel buffer is represented by kCVPixelBufferPixelFormatTypeKey, and the corresponding preset attribute information can be kCVPixelFormatType_32BGRA, which means 32-bit RGBA pixels. Whether it has the ability to generate bitmaps is represented by kCVPixelBufferCGBitmapContextCompatibilityKey. The pixel is used to generate a bitmap. The width of the pixel is represented by kCVPixelBufferWidthKey, and the height of the pixel is represented by kCVPixelBufferHeightKey. According to the preset attribute information, the terminal can call the CVPixelBufferPoolCreate (pixel buffer pool creation) method to create a CVPixelBufferPoolRef (pixel buffer pool reference) , CVPixelBufferPoolRef is a technology in the CoreVideo framework, used to manage CVPixelBufferRef (pixel buffer reference), can operate the pixel buffer in the image, operate various memory storage of the image, use this interface to operate yuv (a color coding technology), pixels and memory space occupied.

Specifically, the terminal will call the capture sample buffer to image buffer method, and try to create a pixel buffer reference based on the sample buffer data. When the sample buffer data contains CVImageBuffer (a reference to the core video image buffer), the pixel buffer The reference can be created successfully, that is, it is not empty, and the terminal will further generate an image information object according to the pixel buffer reference and the preset attribute information of the preset pixel buffer. Among them, the pixel buffer reference contains all pixel information, including the coordinates of each pixel, etc., and the terminal can fill the pixel points of the image according to the pixel buffer reference and preset attribute information, and realize the initialization of the image information object. .

In this embodiment, by calling the method of capturing the sample buffer to the image buffer, the pixel buffer reference can be created according to the sample buffer data, so that when the pixel buffer reference is created successfully, according to the pixel buffer reference and preset The preset attribute information of the pixel buffer generates an image information object.

In one embodiment, after creating a pixel buffer reference according to the sampling buffer data, and generating an image information object according to the successfully created pixel buffer reference, it further includes:

Get the bitmap context corresponding to the image info object;

Render the image information object according to the bitmap context to obtain bitmap data;

Write bitmap data to preset image buffer.

Wherein, the bitmap context refers to basic information for converting an image information object into a UI (User Interface, user interface) image. Bitmap data refers to user interface images produced based on bitmap context and image information objects. Semaphore refers to a facility used in a multi-threaded environment, which can be used to ensure that two or more critical code segments are not called concurrently. Before entering a critical code section, a thread must acquire a semaphore, and once the critical code section is complete, the thread must release the semaphore. The preset image buffer refers to a preset area for storing bitmap data.

Specifically, after the image information object is generated according to the sampling buffer data, the terminal will call the create context function to obtain the bitmap context corresponding to the image information object, and render the image information object according to the bitmap context to complete the drawing of the bitmap data , get the bitmap data, turn on the bitmap write in the semaphore control queue to ensure that the bitmap write is not called concurrently, and write the bitmap data into the preset image buffer. Wherein, the creating context function may specifically refer to a UIGraphicsBeginImageContextWithOptions function, which can use specified options to create a graphics context based on a bitmap. Furthermore, before acquiring the bitmap context, the terminal will also encapsulate the function of creating a pixel cache, and use the encapsulated function of creating a pixel cache to manage the acquired bitmap context. For example, the terminal can be encapsulated by using RPScreenRecorder related functions.

For example, the way to render the image information object according to the bitmap context can be as follows: the terminal first calls the UIGraphicsPushContext function to pass in the bitmap context, and then calls the drawInRect method of UIImage to draw the frame of image according to the bitmap context and the image information object, and obtains Bitmap data, and finally call the CGContextRelease method to release the bitmap context memory. Among them, the UIGraphicsPushContext function is used to make the specified graphics context (that is, the bitmap context in this embodiment) the current context, and the drawInRect method is used to draw the entire image in the specified rectangle (the coordinate system of the bitmap context), and scale it as needed adapt.

In this embodiment, by acquiring the bitmap context corresponding to the image information object, the rendering of the image information object can be realized according to the bitmap context, and the bitmap data can be obtained. By enabling the semaphore to control the writing of the bitmap in the queue, the Write the bitmap data into the preset image buffer when the bitmap write is not called concurrently, so as to realize the correct writing of the bitmap data.

In one embodiment, the page operation process backtracking method further includes:

When confirming that the user page operation fails, generate and display a prompt to obtain bitmap data;

When receiving the consent feedback corresponding to the prompt for obtaining bitmap data, obtain the bitmap data from the preset image buffer;

Upload the problem location prompt carrying the bitmap data to the server, and the problem location prompt is used to prompt the server to confirm the cause of the operation failure based on the bitmap data.

Specifically, when confirming that the operation on the user page fails, the terminal and the server need to confirm the reason for the operation failure. At this time, the terminal will generate and display a prompt for obtaining bitmap data. The bitmap data prompt is used to remind the user of the terminal that the terminal wants to obtain Bitmap data is uploaded to the server. When the consent feedback corresponding to the prompt for obtaining bitmap data is received, it means that the user authorizes the terminal to obtain bitmap data and upload it to the server. The terminal will obtain bitmap data from the preset image buffer , upload the problem location prompt carrying the bitmap data to the server, so as to prompt the server to confirm the cause of the operation failure according to the bitmap data. Furthermore, after the server confirms the reason for the operation failure, it will feed back the reason for the operation failure to the terminal, so that the user using the terminal can confirm the reason for the operation failure, so as to avoid making the same mistake again and causing the operation to fail when performing a page operation next time.

In this embodiment, when the consent feedback is received, the bitmap data is obtained, and the problem location prompt carrying the bitmap data is uploaded to the server to prompt the server to confirm the cause of the operation failure according to the bitmap data, so that the server can feedback that the corresponding operation failed The reason goes to the terminal, so that the user using the terminal can confirm the cause of the operation failure, so as to avoid making the same mistake again and cause the operation to fail when the next page operation is performed, which can realize problem location and improve the success rate of user page operation.

In one embodiment, a pixel buffer reference is created according to the sampling buffer data, an image information object is generated according to the successfully created pixel buffer reference, and after the image information object is cached to the preset pixel buffer, the method further includes:

When no call to the stop recording method is detected, return to the step of recording the screen recording object;

When it is detected that the user page operation time interval is greater than the preset operation time threshold and/or the click event of the target buried point, it is confirmed that the user operation is completed, and the stop recording method is called.

Wherein, the stop recording method is used to stop the recording of the screen recording object. For example, the stop recording method may specifically refer to the stopCaptureWithHandler method, and the stopCaptureWithHandler method is used to stop screen capture. The user page operation time interval refers to the time interval between the user completing one page operation and the next page operation, and the preset operation time threshold can be set according to the needs. The target tracking point refers to the preset tracking point corresponding to the confirmed submission information on the page. For example, the target tracking point may specifically refer to tracking points corresponding to events such as submission and confirmation.

Specifically, when the terminal confirms that the user operation is over, it needs to call the stop recording method. If the call to the stop recording method is not detected, it means that the user operation is not over, and the terminal will return to the step of calling the start recording method to record the screen recording object. , get the image information object again until it detects that the stop recording method is called, indicating that the user operation is over, and the terminal will generate the corresponding backtracking video according to all the image information objects in the preset pixel buffer. Among them, the terminal can confirm the end of the user's operation through the preset operation, and the preset operation can be set according to the needs. Set it up yourself as needed.

Wherein, the terminal can monitor the recording attribute by adding a monitoring method, and monitor the change of the recording attribute, so as to implement the detection and call stop recording method. For example, adding a monitoring method may specifically refer to the addObserver method, and monitoring recording properties may specifically refer to monitoring the recording property of RPScreenRecorder. Furthermore, when monitoring the recording properties, it is mainly to monitor the changes of the recording properties, so in addition to monitoring when the recording ends, you can also monitor the interruption of the recording, for example, a call comes in during the operation of the user page When the recording is interrupted, the terminal will pause the recording and wait for the recording interruption to end before continuing to record.

Specifically, the terminal will detect in real time whether the user's page operation is over. The end of the user's page operation can specifically refer to the user not interacting on the screen for a certain period of time, confirming the submitted information on the page, etc. Therefore, the terminal detects the user's page operation time interval When it is greater than the preset operation time threshold (indicating that the user has not interacted with the screen for a certain period of time) and/or the click event of the target buried point (indicating that the submission information is confirmed on the page), it will confirm that the user operation is over, and call the stop recording method to stop recording.

In this embodiment, by detecting the user page operation time interval and the click event of the target buried point, the confirmation of the end of the user operation can be realized, so that the stop recording method can be called to stop the recording after confirming the end of the user operation.

In one embodiment, when it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, the image information object is encoded, and after the retrospective video is obtained, it also includes:

Obtain the operation information corresponding to the user page operation, and upload the operation information and the backtracking video to the server.

Wherein, the operation information refers to the information filled in by the user when operating the page. For example, when purchasing insurance, the operation information may specifically refer to the form data filled in by the user.

Specifically, when the user operates on the page, the terminal will record the buried point corresponding to the page operation. After obtaining the backtracking video, the terminal will obtain the operation corresponding to the user's page operation according to the recorded buried point corresponding to the page operation Information, upload operation information and backtracking video to the server. Further, when uploading the operation information and the backtracking video to the server, the terminal will generate a URL (Uniform Resource Locator, Uniform Resource Locator) local address, upload the operation information to the server, and simultaneously upload the backtracking video to the server asynchronously. It should be noted that if the backtracking video upload fails, it will try again the next time it starts to respond to user page operations.

In this embodiment, by obtaining the operation information corresponding to the user page operation, and uploading the operation information and the traceback video to the server, the server can realize the traceback of the user page operation according to the traceback video, and confirm the user information according to the operation information.

In one embodiment, the page operation process backtracking method further includes:

Send page data acquisition request;

receiving page data corresponding to the page data acquisition request;

Generate and display pages to be operated based on page data.

Wherein, the page data acquisition request is a request directed to the server to acquire page data, and the page data refers to data used to generate a page to be displayed.

Specifically, when a page operation is required, the user will send a page data acquisition request to the server through the terminal, and the server will send the page data corresponding to the page data acquisition request to the terminal after receiving the page data acquisition request, so that the terminal can Generate and display the page to be operated according to the received page data corresponding to the page data acquisition request.

For example, when the page operation is to purchase insurance, the user will enter the insurance purchase page of the terminal and fill in the user profile information, the terminal will send a page data acquisition request to the server according to the user profile information, and the server will issue the corresponding Insurance form page data, after the terminal receives the insurance form page data, it will create a form serialization parser to analyze different insurance items for different users, and convert insurance plans, insurance details, insurance notices, information disclosure and other information into json (JavaScript Object Notation, JS Object Notation) The fields in the data are established to correspond to the data relationship, and the toDataObj method provided by the application program is used to convert the json field value into an object type, create a tableView (table view), and generate a form based on the object data cell (cell), to further generate the page to be operated according to the form cell. Furthermore, after the form is created, the terminal will configure the current data environment. For example, the configuration on the IOS side may be as shown in FIG. 3 .

In this embodiment, by sending a page data acquisition request, receiving page data corresponding to the page data acquisition request, generating and displaying the page to be operated according to the page data, the display of the page to be operated can be realized.

The present application also provides an application scenario for order confirmation, which applies the above-mentioned backtracking method for the page operation process. As shown in Figure 4, specifically, the application of the page operation process backtracking method in this application scenario is as follows:

The terminal sends a form page data acquisition request to the server, the server publishes the form page data to the terminal, the terminal loads the form page data, creates a parser to generate a form view (that is, the page to be operated), receives user operation form behavior, and records the recording system. Initialize the settings (that is, respond to user page operations, obtain the screen recording object, and record the screen recording object), open the queue and create a pixel buffer, and write each frame of data to the graphics buffer in the block callback (obtaining the screen recording object Sampling buffer data during the recording process, create a pixel buffer reference based on the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to the preset pixel buffer), generate an order, end Record, encode into a video stream, and generate a corresponding relationship with the order (when confirming that the user page operation is successful, obtain the image information object from the preset pixel buffer, encode the image information object, and obtain the backtracking video), upload the video stream, and communicate with The correspondence generated by the order is related to the server (that is, the operation information and the backtracking video are uploaded to the server).

The present application further provides an application scenario, that is, realizing insurance purchase on a terminal based on the IOS system, and this application scenario applies the above-mentioned backtracking method of the page operation process. As shown in Figure 5, specifically, the application of the page operation process backtracking method in this application scenario is as follows:

The user operates the terminal to enter the insurance purchase page, the terminal sends a page data acquisition request carrying user information to the server, the server sends the insurance form data according to the user information, the terminal receives the insurance form data, uses the parser to parse it into data objects, generates a form view, configures Data environment, add observer mode to monitor the recording property of RPScreenRecorder, call the sharedRecorder of the basic recording object of the application (ie TMF in the figure) to obtain the screen recording object, call startRecording (start recording method), and start recording the screen recording object, And preset the kCVPixelBuffer (pixel buffer) attribute (that is, the preset attribute information of the preset pixel buffer).

Start the serial queue, start the asynchronous task to perform rendering through dispatch_async, the terminal will call startCaptureWithHandler (the method of obtaining the recorded image), obtain the sampleBuffer (sampling buffer data) in the block callback, and call CMSampleBufferGetImageBuffer (the method of capturing the sample buffer to the image buffer) to transfer Enter sampleBuffer to try to create CVPixelBufferRef (pixel buffer reference). When CVPixelBufferRef is not empty (that is, when the pixel buffer reference is created successfully), initialize the CIImage object (image information object) according to CVPixelBufferRef, and cache the CIImage object to the preset pixel buffer. Area, the CIImage object includes the information needed to generate the image, generate the bitmap context, call UIGraphicsPushContext to pass in the bitmap context, call the drawInRect method of UIImage to draw the frame image, and call CGContextRelease to release the memory of the bitmap context (that is, the image is processed according to the bitmap context Render the information object to get the bitmap data), open the bitmap write in the semaphore control queue, write the bitmap data into the preset image buffer, judge whether to call stop, and return to call startRecording when the call is not over (Start recording method), start recording the screen recording object, when the call ends, execute the callback of the recording property (that is, when the stop recording method is not detected, return to the step of recording the screen recording object, when detected When the user page operation time interval is greater than the preset operation time threshold and/or the click event of the target buried point, confirm the end of the user operation and call the stop recording method), and when the user page operation is successful, perform video encoding and upload data, ( That is to obtain the image information object from the preset pixel buffer, encode the image information object, obtain the retrospective video, obtain the operation information corresponding to the user page operation, and upload the operation information and the retroactive video to the server).

In one embodiment, as shown in FIG. 6 , the present application also provides a schematic flowchart to illustrate the above-mentioned page operation process backtracking method. The page operation process backtracking method specifically includes the following steps:

Step 602, sending a page data acquisition request;

Step 604, receiving page data corresponding to the page data acquisition request;

Step 606, generating and displaying the page to be operated according to the page data;

Step 608, responding to the user's page operation, and obtaining the screen recording object;

Step 610, recording the screen recording object;

Step 612, acquiring the sampling buffer data during the recording process of the screen recording object;

Step 614, call the method of capturing the sample buffer to the image buffer, and create a pixel buffer reference according to the sample buffer data;

Step 616, when the pixel buffer reference is successfully created, an image information object is generated according to the pixel buffer reference and the preset attribute information of the preset pixel buffer, and the image information object is cached in the preset pixel buffer, and the image information object includes the information needed to generate the image;

Step 618, acquire the bitmap context corresponding to the image information object;

Step 620, render the image information object according to the bitmap context to obtain bitmap data;

Step 622, write the bitmap data into the preset image buffer;

Step 624, return to the step of recording the screen recording object when calling the stop recording method is not detected;

Step 626, when it is detected that the user page operation time interval is greater than the preset operation time threshold and/or the click event of the target buried point, confirm that the user operation is completed, and call the stop recording method;

Step 628, determine whether the user page operation is successful, jump to step 630 when it is confirmed that the user page operation fails, and jump to step 636 when it is confirmed that the user page operation is successful;

Step 630, generating and displaying a prompt for acquiring bitmap data;

Step 632, when receiving the consent feedback corresponding to the prompt for acquiring bitmap data, acquire the bitmap data from the preset image buffer;

Step 634, uploading the problem location prompt carrying the bitmap data to the server, the problem location prompt is used to prompt the server to confirm the cause of the operation failure according to the bitmap data;

Step 636, acquire the image information object from the preset pixel buffer, encode the image information object, and obtain the backtracking video;

Step 638, obtaining the operation information corresponding to the user's page operation, and uploading the operation information and the backtracking video to the server.

It should be understood that although the steps in the flow charts involved in the above embodiments are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be executed at different times. These steps Or the execution sequence of the stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

In one embodiment, as shown in FIG. 7 , a page operation process backtracking device is provided. The device can adopt a software module or a hardware module, or a combination of the two becomes a part of the computer device. The device specifically includes: a response Module 702, recording module 704, acquisition module 706, processing module 708, and encoding module 710, wherein:

Response module 702, configured to respond to user page operations and acquire screen recording objects;

Recording module 704, for recording the screen recording object;

An acquisition module 706, configured to acquire sample buffer data during the recording process of the screen recording object;

The processing module 708 is used to create a pixel buffer reference according to the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, and cache the image information object to a preset pixel buffer. Information;

The encoding module 710 is configured to obtain the image information object from the preset pixel buffer when it is confirmed that the user page operation is successful, and encode the image information object to obtain a retrospective video.

The above-mentioned page operation process backtracking device, by responding to the user page operation, obtains the screen recording object, records the screen recording object, and obtains the sampling buffer data during the recording process of the screen recording object, which can realize the real-time response to the user page operation, obtain To the sampling buffer data corresponding to the user page operation, create a pixel buffer reference according to the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, and cache the image information object to the preset pixel buffer, which can be passed Create a pixel buffer reference to manage the sampling buffer data, and then use the successfully created pixel buffer reference to obtain the image information object corresponding to the user page operation, so that when it is confirmed that the user page operation is successful, from the preset Assuming that the image information object is obtained in the pixel buffer, by encoding the image information object, a backtracking video can be obtained. Through the backtracking video, the backtracking of the user's page operation can be realized, and the operation process when the user fills in the information on the page can be restored, and the page can be realized. Accurate location of operational issues.

In one embodiment, the processing module is also used to call the method of capturing the sample buffer to the image buffer, create a pixel buffer reference according to the sample buffer data, and when the pixel buffer reference is created successfully, according to the pixel buffer reference and preset The preset attribute information of the pixel buffer generates an image information object.

In one embodiment, the page operation process backtracking device further includes a drawing module, the drawing module is used to obtain the bitmap context corresponding to the image information object, render the image information object according to the bitmap context, obtain bitmap data, and convert the bitmap Data is written to the preset image buffer.

In one embodiment, the device for tracing back the page operation process further includes an upload module. The upload module is used to generate and display a prompt for obtaining bitmap data when it is confirmed that the user page operation fails. When feedback, the bitmap data is obtained from the preset image buffer, and the problem location prompt carrying the bitmap data is uploaded to the server. The problem location prompt is used to prompt the server to confirm the cause of the operation failure based on the bitmap data.

In one embodiment, the device for tracing back the page operation process further includes a detection module, and the detection module is used to return to the step of recording the screen recording object when the call to stop recording method is not detected, and when the user page operation is detected When the time interval is greater than the preset operation time threshold and/or the click event of the target buried point, confirm the end of the user operation and call the stop recording method.

In one embodiment, the uploading module is further configured to obtain operation information corresponding to user page operations, and upload the operation information and the backtracking video to the server.

In one embodiment, the device for tracing back the page operation process further includes a page generation module, configured to send a page data acquisition request, receive page data corresponding to the page data acquisition request, generate and display the page to be operated according to the page data.

For the specific limitations of the page operation process backtracking device, refer to the above-mentioned definition of the page operation process backtracking method, which will not be repeated here. Each module in the above-mentioned page operation process backtracking device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure may be as shown in FIG. 8 . The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, an operator network, NFC (Near Field Communication) or other technologies. When the computer program is executed by the processor, a method for backtracking the page operation process is realized. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad, or mouse.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation on the computer equipment to which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, there is also provided a computer device, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, storing a computer program, and implementing the steps in the foregoing method embodiments when the computer program is executed by a processor.

In one embodiment there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the steps in the foregoing method embodiments.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile memory and volatile memory. The non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, and the like. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only represent several implementation modes of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as limiting the scope of the patent for the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims (11)

1. A page operation process backtracking method, characterized in that the method comprises: Get the screen recording object in response to the user page operation; record the screen recording object; Acquiring sample buffer data during recording of the screen recording object; Create a pixel buffer reference according to the sampling buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to a preset pixel buffer, and the image information object includes the information needed to generate the image Information; When it is confirmed that the user page operation is successful, the image information object is acquired from the preset pixel buffer, and the image information object is encoded to obtain a retrospective video. 2. The method according to claim 1, wherein said creating a pixel buffer reference according to said sampling buffer data, and generating an image information object according to a successfully created pixel buffer reference comprises: Call the capture sample buffer to image buffer method, and create a pixel buffer reference according to the sample buffer data; When the pixel buffer reference is created successfully, an image information object is generated according to the pixel buffer reference and the preset attribute information of the preset pixel buffer. 3. The method according to claim 1, wherein, according to the sampling buffer data, creating a pixel buffer reference, after generating an image information object according to the successfully created pixel buffer reference, further comprising: Acquiring a bitmap context corresponding to the image information object; Rendering the image information object according to the bitmap context to obtain bitmap data; Write the bitmap data into a preset image buffer. 4. The method according to claim 3, further comprising: When confirming that the user page operation fails, generate and display a prompt to obtain bitmap data; When receiving the consent feedback corresponding to the prompt for acquiring bitmap data, acquire the bitmap data from the preset image buffer; Uploading a problem location prompt carrying bitmap data to the server, the problem location prompt is used to prompt the server to confirm the cause of the operation failure according to the bitmap data. 5. The method according to claim 1, wherein, according to the sampling buffer data, a pixel buffer reference is created, an image information object is generated according to the successfully created pixel buffer reference, and the image information object is cached After presetting the pixel buffer, also include: When it is not detected that calling the stop recording method, return to the step of recording the screen recording object; When it is detected that the user page operation time interval is greater than the preset operation time threshold and/or the click event of the target buried point, it is confirmed that the user operation is completed, and the stop recording method is called. 6. The method according to claim 1, wherein when it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer, and the image information object is encoded, After getting the backtracking video, also include: Obtain operation information corresponding to user page operations, and upload the operation information and the backtracking video to the server. 7. The method of claim 1, further comprising: Send page data acquisition request; receiving page data corresponding to the page data acquisition request; A page to be operated is generated and displayed according to the page data. 8. A page operation process backtracking device, characterized in that the device comprises: The response module is used to respond to user page operations and obtain screen recording objects; A recording module, configured to record the screen recording object; An acquisition module, configured to acquire sampling buffer data in the process of recording the screen recording object; The processing module is configured to create a pixel buffer reference according to the sample buffer data, generate an image information object according to the successfully created pixel buffer reference, cache the image information object to a preset pixel buffer, and the image information object Include the information needed to generate the image; The encoding module is configured to obtain the image information object from the preset pixel buffer when it is confirmed that the user page operation is successful, and encode the image information object to obtain a retrospective video. 9. A computer device, comprising a memory and a processor, the memory stores a computer program, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program step. 10. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented. 11. A computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images