WO2021128846A1 - Electronic file control method and apparatus, and computer device and storage medium - Google Patents

Abstract

Provided are an electronic file control method and apparatus, and a computer device and a storage medium. The method comprises: capturing a real-time video stream (S201); then acquiring, from the real-time video stream, an image set with a facial image of the current user included therein, and acquiring an action frame set from the real-time video stream in a preset manner (S202); then, comparing the facial image in the image set with a preset facial image, and obtaining a permission verification result (S203); before each instance of control over an electronic file, quickly verifying the security of controlling the electronic file, and when the permission verification result indicates that verification is passed, performing action recognition on the action frame set by means of a trained AU detection network, and obtaining a target action (S204); acquiring, from a preset instruction set, an instruction corresponding to the target action, and taking the instruction as a target instruction (S205); and executing the target instruction on the electronic file (S206). By means of simultaneously performing permission verification and action confirmation on an acquired frame image, an electronic file is rapidly controlled, and the efficiency of controlling the electronic file is improved.

Description

Electronic file control method, device, computer equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on December 23, 2019 with the application number 201911339761.9 and the invention title "Electronic file control method, device, computer equipment and storage medium", and the entire content of it is approved The reference is incorporated in this application.

Technical field

This application relates to the field of information security, and in particular to a method, device, computer equipment, and storage medium for controlling electronic files.

Background technique

With the development of Internet technology and the popularization of smart terminals, people can accomplish more and more work and life matters through smart devices. Some documents that originally needed to be processed offline are slowly being completed through smart devices.

At present, people mainly operate electronic files through touch screens or through physical buttons on smart devices.

technical problem

In the process of realizing this application, the inventor realized that the existing solution has at least the following problems: using this method is more cumbersome, making the control efficiency of electronic files lower. At the same time, in some fields, it involves some safety issues. For higher electronic documents, such as electronic contracts, financial statements, user privacy data, etc., in order to ensure security, the processing of these electronic documents often requires authorization verification, through touch screen (touch screen input password or pattern unlock) or smart device The operation of the physical button on the computer may reveal the unlock password, which makes the security not guaranteed. Finding a method for efficient electronic file control on smart devices has become a problem that needs to be solved urgently.

Technical solutions

The embodiments of the present application provide a method, a device, a computer device, and a storage medium for controlling electronic files, so as to improve the current control efficiency of electronic files.

In order to solve the above technical problems, an embodiment of the present application provides an electronic file control method, including:

Collect real-time video stream;

From the real-time video stream, obtain an image set containing the facial image of the current user, and obtain an action frame set from the real-time video stream according to a preset method;

Comparing the facial images in the image collection with the preset facial images to obtain a permission verification result;

If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action;

Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction;

The target instruction is executed on the electronic file.

In order to solve the above technical problems, an embodiment of the present application also provides an electronic file control device, including:

Data collection module, used to collect real-time video stream;

An image acquisition module, configured to acquire an image collection containing facial images of the current user from the real-time video stream, and acquire an action frame collection from the real-time video stream according to a preset method;

The authorization verification module is used to compare the facial image in the image collection with the preset facial image to obtain the authorization verification result;

An action detection module, configured to perform action recognition on the set of action frames through the trained AU detection network if the permission verification result is a pass, to obtain a target action;

The instruction determining module is used to obtain the instruction corresponding to the target action from a preset instruction set as the target instruction;

The file control module is used to execute the target instruction on the electronic file.

In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, and the processor executes all When the computer-readable instructions are described, the steps of the following electronic file control method are realized:

Collect real-time video stream;

From the real-time video stream, obtain an image set containing the facial image of the current user, and obtain an action frame set from the real-time video stream according to a preset method;

Comparing the facial images in the image collection with the preset facial images to obtain a permission verification result;

If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action;

Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction;

The target instruction is executed on the electronic file.

In order to solve the above technical problems, embodiments of the present application also provide a computer-readable storage medium, the computer-readable storage medium stores computer-readable instructions, the computer-readable instructions are executed by a processor to achieve the following electronic file Steps of the control method:

Collect real-time video stream;

From the real-time video stream, obtain an image set containing the facial image of the current user, and obtain an action frame set from the real-time video stream according to a preset method;

Comparing the facial images in the image collection with the preset facial images to obtain a permission verification result;

If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action;

Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction;

The target instruction is executed on the electronic file.

Beneficial effect

The electronic file control method, device, computer equipment, and storage medium provided by the embodiments of the present application, on the one hand, collect real-time video streams, and then from the real-time video streams, obtain an image collection containing the current user’s facial images, and follow the pre-defined Set up a method to obtain the action frame set from the real-time video stream, and then compare the facial image in the image set with the preset face image to obtain the permission verification result, and quickly verify before each electronic file control The security of electronic file control. On the other hand, when the permission verification result is passed, the action recognition is performed on the action frame set through the trained AU detection network, and the target action is obtained, and the instruction corresponding to the target action is determined. As the target instruction, the target instruction is executed on the electronic file. This method of simultaneously performing authorization check and action confirmation on the acquired frame image realizes the rapid control of the electronic file and improves the efficiency of the electronic file control.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

Figure 2 is a flowchart of an embodiment of the electronic file control method of the present application;

Fig. 3 is a schematic structural diagram of an embodiment of an electronic file control device according to the present application;

Fig. 4 is a schematic structural diagram of an embodiment of a computer device according to the present application.

The best mode of the present invention

Please refer to FIG. 2. FIG. 2 shows an electronic file control method provided by an embodiment of the present application. The method is applied to the server in FIG. 1 as an example for description. The details are as follows:

S201: Collect real-time video streams.

Specifically, when an operation on an electronic file with relatively high security requirements is detected, the camera device is automatically turned on for data collection, and the real-time video stream of the current user is obtained.

For example, in a specific embodiment, a request to view an electronic contract document is detected. At this time, the camera of the electronic device is activated, the user is prompted on the electronic device to open the electronic contract document, and the real-time video of the preset time is taken. flow.

S202: Obtain an image set containing the facial image of the current user from the real-time video stream, and obtain an action frame set from the real-time video stream according to a preset manner.

Specifically, from the real-time video stream, an image collection containing the facial images of the current user is obtained, so that the facial images in the image collection can be used to determine whether the user has the authority to operate the electronic contract. A collection of action video frames is obtained from the real-time video stream.

Among them, the image set contains one or more facial images of the current user, and the basic image set containing the facial images of the current user from the real-time video stream can be specifically obtained by acquiring multiple video frame images from the real-time video stream, and then for each Face detection is performed on two video frame images, and the video frame image containing the complete face is taken as a facial image of the current user, and an image set containing the facial image of the current user is obtained.

Among them, the action frame set refers to a set containing multiple video frame images with a time sequence.

S203: Compare the facial image in the image collection with the preset facial image to obtain a permission verification result.

Specifically, in this embodiment, the facial image with the electronic file operation authority is pre-stored in the terminal. After the image collection is obtained in step S202, the facial image in the image collection is first combined with the facial image preset by the terminal. Make a comparison to determine whether the facial image in the image collection is one of the facial images preset by the terminal. When the facial image in the image collection matches any facial image preset by the terminal successfully, it is determined that the current user has the operation of an electronic file Authority, at this time, the authority verification result is the verification passed.

Understandably, if the facial image in the image collection fails to match with the facial image preset by the terminal, it means that the current user does not have the operation authority of the electronic file. At this time, the operation record is rejected and the current image is collected. Save it and generate related logs.

Among them, the methods of face matching include but are not limited to: face matching algorithm based on Gabor engine, local feature analysis method (Local Face Analysis), geometric feature-based methods and specific face subspace (FSS) algorithms, etc.

S204: If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action.

Specifically, after the permission verification result is that the verification is passed, confirm that the current user has the operation permission of the electronic file, and perform action recognition on the action frame set obtained in step S20 through the trained AU detection network, and obtain the action frame set corresponding The target action.

Among them, the AU detection network is a convolutional neural network model for AU detection. In this embodiment, a pre-trained AU detection network is used to perform action recognition on the action frame set, which is beneficial to improve the speed of action recognition. Ensure the smoothness of electronic file operations.

Among them, AU is the action unit (Facial Action Unit) abbreviation, AU refers to the facial expression unit used to express the actor's facial expression.

Among them, convolutional neural network (Convolutional Neural Network, CNN) is a feed-forward neural network. Its artificial neurons can respond to a part of the surrounding units in the coverage area, and can quickly and efficiently perform image processing. In this embodiment, The pre-trained convolutional neural network can quickly identify the AU contained in the basic image.

It should be noted that when the permission verification result is that the verification is not passed, return to step S201, perform video capture again, and re-verify, and when the number of consecutive failed verifications reaches the preset number of times, the file is locked. Among them, the preset times can be set according to actual needs.

S205: Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction.

Specifically, the terminal is preset with a preset instruction set and a preset number of AU actions. Each preset AU action corresponds to a preset instruction in the preset instruction set. After the target action is obtained, the preset instruction corresponding to the target action is obtained. Set the instruction, and use the preset instruction as the target instruction.

Among them, the preset AU action in this embodiment specifically refers to the actions used to perform electronic file operations, such as: shaking the head left, shaking the head right, blinking, etc., each preset AU action corresponds to the operation of a preset electronic file Instructions, such as: shaking your head left corresponds to the left page instruction of the contract, shaking your head right corresponds to the right page turning instruction, and blinking corresponds to the pause instruction, etc., which can be set according to actual needs, and there are no restrictions here.

S206: Execute the target instruction on the electronic file.

Specifically, the target instruction is executed on the electronic file, so that the electronic file performs a corresponding operation according to the target instruction.

In this embodiment, by collecting a real-time video stream, an image collection containing facial images of the current user is obtained from the real-time video stream, and an action frame collection is obtained from the real-time video stream according to a preset method, and then the image collection Compare the facial image in the file with the preset facial image to obtain the permission verification result. Before each electronic file control, the security of the electronic file control is quickly verified. At the same time, the permission verification result is the verification result. When passing, through the trained AU detection network, action recognition is performed on the action frame set, the target action is obtained, and the instruction corresponding to the target action is determined as the target instruction, and then the target instruction is executed on the electronic file. This will obtain the frame The way that the image performs authorization verification and action confirmation at the same time realizes the rapid control of electronic files and improves the efficiency of electronic file control.

Fig. 3 shows a principle block diagram of an electronic file control device corresponding to the electronic file control method of the above-mentioned embodiment one-to-one. As shown in FIG. 3, the electronic file control device includes a data acquisition module 31, an image acquisition module 32, an authorization verification module 33, an action detection module 34, an instruction determination module 35 and a file control module 36. The detailed description of each functional module is as follows:

The data collection module 10 is used to collect real-time video streams;

The image acquisition module 20 is configured to acquire an image collection containing the facial image of the current user from the real-time video stream, and acquire the action frame collection from the real-time video stream according to a preset method;

The authorization verification module 30 is used to compare the facial image in the image collection with the preset facial image to obtain the authorization verification result;

The action detection module 40 is configured to perform action recognition on the action frame set through the trained AU detection network if the permission verification result is that the verification is passed, to obtain the target action;

The instruction determining module 50 is used to obtain the instruction corresponding to the target action from the preset instruction set as the target instruction;

The file control module 60 is used to execute target instructions on the electronic file.

In order to solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 4 for details. FIG. 4 is a block diagram of the basic structure of the computer device in this embodiment.

The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are connected to each other in communication via a system bus. It should be pointed out that the figure only shows the computer device 4 with the components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the shown components, and alternative implementations can be made. More or fewer components. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA), Digital Signal Processor (DSP), embedded devices, etc.

The computer device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.

This application also provides another implementation manner, that is, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and the computer-readable storage medium stores information for interface Displayed computer-readable instructions, the computer-readable instructions for interface display can be executed by at least one processor, so that the at least one processor executes the steps of the above-mentioned electronic file control method.

Embodiments of the present invention

Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the technical field of the application; the terms used in the specification of the application herein are only for describing specific embodiments. The purpose is not to limit the application; the terms "including" and "having" in the specification and claims of the application and the above-mentioned description of the drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

Please refer to FIG. 1. As shown in FIG. 1, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.

The user can use the terminal devices 101, 102, and 103 to interact with the server 105 through the network 104 to receive or send messages and so on.

The terminal devices 101, 102, 103 can be various electronic devices with display screens and support for web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture E interface displays perts Group Audio Layer III, dynamic image expert compresses standard audio layer 3), MP4 (Moving Picture E interface displays perts Group Audio Layer IV, the dynamic image expert compresses the standard audio layer 4) Players, laptops and desktop computers, etc.

The server 105 may be a server that provides various services, for example, a background server that provides support for pages displayed on the terminal devices 101, 102, and 103.

It should be noted that the electronic file control method provided in the embodiments of the present application is executed by the server, and correspondingly, the electronic file control device is set in the server.

It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. According to implementation needs, there may be any number of terminal devices, networks, and servers. The terminal devices 101, 102, and 103 in the embodiments of the present application may specifically correspond to application systems in actual production.

Please refer to FIG. 2. FIG. 2 shows an electronic file control method provided by an embodiment of the present application. The method is applied to the server in FIG. 1 as an example for description. The details are as follows:

S201: Collect real-time video streams.

Specifically, when an operation on an electronic file with relatively high security requirements is detected, the camera device is automatically turned on for data collection, and the real-time video stream of the current user is obtained.

For example, in a specific embodiment, a request to view an electronic contract document is detected. At this time, the camera of the electronic device is activated, the user is prompted on the electronic device to open the electronic contract document, and the real-time video of the preset time is taken. flow.

S202: Obtain an image set containing the facial image of the current user from the real-time video stream, and obtain an action frame set from the real-time video stream according to a preset manner.

Specifically, from the real-time video stream, an image collection containing the facial images of the current user is obtained, so that the facial images in the image collection can be used to determine whether the user has the authority to operate the electronic contract. A collection of action video frames is obtained from the real-time video stream.

Among them, the image set contains one or more facial images of the current user, and the basic image set containing the facial images of the current user from the real-time video stream can be specifically obtained by acquiring multiple video frame images from the real-time video stream, and then for each Face detection is performed on two video frame images, and the video frame image containing the complete face is taken as a facial image of the current user, and an image set containing the facial image of the current user is obtained.

Among them, the action frame set refers to a set containing multiple video frame images with a time sequence.

S203: Compare the facial image in the image collection with the preset facial image to obtain a permission verification result.

Specifically, in this embodiment, the facial image with the electronic file operation authority is pre-stored in the terminal. After the image collection is obtained in step S202, the facial image in the image collection is first combined with the facial image preset by the terminal. Make a comparison to determine whether the facial image in the image collection is one of the facial images preset by the terminal. When the facial image in the image collection matches any facial image preset by the terminal successfully, it is determined that the current user has the operation of an electronic file Authority, at this time, the authority verification result is the verification passed.

Understandably, if the facial image in the image collection fails to match with the facial image preset by the terminal, it means that the current user does not have the operation authority of the electronic file. At this time, the operation record is rejected and the current image is collected. Save it and generate related logs.

Among them, the methods of face matching include but are not limited to: face matching algorithm based on Gabor engine, local feature analysis method (Local Face Analysis), geometric feature-based methods and specific face subspace (FSS) algorithms, etc.

S204: If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action.

Specifically, after the permission verification result is that the verification is passed, confirm that the current user has the operation permission of the electronic file, and perform action recognition on the action frame set obtained in step S20 through the trained AU detection network, and obtain the action frame set corresponding The target action.

Among them, the AU detection network is a convolutional neural network model for AU detection. In this embodiment, a pre-trained AU detection network is used to perform action recognition on the action frame set, which is beneficial to improve the speed of action recognition. Ensure the smoothness of electronic file operations.

Among them, AU is the action unit (Facial Action Unit) abbreviation, AU refers to the expression unit used to express the actor's facial expression on the actor's face.

Among them, convolutional neural network (Convolutional Neural Network, CNN) is a feed-forward neural network. Its artificial neurons can respond to a part of the surrounding units in the coverage area, and can quickly and efficiently perform image processing. In this embodiment, The pre-trained convolutional neural network can quickly identify the AU contained in the basic image.

It should be noted that when the permission verification result is that the verification is not passed, return to step S201, perform video capture again, and re-verify, and when the number of consecutive failed verifications reaches the preset number of times, the file is locked. Among them, the preset times can be set according to actual needs.

S205: Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction.

Specifically, the terminal is preset with a preset instruction set and a preset number of AU actions. Each preset AU action corresponds to a preset instruction in the preset instruction set. After the target action is obtained, the preset instruction corresponding to the target action is obtained. Set the instruction and use the preset instruction as the target instruction.

Among them, the preset AU action in this embodiment specifically refers to the actions used to perform electronic file operations, such as: shaking the head left, shaking the head right, blinking, etc., each preset AU action corresponds to the operation of a preset electronic file Instructions, such as: shaking your head left corresponds to the left page instruction of the contract, shaking your head right corresponds to the right page turning instruction, and blinking corresponds to the pause instruction, etc., which can be set according to actual needs, and there are no restrictions here.

S206: Execute the target instruction on the electronic file.

Specifically, the target instruction is executed on the electronic file, so that the electronic file performs a corresponding operation according to the target instruction.

In this embodiment, by collecting a real-time video stream, an image collection containing facial images of the current user is obtained from the real-time video stream, and an action frame collection is obtained from the real-time video stream according to a preset method, and then the image collection Compare the facial image in the file with the preset facial image to obtain the permission verification result. Before each electronic file control, the security of the electronic file control is quickly verified. At the same time, the permission verification result is the verification result. When passing, through the trained AU detection network, action recognition is performed on the action frame set, the target action is obtained, and the instruction corresponding to the target action is determined as the target instruction, and then the target instruction is executed on the electronic file. This will obtain the frame The way that the image performs authorization verification and action confirmation at the same time realizes the rapid control of electronic files and improves the efficiency of electronic file control.

In some optional implementation manners of this embodiment, in step S202, obtaining an image set containing the facial image of the current user from the real-time video stream includes:

Obtain frame images from the real-time video stream according to a preset time interval, and obtain a frame image set containing a preset number of frame images;

Use face detection algorithm to perform face detection on the frame images in the frame image collection to obtain the detection result;

In the detection result, each frame image containing complete facial features is taken as a facial image, and an image set containing at least one facial image is obtained.

Specifically, the terminal extracts video frames from the received real-time video stream at a preset time interval to obtain a frame image set containing multiple frame images, and then uses face detection technology to determine Face detection is performed on each frame image. In this embodiment, the frame image in the real-time video stream is obtained for the purpose of verifying the operation authority of the electronic file. Therefore, the face detection is mainly to detect whether the frame image includes clearness A complete face image. Therefore, the detection result includes two situations: including complete facial features and not including complete facial features. The frame image containing complete facial features is used as the current user’s facial image. The user's facial image can be one or multiple.

For example, in a specific embodiment, the preset time interval is the time corresponding to 8 continuous video frames. After receiving the real-time video stream sent by the client, the terminal will obtain the real-time video containing 128 continuous video frames. In the stream, 8 video frames are regarded as a video frame set, and the last video frame image in each video frame set is obtained, a total of 128/8=16 video frame images are obtained, and these 16 frame images are regarded as frame images set.

Among them, the face detection technology is mainly used to detect whether the facial features, contours, and important facial arcs in the frame image are clear and complete. The specific implementation methods include but not limited to: Adaboost algorithm, facial feature point recognition algorithm (Feature-based recognition algorithms), recognition algorithms using neural network (Recognition algorithms using neural network), and eigenface algorithms based on Principal Component Analysis (PCA).

It is worth noting that there is at least one facial image of the current user acquired. If the facial image of the current user is not acquired, it is considered that the current user's state acquisition is abnormal. At this time, the corresponding prompt information is sent to the display interface of the electronic device. And reacquire the real-time video stream.

In this embodiment, frame images are obtained from the real-time video stream at a preset time interval to obtain a frame image set containing a preset number of frame images, and then a face detection algorithm is used to determine the frames in the frame image set. The image is subjected to face detection, and the detection result is obtained. Each frame image of the complete face feature in the detection result is regarded as a facial image, and an image set containing at least one facial image is obtained, so as to realize fast authorization through the collected video stream At the same time, it avoids the risk of leakage and cracking that may exist in traditional password verification, and improves the security and efficiency of authorization verification.

In some optional implementation manners of this embodiment, in step S202, acquiring an action frame set from a real-time video stream according to a preset manner includes:

Obtain frame images from the real-time video stream according to a preset time interval, and obtain a frame image set containing a preset number of frame images;

Sort the frame images according to the sequence of appearance of each frame image in the frame image collection in the real-time video stream to obtain the sorted frame image sequence;

Annotate each frame image in the frame image sequence to obtain the action frame set.

Specifically, the terminal extracts video frames from the received real-time video stream at a preset time interval to obtain a frame image set containing multiple frame images, and sorts and labels the frame images according to the timing relationship corresponding to the frame images , Get a set of action frames with annotation information.

Among them, labeling refers to assigning a sequential mark to each frame image. The association between the images can be determined according to the labeling. The labeling can be numbers, letters, or a combination of numbers and letters, etc., which can be set according to actual needs. No restrictions.

In this embodiment, frame images are obtained from the real-time video stream at a preset time interval to obtain a frame image set containing a preset number of frame images, and then according to each frame image in the frame image set in the real-time video stream Sort the frame images in the order of appearance in the sequence to obtain the sorted frame image sequence, and mark each frame image in the frame image sequence to obtain the action frame set, so that these labeled action frames can be subsequently passed Collecting and determining the change content of the frame image, and using the change content for action recognition can not only improve the accuracy of action recognition, but also improve the efficiency of action recognition.

In some optional implementations of this embodiment, in step S204, if the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network, and the target action is obtained including:

If the permission verification result is that the verification is passed, input each frame image in the acquired action frame set into the trained AU detection network;

Calculate the pixel difference between adjacent frames in sequence according to the order of action frame labeling, and obtain the difference content between adjacent frames;

Through the convolutional layer of the trained AU detection network, feature extraction is performed on the difference content in turn to obtain the corresponding feature change content;

Each feature change content is input to the AU action recognition layer, and the feature change content is classified and recognized according to the AU action recognition layer to obtain the target action.

Specifically, when the permission verification is passed, each frame image in the acquired action frame set is used as input data, and input into the trained AU detection network, and the previous frame image is determined according to the label of the frame image The difference content with the next frame image is then extracted through the AU detection network to extract the characteristic information of the difference content, and the characteristic information is passed to the AU action recognition layer for action recognition.

It should be noted that, in this embodiment, the action recognition is performed by changing the content of the feature, compared with the traditional method to directly recognize the frame image, reducing a large number of features, which can effectively improve the recognition efficiency, and at the same time, the changed content can more reflect the action. Some of the processes help to improve the accuracy of action recognition.

Among them, to determine the difference content between the previous frame image and the next frame image, specific methods that can be used include, but are not limited to: optical flow detection, frame difference, edge detection, and motion vector detection.

Preferably, this embodiment adopts the frame difference method to determine the difference content.

The feature change content is classified and recognized through the AU action recognition layer, and the specific process of obtaining the target action can be referred to the description of the subsequent embodiments. To avoid repetition, details are not described here.

In this embodiment, when the permission check result is that the check is passed, each frame image in the acquired action frame set is input into the trained AU detection network and the difference content between adjacent frames is obtained. Then through the convolutional layer of the trained AU detection network, feature extraction is performed on the difference content in turn to obtain the corresponding feature change content, and then each feature change content is input to the AU action recognition layer, and the features are compared according to the AU action recognition layer The change content is classified and identified, and the target action is obtained. Conducive to improving the efficiency and accuracy of action recognition.

In some optional implementations of this embodiment, each feature change content is input to the AU action recognition layer, and the feature change content is classified and recognized according to the AU action recognition layer, and the target action obtained includes:

Use the n AU classifiers of the AU action recognition layer to calculate the similarity of the feature change content, and obtain the probability that the feature change content belongs to the action category corresponding to the AU classifier. A total of n probabilities are obtained, where each AU classifier corresponds to An action category;

From n probabilities, the action category with the highest probability is selected as the target action corresponding to the feature change content.

Specifically, the AU detection network model includes but is not limited to: input layer, convolutional layer and AU action recognition layer, etc. There are n trained AU classifiers in the AU action recognition layer, and each AU classifier is changed with the feature The content similarity calculation is performed to obtain the probability that the feature change content belongs to the action category corresponding to the AU classifier, and a total of n probabilities are obtained. From the n probabilities, the action category with the highest probability is selected as the target action corresponding to the feature change content.

Among them, the behavior category corresponding to the AU classifier can be trained according to actual needs, such as page up, page down, automatic page turning, zoom in, and pause. The number n of AU classifiers can also be set as required, and there is no specific limitation here. For example, n is set to 14.

Among them, the realization method of AU classifier includes but not limited to: Logistic regression (Logistic Regression, LR), support vector machine (Support Vector Machine, SVM), cross entropy (Corss Entropy) and softmax regression, etc.

Preferably, the embodiment of the present application adopts softmax regression to realize the classification and recognition of multiple AU classifiers.

In this embodiment, by using n AU classifiers of the AU action recognition layer to calculate the similarity of the feature change content, the probability that the feature change content belongs to the action category corresponding to the AU classifier is obtained, and the action category with the highest probability is selected as The target action corresponding to the feature change content improves the recognition accuracy of the target action.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

Fig. 3 shows a principle block diagram of an electronic file control device corresponding to the electronic file control method of the above-mentioned embodiment one-to-one. As shown in FIG. 3, the electronic file control device includes a data acquisition module 31, an image acquisition module 32, an authorization verification module 33, an action detection module 34, an instruction determination module 35 and a file control module 36. The detailed description of each functional module is as follows:

The data collection module 10 is used to collect real-time video streams;

The image acquisition module 20 is configured to acquire an image collection containing the facial image of the current user from the real-time video stream, and acquire the action frame collection from the real-time video stream according to a preset method;

The authorization verification module 30 is used to compare the facial image in the image collection with the preset facial image to obtain the authorization verification result;

The action detection module 40 is configured to perform action recognition on the action frame set through the trained AU detection network if the permission verification result is that the verification is passed, to obtain the target action;

The instruction determining module 50 is used to obtain the instruction corresponding to the target action from the preset instruction set as the target instruction;

The file control module 60 is used to execute target instructions on the electronic file.

Further, the image acquisition module includes:

The frame image selection unit is used to obtain frame images from the real-time video stream according to a preset time interval, and obtain a frame image set containing a preset number of frame images;

The face detection unit is used to use a face detection algorithm to perform face detection on the frame images in the frame image set to obtain the detection result;

The facial image determining unit is used to treat each frame image containing complete facial features in the detection result as a facial image to obtain an image set containing at least one facial image.

Further, the image acquisition module also includes:

The frame image selection unit obtains frame images from the real-time video stream according to a preset time interval, and obtains a frame image set containing a preset number of frame images;

The image sorting unit is used to sort the frame images according to the sequence of appearance of each frame image in the frame image collection in the real-time video stream to obtain a sequence of frame images after sorting;

The image tagging unit is used to tag each frame image in the frame image sequence to obtain an action frame set.

Further, the motion detection module includes:

The data input unit is used for inputting each frame image in the acquired action frame set into the trained AU detection network if the permission verification result is that the verification is passed;

The difference content obtaining unit is used to sequentially calculate the pixel difference between adjacent frames according to the order in which the action frames are marked to obtain the difference content between adjacent frames;

The difference feature extraction unit is used to detect the convolutional layer of the network through the trained AU, and then perform feature extraction on the difference content in turn to obtain the corresponding feature change content;

The action recognition unit is used to input each feature change content to the AU action recognition layer, and classify and recognize the feature change content according to the AU action recognition layer to obtain the target action.

Further, the difference content obtaining unit includes:

The frame difference method calculation subunit is used to calculate the gray level difference between adjacent frames by using the frame difference method to obtain the difference content.

Further, the action recognition unit includes:

The probability calculation subunit is used to use n AU classifiers of the AU action recognition layer to calculate the similarity of the feature change content to obtain the probability that the feature change content belongs to the action category corresponding to the AU classifier, and obtain a total of n probabilities, where , Each AU classifier corresponds to an action category;

The target action determination subunit is used to select the action category with the highest probability from n probabilities as the target action corresponding to the feature change content.

For the specific limitation of the control device of the electronic file, please refer to the above limitation of the control method of the electronic file, which will not be repeated here. Each module in the above electronic file control device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In order to solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 4 for details. FIG. 4 is a block diagram of the basic structure of the computer device in this embodiment.

The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are connected to each other in communication via a system bus. It should be pointed out that the figure only shows the computer device 4 with the components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the shown components, and alternative implementations can be made. More or fewer components. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA), Digital Signal Processor (DSP), embedded devices, etc.

The computer device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.

The memory 41 includes at least one type of readable storage medium, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM) , Static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disks, optical disks, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), and a secure digital device equipped on the computer device 4. (Secure Digital, SD) card, flash memory card (Flash Card) and so on. Of course, the memory 41 may also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed in the computer device 4, such as computer-readable instructions for controlling electronic files. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

The processor 42 may be a central processing unit (Central Processing Unit) in some embodiments. Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to run computer-readable instructions or process data stored in the memory 41, for example, run computer-readable instructions for controlling electronic files.

The network interface 43 may include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

This application also provides another implementation manner, that is, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and the computer-readable storage medium stores information for interface Displayed computer-readable instructions, the computer-readable instructions for interface display can be executed by at least one processor, so that the at least one processor executes the steps of the above-mentioned electronic file control method.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the method described in each embodiment of the present application.

Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The drawings show preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these examples is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it is still possible for those skilled in the art to modify the technical solutions described in each of the foregoing specific embodiments, or equivalently replace some of the technical features. . All equivalent structures made by using the contents of the description and drawings of this application, directly or indirectly used in other related technical fields, are similarly within the scope of patent protection of this application.

Claims (20)

An electronic file control method, wherein the electronic file control method includes: Collect real-time video stream; From the real-time video stream, obtain an image set containing the facial image of the current user, and obtain an action frame set from the real-time video stream according to a preset method; Comparing the facial images in the image collection with the preset facial images to obtain a permission verification result; If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action; Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction; The target instruction is executed on the electronic file. 5. The method for controlling electronic files according to claim 1, wherein said acquiring from said real-time video stream an image set containing the facial image of the current user comprises: Obtaining frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; Using a face detection algorithm to perform face detection on the frame images in the frame image set to obtain a detection result; In the detection result, each frame image containing complete facial features is used as one facial image, and an image set containing at least one facial image is obtained. 5. The method for controlling electronic files according to claim 1, wherein said acquiring a set of action frames from said real-time video stream according to a preset method comprises: Obtaining frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; Sorting the frame images according to the sequence of appearance of each frame image in the frame image set in the real-time video stream to obtain a sequence of frame images after sorting; Annotate each frame image in the frame image sequence to obtain the action frame set. The electronic document control method according to claim 3, wherein if the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target Actions include: If the permission verification result is that the verification is passed, input each frame image in the acquired action frame set into the trained AU detection network; Calculate the pixel difference between adjacent frames in sequence according to the order of action frame labeling, and obtain the difference content between adjacent frames; Through the convolutional layer of the trained AU detection network, feature extraction is performed on the difference content in sequence to obtain corresponding feature change content; Input each feature change content to the AU action recognition layer, and classify and recognize the feature change content according to the AU action recognition layer to obtain a target action. 5. The electronic document control method according to claim 4, wherein the step of sequentially calculating the pixel difference between adjacent frames according to the order in which the action frames are marked to obtain the difference content between the adjacent frames comprises: The frame difference method is used to calculate the gray scale difference between adjacent frames to obtain the difference content. The electronic document control method according to claim 4, wherein the AU action recognition layer includes n preset AU classifiers, wherein n is a positive integer greater than 1, and the AU action recognition layer The classification and recognition of the feature change content to obtain the target action includes: Use the n AU classifiers of the AU action recognition layer to calculate the similarity of the feature change content to obtain the probability that the feature change content belongs to the action category corresponding to the AU classifier, and obtain a total of n of the probabilities, Among them, each AU classifier corresponds to an action category; From the n probabilities, the action category with the highest probability is selected as the target action corresponding to the feature change content. An electronic file control device, wherein the electronic file control device includes: Data collection module, used to collect real-time video stream; An image acquisition module, configured to acquire an image collection containing facial images of the current user from the real-time video stream, and acquire an action frame collection from the real-time video stream according to a preset method; The authorization verification module is used to compare the facial image in the image collection with the preset facial image to obtain the authorization verification result; An action detection module, configured to perform action recognition on the set of action frames through the trained AU detection network if the permission verification result is a pass, to obtain a target action; The instruction determining module is used to obtain the instruction corresponding to the target action from a preset instruction set as the target instruction; The file control module is used to execute the target instruction on the electronic file. 8. The electronic document control device according to claim 7, wherein the image acquisition module comprises: The frame image selection unit is configured to obtain frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; The face detection unit is configured to use a face detection algorithm to perform face detection on the frame images in the frame image set to obtain a detection result; The facial image determining unit is configured to use, in the detection result, each of the frame images containing complete facial features as one of the facial images, to obtain an image set containing at least one of the facial images. A computer device includes a memory, a processor, and computer readable instructions stored in the memory and capable of running on the processor, wherein the processor implements the following electronic file when the computer readable instruction is executed Control method: Collect real-time video stream; From the real-time video stream, obtain an image set containing the facial image of the current user, and obtain an action frame set from the real-time video stream according to a preset method; Comparing the facial images in the image collection with the preset facial images to obtain a permission verification result; If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action; Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction; The target instruction is executed on the electronic file. 9. The computer device according to claim 9, wherein said acquiring, from said real-time video stream, an image collection containing a facial image of the current user comprises: Obtaining frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; Using a face detection algorithm to perform face detection on the frame images in the frame image set to obtain a detection result; In the detection result, each frame image containing complete facial features is used as one facial image, and an image set containing at least one facial image is obtained. 9. The computer device according to claim 9, wherein said acquiring an action frame set from said real-time video stream in a preset manner comprises: Obtaining frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; Sorting the frame images according to the sequence of appearance of each frame image in the frame image set in the real-time video stream to obtain a sequence of frame images after sorting; Annotate each frame image in the frame image sequence to obtain the action frame set. The computer device according to claim 11, wherein, if the permission verification result is a pass, performing action recognition on the action frame set through the trained AU detection network to obtain the target action comprises: If the permission verification result is that the verification is passed, input each frame image in the acquired action frame set into the trained AU detection network; Calculate the pixel difference between adjacent frames in sequence according to the order of the action frame labeling, and obtain the difference content between adjacent frames; Through the convolutional layer of the trained AU detection network, feature extraction is performed on the difference content in sequence to obtain the corresponding feature change content; Input each feature change content to an AU action recognition layer, and classify and recognize the feature change content according to the AU action recognition layer to obtain a target action. 11. The computer device according to claim 12, wherein said sequentially calculating the pixel difference between adjacent frames in the order of labeling the action frames, and obtaining the content of the difference between adjacent frames comprises: The frame difference method is used to calculate the gray scale difference between adjacent frames to obtain the difference content. The computer device of claim 12, wherein the AU action recognition layer includes n preset AU classifiers, wherein n is a positive integer greater than 1, and the AU action recognition layer performs The feature change content is classified and identified, and the target actions obtained include: Use the n AU classifiers of the AU action recognition layer to calculate the similarity of the feature change content to obtain the probability that the feature change content belongs to the action category corresponding to the AU classifier, and obtain a total of n of the probabilities, Among them, each AU classifier corresponds to an action category; From the n probabilities, the action category with the highest probability is selected as the target action corresponding to the feature change content. A computer-readable storage medium, the computer-readable storage medium stores computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, the following electronic file control method is implemented: Collect real-time video stream; From the real-time video stream, obtain an image set containing the facial image of the current user, and obtain an action frame set from the real-time video stream according to a preset method; Comparing the facial images in the image collection with the preset facial images to obtain a permission verification result; If the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target action; Obtain the instruction corresponding to the target action from the preset instruction set as the target instruction; The target instruction is executed on the electronic file. 15. The computer-readable storage medium according to claim 15, wherein said acquiring, from said real-time video stream, an image collection containing a facial image of a current user comprises: Obtaining frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; Using a face detection algorithm to perform face detection on the frame images in the frame image set to obtain a detection result; In the detection result, each frame image containing complete facial features is used as one facial image, and an image set containing at least one facial image is obtained. 15. The computer-readable storage medium according to claim 15, wherein said acquiring a set of action frames from said real-time video stream in a preset manner comprises: Obtaining frame images from the real-time video stream according to a preset time interval to obtain a frame image set containing a preset number of frame images; Sorting the frame images according to the sequence of appearance of each frame image in the frame image set in the real-time video stream to obtain a sequence of frame images after sorting; Annotate each frame image in the frame image sequence to obtain the action frame set. The computer-readable storage medium according to claim 17, wherein, if the permission verification result is that the verification is passed, the action recognition is performed on the action frame set through the trained AU detection network to obtain the target Actions include: If the permission verification result is that the verification is passed, input each frame image in the acquired action frame set into the trained AU detection network; Calculate the pixel difference between adjacent frames in sequence according to the order of the action frame labeling, and obtain the difference content between adjacent frames; Through the convolutional layer of the trained AU detection network, feature extraction is performed on the difference content in sequence to obtain the corresponding feature change content; Input each feature change content to an AU action recognition layer, and classify and recognize the feature change content according to the AU action recognition layer to obtain a target action. 18. The computer-readable storage medium according to claim 18, wherein said sequentially calculating the pixel difference between adjacent frames according to the order of action frame labeling, and obtaining the difference content between adjacent frames comprises: The frame difference method is used to calculate the gray scale difference between adjacent frames to obtain the difference content. The computer-readable storage medium of claim 18, wherein the AU action recognition layer includes n preset AU classifiers, wherein n is a positive integer greater than 1, and the AU action recognition layer The classification and recognition of the feature change content to obtain the target action includes: Use the n AU classifiers of the AU action recognition layer to calculate the similarity of the feature change content to obtain the probability that the feature change content belongs to the action category corresponding to the AU classifier, and obtain a total of n of the probabilities, Among them, each AU classifier corresponds to an action category; From the n probabilities, the action category with the highest probability is selected as the target action corresponding to the feature change content.