CN107194968B - Image recognition and tracking method, device, smart terminal and readable storage medium - Google Patents

Abstract

The invention discloses an image identification tracking method, an image identification tracking device, an intelligent terminal and a readable storage medium. The method comprises the following steps: obtaining a recognition result of a mark pattern in the image; positioning a mark pattern in the image according to the identification result of the mark pattern, and carrying out target tracking by the positioned mark pattern to obtain translation information of the mark pattern in the image in space; and fusing the translation information with a plurality of sensors in the intelligent terminal to output rotation angles to form a pose matrix of the mark pattern in the image. Under the effect of multi-sensor fusion in the intelligent terminal, the rotation angle is obtained without adopting the process of target tracking, the time performance is improved as a whole, the limitation that the rotation angle is inaccurate and even cannot be calculated when the rotation angle is obtained by adopting the process of target tracking is avoided, and the combination of the target tracking process and the multi-sensor fusion ensures the fast tracking speed, has stronger stability and accuracy and can give consideration to the tracking effect and the time performance.

Description

Image recognition and tracking method, device, smart terminal and readable storage medium

Technical field

The invention relates to the field of Internet application technology, and in particular to an image recognition and tracking method, device, intelligent terminal and readable storage medium.

Background technique

With the rapid development of Internet application technology, smart terminals estimate the posture corresponding to the image by tracking the target of the obtained image, and implement various interactive applications based on the obtained image based on the obtained posture. The posture corresponding to this image is used to describe the translation and rotation of the entity target corresponding to the captured image in physical space.

Existing target tracking is based on feature point tracking. Generally speaking, the greater the number of tracked feature points, the more complex the descriptors of the feature points, the better the tracking effect, and the more accurate the estimated posture. However, the slower the running speed, that is, there is a time-consuming process for existing target tracking. Performance and tracking effects are contradictory limitations.

Tracking based solely on feature points will inevitably bring about trade-offs between time performance and tracking effects. For existing target tracking applications on smart terminals, in order to obtain better time performance and limit processing resources, simple feature points are mostly used to obtain faster feature point extraction speed and tracking speed. However, the tracking accuracy However, it is very low and cannot achieve both tracking effect and time performance for target tracking on smart terminals.

Contents of the invention

In order to solve the technical problem in the related art that the implementation of target tracking in images cannot take into account the tracking effect and time performance, one object of the present invention is to provide an image recognition and tracking method and device to solve the problems existing in the existing technology. The defect of tracking effect and time performance cannot be guaranteed at the same time.

An image recognition and tracking method, the method includes:

Obtain the recognition results of the mark pattern in the image captured by the smart terminal;

Locate the marking pattern in the image according to the recognition result of the marking pattern, and perform target tracking based on the positioned marking pattern to obtain the spatial translation information of the marking pattern in the image;

The translation information and the rotation angle output by fusion of multiple sensors in the smart terminal form a pose matrix of the marking pattern in the image.

An image recognition and tracking device, characterized in that the device includes:

The recognition result acquisition module is used to obtain the recognition result of the mark pattern in the image captured by the smart terminal;

A target tracking module, configured to locate the mark pattern in the image according to the recognition result of the mark pattern, and perform target tracking based on the positioned mark pattern to obtain the spatial translation information of the mark pattern in the image;

A pose acquisition module is configured to combine the translation information and the rotation angle output by fusion of multiple sensors in the smart terminal to form a pose matrix of the marking pattern in the image.

An intelligent terminal includes:

processor; and

A memory on which computer readable instructions are stored. When the computer readable instructions are executed by the processor, the image recognition and tracking method as described above is implemented.

A computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the image recognition and tracking method as described above is implemented.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

For the captured image, first obtain the recognition result of the mark pattern in the image captured by the smart terminal, then locate the mark pattern in the image based on the recognition result of the mark pattern, and perform target tracking based on the positioned mark pattern to obtain the spatial position of the mark pattern in the image. The translation information is finally fused with the rotation angle output by the multi-sensors in the smart terminal to form the pose matrix of the image mark pattern. With the multi-sensor fusion in the smart terminal, it is no longer necessary to use the target tracking process to obtain the rotation. Angle, improves the time performance as a whole, and also avoids the limitation that the rotation angle is very inaccurate or even impossible to calculate when obtaining the rotation angle using the target tracking process. The combination of the target tracking process and multi-sensor fusion ensures fast While tracking speed, it also has strong stability and accuracy, and can take into account tracking effect and time performance.

It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present invention.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

Figure 1 is a flow chart of an image recognition and tracking method according to an exemplary embodiment;

Figure 2 is a flowchart describing details of the rotation angle of a smart terminal in space based on sensor data according to an exemplary embodiment;

FIG. 3 is a flowchart describing details of step 230 according to an exemplary embodiment;

Figure 4 is a flowchart describing the details of step 130 according to the corresponding embodiment shown in Figure 1;

FIG. 5 illustrates, according to an exemplary embodiment, performing transmission transformation preprocessing on the currently captured image relative to the image in which the mark pattern is first recognized to obtain a transmission image, and the transmission image is used to perform a target tracking step on the currently captured image. A flowchart describing the details;

Figure 6 is a flowchart describing the details of step 110 according to the corresponding embodiment of Figure 1;

Figure 7 is a framework diagram of a reality augmentation system in a smart terminal according to an exemplary embodiment;

Figure 8 is an implementation framework diagram of the image tracking step in the embodiment corresponding to Figure 7;

Figure 9 is a block diagram of an image recognition and tracking device according to an exemplary embodiment;

Figure 10 is a block diagram describing details of the target tracking module according to the embodiment corresponding to Figure 9;

Figure 11 is a block diagram of an image recognition and tracking device according to another exemplary embodiment;

Figure 12 is a block diagram illustrating details of a transmission transformation module according to an exemplary embodiment;

Figure 13 is a block diagram of a device according to an exemplary embodiment.

Detailed ways

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the appended claims.

FIG. 1 is a flow chart of an image recognition and tracking method according to an exemplary embodiment. In an exemplary embodiment, the image recognition and tracking method, as shown in Figure 1, may include the following steps.

In step 110, the recognition result of the mark pattern in the image captured by the smart terminal is obtained.

Among them, the smart terminal is used to perform the image recognition and tracking process of the present invention. Here, the smart terminal will first obtain the image it captured. It should be noted here that the image capturing process of the smart terminal may be currently executed or may be executed in advance. Through the image recognition and tracking process of the present invention, the recognition and tracking process is performed for images captured instantly or pre-captured by the smart terminal.

Of course, it can be understood that the smart terminal that performs image capture and the smart terminal that currently performs the recognition and tracking process on the captured image can be the same smart terminal, or they can be two different smart terminals. For different smart terminals, the smart terminal that performs image capture will need to transfer the captured image and corresponding other data, such as the sensor data mentioned later, to the smart terminal that performs the identification and tracking process.

In an exemplary embodiment, the image captured by the smart terminal is a shooting component configured by the smart terminal, for example, various built-in or external camera implementations. Thus, the captured image is a real scene image.

Specifically, as the smart terminal captures the real environment, the images captured by the smart terminal are used to reflect the real scene in the form of images. It can be understood that real scenes are scenes in real environments. That is to say, the captured image is captured by the smart terminal carried by the user against the real environment. Of course, it should be further explained that it can be captured currently or obtained in advance. This is not limited.

The marker pattern is a pre-specified pattern. As for the image, the marker pattern exists in the image by photographing markers laid out in the real environment.

The recognition result of the mark pattern in the image is used to indicate whether the mark pattern exists in the image and the position of the mark pattern in the image. Specifically, for the case where the mark pattern exists in the image, the obtained recognition result indicates that the mark pattern is recognized. , at this point, target tracking can be performed on this image.

It should be noted here that the recognition result of the mark pattern in the image can be output by matching the image with the preset mark image, or can be output after the recognition of the mark image in the image is completed in other ways. Here There is no limitation, as long as the recognition result of the mark pattern in the image can be obtained from the output information.

In a specific implementation of an exemplary embodiment, the recognition result of the mark pattern in the image captured by the smart terminal is obtained by matching the image with the preset mark pattern or by a user-specified trigger of the mark pattern in the captured image.

Through this configuration of the method of obtaining the recognition result of the mark pattern in the captured image, multiple methods are provided for the image recognition and tracking process of the smart terminal, so that it can be flexibly adapted to various scenarios and the performance of the smart terminal, and improve the efficiency of image recognition and tracking. reliability.

In step 130, the marking pattern in the image is located according to the recognition result of the marking pattern, and target tracking is performed based on the positioned marking pattern to obtain spatial translation information of the marking pattern in the image.

Among them, image target tracking refers to performing a target tracking process on the image, and the target referred to here is the mark pattern in the image. Therefore, for image target tracking, it must be triggered by the recognition result of the mark pattern, so as to avoid invalid target tracking, waste of computing resources, and improve processing efficiency.

In the triggered target tracking of the image, it is a process of calculating the pose of the mark pattern in the image. In an exemplary embodiment of the present invention, the pose of the mark pattern obtained by target tracking will include the mark pattern in the image. Translation information of the pattern in space.

The space referred to corresponds to the space where the real scene is located, that is, the physical space, which is the three-dimensional space constructed by the smart terminal where it is located. Therefore, the translation information of the mark pattern in the image in space is relative to the three orientations in the space.

The three orientations in space are the directions pointed by the three coordinate axes in the constructed spatial coordinate system. Relative to the physical space, a spatial coordinate system is constructed. There is a conversion between the physical coordinate system and the constructed spatial coordinate system between physical space and space. Therefore, the captured real environment can be accurately mapped to the space. Get translation information in space.

The spatial coordinate system is a three-dimensional coordinate system, which includes mutually perpendicular x-coordinate axes, y-coordinate axes, and z-coordinate axes. The translation information of the marking pattern in the image in space is based on the x-coordinate axes, y-coordinate axes, and z-coordinate axes. And the translation distance obtained by the operation.

Specifically, the translation information of the marking pattern in the image in space is used to indicate the movement of the marking pattern in space. The translation information includes the translation distance and the vertical distance on the horizontal plane of space. The translation distance on the horizontal plane of space corresponds to the orientation of the two coordinate axes in space.

For target tracking in images, it is possible to quickly and stably predict the translation of the marking pattern in three directions in space, so the target tracking in the image will still be performed, but the calculation of the relevant rotation information of the marking pattern in the image will no longer be performed. As a result, the time performance and speed of target tracking will be greatly improved and improved.

In a specific implementation of an exemplary embodiment, the algorithm used to implement target tracking in the image can be a single target tracking algorithm, or can use continuous matching when the matching speed is fast enough, or can also be implemented using deep learning. , and even multi-target tracking algorithms, we will not list them one by one here.

In step 150, the translation information and the rotation angle output by fusion of multiple sensors in the smart terminal form a pose matrix of the marking pattern in the image.

The smart terminal refers to the terminal device that performs the image recognition and tracking process of the present invention. For example, the smart terminal can be a portable mobile terminal such as a smart phone or a tablet computer. Smart terminals are equipped with various sensors. Therefore, the sensor data output by multiple sensors in the smart terminal that performs image capture can be fused to obtain a rotation angle that reflects the rotation of the smart terminal. Since the smart terminal captures images and The installed sensors output sensor data. Therefore, the rotation angle output by multi-sensor fusion can be used to describe the occurrence of rotation in the pose corresponding to the marker pattern in the image.

Multi-sensor fusion in smart terminals can quickly and accurately calculate the rotation angle, and cooperate with the target tracking of the image to avoid the problem of inaccurate or even impossible calculation of the rotation angle in target tracking, thereby ensuring efficiency and accuracy.

After the translation information of the marking pattern in space is obtained through target tracking of the image, this translation information is combined with the rotation angle output by multi-sensor fusion to form a pose matrix of the marking pattern in the image.

The pose matrix that combines the translation information and the rotation angle to form the marking pattern in the image means that the movement of the marking pattern in the space indicated by the translation information and the rotation indicated by the rotation angle are formed together as elements respectively. The pose matrix of the marking pattern.

For example, the movement of the mark pattern in space indicated by the translation information can be reflected by the distance obtained respectively relative to the orientation of the three coordinate axes in the space; the rotation angle can also be represented by the distance corresponding to the three coordinate axes in the space. The rotation angle of the coordinate axis, therefore, the marking pattern pose matrix with six degrees of freedom can be obtained.

At this point, the distances corresponding to the three coordinate axes in the space and the rotation angles of the coordinate axes corresponding to the three coordinate axes are respectively used as elements in the matrix to form the pose matrix of the marking pattern.

The pose matrix of the marker pattern in the image is used to describe the pose changes of the marker pattern relative to its initial pose. The pose matrix obtained through calculation will make the subsequent business implementation scenario match the pose matrix, and then be adapted to the image and the marking pattern in the image.

For example, for the subsequently implemented augmented reality business scene, the virtual scene image is projected according to the calculated pose matrix, so that the virtual scene image projected to the image is adapted to its pose matrix, ensuring that The accuracy and adaptability of subsequent business implementation scenarios.

As described above, in the exemplary embodiments, with the help of image target tracking and multi-sensor fusion, the implementation of image recognition and tracking in smart terminals can complete operations at a very fast speed while maintaining good tracking accuracy. It can not only ensure the real-time performance of the smart terminal, that is, the mobile terminal, but also ensure that the tracking effect will not be affected, making the tracking of local real-time mark patterns on the mobile terminal a reality and with better stability.

In an exemplary embodiment, before step 350, the image recognition and tracking method may further include the following steps.

Obtain sensor data output by multiple sensors when the smart terminal captures images;

A multi-sensor fusion algorithm is executed on the sensor data to calculate the rotation angle of the smart terminal in space. The rotation angle is output by multi-sensor fusion and used to form the pose matrix of the marking pattern in the image.

Among them, as mentioned above, multi-sensor fusion is aimed at the data output by multiple sensors in the smart terminal, that is, sensor data. A multi-sensor fusion algorithm is executed in the sensor data to calculate the rotation angles of the smart terminal corresponding to three directions in space. The calculated rotation angles correspond to the rotation of the mark pattern in the image in space.

Further, FIG. 2 is a flowchart describing details of the rotation angle of a smart terminal in space based on sensor data according to an exemplary embodiment. This step, as shown in Figure 2, may include the following steps.

In step 210, sensor data output by multiple sensors when the smart terminal captures images is obtained.

Among them, in order to realize the pose prediction of the mark pattern in the image under multi-sensor fusion, the capture of this image is carried out simultaneously with the collection of sensor data in the smart terminal to ensure that the collected sensor data can correspond to the pose of the image captured by the smart terminal. , so that the calculation based on the sensor data is accurate relative to the image.

In addition, the data collection by multiple sensors in the smart terminal can also be performed while maintaining the posture when the smart terminal captures the image, or it can be performed before the smart terminal captures the image and maintain the posture to capture, which is not limited here. .

The multiple sensors in the smart terminal used to output sensor data when capturing real images refer to all sensors in the smart terminal that can be used to calculate the rotation angle of the smart terminal. In an exemplary embodiment, the number of sensors is three.

The sensor data is related to multiple sensors related to the rotation of the smart terminal. For example, the sensor data may be data output by an angular velocity meter, an accelerometer, and a gravity sensor meter in the smart terminal.

When the smart terminal captures an image, the sensor collects data to output the sensor data associated with the image.

In step 230, the sensor data is used to calculate the rotation angle of the smart terminal itself in space.

Among them, the rotation angle in space is calculated through sensor data, and the rotation angle relative to each orientation is obtained. It should be noted here that the orientation referred to is the direction of the coordinate axis of the three-dimensional coordinate system constructed in space.

Specifically, FIG. 3 is a flowchart describing the details of step 230 according to an exemplary embodiment. This step 230, as shown in Figure 3, may include the following steps.

In step 231, the angular velocity in the sensor data is integrated to obtain a rough rotation value of the smart terminal relative to each orientation in space.

Among them, the sensor data includes angular velocity, acceleration and gravity direction information. The angular velocity is collected by the angular velocity meter in the smart terminal, the acceleration is collected by the accelerometer, and the gravity direction information is collected by the gravity sensor.

As the image capture proceeds, sensor data is also obtained by multiple sensors specified in the smart terminal. To extract the angular velocity in the sensor data, first integrate the angular velocity in the sensor data to obtain the rough rotation value of the smart terminal relative to each orientation in space.

That is to say, during the integration process of angular velocity, the equipment error of the angular velocity meter is always accumulated. Therefore, the accurate value cannot be obtained, but only the rough value of the rotation relative to each orientation is obtained.

For example, it can be understood that there are three coordinate axis directions in space, that is, the three directions pointed by the x coordinate axis, y coordinate axis, and z coordinate axis of the coordinate system established in this space. These three directions are the Referring to the orientation, the angular velocity in the sensor data is integrated for each orientation to obtain a rough rotation value for each orientation.

In step 233, an auxiliary calculation of the rotation angle is performed on the rough rotation value based on the acceleration and gravity direction information in the sensor data to obtain the rotation angle of the smart terminal relative to each orientation in space.

Among them, after obtaining the rough rotation value of the smart terminal relative to each orientation in space, the rough rotation value will be accurately calculated using the acceleration and gravity direction information in the sensor data to finally obtain the accurate rotation angle.

The acceleration and gravity direction information in the sensor data are information that records the position and movement of the smart terminal. Therefore, the acceleration and gravity direction information will be used as an aid to perform Kalman filtering, thereby reducing the error in the rough rotation value, and then A rotation angle is obtained at which the error is greatly reduced.

Specifically, after calculating the rough rotation values of the smart terminal relative to various directions in space, the acceleration and gravity direction information in the sensor data are used as assistance and sent to the Kalman filter together with the angular velocity. The Kalman filter The device outputs the rotation angle of the smart terminal itself relative to each orientation in space.

FIG. 4 is a flowchart describing the details of step 130 according to the corresponding embodiment shown in FIG. 1 . This step 130, as shown in Figure 4, may include the following steps.

In step 131, the mark pattern indicated by the recognition result is recognized to locate the mark pattern in the image.

In step 133, target tracking is performed according to the positioned marking pattern, and the translation distance of the marking pattern in the image on the spatial horizontal plane and the scaling size of the marking pattern relative to the pre-stored marking image are obtained from the performed target tracking.

After the recognition result of the mark pattern in the image is obtained through step 110, under the instruction that the mark pattern in the image is recognized, the target tracking process of the image is triggered and the mark pattern in the image is located.

During the target tracking process of the image, the moving distance of the image corresponding to the left and right, up and down directions on the horizontal plane of space can be obtained, which is the translation distance of the marking pattern on the horizontal plane of space.

As mentioned above, the mark image is an image that is pre-stored and contains the mark pattern. Therefore, the mark pattern in the image is in a certain proportional relationship with the mark image, that is, corresponds to a zoom size, and is determined by the execution of the target tracking process. Get this scaled size.

In step 135, the vertical distance in space of the marking pattern in the image is calculated based on the zoom size and the size of the marking image, and the vertical distance and the translation distance form translation information.

Among them, the size of the pre-stored mark image is obtained, and the vertical distance of the mark pattern in the space is calculated with the cooperation of the zoom size, that is, the translation distance in the vertical direction in the space, thereby forming the vertical distance and the translation distance of the horizontal plane of the space together. Pan information.

Through the exemplary embodiments described above, a multi-sensor fusion algorithm is implemented in a smart terminal, and the rotation angle of the smart terminal in three directions in space can be quickly and accurately calculated, which will also be the captured image and the mark pattern in the image. posture.

In another exemplary embodiment, step 110 further includes: the smart terminal continues to capture images and obtains a recognition result of the mark pattern in the currently captured image.

Correspondingly, before step 130, the image recognition and tracking method also includes the following steps.

Relative to the image in which the mark pattern is recognized for the first time, a transmission transformation preprocessing of the currently captured image is performed to obtain a transmission image, and the transmission image is used for target tracking of the currently captured image.

Among them, it should be noted here that the recognition and tracking implemented for the image is performed during the continuous image capture by the smart terminal. In other words, the image to be recognized and tracked is every image continuously captured by the smart terminal.

For example, when shooting a real scene in frame units, the image obtained is one frame of the image being shot. As the shooting continues, the next frame of image will also be subjected to the recognition and tracking process.

As mentioned above, the translation information of the mark pattern in the image in space will be obtained through the target tracking process of the image. In order to ensure the accuracy of the translation information, further simplify the target tracking process, and improve the processing speed and efficiency, we will Before the image performs the target tracking process, the image is optimized, that is, transmission transformation preprocessing is performed.

The transmission image obtained through transmission transformation preprocessing is performed to perform the target tracking process of this image, so that the transmission image used for target tracking is consistent with the spatial angle and posture of the image where the marking pattern is first recognized, thus enabling target tracking to be achieved. There is no need to consider the rotation angle anymore, and the translation information can be obtained directly.

It should be added here that with the image recognition and tracking method, the pose matrix obtained is relative to the initial pose, that is, relative to the pose corresponding to the image in which the mark pattern is first recognized.

Further, FIG. 5 shows, according to an exemplary embodiment, a transmission image obtained by performing transmission transformation preprocessing on the currently captured image relative to the image in which the mark pattern is first recognized, and the transmission image is used to perform preprocessing of the currently captured image. A flowchart describing the details of the goal tracking steps.

This step, as shown in Figure 5, may include the following steps.

In step 301, the rotation angle corresponding to the marking pattern in the image in which the marking pattern is recognized for the first time is obtained, and the rotation angle is used as the initial rotation angle.

In step 303, based on the rotation angle output by multi-sensor fusion in the smart terminal and the initial rotation angle, the angle between the currently captured image and the image in which the mark pattern is first recognized is calculated to obtain.

Among them, for an image, after completing multi-sensor fusion in the smart terminal, the rotation angle of the image is obtained. For this image, the initial rotation angle corresponding to the marking pattern in the image in which the marking pattern is recognized for the first time is obtained, and the included angle is obtained by calculating the difference between the rotation angle of the image and the initial rotation angle.

In step 305, a transmission image is obtained by performing transmission transformation of the currently captured image through the included angle.

Among them, performing transmission transformation on the captured image is essentially to correct the captured image and eliminate the rotation and distortion errors between it and the image where the mark pattern is first recognized, thereby facilitating subsequent target tracking.

After the transmission image is obtained, the transmission image is used to perform the object tracking process of the image without using the image directly.

To obtain the pose matrix of the marking pattern in the image, the obtained included angle can be combined. This included angle also corresponds to the three orientations in the space. The obtained included angle and translation information are combined to form the pose matrix, as follows: In this way, the camera pose matrix with six degrees of freedom can be quickly obtained.

In an exemplary embodiment, step 110 in the embodiment shown in Figure 1 may include:

Match the image captured by the smart terminal and the pre-stored marked image, identify whether there is a marked pattern in the image, and obtain the recognition result of the marked pattern in the image.

Among them, before the target tracking of the image, the image will be matched with the preset marked image. If the image matches the marked image, it means that there is a marked pattern in the image, and the marked pattern in the image is recognized, thereby obtaining the corresponding Recognition results.

If the image does not match the marked image, it means that there is no marked pattern in the image, so subsequent target tracking will not be performed and the next image will be waited.

FIG. 6 is a flowchart describing the details of step 110 according to the embodiment corresponding to FIG. 1 . This step 110, as shown in Figure 6, may include the following steps.

In step 111, a user instruction for specifying a mark pattern in the captured image is received.

Among them, whether there is a mark pattern in the image can be realized by interactive recognition with the user. Specifically, the captured image is displayed. At this time, the user can view the captured image and confirm whether there is a mark pattern in the image. If there is a mark pattern, trigger the specified operation of the mark pattern in the image. , correspondingly, the smart terminal will respond to this specifying operation and generate a user instruction for specifying the mark pattern in the image.

For example, the user's operation of specifying a mark pattern in the image may be triggered by the user's operation of selecting a mark pattern in the displayed image. Of course, other operations may also be possible, which are not limited here.

In step 113, the recognition result of the mark pattern in the image is obtained according to the user instruction.

Through the exemplary embodiments described above, the marked image in the image can be realized simply and quickly through user interaction, which will further improve the accuracy and efficiency of image recognition and tracking.

In an exemplary embodiment, after step 150, the image recognition and tracking method further includes:

The preset virtual scene image is projected into the image according to the pose matrix of the marking pattern in the image.

Among them, after obtaining the pose matrix of the mark pattern in the image, you can use this as a basis to project the virtual scene image in the image, and then realize various business scenarios, and then build a personal-oriented reality augmentation system on the smart terminal, or you can build Reality augmented auxiliary office system for enterprises, etc.

For example, for the implementation of an enterprise-oriented augmented reality auxiliary office system, a marked image can be identified and tracked. This marked pattern corresponds to a desk in the real environment or a specific location in the office, and an email is generated based on the pose matrix of the marked image. , meeting notifications, video conferencing and other business scenarios can also be applied to remote meetings to create a real sense that meeting partners are around you.

Through the exemplary embodiments described above, rapid mark recognition and tracking can be realized on the mobile terminal, that is, the smart terminal, making local real-time mark pattern recognition and tracking of the smart terminal a reality, and having better performance than traditional feature point tracking. Stability and higher tracking accuracy.

Taking the implementation of the reality augmentation system in a smart terminal as an example, the exemplary embodiments described above are described in combination with products.

Figure 7 is a framework diagram of a reality augmentation system in a smart terminal according to an exemplary embodiment. The smart terminal continues to shoot the real environment and continuously captures images to form video images.

In other words, the continuously captured image is one frame of the video image displayed in the smart terminal with augmented reality.

As shown in Figure 7, after a frame of image is captured, the marked image is matched, that is, as shown in step 410 in the system framework. If matched, step 430 is executed to perform image tracking, and finally obtain The pose matrix of this frame of image can realize the projection of the virtual scene image for this frame of image in the reality augmentation system of the smart terminal, and then perform reality augmentation display.

When step 410 is performed but the marked image is not matched, the next frame of image will be waited for; if the tracking image step can be successfully performed, the next frame of image will also be waited for.

By analogy, recognition tracking is continuously performed for the captured images.

FIG. 8 is an implementation framework diagram of the image tracking step in the embodiment corresponding to FIG. 7 . For each frame of the video image, a corresponding transmission image will be obtained with the cooperation of multiple sensors, as shown in step 510 in the implementation framework.

Then, single target tracking is performed using the transmission image as input, and the pose matrix of the transmission transformation is obtained, that is, step 530. The pose matrix of the transmission transformation is obtained by eliminating rotation. Therefore, it only describes the translation information of the marker pattern in the image, but does not include the rotation angle.

At this time, with the cooperation of multiple sensors, the rotation angle obtained through multi-sensor fusion, together with the pose matrix of this transmission transformation, forms the pose matrix of the mark pattern in the image in space. At this point, it is the reality in the smart terminal. The implementation of the enhanced system provides the fastest and most stable implementation. While further accelerating performance, there is no longer a need to make a trade-off between time performance and recognition effect.

Through the above-mentioned implementation, multiple sensors in the smart terminal are fully integrated, and the tracking stability and accuracy are maintained at a high level.

The following are device embodiments of the present invention, which can be used to perform the above-mentioned image recognition and tracking method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the image recognition and tracking method embodiments of the present invention.

FIG. 9 is a block diagram of an image recognition and tracking device according to an exemplary embodiment. The image recognition and tracking device, as shown in Figure 9, may include but is not limited to: a recognition result acquisition module 710, a target tracking module 730, and a pose acquisition module 750.

The recognition result obtaining module 710 is used to obtain the recognition result of the mark pattern in the image captured by the smart terminal.

The target tracking module 730 is used to locate the mark pattern in the image according to the recognition result of the mark pattern, and perform target tracking based on the positioned mark pattern to obtain the spatial translation information of the mark pattern in the image.

The pose acquisition module 750 is used to fuse the translation information and the rotation angle output by fusion of multiple sensors in the smart terminal to form a pose matrix of the marking pattern in the image.

FIG. 10 is a block diagram describing details of the target tracking module according to the embodiment corresponding to FIG. 9 . The target tracking module 730, as shown in FIG. 10, may include but is not limited to: a mark positioning unit 731, a tracking execution unit 733, and a translation information forming unit 735.

The mark positioning unit 731 is configured to identify and position the mark pattern in the image based on the mark pattern indicated by the recognition result.

The tracking execution unit 733 is configured to perform target tracking according to the positioned marking pattern, and obtain the translation distance of the marking pattern in the image on the spatial horizontal plane and the scaling size of the marking pattern relative to the pre-stored marking image from the performed target tracking.

The translation information forming unit 735 is used to calculate the vertical distance of the marking pattern in the image in space according to the zoom size and the size of the marking image. The vertical distance and the translation distance form translation information.

Figure 11 is a block diagram of an image recognition and tracking device according to another exemplary embodiment. The image recognition and tracking device also includes but is not limited to: a data acquisition module 810 and a multi-sensor fusion module 830.

The data acquisition module 810 is used to obtain sensor data output by multiple sensors when the smart terminal captures images.

The multi-sensor fusion module 830 is used to perform a multi-sensor fusion algorithm on sensor data to calculate the rotation angle of the smart terminal in space. The rotation angle is output by multi-sensor fusion and used to form a pose matrix of the marking pattern in the image.

In another exemplary embodiment, the recognition result obtaining module 710 is further used for the smart terminal to continuously perform image capture and obtain the recognition result of the mark pattern in the currently captured image.

The image recognition and tracking device also includes a transmission transformation module. The transmission transformation module is used to perform transmission transformation preprocessing on the currently captured image relative to the image in which the mark pattern is first recognized to obtain a transmission image, and the transmission image is used for target tracking of the currently captured image.

Further, FIG. 12 is a block diagram illustrating details of the transmission transformation module according to an exemplary embodiment. The transmission transformation module 910, as shown in Figure 12, may include but is not limited to: an initial rotation acquisition unit 911, a rotation transformation unit 913 and an image transmission transformation unit 915.

The initial rotation obtaining unit 911 is used to obtain the rotation angle corresponding to the marking pattern in the image in which the marking pattern is recognized for the first time, and use the rotation angle as the initial rotation angle.

The rotation transformation unit 913 is used to calculate the angle between the currently captured image and the image in which the mark pattern is first recognized based on the rotation angle output by multi-sensor fusion in the smart terminal and the initial rotation angle.

The image transmission transformation unit 915 is configured to perform transmission transformation of the currently captured image through the included angle to obtain a transmission image.

In an exemplary embodiment, the recognition result obtaining module 710 shown in Figure 9 is further used to perform matching between the captured image and the marked image, identify whether there is a marked pattern in the image, and obtain the identification of the marked pattern in the image. result.

In another exemplary embodiment, the image recognition and tracking device also includes, but is not limited to, a projection module. The projection module is used to project the preset virtual scene image in the image according to the pose matrix of the marking pattern in the image.

Figure 13 is a block diagram of a device according to an exemplary embodiment. For example, the device 900 may be the smart terminal 110 in the implementation environment shown in FIG. 1 . For example, the smart terminal 110 may be a terminal device such as a smartphone or a tablet computer.

Referring to Figure 13, apparatus 900 may include one or more of the following components: processing component 902, memory 904, power supply component 906, multimedia component 908, audio component 910, sensor component 914, and communication component 916.

Processing component 902 generally controls the overall operations of device 900, such as operations associated with display, phone calls, data communications, camera operations, recording operations, and the like. Processing component 902 may include one or more processors 918 to execute instructions to complete all or part of the methods described below. Additionally, processing component 902 may include one or more modules that facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

Memory 904 is configured to store various types of data to support operations at device 900 . Examples of such data include instructions for any application or method operating on device 900 . The memory 904 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read -Only Memory (EEPROM for short), Erasable Programmable Read Only Memory (EPROM for short), Programmable Red-Only Memory (PROM for short), Read-Only Memory , referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. One or more modules are also stored in the memory 904, and the one or more modules are configured to be executed by the one or more processors 918 to complete any of the following Figures 3, 4, 5 and 6. Show all or part of the steps in the method.

Power supply component 906 provides power to the various components of device 900 . Power supply components 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900 .

Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD for short) and a touch panel. If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. The screen may also include an Organic Light Emitting Display (OLED for short).

Audio component 910 is configured to output and/or input audio signals. For example, the audio component 910 includes a microphone (Microphone, referred to as MIC). When the device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signals may be further stored in memory 904 or sent via communications component 916 . In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

Sensor component 914 includes one or more sensors for providing various aspects of status assessment for device 900 . For example, sensor component 914 may detect an open/closed state of device 900, the relative positioning of components, sensor component 914 may also detect a change in position of device 900 or a component of device 900, and a change in temperature of device 900. In some embodiments, the sensor assembly 914 may also include a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 916 is configured to facilitate wired or wireless communication between apparatus 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi (WIreless-Fidelity). In one exemplary embodiment, the communication component 916 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth technology and other technologies.

In an exemplary embodiment, the device 900 may be controlled by one or more application specific integrated circuits (Application Specific Integrated Circuits, ASICs for short), digital signal processors, digital signal processing equipment, programmable logic devices, field programmable gate arrays, device, microcontroller, microprocessor or other electronic components for executing the following methods.

Optionally, the present invention also provides a smart terminal. The TV terminal can be used in the implementation environment shown in Figure 1 to perform any of the steps shown in Figures 1, 2, 3, 4, 5 and 6. All or part of the steps of the image recognition and tracking method. The smart terminal includes:

processor;

Memory used to store instructions executable by the processor;

Wherein, the processor is configured to perform:

Obtain the recognition results of the mark pattern in the image captured by the smart terminal;

Locate the marking pattern in the image according to the recognition result of the marking pattern, and perform target tracking based on the positioned marking pattern to obtain the spatial translation information of the marking pattern in the image;

The translation information and the rotation angle output by fusion of multiple sensors in the smart terminal form a pose matrix of the marking pattern in the image.

The specific manner in which the processor of the device in this embodiment performs operations has been described in detail in the embodiment of the method for identifying and tracking images of the smart terminal, and will not be described in detail here.

In an exemplary embodiment, a storage medium is also provided, and the storage medium is a computer-readable storage medium, such as a transitory and non-transitory computer-readable storage medium including instructions. The storage medium refers to, for example, the memory 904 including instructions that can be executed by the processor 918 of the device 900 to perform the above method.

It is to be understood that the present invention is not limited to the precise construction described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims (6)

1. An image recognition and tracking method, characterized in that the method includes: Obtain the recognition result of the mark pattern in the image captured by the smart terminal, the recognition result is used to indicate whether the mark pattern exists in the image and the position of the mark pattern in the image; The mark pattern indicated by the recognition result is recognized to locate the mark pattern in the image; Obtain the rotation angle corresponding to the marking pattern in the image in which the marking pattern is recognized for the first time, and use the rotation angle as the initial rotation angle; Obtain sensor data output by multiple sensors when the smart terminal captures the image; Integrate the angular velocity in the sensor data to obtain the rough rotation value of the smart terminal relative to various directions in space. Based on the acceleration and gravity direction information in the sensor data, the rotation angle is auxiliary calculated to obtain the respective positions of the smart terminal in space. The rotation angle relative to each orientation is output by multi-sensor fusion and used to form the pose matrix of the marking pattern in the image; According to the rotation angle output by multi-sensor fusion in the smart terminal and the initial rotation angle, calculate the angle between the currently captured image and the image in which the mark pattern is recognized for the first time; Perform transmission transformation of the currently captured image through the included angle to obtain a transmission image, and the transmission image is used for target tracking of the currently captured image; The target tracking of the marking pattern positioned through the obtained transmission image is performed so that the transmission image used for target tracking is consistent with the spatial angle posture of the image in which the marking pattern is first recognized, so as to directly obtain the The translation distance of the spatial horizontal plane and the scaling size of the marking pattern relative to the pre-stored marking image; The vertical distance in space of the marking pattern in the image is calculated according to the zoom size and the size of the marking image. The vertical distance and the translation distance form translation information. The translation information is used to indicate the translation distance of the marking pattern in space. and vertical distance; The translation information indicates the distance obtained by the mark pattern relative to the orientation of the three coordinate axes in the space, and the rotation angle output by the fusion of multiple sensors in the smart terminal is relative to the rotation angle of the coordinate axes corresponding to the three coordinate axes in the space, The pose matrix of the marking pattern in the image is formed as elements in the matrix respectively, and the rotation angle output by the multi-sensor fusion is used to describe the occurrence of rotation in the pose corresponding to the marking pattern in the image. 2. The method according to claim 1, characterized in that after the translation information and the rotation angle output by fusion of multiple sensors in the smart terminal form a pose matrix of the marking pattern in the image, the method Also includes: The projection of the preset image in the image is performed according to the pose matrix of the marking pattern in the image. 3. An image recognition and tracking device, characterized in that the device includes: The recognition result acquisition module is used to obtain the recognition result of the mark pattern in the image captured by the smart terminal; A target tracking module, configured to locate the mark pattern in the image according to the recognition result of the mark pattern, and perform target tracking based on the positioned mark pattern to obtain the spatial translation information of the mark pattern in the image; The target tracking module includes: a mark positioning unit, a tracking execution unit and a translation information forming unit; a mark positioning unit, configured to locate the mark pattern in the image after the mark pattern indicated by the recognition result is recognized; The device also includes a transmission transformation module, which is used to perform transmission transformation preprocessing of the currently captured image with respect to the image in which the mark pattern is first recognized to obtain a transmission image, and the transmission image is used to perform the currently captured image. Target tracking of captured images; The transmission transformation module includes: An initial rotation obtaining unit is used to obtain the rotation angle corresponding to the marking pattern in the image in which the marking pattern is recognized for the first time, and use the rotation angle as the initial rotation angle; A data acquisition module, used to obtain sensor data output by multiple sensors when the smart terminal captures the image; The multi-sensor fusion module is used to integrate the angular velocity in the sensor data to obtain the rough rotation value of the smart terminal relative to various directions in space. The rough rotation value is obtained by auxiliary calculation of the rotation angle based on the acceleration and gravity direction information in the sensor data. The rotation angle of the smart terminal relative to each orientation in space, which is output by multi-sensor fusion and used to form the pose matrix of the marking pattern in the image; A rotation transformation unit, configured to calculate the angle between the currently captured image and the image in which the mark pattern is recognized for the first time based on the rotation angle output by multi-sensor fusion in the smart terminal and the initial rotation angle; An image transmission transformation unit, configured to perform transmission transformation of the currently captured image through the included angle to obtain a transmission image; A tracking execution unit configured to perform target tracking of the marking pattern positioned through the obtained transmission image, and perform target tracking such that the transmission image used for target tracking is at a spatial angle with the image in which the marking pattern is first recognized. The posture is consistent to directly obtain the translation distance in the horizontal plane of space and the scaling size of the marking pattern relative to the pre-stored marking image; a translation information forming unit, configured to calculate the vertical distance of the marking pattern in the image in space according to the zoom size and the size of the marking image, and the vertical distance and the translation distance form translation information; The posture acquisition module is used to obtain the distances obtained by the translation information indicating the mark pattern relative to the orientation of the three coordinate axes in the space, and the rotation angle output by the multi-sensor fusion in the smart terminal relative to the three coordinate axes in the space. The corresponding coordinate axis rotation angles are used as elements in the matrix to form the pose matrix of the marking pattern in the image. The rotation angle output by the multi-sensor fusion is used to describe the occurrence of rotation in the pose corresponding to the marking pattern in the image. . 4. The device according to claim 3, characterized in that the device further comprises: A projection module, configured to project a preset virtual scene image in the image according to the pose matrix of the marking pattern in the image. 5. An intelligent terminal, characterized by including: processor; and A memory on which computer readable instructions are stored. When the computer readable instructions are executed by the processor, the image recognition and tracking method according to any one of claims 1 to 2 is implemented. 6. A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image recognition and tracking method according to any one of claims 1 to 2 is implemented.