TWI746417B - Computer vision positioning method and device - Google Patents

Abstract

The present application provides a computer vision positioning method, including: performing posture and motion detection; obtaining multiple photos taken at one location; inputting the multiple photos into a trained deep neural network model for inference, so as to obtain Whether each photo corresponds to a location in a positioning area; locate according to the multiple corresponding locations; obtain one or more corresponding second locations of one or more second photos taken at another location When, perform positioning according to the one or more corresponding second positions to obtain a second positioning position; and determine a corrected positioning position according to the positioning position, the second positioning position, and the posture and dynamic detection results .

Description

Computer vision positioning method and device

The present invention relates to positioning, especially to computer vision positioning.

Positioning and navigation is one of the functions that modern people often use electronic products to perform in daily life. Most modern smart phones have radio navigation functions, such as satellite signal positioning chips, mobile phone communication chips, and local wireless network communication chips. With the navigation signal sent by the satellite, the base station number sent by the wireless base station, and/or the wireless service name of the local wireless network base station, the radio can be used for navigation.

But if you want to achieve higher accuracy, or when you can't receive radio signals, or when radio propagation is multipath, you must rely on computer images with shorter wavelengths of infrared or visible light for positioning. In another case, the user's mobile electronic equipment may be very thin and light, and cannot be equipped with the above-mentioned radio receiving chip and antenna. Or the environment and place where it is located does not allow users to carry chips with radio communication functions, such as tightly-protected military camps and factories, but they also need the functions of positioning and navigation.

Accordingly, there is an urgent need for a computer vision positioning mechanism that does not require radio positioning, at least achieves the accuracy of indoor navigation, and can be implemented in mobile electronic devices to facilitate user applications.

According to an embodiment of the present application, a computer vision positioning method is provided, including: performing posture and motion detection; obtaining multiple photos taken at one location, wherein each photo of the multiple photos corresponds to a different orientation Angle, the azimuth angle information is provided by the detection step; input the multiple photos into a trained deep neural network model for inference, so as to obtain whether each photo corresponds to a location in a localization area Position; and when the inference step obtains a plurality of corresponding positions in the positioning area, positioning is performed according to the plurality of corresponding positions.

According to an embodiment of the present application, a computer vision positioning device is provided, which includes: a camera module for taking photos; a posture and motion detection module for posture and motion detection; and a central processing unit The device module is used to execute the code stored in the non-volatile memory, and is used to perform the following steps: make the attitude and motion detection module perform attitude and motion detection; obtain the camera module to shoot at a location Multiple photos of, where each photo of the multiple photos corresponds to different azimuth information, the azimuth information is provided by the detection step; input the multiple photos to a trained deep neural network The road model performs inference to obtain whether each photo corresponds to a position in a positioning area; and when the inference step obtains multiple corresponding positions in the positioning area, perform positioning according to the multiple corresponding positions .

According to the computer vision positioning device and method provided in the present application, a computer vision positioning mechanism that does not require radio positioning can be provided, the accuracy of indoor navigation can be achieved, and it can also be implemented in mobile electronic equipment to facilitate user applications.

This application provides a computer vision positioning device and a positioning method thereof, which can be implemented on mobile electronic equipment. Before positioning, the computer vision positioning device needs to load the data related to the positioning map before it can perform computer vision positioning in the positioning map. The following will explain how to make location map data. Map data can be referred to as map data for short.

Please refer to FIG. 1, which is a schematic diagram of a computer vision positioning map prepared according to an embodiment of the present application. FIG. 1 shows a schematic diagram of a bird's-eye view of the positioning area 100. As shown in FIG. 1, the positioning area 100 may be located inside a building, especially a building that is difficult for radio waves to penetrate. However, the positioning area 100 of the present application is not limited to being inside a building, it can also be outside the building, or other areas with different visual characteristics.

When preparing the location map data, multiple photos need to be taken in the location area 100. In the embodiment shown in FIG. 1, at least four targets 110, 120, 130, and 140 are included. The four targets 110 to 140 in the positioning area 100 can all correspond to a set of coordinates. This application does not limit the size of the coordinate system and its origin. However, in one embodiment, if a local coordinate system is used, its origin or a specific point may have coordinates in a generalized coordinate system, so that the coordinates can be converted between the local and generalized coordinate system. The aforementioned generalized coordinate system may be one of the coordinate systems used by various satellite positioning systems.

The target object in the positioning area 100 may be a specific object, such as a beaulieu vase, a tree head art table, and a Godzilla toy model. The target can be an axisymmetric or surface symmetric object, such as an axisymmetric round table or a square table with surface symmetry. The target can also be a completely asymmetrical object. The target in the positioning area 100 may be a scene, for example, a wall, a corridor, or a window scene. The so-called scene is a collection of some objects, with a fixed relative position.

To allow the target to be identified, one or more pictures of a target can be taken in the positioning area 100 first, so that the neural network model of artificial intelligence can learn the target. When the target object is a symmetrical object, the photo collection formed by the photos of a certain target object may be taken at the target object in a range. For example, when the target object is a beaulieu vase, the photos in the above photo collection can be taken within an area of two meters around the vase.

When the target is a Godzilla toy model, the Godzilla toy model can be regarded as two target faces at the same position. The first target face is the Godzilla toy model seen from a position near the head end. The second target object is the Godzilla toy model seen from near the end. Although it is the same real target, the corresponding photo sets are different, and the identified target faces can also be different. The first target face can correspond to the first area near the head end. The second target object can correspond to the second area near the end.

For example, the target 140 shown in FIG. 1 may have three different aspects. Each aspect corresponds to a regional range. For example, the target 140 corresponds to three area ranges 141, 142, and 143. A person of ordinary skill in the art can understand that the three aspects and their corresponding area ranges may not overlap each other, or may overlap. For example, the area ranges 141 and 142 do not overlap. On the contrary, the area ranges 141 and 143 overlap in some areas.

The targets 120 and 130, because they are axisymmetric, only have a single face, and only correspond to the area ranges 121 and 131 respectively. The target 110 is a wall window scene. Since there is no method to identify the target 110 from the reverse direction, the target 110 also has only one face, which corresponds to a region 111.

In the embodiment of FIG. 1, only six area ranges are shown, but those of ordinary skill in the art can understand that this is for the sake of simplifying the description. If you want to locate in each room or area of the building, at least you need to pick a target in each room or aisle area and take a set of photos for it. For example, the target position 110 in FIG. 1 is in a space with three walls, and the left and right walls do not have any decorative objects, which are difficult to identify. There are windows and installations on the upper wall, suitable for computer vision identification. Therefore, the preparer of the positioning map can take a set of photos at the position 110 of the target and toward the north. The photos included in the photo collection may be taken in the area range 111. However, this application is not limited, and every photo in the photo set when the neural network model is trained is all taken within the region. A certain target does not necessarily need to correspond to an area range. In other words, the orientation and area of the target are optional data, but the target must correspond to a position on the map.

In addition, the area ranges 111 to 143 of the embodiment shown in FIG. 1 are all rectangular. However, this application does not limit the area range to necessarily be rectangular, and the area range corresponding to a certain target object can be a circle, an ellipse, a sector, and a polygon. It can be symmetrical or asymmetrical.

Furthermore, the area ranges 111 to 143 of the embodiment shown in FIG. 1 all include the position of the corresponding target. However, this application does not require that the area must cover the location of the target. For example, when the aforementioned Beaulieu vase is two people tall, it cannot be distinguished if it is too close. It may take a distance of more than one meter to identify the vase.

The shooting range, shooting time, focal length, contrast, brightness, depth of field and other parameters of each photo in each group of photo collections can be different. For example, you can shoot in the morning and dusk, noon, and night. The windows in the scene of the target 110 will present different views, and the light changes in the room will also be different. For example, when there is a cup on the table in the target scene. When taking photos at different times, the positions of the cups may be different. There may even be multiple cups or no cups. The lighting of the indoor desk lamp may also vary in brightness.

Since the human brain can filter out the above-mentioned changes, it recognizes that these photos are taken within the region corresponding to the target. But for computers to have the same degree of cognition, they must rely on a larger number of photos for learning. In this application, it is not necessary to use a photo in the visible light waveband, and a photo can be taken using a waveband that contains part or all of invisible light, such as infrared waveband or ultraviolet waveband. You can also use a narrower visible light band to take photos, such as black and white photos. However, the band of the photo taken must be able to match the camera module on the positioning device, so that the two have the same band.

After collecting multiple sets of photos, or after collecting additional areas, the computer can be used for visual learning. Please refer to FIG. 2, which is a schematic flowchart of a method 200 for preparing a computer vision positioning map according to an embodiment of the present application. The flowchart shown in FIG. 2 can be a program code executed by an ordinary computer, or a program code executed by a special computer with a hardware acceleration function, so as to improve the learning speed of the artificial intelligence module.

A person of ordinary skill in the art can understand that there are many types of artificial intelligence modules, and a Deep Neural Network (DNN) model is currently more commonly used. In the deep neural network model, there are also various layers, connection methods, weights, and filters used in each layer. This application does not limit which type of deep neural network model to use, as long as it is a model that can obtain a certain accuracy rate or more after supervised learning, it can be used as the deep neural network model used in this application. In addition, this application may not be limited to artificial intelligence modules based on deep neural network models.

Step 210: Provide multiple photo collections about multiple targets.

Step 220: Mark the positions of the multiple targets, or additionally mark multiple regions corresponding to the multiple targets.

Step 230: Provide a deep neural network model.

Step 240: Perform supervised training on the deep neural network model according to the multiple labeled photo sets. In other words, the purpose of this supervised training is to allow the deep neural network model to learn the connection relationship between each photo set and its corresponding location and area range.

Step 250: Test the deep neural network model. For example, input an unlabeled picture into the deep neural network model to make an inference, and test whether the inference result is consistent with the target in the picture. Multiple tests can be repeated to test its success rate.

Step 260: Determine whether the success rate exceeds a preset value. When the success rate is higher than the preset value, the process goes to step 270. Otherwise, the process goes back to step 210, and it may be necessary to provide more pictures to deepen the learning.

Step 270: The training is completed. The deep neural network model can be saved for copying to the computer vision positioning device.

Please refer to FIG. 3, which is a schematic diagram of a computer vision positioning method according to an embodiment of the present application. When the user uses a computer vision positioning device to enter the positioning area 100 for positioning, for example, at the actual location 310, a horizontal 360-degree panoramic view of the surrounding environment can be photographed or recorded. The computer vision positioning device can also take multiple photos at different azimuths at the same or similar locations. The computer vision positioning device can include a posture and motion detection module. For example, the posture and motion detection module may include one of the accelerometer, angular accelerometer, magnetometer, gyroscope, etc. implemented by the micro-electromechanical device, or any combination thereof. When the computer vision positioning device starts to take a picture of the surrounding scene, it can know the azimuth angle of the surrounding scene at the beginning through the data detected by the posture and motion detection module.

In one embodiment, the computer vision positioning device may include two or more wide-angle camera lenses, which can take multiple photos with different azimuth angles at the same time.

In the embodiment shown in FIG. 3, multiple photos with corresponding azimuth angle information taken by the computer vision positioning device at position 310 are input into the deep neural network model that has completed supervised learning for inference. Determine whether each photo can meet one of the four targets 110~140.

When the panoramic photos taken at the location 310 or multiple photos with different azimuth angles are compared to the photo group 111, the deep neural network model cannot infer the photo group 111 due to the long distance. In other words, compared with the photo group 111, its degree of confidence in conformity did not exceed the threshold.

When two photos with different azimuth angles are compared to the targets 120 and 140, the computer vision positioning device can find the two positions of the targets 120 and 140 from the images. In one embodiment, a current position can be taken out of the connection line 330 of the two targets 120 and 140. For example, the current position may be between the target 120 and 140.

In another embodiment, when the targets 120 and 140 have additional information of the area ranges 121 and 143, the intersection area of the area ranges 121 and 143, that is, the area range 320, can be extracted. It can be considered that the current position is in the area range 320.

In a further embodiment, it can be further calculated whether the connection line 330 is in the area range 320 of the intersection. If the result of the judgment is yes, a current position can be retrieved from the connection 330 in the area range 320. For example, it is a point closest to the midpoint of the line 330 among the lines 330 in the area range 320.

In other words, the current position or the area range in which the current position is located can be found according to the positions corresponding to at least two matching targets, or more according to the area ranges corresponding to the targets.

Please refer to FIG. 4, which is a schematic diagram of the triangle positioning step of the computer vision positioning method according to an embodiment of the present application. When multiple photos with different azimuth angles taken at the location 450 are compared to the targets 410 to 430, their conforming confidence levels exceed the threshold value respectively. Therefore, the deep neural network model can infer that multiple photos taken at location 430 with different azimuth angles are related to locations 410, 420, and 430. The positions 410, 420, 430 may form a triangle 470.

In one embodiment, it is assumed that when there are more than three positions of objects related to the photo, the three with the highest degree of confidence or conformity can be selected first. Then, three positions are used to obtain a triangle 470. Finally, the triangle can be used to obtain a positioning position 450.

In an embodiment, the positioning position 450 may be one of the inner center, the outer center, the center of gravity, and the vertical center of the triangle 470. In an embodiment, the positioning position 450 may be related to the aforementioned four centers. For example, it can be the midpoint of any two hearts.

In one embodiment, when the target has a corresponding area range, three position ranges 411, 421, and 431 can be obtained. Then, the intersection of these three position ranges, that is, the position range 460 is calculated.

In an embodiment, the intersection of the position range 460 and the triangle 470 may be further intersected, and it is determined that the current position is in the area range obtained by the intersection.

In one embodiment, it is assumed that when the position of the target object related to the photo exceeds three, it can be divided into a plurality of triangles first. Then, calculate a position of each triangle, such as one of the aforementioned four centers, or a position related to the four centers. Finally, calculate the intermediate point according to the position corresponding to each triangle. For example, when there are four related positions, it can be divided into two triangles first. Then calculate the center of gravity of the two triangles. Finally, calculate the midpoint of the two centers of gravity.

After the positioning position is obtained, a map azimuth angle of a certain matching target relative to the positioning position can be calculated on the map information. Then, according to the comparison of the phase angle when the photo of the target is found and the azimuth angle of the map, the azimuth angle when the photo is taken can be regarded as the azimuth angle of the map.

Please refer to FIG. 5, which is a block diagram of a computer vision positioning device 500 according to an embodiment of the present application. In one embodiment, the computer visual positioning device 500 may be a mobile device, especially a wearable device, which is convenient for the user to carry. Although not shown in FIG. 5, the computer visual positioning device 500 may have a battery to supply power.

The computer vision positioning device 500 may include a camera module 510, a posture and motion detection module 520, a display module 530, a network module 540, a memory 550, an inference module 560, and connection to the above-mentioned modules. Group of a central processing unit 570. The central processing unit 570 is used to control the computer visual positioning device 500. The CPU module 570 may include at least one processor core for executing the operating system and application programs stored in the non-volatile memory, so as to realize the technical solutions and features provided in this application through the remaining modules . A person of ordinary skill in the art should have general knowledge of computer structure and computer organization, and can understand and implement the above-mentioned computer architecture.

The camera module 510 may include at least one camera lens group, which includes optical elements, photosensitive elements, filters, amplifiers, encoders and other components and equipment, used to transmit the captured photos to the central processing unit 570, Or it can be stored directly in the memory 550.

The attitude and motion detection module 520 may include one of an accelerometer, an angular accelerometer, a magnetometer, a gyroscope and other devices implemented by a microelectromechanical device, or any combination thereof. Through the attitude and motion detection module 520, six degrees of freedom information can be output, such as the angular offset and angular acceleration of the pitch axis, roll axis, and yaw axis, and the displacement vector and acceleration of the three vertical axes. . According to the information of the aforementioned six degrees of freedom, the attitude and motion detection module 520 can further calculate the angular offset and displacement of two consecutive timing points. According to this, the information measured at more than two timing points can be accumulated, and the corresponding attitude and dynamic changes can be obtained.

Since the camera module 510 and the posture and motion detection module 520 are fixed in the computer vision positioning device 500, the posture and motion detection module 520 can record the azimuth and position at the time of taking a photo. After the computer vision positioning device 500 moves, the included angle relative to the azimuth angle of the photographing location and the movement vector between the two positions can be known. However, unless an initial coordinate system position and attitude are given by the outside world, the attitude and motion detection module 520 can only calculate the relative position and relative azimuth change of the two timing points.

Those of ordinary skill in the art can understand that the corresponding posture and dynamic change amount can be calculated by the central processing unit 570, or can be calculated by the posture and motion detection module 520. This application does not limit which module is used for calculation.

The display module 530 can be used to display photos taken by the camera module 510 and output images presented by the operating system and the application program. In an embodiment, the display module 530 may include at least one thin display screen, such as various liquid crystal screens. In another embodiment, the display module 530 may include a transmissive optical head-mounted display. There are currently six waveguide-type head-mounted displays, and this application does not limit which type of display.

The optional network module 540 may include a wireless communication module, such as a Bluetooth, a local wireless network, or a fourth, fifth, and sixth-generation mobile communication network module. The network module 540 can be used to download and update the information of the positioning area, that is, the map data.

The memory 550 may include volatile and non-volatile storage media for use as a system memory and long-term storage media. The memory 550 can store graphics and models for use by the CPU module 570 and the inference module 560, respectively.

The optional inference module 560 may be a software module, a hardware module, or a module that cooperates with software and hardware. The inference module 560 is used to make inferences using the deep neural network model that has been supervised and learned, so as to determine and find out whether the input photo is related to a set of positions and/or directions, and may also output its related confidence level. Between 0~100%. The inference module 760 implemented with hardware or software and hardware is used for accelerating calculations. When the calculation amount provided by the central processing unit 570 cannot provide the inference calculation of the deep neural network model in time, the inference module 560 implemented by hardware or software and hardware can be used. When the amount of calculation provided by the central processing unit 570 is sufficient for the inference of the deep neural network model, the inference module 560 implemented by hardware or software and hardware may not be used.

Please refer to FIG. 6, which is a schematic flowchart of a computer vision positioning method 600 according to an embodiment of the present application. The computer vision positioning method 600 can be applied to the computer vision positioning device 500 shown in FIG. 5. In one embodiment, the computer vision positioning method 600 may be a program executed by the central processing unit 570. In the embodiment shown in FIG. 6, if the causal relationship between any two steps is not mentioned, the application does not limit the execution sequence between these two steps.

Step 610: Turn on dynamic posture detection.

Step 620: Obtain multiple photos with azimuth angle information. These photos can have different azimuths and be taken at the same place at the same time or in time sharing.

Step 630: Infer each photo to obtain relevant information. The CPU module 570 or the inference module 560 can be used to make inferences on each photo to obtain relevant information. The relevant information referred to here is whether a certain target is judged in the photo.

Step 640: Determine whether the information of multiple targets is obtained. It is determined whether there is at least one photo corresponding to a target among the inference results of step 640, and the position of the target, or even the corresponding area range, is obtained from the image data. When there are no two photos that correspond to the target object, the process proceeds to optional step 650 or returns to step 610. When there are at least two photos corresponding to the target, step 660 can be performed.

Optional step 650: prompt the user to shift. Since the corresponding position is not found, the display module 530 can be used to ask the user to move and perform the photographing step 620 again.

Step 660: Determine the information of N targets? N is a positive integer greater than or equal to 2. When N is 2, the flow proceeds to step 665. When N is greater than 2, the flow proceeds to step 666.

Step 665: Use the positions of the two targets to locate. Or use the positions of the two targets and their corresponding two area ranges for positioning. After positioning, the map azimuth angle of a target object relative to the positioning position can be used to correspond to the azimuth angle of the photograph of the target object. You can know the current azimuth. For this step, reference may be made to the previous description of FIG. 3, which will not be described in detail here. Then, the flow proceeds to step 670.

Step 666: Use the positions of the N targets to locate. Or use the positions of the N targets and their corresponding two area ranges for positioning. After positioning, the map azimuth angle of a target object relative to the positioning position can be used to correspond to the azimuth angle of the photograph of the target object. You can know the current azimuth. For this step, reference may be made to the previous description of the embodiment shown in FIG. 4. Then, the flow proceeds to step 670.

Step 670: Detect motion for navigation. Since the position and azimuth angle during step 620 are known, and the attitude and motion detection module 720 can detect the angle between the current displacement vector and the azimuth angle during step 620, it can be performed on the relevant map. Positioning and navigation. Then, the process can proceed to optional step 680 or optional step 690.

Optional step 680: display the augmented reality object. In one embodiment, the map may include augmented reality information. For example, the map can contain information that there is something somewhere. Therefore, the user can be guided how to get there and find the object.

In one embodiment, the target object may correspond to a virtual treasure. When it is judged that there is the target object in the photo, it can be considered that the virtual treasure is obtained. In another embodiment, when the positioning position is in a certain area corresponding to a virtual treasure, it can be considered that the virtual treasure is obtained. The virtual treasure can be used to exchange for a physical item, or a discount coupon, entertainment coupon, exchange coupon or a voucher. The number of virtual treasures may have a limit. In other words, only the first few who reach the area or find the target can get the virtual treasure. The network module 540 and a virtual treasure management server can be used to register for rankings, virtual treasures, discount coupons, entertainment coupons, redemption coupons and other vouchers.

Optional step 690: correct positioning. Since the accuracy of the position obtained for the first time is not ideal, subsequent corrections can be made after the positioning information is obtained.

Please refer to FIG. 7, which is a schematic flowchart of the calibration step 690 of the computer vision positioning method according to an embodiment of the present application. In the embodiment shown in FIG. 7, if the causal relationship between any two steps is not mentioned, the application does not limit the execution order between these two steps.

Step 710: Obtain one or more photos. This step 710 is similar to step 620, but only one photo can be obtained.

Step 720: Infer each photo to obtain relevant information. Since the shooting location and azimuth angle of the photo in step 710 are known, it can be compared with the area range corresponding to the target on the map information. For example, when a photo is taken in a region corresponding to a certain target, it can be inferred whether the photo corresponds to the target. In this way, it may save time for inference.

Step 730: Determine whether the information of at least one target object is obtained. When the information of any target is not obtained, the process can return to step 710. Otherwise, the process goes to step 740.

Step 740: Determine and obtain information about N targets. When N=1, proceed to step 751. When N=2, proceed to step 752. When N=3, proceed to step 753.

Step 751: Use the position corresponding to the target to locate. When only one target is found, it can be judged whether the distance between the position of the target and the current position is reasonable. For example, when the distance between two positions is within an error range, there is no need to correct the current position. When the distance between the two positions exceeds the error range, the midpoint of the two positions can be calculated as the new position.

Step 752: Use the positions corresponding to the two targets to locate. In step 665, two target positions can be used to find a new position. Then, it can be judged whether the distance between the new position and the current position is reasonable. For example, when the distance between two positions is within an error range, there is no need to correct the current position. When the distance between the two positions exceeds the error range, the midpoint of the two positions can be calculated as the new position. Or use the new position as the current position.

Step 753: Use the positions corresponding to the N targets to locate. In step 666, N target positions can be used to find a new position. Then, it can be judged whether the distance between the new position and the current position is reasonable. For example, when the distance between two positions is within an error range, there is no need to correct the current position. When the distance between the two positions exceeds the error range, the midpoint of the two positions can be calculated as the new position. Or use the new position as the current position.

Step 760: Update the positioning information according to the old positioning information and the new positioning information.

Please refer to FIG. 8, which is a schematic flowchart of an embodiment of step 666 shown in FIG. 6. In the embodiment shown in FIG. 8, if the causal relationship between any two steps is not mentioned, the application does not limit the execution order between these two steps. For the embodiment shown in FIG. 8, reference may be made to the related description of FIG. 4.

Step 810: Form multiple adjacent triangles according to more than four corresponding positions. For example, four points can form two adjacent triangles. Five points can form three adjacent triangles.

Step 820: Calculate the relative position of each triangle. As mentioned above, the relative position can be related to one of the center of gravity, the inner center, the outer center, and the vertical center of the triangle, or it can be related to any two or more centers.

Step 830: Obtain a positioning position according to the multiple related positions. In an embodiment, the first axis coordinate value of the positioning position may be the average of the first axis coordinate values of all related positions, and the second axis coordinate value of the positioning position may be the average of the second axis coordinate values of all related positions.

Please refer to FIG. 9, which is a schematic flowchart of an embodiment of step 666 shown in FIG. 6. In the embodiment shown in FIG. 9, if the causal relationship between any two steps is not mentioned, the application does not limit the execution sequence between these two steps. For the embodiment shown in FIG. 9, reference may be made to the related description of FIG. 4.

Step 910: According to the four or more corresponding positions, three corresponding positions with the highest confidence level are selected.

Step 920: According to the three corresponding positions, a triangle is formed.

Step 930: Calculate the relevant position of the triangle as a positioning position.

According to an embodiment of the present application, a computer vision positioning method is provided, including: performing posture and motion detection; obtaining multiple photos taken at one location, wherein each photo of the multiple photos corresponds to a different orientation Angle, the azimuth angle information is provided by the detection step; input the multiple photos into a trained deep neural network model for inference, so as to obtain whether each photo corresponds to a location in a localization area Position; and when the inference step obtains a plurality of corresponding positions in the positioning area, positioning is performed according to the plurality of corresponding positions.

In an embodiment, when the inference step obtains two corresponding positions in the positioning area, it further includes: obtaining a line segment passing through the two corresponding positions according to the two corresponding positions; and according to the line segment A positioning position on the top for positioning.

In an embodiment, when the inference step obtains three corresponding positions in the positioning area, it further includes: forming a triangle based on the three corresponding positions; and calculating a positioning position for positioning based on the triangle .

In one embodiment, if the neural network model can provide the degree of inference confidence, when the inference step obtains more than four corresponding positions in the positioning area, it further includes: inference confidence based on the four or more corresponding positions Select three corresponding positions with the highest degree of inference confidence; form a triangle based on the three corresponding positions; and calculate a positioning position for positioning based on the triangle.

In one embodiment, when the inference step obtains more than four corresponding positions in the positioning area, it further includes: forming a plurality of adjacent triangles according to the four or more corresponding positions; calculating the value of each triangle Related positions; and calculating a positioning position for positioning based on a plurality of the related positions.

In one embodiment, in order to continue the calibration after obtaining the position for the first time, the computer vision positioning method further includes: obtaining one or more second photos taken at another location; and inputting the one or more second photos into the The deep neural network model makes inferences to obtain whether each second photo corresponds to a second position in the location area; when the inference step related to the one or more second photos obtains the location area When one or more corresponding second positions are used, perform positioning according to the one or more corresponding second positions to obtain a second positioning position; and according to the positioning position, the second positioning position, and the attitude and dynamic detection As a result, a positioning position after correction is determined.

In one embodiment, the inference step further includes: inputting the multiple photos or the second photos into the deep neural network model for inference, so as to obtain whether each photo or the second photo corresponds to the location area When the photo corresponds to the target, find the location corresponding to the target in the map information; and when the second photo corresponds to the target, find the target in the map information Object corresponding to the second position.

In one embodiment, in order to support the function of obtaining a certificate with a virtual object, the computer visual positioning method further includes: determining whether the target object corresponds to a virtual object; A server registers to obtain a certificate of the virtual object.

According to an embodiment of the present application, a computer vision positioning device is provided, which includes: a camera module for taking photos; a posture and motion detection module for posture and motion detection; and a central processing unit The device module is used to execute the code stored in the non-volatile memory, and is used to perform the following steps: make the attitude and motion detection module perform attitude and motion detection; obtain the camera module to shoot at a location Multiple photos of, where each photo of the multiple photos corresponds to different azimuth information, the azimuth information is provided by the detection step; input the multiple photos to a trained deep neural network The road model performs inference to obtain whether each photo corresponds to a position in a positioning area; and when the inference step obtains multiple corresponding positions in the positioning area, perform positioning according to the multiple corresponding positions .

In one embodiment, when the inference step obtains two corresponding positions and azimuth angles in the positioning area, the central processing unit is further configured to: according to the two corresponding positions, obtain the two corresponding positions and azimuth angles. A line segment of the position of the line segment; and the positioning is performed according to a positioning position on the line segment.

In one embodiment, when the inference step obtains three corresponding positions in the positioning area, the CPU module is further used to: form a triangle according to the three corresponding positions; and according to the triangle, Calculate a positioning position for positioning.

In one embodiment, if the neural network model can provide the degree of confidence in inference, when the inference step obtains four corresponding positions in the positioning area, the CPU module is further used to: According to the degree of inference confidence of the corresponding position, three corresponding positions with the highest degree of inference confidence are selected; based on the three corresponding positions, a triangle is formed; and based on the triangle, a positioning position is calculated for positioning.

In an embodiment, when the inference step obtains more than four corresponding positions in the positioning area, the central processing unit module is further configured to: form a plurality of adjacent triangles according to the four or more corresponding positions ; Calculate the relevant position of each of the triangles; and calculate a positioning position for positioning based on a plurality of the relevant positions.

In one embodiment, in order to continue the calibration after obtaining the position for the first time, the CPU module is further used to: obtain one or more second photos taken by the camera module at another location; The second photo is input to the deep neural network model for inference, so as to obtain whether each second photo corresponds to a second position in the positioning area; when it is related to the inference step of the one or more second photos When obtaining one or more corresponding second positions in the positioning area, perform positioning according to the one or more corresponding second positions to obtain a second positioning position; and according to the positioning position, the second positioning position, and The posture and motion detection results determine a positioning position after correction.

In one embodiment, the CPU module is further used to input the multiple photos or the second photos into the deep neural network model for inference, so as to obtain whether each photo or the second photo corresponds to To a target in the location area; when the photo corresponds to the target, look for the position corresponding to the target in the map information; and when the second photo corresponds to the target, in the map information To find the second position corresponding to the target.

In one embodiment, in order to support the function of obtaining a certificate with a virtual object, the computer visual positioning device further includes a network module connected to a network, wherein the CPU module is further used to: determine the target object Whether it corresponds to a virtual object; and when the target object corresponds to the virtual object, the network module is made to register with a server through the network to obtain a certificate of the virtual object.

According to the computer vision positioning device and method provided in the present application, a computer vision positioning mechanism that does not require radio positioning can be provided, the accuracy of indoor navigation can be achieved, and it can also be implemented in mobile electronic equipment to facilitate user applications.

100: positioning area 110: target 111: Area scope 120: Target 121: regional scope 130: target 131: Area Range 140: target 141: Area Range 142: Area Range 143: Area Range 200: Computer vision positioning map preparation method 310: Actual location 320: area range 330: connection 410: Target 411: Area Range 420: target 421: Area Range 430: target 431: Area Range 450: location 460: position range 470: Triangle 500: Computer vision positioning device 510: camera module 520: Attitude and motion detection module 530: display module 540: Network Module 550: Memory 560: Inference Module 570: CPU module 600: Computer vision positioning method

[Figure 1] is a schematic diagram of a computer vision positioning map prepared according to an embodiment of the present application. [Fig. 2] is a schematic flowchart of a method for preparing a computer vision positioning map according to an embodiment of the present application. [Fig. 3] is a schematic diagram of a computer vision positioning method according to an embodiment of the present application. [Fig. 4] is a schematic diagram of the triangle positioning step of the computer vision positioning method according to an embodiment of the present application. [Fig. 5] is a block diagram of a computer vision positioning device according to an embodiment of the present application. [Fig. 6] is a schematic flowchart of a computer vision positioning method according to an embodiment of the present application. [FIG. 7] is a schematic flow chart of the calibration steps of the computer vision positioning method according to an embodiment of the present application. [Fig. 8] is a schematic flowchart of an embodiment of step 666 shown in Fig. 6. [Fig. [Fig. 9] is a schematic flowchart of another embodiment of step 666 shown in Fig. 6. [Fig.

600: Computer vision positioning method

Claims (6)

A computer vision positioning method includes: performing posture and motion detection; obtaining multiple photos taken at one location, wherein each photo of the multiple photos corresponds to a different azimuth angle, and the azimuth angle information is determined by the Provided by the detection step; input the multiple photos into a trained deep neural network model for inference, so as to obtain whether each photo corresponds to a location in a location area; when the inference step obtains the location When there are multiple corresponding locations in the area, perform positioning based on the multiple corresponding locations to obtain a positioning location; obtain one or more second photos taken at another location; input the one or more second photos into The deep neural network model makes inferences to obtain whether each second photo corresponds to a second position in the location area; when the inference step related to the one or more second photos obtains the location area When one or more corresponding second positions of, perform positioning according to the one or more corresponding second positions to obtain a second positioning position; and according to the positioning position, the second positioning position and the posture and dynamic detection The measurement result determines a positioning position after correction. The computer vision positioning method according to claim 1, wherein the inference step further comprises: inputting the multiple photos or the second photos into the deep neural network model for inference, so as to obtain each photo or the second photo Whether the photo corresponds to a target in the positioning area; when the photo corresponds to the target, search for the location corresponding to the target in the map information; and when the second photo corresponds to the target, Find the second location corresponding to the target in the map information. The computer vision positioning method according to claim 2, further comprising: judging whether the target object corresponds to a virtual object; and when the target object corresponds to the virtual object, registering with a server through the network to obtain the virtual object A certificate of the object. A computer vision positioning device includes: a camera module for taking photos; a posture and motion detection module for posture and motion detection; and a central processing unit module for performing non-volatile The program code stored in the sexual memory is used to perform the following steps: make the attitude and motion detection module perform attitude and motion detection; obtain multiple photos taken by the camera module at a location, of which the multiple Each photo of the photo corresponds to different azimuth information, which is provided by the detection step; input the multiple photos into a trained deep neural network model for inference, so as to obtain each Whether the photo corresponds to a position in a positioning area; when the inference step obtains a plurality of corresponding positions in the positioning area, perform positioning according to the plurality of corresponding positions to obtain a positioning position; obtain the camera module One or more second photos taken at another location; input the one or more second photos into the deep neural network model for inference, so as to obtain whether each second photo corresponds to the location area When the inference step related to the one or more second photos obtains one or more corresponding second positions in the positioning area, perform positioning according to the one or more corresponding second positions , Obtaining a second positioning position; and determining a corrected positioning position according to the positioning position, the second positioning position, and the posture and dynamic detection results. The computer vision positioning device according to claim 4, wherein the central processing unit module is further used for: inputting the multiple photos or the second photos into the deep neural network model for inference, so as to obtain each photo Or whether the second photo corresponds to a target in the positioning area; when the photo corresponds to the target, search for the location corresponding to the target in the map information; and when the second photo corresponds to the target When the target object, the second location corresponding to the target object is searched for in the map information. The computer visual positioning device according to claim 5, further comprising a network module connected to a network, wherein the CPU module is further used for determining whether the target object corresponds to a virtual object; and When the target object corresponds to the virtual object, the network module is made to register with a server through the network to obtain a certificate of the virtual object.

Images