TWI731624B - Method for estimating position of electronic device, electronic device and computer device - Google Patents

Abstract

The invention provides a method for estimating a position of a head-mounted display (HMD), the HMD and a computer device. The method includes: obtaining scene images at an i-th time point, and extracting feature points in each scene image; identifying specific object regions in each scene image, and retrieving a specific feature point failing to correspond to each specific object region; obtaining a feature point position of each scene feature point; obtaining a first device position of the HMD at the i-th time point, and obtaining a moving distance of the HMD; obtaining a measured position of the HMD at the (i+1)-th time point; estimating a second device position of the HMD at the (i+1)-th time point based on the first device position, the moving distance, and the measured position.

Description

Method for estimating the position of head mounted display, computer device and head mounted display

The present invention relates to a spatial positioning technology, and particularly relates to a method for estimating the position of a head-mounted display, a computer device, and a head-mounted display (HMD).

Please refer to FIG. 1, which is a conventional head-mounted display that captures different images of the same scene by different lenses. In the conventional technology, because there are objects such as lamps or animals in the captured scene 199, it is easy to make the scene images 111 and 112 shot by different lenses on the head-mounted display (such as a robot or a head-mounted display) exist between the scene images 111 and 112. Larger variation, and it is easy to cause overexposure in some image areas. Under this circumstance, it is easy to make mistakes when establishing the link between the feature points in the prior art, which may lead to larger errors in the subsequent estimation of the position/coordinates of the head mounted display.

In view of this, the present invention provides a method for estimating the position of a head-mounted display, a computer device, and a head-mounted display, which can be used to solve the above-mentioned technical problems.

The present invention provides a method for estimating the position of a head-mounted display, wherein a plurality of lenses are arranged on the head-mounted display, and the method includes: obtaining a plurality of scene images taken by the aforementioned lens on a reference scene at the i-th time point, and Capture at least one feature point in each scene image; identify at least one specific object area corresponding to at least one specific object in each scene image, and extract at least one specific feature point that does not correspond to each specific object area from each scene image , Wherein at least one specific feature point corresponds to at least one scene feature point in the reference scene; obtains a feature point position of each scene feature point in the reference scene based on at least one specific feature point of each scene image; The position of a first device in the reference scene at the i-th time point, and a moving distance of the head-mounted display between the i-th time point and the (i+1)-th time point is obtained; based on each scene feature The characteristic point position of the point obtains the measurement position of the head-mounted display in the reference scene at the i+1th time point; based on the position of the first device, the moving distance, and the measured position, it is estimated that the head-mounted display is at the i+1th time Click on a second device location in the reference scene.

The invention provides a head-mounted display, which includes a plurality of lenses, a storage circuit and a processor. The storage circuit stores multiple modules. The processor is coupled to the aforementioned lens and storage circuit, and accesses the aforementioned module to perform the following steps: obtain a plurality of scene images captured by the aforementioned lens on a reference scene at the i-th time point, and capture each scene image At least one feature point; identifying at least one specific object region corresponding to at least one specific object in each scene image, and extracting at least one specific feature point not corresponding to each specific object region from each scene image, wherein at least one specific feature point Corresponding to at least one scene feature point in the reference scene; obtain a feature point position of each scene feature point in the reference scene based on at least one specific feature point of each scene image; obtain when the head mounted display is at the i-th time point A first device position in the reference scene, and a moving distance of the head mounted display between the i-th time point and the i+1-th time point is obtained; the head is obtained based on the feature point position of each scene feature point The measured position of the head-mounted display in the reference scene at the i+1th time point; based on the position of the first device, the moving distance, and the measured position, the estimated position of the head-mounted display in the reference scene at the i+1th time point A second device location.

The invention provides a computer device including a storage circuit and a processor. The storage circuit stores multiple modules. The processor is coupled to the storage circuit, and accesses the aforementioned module to perform the following steps: obtain multiple scene images taken by multiple lenses on a head-mounted display on a reference scene at the i-th time point, and capture each At least one feature point in the scene image; identifying at least one specific object area corresponding to at least one specific object in each scene image, and extracting at least one specific feature point not corresponding to each specific object area from each scene image, wherein at least A specific feature point corresponds to at least one scene feature point in the reference scene; a feature point position of each scene feature point in the reference scene is obtained based on at least one specific feature point of each scene image; A first device position in the reference scene at two time points, and a moving distance of the head-mounted display between the i-th time point and the i+1-th time point is obtained; based on the characteristics of each scene feature point The point position obtains the measured position of the head mounted display in the reference scene at the i+1th time point; based on the position of the first device, the moving distance, and the measured position, it is estimated that the head mounted display is at the i+1th time point The location of a second device in the reference scene.

Please refer to FIG. 2, which is a schematic diagram of a head-mounted display according to an embodiment of the present invention. In different embodiments, the head-mounted display 200 is, for example, a robot, or a head-mounted display that can be used to provide virtual reality (VR)/augmented reality (AR) services, but it may not be limited to this. As shown in FIG. 2, the head-mounted display 200 includes a storage circuit 202, a processor 204, and lenses 2061 to 206N (N is the total number of lenses).

The storage circuit 202 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk Disk or other similar devices or a combination of these devices can be used to record multiple codes or modules.

The processor 204 is coupled to the storage circuit 202 and the lenses 2061~206N, and can be a general purpose processor, a special purpose processor, a traditional processor, a digital signal processor, multiple microprocessors, one or more A microprocessor, controller, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), any other Types of integrated circuits, state machines, processors based on Advanced RISC Machine (ARM) and similar products.

The lenses 2061 to 206N may be charge coupled device (CCD) lenses, complementary metal oxide semiconductor transistors (CMOS) lenses, or other lenses that can be used to shoot images. In the embodiment of the present invention, the positions of the lenses 2061 to 206N on the head mounted display 200 can be assumed to be fixed. In other words, the relative positions of the lenses 2061 to 206N to each other can be understood as known.

In the embodiment of the present invention, the processor 204 can access the modules and program codes recorded in the storage circuit 202 to implement the method for estimating the position of the head mounted display proposed by the present invention. The details are described in detail as follows.

Please refer to FIG. 3, which is a flowchart of a method for estimating the position of a head mounted display according to an embodiment of the present invention. The method of this embodiment can be executed by the head-mounted display 200 in FIG. 2. The details of each step in FIG. 3 are described below in conjunction with the components shown in FIG. 2.

First, in step S310, the processor 204 may obtain multiple scene images taken by the lenses 2061~206N of the reference scene at the i-th time point, and capture the feature points in each scene image, where i is the time point Index value. In an embodiment, assuming that the head-mounted display 200 includes only two lenses 2061 and 2062, and they are used to photograph the scene 199 (ie, the reference scene) shown in FIG. 1, the scene obtained by the processor 204 The images are, for example, the images 111 and 112 shown in FIG. 1, but it is not limited to this.

After obtaining the above-mentioned scene images, the processor 204 may, for example, capture the information in each scene image based on the Scale-invariant Feature Transform (SIFT) algorithm or the Speeded Up Robust Features (SURF) algorithm. Feature points, but not limited to this.

After that, in step S320, the processor 204 can identify specific object regions corresponding to specific objects in each scene image, and extract specific feature points that do not correspond to each specific object region from each scene image. In different embodiments, the specific object considered by the present invention is, for example, at least one of a lamp and an animal (such as a human, a cat, a dog, etc.), but it may not be limited thereto.

In one embodiment, the processor 204 may, for example, use certain machine learning algorithms trained to identify lamps and animals to identify specific object regions corresponding to specific objects in each scene image, and will be located in each specific object region. The feature points are excluded in each scene image, so that only specific feature points that do not correspond to each specific object area are left in each scene image, but it is not limited to this.

Furthermore, as mentioned earlier, when there are lamps or animals in the obtained scene image, it is easy to cause errors in the subsequent feature point linkage. Therefore, through step S304, the present invention can directly exclude the characteristic points corresponding to specific objects such as lamps and animals, thereby avoiding the occurrence of the above-mentioned situation and improving the subsequent positioning accuracy of the head-mounted display 200, but it is not limited to this.

After that, in step S330, the processor 204 may obtain the feature point position of each scene feature point in the reference scene based on the specific feature point of each scene image. In one embodiment, the processor 204 can learn images of different scenes based on related existing technologies (for example, " Lowe, DG Distinctive Image Features From Scale-Invariant Keypoints. International Journal of Computer Vision 60, 91-110 (2004)"). Which specific feature points in the reference scene correspond to the same scene feature point in the reference scene (the reference scene can include multiple scene feature points), and link these feature points, and the result of the linking can be Roughly refer to the content illustrated in Figure 1 (but the characteristic points corresponding to specific objects such as lamps and animals have been excluded).

After knowing which specific feature points in different scene images correspond to the same scene feature point, the processor 204 can further estimate the location of the feature point of the scene feature point in the reference scene. For example, in one embodiment, it is assumed that the aforementioned scene image includes a first scene image taken by the lens 2061 and a second scene image taken by the lens 2062, and the first scene image includes a first specific feature point, so The second scene image includes a second specific feature point, and both the first specific feature point and the second specific feature point correspond to the first scene feature point in the scene feature points.

In this case, the processor 204 can use the triangular parallax technology to estimate the position of the feature point of the first scene feature point in the reference scene based on the relative positions of the first specific feature point, the second specific feature point, and the lenses 2061 and 2062. Since the relative positions of the lenses 2061 and 2062 are assumed to be known, the processor 204 only needs to simply estimate the position of the feature points of the first scene in the reference scene based on the triangular parallax method, and the correlation of the triangular parallax method For details, please refer to the relevant technical literature (for example, " R. Mur-Artal and JD Tardós, "ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras," in IEEE Transactions on Robotics, vol. 33 , no. 5, pp. 1255-1262, Oct. 2017.” ), I will not repeat it here.

表示）。應了解的是，本發明的技術概念大致可理解為持續地基於頭戴顯示器200在前一個時間點的裝置位置以及各場景特徵點的特徵點位置來估計頭戴顯示器200在當下時間點的裝置位置。因此，若所考慮的時間點是第i+1個時間點，則對於處理器204而言，頭戴顯示器200在第第i個時間點時在參考場景中的裝置位置（即上述第一裝置位置）可視為是已知的，但可不限於此。 Next, in step S340, the processor 204 may obtain the first device position of the head-mounted display 200 in the reference scene at the i-th time point, and obtain the head-mounted display 200 at the i-th time point and the i+1th time point. Moving distance between time points (in

Said). It should be understood that the technical concept of the present invention can be roughly understood as a device that continuously estimates the head-mounted display 200 at the current time point based on the device position of the head-mounted display 200 at the previous time point and the feature point positions of each scene feature point. position. Therefore, if the considered time point is the i+1th time point, for the processor 204, the head mounted display 200 is at the i-th time point in the device position in the reference scene (that is, the first device The location) can be regarded as known, but it is not limited to this.

In addition, the head mounted display 200 of the present invention may be provided with an inertial measurement unit (IMU) (not shown), and the processor 204 may obtain the IMU at the i-th time point and the i+1-th time point According to the measured acceleration, the moving distance of the head mounted display 200 between the i-th time point and the (i+1)-th time point is estimated.

Next, in step S350, the processor 204 may obtain the measurement position of the head mounted display 200 in the reference scene at the i+1th time point based on the feature point position of each scene feature point. In one embodiment, the measurement position of the head-mounted display 200 in the reference scene at the i+1th time point can be assumed to be the origin, so the feature point position of each scene feature point (which can be expressed as a vector In the case of form), the processor 204 can easily calculate the position of the origin, that is, the measurement position of the head mounted display 200 at the i+1th time point, but it is not limited to this.

After that, in step S360, the processor 204 may estimate the position of the second device in the reference scene of the head mounted display at the i+1th time point based on the position of the first device, the moving distance, and the measured position.

. In an embodiment, the processor 204 may use a Kalman filter to estimate that the head mounted display 200 is in the reference scene at the i+1th time point based on the position of the first device, the moving distance, and the above-mentioned measured position. The position of the second device and the related technical details when using the Kalman filter are described below.

的狀態向量（state vector），其中

，且

及

為

的一次及二次微分。此外，

，其分別對應於三度空間中的x、y、z座標。 In an embodiment, the first device position of the head mounted display 200 in the three-dimensional space can be expressed as

State vector (state vector), where

And

and

for

The first and second derivative of. In addition,

, Which respectively correspond to the x, y, and z coordinates in the three-dimensional space.

為轉換矩陣（transition matrix），其可表徵為：

，其中

，

為第i個時間點及第i+1個時間點之間的時間間隔。 In addition, another definition

It is the transition matrix, which can be characterized as:

,among them

,

Is the time interval between the i-th time point and the i+1-th time point.

為狀態向量的共變異數矩陣（covariance matrix）。在一實施例中，

，其中

，

為資料的變異數，且

的值可隨時間而改變。 In addition, another definition

Is the covariance matrix of the state vector. In one embodiment,

,among them

,

Is the variance of the data, and

The value of can change over time.

為雜訊的共變異數矩陣，其可表徵為

，其中

，而

為白雜訊（white noise）的雜訊功率。 In addition, another definition

Is the covariance matrix of noise, which can be characterized as

,among them

,and

It is the noise power of white noise.

為用於將

轉換為所需結果（例如測量位置）的測量矩陣，其可表徵為

，其中

。 In addition, another definition

For the

The measurement matrix converted to the desired result (such as the measurement position), which can be characterized as

,among them

.

為測量的共變異數矩陣，其可表徵為

，其中

，

為即時定位與地圖構建（Simultaneous localization and mappingSLAM）的測量變異數。 In addition, another definition

Is the measured covariance matrix, which can be characterized as

,among them

,

Measured variance for Simultaneous localization and mappingSLAM (Simultaneous localization and mappingSLAM).

表示）：

，其中控制因子（control factor）

可作為

代換至

中。 In one embodiment, in the predict stage of the Kalman filter, the processor 204 may generate a predicted position according to the following formula (in order to

Said):

, Where control factor

can be used as

Substitute to

in.

以將預測的不確定性納入考量，其中調整後的

（表徵為

）可依「

」的式子計算而得。 Then, the processor 204 can adjust

In order to take into account the uncertainty of the forecast, the adjusted

(Characterized as

) May follow "

”Is calculated by the formula.

，以獲得關於其準確性的測量值和相關協方差矩陣。 After that, in the update phase of the Kalman filter, the processor 204 can calculate

, To obtain the measurement value and the correlation covariance matrix about its accuracy.

表示）與預測位置之間的差距（以

表示）：

。 Then, the processor 204 can calculate the measurement position with the following formula (in

Expressed) and the gap between the predicted position (in

Said):

.

表示），其中

。 After that, the processor 204 may calculate a scaling factor (scaling factor) based on the measured position or the predicted position, whichever is more accurate.

Means), where

.

表示），其中

。並且，處理器204可再根據在測量中的確定性來更新為狀態向量的共變異數矩陣（即

），其中

，且此步驟可體現出

隨時間而變的特性。以上與卡爾曼濾波器有關的進一步技術細節可參照相關的現有技術文獻（例如「 Kalman, Rudolph Emil. "A new approach to linear filtering and prediction problems." (1960): 35-45.」），於此不另贅述。 After that, the processor 204 may estimate the second device position of the head-mounted display 200 based on the zoom factor, the predicted position, and the aforementioned gap (in order to

Means), where

. In addition, the processor 204 may then update the covariance matrix of the state vector according to the certainty in the measurement (that is,

),among them

, And this step can reflect

Characteristics that change over time. For further technical details related to the Kalman filter above, please refer to related prior art documents (for example, " Kalman, Rudolph Emil. "A new approach to linear filtering and prediction problems." (1960): 35-45. "). This will not be repeated here.

Please refer to FIG. 4, which is a schematic diagram of a computer device according to an embodiment of the present invention. In different embodiments, the computer device 400 is, for example, a personal computer, a notebook computer, a server or other similar devices that can obtain scene images captured by the head-mounted display 200, but it is not limited to this. As shown in FIG. 4, the computer device 400 may include a storage circuit 402 and a processor 404. For various possible implementations of the storage circuit 402 and the processor 404, please refer to the relevant descriptions of the storage circuit 202 and the processor 204, which are not described here. Go into details.

In an embodiment, the processor 404 can access the modules and program codes recorded in the storage circuit 402 to implement the method in FIG. 3. In other words, the various steps/operations performed by the processor 204 of the head-mounted display 200 described above can be changed to be executed by the processor 404 of the computer device 400 based on the obtained scene image to estimate the second device of the head-mounted display 200 position. For the relevant details, please refer to the description in the previous embodiment, which will not be repeated here.

To sum up, by excluding the feature points corresponding to specific objects (such as animals, lamps, etc.), the method of the present invention can make the linking of feature points between images of different scenes more accurate, thereby achieving more accurate headwear Display positioning.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.

111,112: image 199: Scene 200: Head-mounted display 202, 402: storage circuit 204,404: Processor 2061~206N: lens 400: computer device S310~S360: steps

Figure 1 is a conventional head-mounted display with different images shot on the same scene by different lenses. FIG. 2 is a schematic diagram of a head-mounted display according to an embodiment of the present invention. FIG. 3 is a flowchart of a method for estimating the position of a head mounted display according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a computer device according to an embodiment of the present invention.

S310~S360: steps

Claims (8)

A method for estimating the position of a head-mounted display, wherein a plurality of lenses are arranged on the head-mounted display, and the method includes: Acquiring a plurality of scene images taken by the lenses on a reference scene at the i-th time point, and capturing at least one feature point in each of the scene images; Identify at least one specific object area corresponding to at least one specific object in each of the scene images, and extract at least one specific feature point that does not correspond to each specific object area from each of the scene images, wherein the at least one specific feature point corresponds to At least one scene feature point in the reference scene; Obtaining a feature point position of each scene feature point in the reference scene based on the at least one specific feature point of each scene image; Obtain a first device position of the head-mounted display in the reference scene at the i-th time point, and obtain the head-mounted display between the i-th time point and the i+1-th time point A moving distance of; Obtaining a measurement position of the head-mounted display in the reference scene at the i+1th time point based on the feature point position of each feature point of the scene; Estimating a second device position of the head mounted display in the reference scene at the (i+1)th time point based on the first device position, the moving distance, and the measured position. The method according to claim 1, wherein the step of capturing the at least one characteristic point in each of the scene images includes: The at least one feature point in each scene image is extracted based on a scale-invariant feature conversion algorithm or an accelerated robust feature algorithm. The method according to claim 1, wherein the at least one specific object includes at least one of a lamp and an animal. The method according to claim 1, wherein the lenses include a first lens and a second lens, and the scene images include a first scene image taken by the first lens and a first scene image taken by the second lens A second scene image, the first scene image includes a first specific feature point, the second scene image includes a second specific feature point, the first specific feature point and the second specific feature point both correspond to the scenes A first scene feature point in the feature points, and the step of obtaining the feature point position of each scene feature point in the reference scene based on the at least one specific feature point of each scene image includes: Using triangular parallax technology to estimate the position of the feature point of the first scene feature point in the reference scene based on the first specific feature point, the second specific feature point, and a relative position of the first lens and the second lens . The method according to claim 1, wherein the head-mounted display is provided with an inertial measurement unit, and the step of obtaining the moving distance of the head-mounted display between the i-th time point and the i+1-th time point include: Obtain the acceleration measured by the inertial measurement unit between the i-th time point and the i+1-th time point, and then estimate the head-mounted display at the i-th time point and the The moving distance between the i+1th time point. The method according to claim 1, wherein the head-mounted display is estimated to be at the reference point at the i+1th time point based on the location of the feature point, the location of the first device, and the moving distance of each feature point of the scene The steps of the second device position in the scene include: A Kalman filter is used to estimate the location based on the feature point location of each scene feature point, the first device location, the moving distance, and a measurement location of the head-mounted display at the i+1th time point. The head-mounted display is a second device position in the reference scene at the i+1th time point. A head-mounted display, including: Multiple shots A storage circuit for storing multiple modules; and A processor, coupled to the lenses and the storage circuit, accesses the modules to perform the following steps: Acquiring a plurality of scene images taken by the lenses on a reference scene at the i-th time point, and capturing at least one feature point in each of the scene images; Identify at least one specific object area corresponding to at least one specific object in each of the scene images, and extract at least one specific feature point that does not correspond to each specific object area from each of the scene images, wherein the at least one specific feature point corresponds to At least one scene feature point in the reference scene; Obtaining a feature point position of each scene feature point in the reference scene based on the at least one specific feature point of each scene image; Obtain a first device position of the head-mounted display in the reference scene at the i-th time point, and obtain the head-mounted display between the i-th time point and the i+1-th time point A moving distance of; Obtaining a measurement position of the head-mounted display in the reference scene at the i+1th time point based on the feature point position of each feature point of the scene; Estimating a second device position of the head mounted display in the reference scene at the i+1th time point based on the first device position, the moving distance, and the measured position. A computer device including: A storage circuit for storing multiple modules; and A processor, coupled to the storage circuit, accesses the modules to perform the following steps: Obtain a plurality of scene images taken by a plurality of lenses on a head-mounted display on a reference scene at the i-th time point, and capture at least one characteristic point in each of the scene images; Identify at least one specific object area corresponding to at least one specific object in each of the scene images, and extract at least one specific feature point that does not correspond to each specific object area from each of the scene images, wherein the at least one specific feature point corresponds to At least one scene feature point in the reference scene; Obtaining a feature point position of each scene feature point in the reference scene based on the at least one specific feature point of each scene image; Obtain a first device position of the head-mounted display in the reference scene at the i-th time point, and obtain the head-mounted display between the i-th time point and the i+1-th time point A moving distance of; Obtaining a measurement position of the head-mounted display in the reference scene at the i+1th time point based on the feature point position of each feature point of the scene; Estimating a second device position of the head mounted display in the reference scene at the i+1th time point based on the first device position, the moving distance, and the measured position.

Images