WO2024061654A1 - Method and device for determining a glasses pose of pair of smart glasses by means of outside-in tracking - Google Patents

Abstract

The invention relates to a method for determining a pose of at least one pair of smart glasses in an interior of a mobile machine, comprising the following steps: - capturing a camera image of an interior of the mobile machine; - evaluating a data-based glasses-type recognition model on the basis of the captured camera image in order to determine a relevant glasses type for at least one pair of smart glasses imaged in the camera image, the data-based glasses-type recognition model being trained to assign a camera image to a relevant glasses type of one or more pairs of smart glasses; - selecting, in each case, one of a plurality of data-based pose recognition models depending on the recognized glasses type, the data-based pose recognition models each being trained for one glasses type, in order to determine a glasses pose of the corresponding pair of smart glasses according to the camera image; - determining, in each case, the glasses pose of the pair of smart glasses of the corresponding glasses type by means of the selected pose recognition model.

Description

Description

Method and device for determining a glasses pose of data glasses using outside-in tracking

The invention relates to data glasses, particularly for use in mobile devices. The invention further relates to measures for tracking a glasses pose based on a pose recognition system external to the glasses.

Data glasses, also called head-mounted displays, are known which, using a display device, can display an image on one or two display surfaces in the field of vision of the wearer of the data glasses. In so-called augmented reality data glasses, the display surfaces can correspond to reflective surfaces that direct images into the eyes of the wearer of the data glasses. The viewing openings of the data glasses are transparent, so that the real environment can be perceived in the usual way through the data glasses. The display surfaces are located in the viewing openings, so that information to be displayed, such as text, symbols, graphics, video displays and the like, can be displayed overlaying the perception of the surroundings.

The information is usually presented to the wearer of the data glasses as information objects in a contact-like manner, i.e. presented in such a way that these information objects are superimposed on a specific assigned object in the real environment or are oriented to this or that the information object to be displayed is in a specific orientation of the data glasses or whose carrier is displayed. In order to display the information objects in a contact-like manner on the display surfaces of the data glasses, it is necessary to know the position of the object in the environment and the user's viewing direction. When wearing the data glasses, the user's viewing direction is permanently assigned to their pose, ie the 3D position as well as the 3D orientation of the data glasses.

The glasses pose of the data glasses can be determined by an external pose recognition device, in which an interior camera is directed at the head of the wearer of the data glasses and the data glasses and determines the glasses pose of the data glasses by evaluating the camera image. The interior camera is usually an infrared camera to enable good coverage of the area of the data glasses even in poor lighting conditions. Such a tracking method is called outside-in tracking and is usually based on a data-based pose recognition model, which is designed as a data-based pattern or object recognition model.

The pose recognition model can be designed in a manner known per se as an artificial neural network or the like and can be trained to assign one or more camera images on which data glasses are displayed to a glasses pose using a per se known pattern recognition method. The glasses pose has a reference to a vehicle coordinate system, which is considered the reference reference system. This glasses pose is then transmitted to the data glasses so that a contact-like display of an information object can be made. In this regard, the glasses pose provided externally to the glasses is often improved using sensor data fusion with detections from motion sensors internal to the glasses in order to provide glasses pose information that is as latency-free as possible in the data glasses.

To recognize the glasses pose of smart glasses from camera images, the data-based pose recognition model can be trained to determine the glasses pose from a camera image in which one is on a user's head data glasses worn are shown. Such pose recognition models are trained with training data sets that reflect reality in the best possible way. However, data glasses for use in a vehicle are offered by different manufacturers and have individual, slightly different shapes, which are characterized by different shapes of the lenses, the frame, the color of the frame and the like. This requires pose recognition models, which are based on pattern recognition based on the camera image, to recognize the different characteristics of the data glasses in order to reliably determine a glasses pose. To do this, the different types of data glasses from different manufacturers must be taken into account when training the pose recognition model.

Therefore, a pose recognition model must be trained with the different glasses shapes from different manufacturers in order to enable reliable pose determination.

It is an object of the present invention to provide an improved method for determining a glasses pose based on a pose recognition model, which is easy to train and has a reduced complexity.

This task is achieved by the method for determining a glasses pose of data glasses in a mobile device using a data-based model system according to claim 1 and by the method for training a model system with a glasses type recognition model for the individual glasses type recognition of data glasses and a variety of pose recognition models for a pose determination of the data glasses solved according to the independent claim.

Further refinements are specified in the dependent claims. According to a first aspect, a method for determining a pose of at least one pair of data glasses in an interior of a mobile device is provided, with the following steps;

Capturing a camera image of an interior of the mobile device, in particular using an interior camera;

Evaluating a data-based glasses type recognition model based on the captured camera image in order to determine a respective glasses type for at least one data glasses depicted in the camera image, the data-based glasses type recognition model being trained to assign a camera image to a respective glasses type of one or more data glasses;

Respectively selecting one of a plurality of data-based pose recognition models depending on the respective type of glasses recognized, the data-based pose recognition models each being trained on a type of glasses in order to determine a glasses pose of the corresponding data glasses depending on the camera image;

Determining the respective glasses pose of the data glasses of the corresponding glasses type based on the selected pose recognition model.

One idea of the above method is to provide a model system that determines a glasses pose of a data glasses in an interior of a mobile device using an interior camera external to the glasses. For this purpose, the model system includes a glasses type recognition model, which first searches the camera image for data glasses located thereon and assigns a type of glasses depending on a shape and/or other characteristics of a recognized data glasses. The glasses type recognition model therefore only needs to be trained to recognize data glasses of a certain type of glasses and to assign them to the recognized glasses type. Depending on the type of glasses detected, a corresponding pose recognition model can be selected from a large number of data-based pose recognition models, each assigned to a specific type of glasses, and then can be used to determine a glasses pose of the data glasses shown.

The data-based pose recognition model is trained precisely on the specific type of glasses and therefore does not take other types or shapes of glasses into account. Since only one type of glasses is used to train the data-based pose recognition model, i.e. H. a specific form of data glasses must be taken into account, this corresponding pose recognition model can be simpler and less complex and requires a smaller number of training data sets for training, which only take poses of the specific type of glasses into account. The implementation for evaluating such a pose recognition model is also less complex.

According to a further embodiment, a bounding box of the data glasses recognized in each case can be determined using the glasses type recognition model at the same time as the glasses type recognition, with the respective pose recognition model being applied to a section of the camera image that is determined by the bounding box.

Thus, the glasses type recognition model can determine a bounding box and thus enable a search space restriction, so that the pose recognition model is selected by specifying the glasses type and can then perform the pose determination in a simpler manner in a corresponding section of the captured camera image by specifying the search frame.

It can be provided that the pose of the at least one data glasses is transmitted to the respective data glasses, so that a contact-analog display can be displayed in the respective data glasses depending on their pose.

Furthermore, model parameters of the selected pose recognition model can be retrieved from a database and implemented to provide the corresponding pose recognition model. The database can be located in a cloud, ie outside the mobile device.

According to a further aspect, a device for determining a pose of at least one pair of data glasses is provided in an interior of a mobile device, comprising: an interior camera that is designed to capture a camera image of an interior of the mobile device; a processor unit which is designed to evaluate a data-based glasses type recognition model based on the captured camera image in order to determine a respective glasses type for at least one data glasses depicted in the camera image, wherein the data-based glasses type recognition model is trained to assign a camera image to a respective one to assign the glasses type to one or more data glasses; o to select one of several data-based pose recognition models depending on the respective type of glasses recognized, the data-based pose recognition models each being trained on a specific type of glasses in order to determine a glasses pose of the corresponding data glasses depending on the camera image; and o to determine the glasses pose of the data glasses of the corresponding glasses type based on the selected pose recognition model.

Furthermore, a communication unit can be designed to transmit the glasses pose to the data glasses.

Embodiments are explained in more detail below with reference to the accompanying drawings. Show it: Figure 1 shows a display system with data glasses for use in a vehicle;

Figure 2 shows a flowchart to illustrate a method for pose recognition of data glasses in an interior of a vehicle; and

Figure 3 different shapes of data glasses.

Figure 1 shows a schematic representation of a display system 1, particularly for use in a motor vehicle. The display system 1 includes an assistance system 2, which is in communication connection 4 with data glasses 3. The communication connection 4 is designed as a data transmission channel, for example in the form of a wireless communication connection or a wired communication connection. The communication connection 4 is able to transmit any type of data and information between the assistance system 2 and the data glasses 3, for example based on packet-bound data transmission. The communication connection 4 can be based, for example, on WiFi, Bluetooth, Bluetooth Low Energy or a comparable standardized radio protocol.

The assistance system 2 can be part of a vehicle assistance system and, in particular, can be provided in a stationary manner in the motor vehicle. The assistance system 2 can be equipped with a communication unit 21, which enables the communication connection 4 between data glasses 3 and the assistance system 2.

The assistance system 2 can also be provided with an interior camera 22 as a camera aimed at the data glasses 3, which points to a likely area of the head of a user/wearer of the data glasses 3 is directed. The interior camera 22 can z. B. include an RGB camera or an infrared camera.

The assistance system 2 can have a processor unit 23. The processor unit 23 can implement a data-based model system, which is described in more detail below. The model system includes a glasses type recognition model and a plurality of pose recognition models, each of which is trained to determine a glasses pose based on a camera image from the interior camera 22. The glasses pose is then transmitted to the data glasses 3 via the communication unit 21.

For display in the data glasses 3, object information is generated which indicates the corresponding object position, object content and object type of the representation of the at least one virtual display object in the data glasses 3.

The data glasses 3 comprise two transparent viewing panes 32 which are enclosed in a frame 31 in a manner known per se. The frame 31 is provided with temples 33 so that the data glasses 3 can be worn on the head of a user in a manner known per se.

One or both viewing panes 32 (glass lenses) are furthermore provided with a transparent display surface 35, through which a display image for displaying virtual display objects is brought into the eye of the wearer of the data glasses 3 by a suitable device, such as a display device 36 arranged on the frame 31 can be projected. The display device 36 can have a microprocessor or a comparable computing unit and a display unit, such as a projection device or the like. The display unit can be designed to direct the electronically generated display image onto the display surface 35 and to image/display it there.

Due to the transparent design of the display surface 35, the electronically generated image can reflect the real environment that can be perceived through the display surface 35 overlay (represent augmented). Using the display device 36, a virtual display object, such as text, a symbol, video information, a graphic or the like, can be displayed on one or both display surfaces 35. The object information defines the object position and representation of the at least one virtual display object in relation to the motor vehicle, ie in the vehicle coordinate system (reference system of the motor vehicle).

Furthermore, the data glasses 3 can be provided with a control unit 37. The control unit 37 can be designed separately or together with the microprocessor of the display device 36. The control unit 37 can be designed in a suitable manner to carry out or support data glasses functions and functions of the display system 1. For this purpose, the assistance system 2 can be connected to the data glasses 3 in order to transmit the object information regarding virtual display objects to be displayed in a contact-analog or non-contact-analog manner and the glasses pose determined externally of the glasses in relation to the vehicle coordinate system to the data glasses 3. For this purpose, the data glasses 3 can include a communication unit 39, which enables communication with the assistance system 2, in particular to detect a glasses pose ui determined externally to the glasses.

For example, the control unit 37 can read out movement information using a glasses inertial sensor system 38. The movement information can be used to correct the latency information of the glasses pose determined externally of the glasses by sensor fusion and to provide a corrected glasses pose in the data glasses 3. This can then be used to display information objects in a contact-like manner.

Figure 2 schematically illustrates a method for pose recognition of data glasses 3 in a vehicle as a mobile device using a flowchart. The pose recognition is carried out remotely and in particular in the assistance system 2. The model system consists of a glasses type recognition model and a large number of glasses each assigned to a type of glasses Pose recognition models are stored in a database of the assistance system 2 in the form of model parameters.

The method initially provides for a camera image to be captured using the interior camera 22 in step S1.

In step S2, this camera image is checked or searched for one or more data glasses. Using an image recognition method or pattern recognition method in the form of the data-based glasses type recognition model, a glasses type of at least one data glasses can be recognized in the camera image based on predetermined glasses shapes of data glasses, as shown, for example, in Figures 3a to 3c. With the help of the implemented pattern recognition method, the type of glasses of the at least one data glasses 3 can be classified in the camera image, which is worn by one or more users in the vehicle.

The glasses type recognition model can be designed as an artificial neural network, in particular as a convolutional neural network (CNN), in order to recognize the type of glasses based on the shape of the glasses or the characteristics of the data glasses 3.

In addition, using the glasses type recognition model, a corresponding position in the interior of the vehicle can be recognized where the respective data glasses 3 are located, for example in the form of a bounding box that surrounds the recognized data glasses. This makes it possible to provide information about the at least one position of the at least one data glasses 3.

In a subsequent step S3, depending on the type of glasses recognized, a pose recognition model trained on exactly this type of glasses is selected from a plurality of pose recognition models each assigned to a type of glasses. The pose recognition model is also data-based and makes it possible to determine a pose of the data glasses 3 of the specific type of glasses based on one or more camera images in the interior of the vehicle. The pose of the data glasses 3 is specified in relation to a vehicle coordinate system.

In step S4, a pose of the data glasses 3 can be determined in a manner known per se using the respectively selected pose recognition model based on the camera image.

By specifying the bounding box for the specific data glasses of the corresponding type of glasses, the selected pose recognition model can carry out the pose determination more reliably and quickly, since the search area in the camera image can be limited to the bounding box.

The previously selected pose recognition model can now be used continuously to determine the glasses pose.

It can be provided that the type of glasses is only determined when data glasses are recognized in the camera image for the first time. Alternatively, the recognition of the type of glasses can be carried out regularly, so that in the event of a change, a correspondingly different pose recognition model can be selected immediately. In particular, the glasses type can be determined if the pose recognition model was unable to determine a glasses pose of the data glasses 3 for a predetermined period of time, such as 5 seconds.

Reference symbol list

Display system

Assistance system

Smart glasses

Communication link

Communication unit

Interior camera

Processor unit

Frame

viewing windows

Glasses temples

Display area

Display device

Control unit

Glasses inertial sensors

Communication unit

Claims

Claims Method for determining a pose of at least one pair of data glasses (3) in an interior of a mobile device, with the following steps: - Capturing (S1) a camera image of an interior of the mobile device; - Evaluating (S2) a data-based glasses type recognition model based on the captured camera image in order to determine a respective glasses type for at least one data glasses depicted in the camera image, the data-based glasses type recognition model being trained to assign a camera image to a respective glasses type of one or more data glasses to assign; - Respectively selecting (S3) one of several data-based pose recognition models depending on the respective type of glasses recognized, the data-based pose recognition models each being trained on a type of glasses in order to determine a glasses pose of the corresponding data glasses depending on the camera image; - Respectively determining (S4) the glasses pose of the data glasses (3) of the corresponding glasses type based on the selected pose recognition model. Method according to claim 1, wherein a bounding box of the respectively recognized data glasses (3) is further determined using the glasses type recognition model, the respective pose recognition model being applied to a section of the camera image which is determined by the bounding box. Method according to claim 1 or 2, wherein the pose of the at least one data glasses (3) is transmitted to the respective data glasses (3), so that a contact-analog display can be displayed in the respective data glasses depending on their pose. Method according to one of claims 1 to 3, wherein model parameters of the respectively selected pose recognition model are retrieved from a database and implemented in order to provide the corresponding pose recognition model. Method according to one of claims 1 to 4, wherein the model parameters of the pose recognition model are retrieved from a cloud database depending on a recognized type of glasses. Device for determining a pose of at least one pair of data glasses (3) in an interior of a mobile device, comprising: - an interior camera (22) which is designed to capture a camera image of an interior of the mobile device; - a processor unit (23) which is designed to evaluate a data-based glasses type recognition model based on the captured camera image in order to determine a respective glasses type for at least one data glasses depicted in the camera image, the data-based glasses type recognition model being trained to to assign a camera image to a respective type of glasses of one or more data glasses; o to select one of several data-based pose recognition models depending on the respective type of glasses recognized, the data-based pose recognition models each being trained on a specific type of glasses in order to determine a glasses pose of the corresponding data glasses depending on the camera image; and o to determine the glasses pose of the data glasses (3) of the corresponding glasses type based on the selected pose recognition model. Device according to claim 6, wherein a communication unit (21) is designed to transmit the glasses pose to the data glasses (3).