DE102023123406A1 - Method for displaying user support information in an augmented reality application and augmented reality device - Google Patents

Abstract

The present invention relates to a method for displaying at least one item of user support information relating to at least one device or one of its components in an augmented reality application. The present invention also relates to an augmented reality device for providing at least one item of user support information relating to at least one device or one of its components to a user.

Description

field of technology

The present invention relates to a method for displaying at least one item of user support information relating to at least one device or one of its components in an augmented reality application. The present invention also relates to an augmented reality device for providing at least one item of user support information relating to at least one device or one of its components to a user.

State of the art

In the industrial environment, as the machine park becomes increasingly digitalized, more and more information about the devices used and their components is being recorded, processed and stored. Error reports or maintenance information, for example, can be generated based on the data. This information can be used to support on-site service employees in their service operations.

However, as the amount of data increases, it becomes more difficult to maintain an overview of the individual information.

Summary of the Invention

It is therefore an object of the present invention to overcome the described disadvantages of the prior art and in particular to provide means by which a reliable provision of information to service employees is possible even with a larger amount of data.

The object is achieved by the invention according to a first aspect in that a method for displaying user support information with respect to at least one device or one of its components in an augmented reality application,
wherein at least one image of a section of the environment in which the device is at least partially visible is recorded,
wherein the user support information to be displayed is received and displayed in the image or in a field of view of a user of the augmented reality application that at least partially contains the environmental section,
wherein a marker associated with the device is identified in the image and the identified marker is evaluated, and wherein, at least in part based on at least one result of the evaluation, (i) a position at which the user support information is displayed in the image or in the field of view of the user is determined as a display position and/or (ii) a perspective adjustment of the user support information or parts thereof is made.

The invention is therefore based on the surprising finding that, based on structures defined in the image and associated with the device, further information can be derived from the structures, which make it possible to display user support information at a desired location and with a spatial impression in an image of the device or in a user's field of vision.

By determining the display position of the user support information in the image accordingly, the user support information can be displayed in the image or the field of view with reference to a specific component of the device, for example in its vicinity and/or by an allocation line to this component. This makes it very reliable for the user to assign the user support information to be provided to a physical component of the device. By also adjusting the perspective of the user support information, it is made even easier for the user to assign the information to be displayed to the correct component, especially when looking at the device at an angle.

As a result, the proposed method can advantageously reliably display maintenance information on the component to be maintained as user support information. This makes maintenance of the device and its components easier, and in particular maintenance can be carried out more reliably and quickly.

Even if several identical or identical devices or several identical or identical components are in use, the proposed method can be used to display information about one of these components in the image or in the user's field of vision with reference to the correct component. This enables reliable assignment of the information and the risk of accidentally swapping the components is eliminated or at least greatly reduced. This means that information can be provided reliably even with increasing amounts of data.

The user of the augmented reality application can provide the user support information information can therefore be provided very reliably by overlaying the user assistance information in the user's field of view and/or by overlaying the user assistance information in the image that the user can view.

Preferably, in the sense of this application, a perspective adaptation of the user support information or parts thereof is understood to mean an adaptation of the representation of the user support information or parts thereof such that the user support information displayed in the image or in the field of view creates at least a partial spatial effect (in particular for the user of the augmented reality application, who may be a viewer of the image, for example, or in whose field of view the user support information is displayed).

By using the marker at least partially for both determining the display position and for adjusting the perspective of the user support information, the marker can be used for several purposes at once. This makes it possible to implement the method with less effort. In the real world, only a corresponding marker assigned to the device needs to be provided in such a way that it can be recorded together with the device in the image. Further adjustments to the device are therefore not necessary.

The marker contained in the image is advantageously used to determine information for determining the fade-in position. The fade-in position can be determined directly using the marker, or intermediate information can be determined using the marker, which in turn is used to determine the fade-in position, particularly by taking other sources into account.

Information for perspective adjustment is advantageously determined using the marker contained in the image. The information required for perspective adjustment can be determined directly using the marker (as will be explained in more detail later) or intermediate information can be determined using the marker, which in turn can be used to determine the information required for perspective adjustment, particularly by including other sources.

The recorded image with the marker visible in it serves as a starting point for obtaining further information for displaying the user support information. To do this, the marker is identified in the image and then evaluated. The user support information can then be displayed in the user's field of vision. Or the image can also be used to display the user support information.

This also makes it very easy to retrofit the method to existing devices. This means that the advantages of the proposed method can also be exploited in existing environments.

Preferably, the marker is at least partially, preferably completely, visible in the surrounding area. This means that the marker is also reliably completely or at least partially visible in the recorded image.

For example, the marker can be assigned to the device by attaching the marker to the device and/or by positioning the marker such that it is visible in the surrounding area of the image.

A physical object that has a surface area provided with a pattern that can be read out in particular in the image and/or that is arranged on the device can advantageously be provided as a marker. The marker can also be formed in one piece with the device. For example, a surface area of the device can be formed as a marker, which surface area has a surface structure (which is formed or can be formed, for example, by engraving, for example by means of a laser or a mechanical process) or physical properties (such as a refractive index of a material), whereby a pattern is formed that can be read out in particular in the image. Examples of marker patterns are one- or multi-dimensional codes, such as one-dimensional barcodes or two-dimensional data codes, in particular QR codes. Another example of advantageous marker patterns are AprilTags.

Preferably, the identification and evaluation of the marker in the image is carried out at least partially using means of digital image recognition and/or processing. For example, the marker or an image area that at least partially contains the marker can first be identified in the image and then the identified marker and/or image area can be evaluated accordingly.

The at least one result of the evaluation can have several partial results, such as a first, second, third, etc. result of the evaluation. The individual partial results can at least at least partially have different processing steps and/or at least partially build on each other (so that a partial result is at least partially based on another partial result).

Preferably, the position at which the user support information is displayed in the image or in the field of view of the user is determined as the display position, at least partially based on at least a first result of the evaluation. Optionally, the perspective adjustment of the user support information or parts thereof can also be carried out, at least partially based on the first result of the evaluation.

Preferably, the perspective adjustment of the user support information or parts thereof is carried out at least partially based on at least a second result of the evaluation. Optionally, the display position can also be determined at least partially based on the second result of the evaluation.

Preferably, the user support information to be displayed is displayed at the specific display position in the image or in the user's field of vision.

Advantageously, the user support information may contain status data, performance data, parameters and/or a designation of a component of the device or of the device itself.

An advantageous procedure can be as follows, for example: For an image of a device taken with a camera, an orientation (such as the position and/or orientation) of a marker attached to the device is estimated. To do this, a rotation matrix and a translation vector are defined mathematically relative to the camera. The corner points of the user support information to be displayed are defined in a coordinate system of the marker. The corner points of the user support information are displayed in the image using the rotation matrix and the translation vector and optionally a camera intrinsic (for example, world coordinates can be converted into coordinates of the image). The contents of the user support information (bar chart, line charts, texts, values) are available as 2D images and are displayed in perspective in the image. To do this, a linear geometric transformation is calculated based on the corner points of the user support information. This makes it possible to obtain a representation of the user support information with a spatial impression.

Alternatively or additionally, it can also be provided that the display position is also determined depending on the user support information to be displayed, in particular a categorization of the user support information to be displayed, and/or the component to which the user support information to be displayed is assigned.

For example, the user support information to be displayed can be categorized according to its display position. Possible categories include "top", "bottom", "left", "right" and "center" and combinations thereof.

The assignment of the user support information to the respective component can be obtained separately and/or in parallel with the user support information to be displayed.

Alternatively or additionally, it can also be provided that at least some, preferably all, components of the device are stored together with the information associated with them in a data-based collection and the data-based collection is included in the determination of the display position and/or in the perspective adjustment, wherein the information associated with the components preferably represents or makes it possible to determine at least one display position for a user support information associated with the respective component.

This makes it advantageously possible to determine the display position for the user support information for the component to which the user support information is assigned in the data-based collection (for example, the output position described later can be stored in the data-based collection).

Preferably, the fade-in position is specified relative to a position of the marker or it is determined as a position relative to the marker position.

The display positions assigned to the components (or determinable for them) can be display positions dependent on user support information. Then a single component can be assigned (or determinable for) several display positions, which can be selected depending on the user support information.

The collection can be organized in the form of a tree structure, for example. Dependencies between components can also be displayed.

Alternatively or additionally, it can also be provided that the marker contains readable and/or decodable information, such as a unique identifier of the device, as marker information and the marker information is read and/or decoded from the marker and the user support information is requested at least partially on the basis of the marker information, in particular from a memory and/or from a remote computer.

The marker information can advantageously be used to clearly identify the device.

The marker information can advantageously be used to request user support information.

Using the marker information, an overview of available user support information can be requested. Then, after performing a user selection, the actual user support information can be requested in a further step.

The readable and/or decodable information can be provided by the pattern described above on the marker surface.

Alternatively or additionally, it may also be provided that the user support information is requested at least partially based on a user selection and/or from a memory and/or from a remote computer.

For example, the user selection may include selecting one of the available user support information from the requested overview as previously described.

Alternatively or additionally, it may also be provided that the user support information is obtained as image information, in particular by supplementing an image template with further information.

The image information can be particularly efficiently integrated into the image (which in turn preferably contains image information) or into the user's field of vision.

Preferably, the image information is two-dimensional image information.

Alternatively or additionally, it can also be provided that information on the geometry of the device and/or its components is stored for the marker, in particular in the data-based collection, and the geometry information is included in the determination of the fade-in position and/or in the perspective adjustment.

The geometry information can be obtained or retrieved, for example from a memory or from a remote unit, using, for example, the marker information.

The information on the geometry can include, for example, relative positions (in particular to a reference point of the device), dimensions, orientations and/or surface profiles of the device and/or its components. It is advantageous if the marker is oriented in a known manner and/or positioned relative to the reference point, in particular arranged on the device.

This allows user support information to be displayed very reliably at the location or component to which it belongs based on context.

The geometry information can be a model, such as a grid model or a CAD model, of at least parts of the device and/or its components. The model can be a simplified representation of the device and/or its components. This allows the model to be stored and used more efficiently in memory. Often, a model with only a rough representation of the components (approximated as cuboid objects, for example) is sufficient.

The geometry information and/or the model can be part of the data-based collection, in particular part of the information stored in the data-based collection on the individual components.

Alternatively or additionally, it can also be provided that the marker has a surface and the user support information is displayed in the image or in the field of vision of the user in such a perspectively correct manner, in particular as a two-dimensional representation, that at least part of the user support information is perceived by the user as lying within a plane, which plane runs parallel to the marker surface or encloses a defined solid angle with it.

It is advantageously possible to evaluate the marker in the image to create a perspective view. distortion of the marker in the image and to adjust the perspective of the user support information based on the perspective distortion determined. This allows the user support information or parts thereof to be displayed in the image or in the user's field of vision with a perspectively correct representation.

For the purposes of this application, a plane runs parallel to the marker surface even if the marker surface lies in the respective plane.

Alternatively or additionally, it can also be provided that the dimensioning of the user support information within the image or the field of view of the user is carried out at least partially on the basis of a structure of the marker contained in the image, such as an edge of the marker to which a previously known length value is assigned.

This makes it particularly easy to adapt the display behavior of the user support information (in the image or in the user's field of vision) by adjusting the marker size (in reality).

Alternatively or additionally, it can also be provided that the user support information is assigned to a specific component of the device.

The specific component can thus be distinguished from other components of the device.

Alternatively or additionally, it may also be provided that the specific component is a component of the device with a defect or impending maintenance.

Alternatively or additionally, it can also be provided that the device has several components identical to the specific component.

This allows the user support information to be reliably displayed in relation to the correct one of several identical components in the image or in the user's field of vision.

Alternatively or additionally, it can also be provided that the evaluation of the marker comprises determining, at least partially based on the representation of the marker in the image, at least some or all of the necessary information for defining a reference coordinate system, in particular a three-dimensional one, in relation to the marker, in particular viewed from the perspective selected for recording the image, and wherein the determined information is used at least partially in each case to determine the display position of the user support information and/or to adjust the perspective of the user support information.

The representation of the marker in the image can be characterized, for example, by a position, an orientation and/or a geometric course of the marker or parts thereof in the image. Therefore, corresponding features can be determined, at least partially based on determining the information necessary for defining the reference coordinate system. For example, the course of two edges of the marker could be evaluated as the geometric course of the marker in the image, wherein it can preferably be known in advance that the two edges on the physical marker run parallel to each other. If the edges in the image have an oblique course to each other, a viewing angle (i.e. an angle at which the image was taken relative to a frontal direction) can be determined from this and, for example, a perspective adjustment of the user support information can be carried out at least partially.

The reference coordinate system can be viewed as a coordinate system of the marker. For example, the reference coordinate system can have its origin on the marker surface, in particular in the middle of the marker surface. This means that a coordinate system can be set up as a reference coordinate system based on the image, so that the reference coordinate system and the marker can be perceived correctly in perspective in the image. In other words, for several images from different perspectives, the marker would be perceived as being stationary in relation to the respective reference coordinate system displayed in the image.

However, the person skilled in the art understands that the reference coordinate system does not have to appear in itself and is certainly not a physical part of the device. Rather, the display position can be determined at least in part and/or the perspective adjustment can be carried out on the basis of the information determined, with which the reference coordinate system can in principle be defined.

Preferably, the position of the marker on the device and/or the position of the component relative to the marker is known (for example, in the same world coordinate system).

Preferably, a "marker coordinate system" can be set up with the information determined. Above all, however, the information determined can be used to advantageously determine the display position and/or to adjust the perspective of the user support information.

For example, a location specified in coordinates of the reference coordinate system can be mapped to a location in the image (in particular taking into account the information determined to define the reference coordinate system) by projection onto the image plane of the image.

Alternatively or additionally, it can also be provided that the reference coordinate system is a coordinate system of the device and/or the reference coordinate system can be converted in a previously known manner into a coordinate system of the device and vice versa.

Within the coordinate system of the device, for example, positions and/or extensions of the individual device components can be defined as geometry information. For example, corresponding geometry information can be stored in the data-based collection for the individual components. Information about the position and/or extension of a specific component (to which the user support information to be displayed is assigned) can then be retrieved from the data-based collection in order to at least partially determine the display position based on this. This means, for example, that a display position that is "near" the respective component can be reliably determined. Conversely, the component intended there can also be advantageously determined using the data-based collection based on a given position.

Alternatively or additionally, it can also be provided that the user support information and/or the specific component is assigned an output position which indicates a position within the reference coordinate system or which is converted into a position within the reference coordinate system, and wherein the display position of the user support information is determined based on the position within the reference coordinate system.

Specifying the output position in a coordinate system other than the reference coordinate system is advantageous because when the origin of the reference coordinate system changes, for example due to a change in the position of the marker on the device, the fade-in positions stored for the individual components do not have to be adjusted, but only a reference between the two coordinate systems needs to be adjusted. This allows for greater flexibility.

Alternatively or additionally, it can also be provided that (i) the reference coordinate system has its origin on a surface of the marker and/or (ii) the surface of the marker lies in a plane spanned by two coordinate axes of the reference coordinate system and preferably the third coordinate axis is parallel or antiparallel to a normal vector of the surface of the marker.

The reference coordinate system can therefore be the coordinate system of the marker mentioned above.

Alternatively or additionally, it can also be provided that the user support information is assigned to at least one information class of several possible information classes and the form of presentation of the user support information is selected at least partially depending on the information class.

By means of appropriate information classes, different forms of representation can be provided for user support information of different levels of importance, for example. This makes it advantageously possible to encourage the user's attention to the augmented reality application if necessary. For example, user support information that represents a general note ("low priority") can be represented differently than user support information that represents an error state of the device or one of its components ("high priority").

For example, user support information indicating a critical state of the device or one of its components can be emphasized by an appropriate form of representation, for example by displaying it with a prominent font, with a prominent font color, with a prominent font size and/or with prominent symbols. For example, user support information indicating additional information about the device or one of its components can be de-emphasized by an appropriate form of representation, for example by displaying it with a subdued font, with a subdued font color, with a subdued font size and/or with subdued symbols.

For example, different forms of representation can be provided for different types of user support information using appropriate information classes. This makes it easier for the user of the augmented reality application to understand the user support information.

For example, for user support information indicating a value of an operating parameter of the device, a form of presentation resembling an instrument display may be selected. For example, for user support information representing historical maintenance information, a form of presentation in the form of a note sheet may be selected.

Alternatively or additionally, it can also be provided that, depending on the information class, one of the following forms of representation is selected as the form of representation of the user support information in the image or in the field of vision of the user: an information panel, a diagram, an instrument display, a highlight, in particular a border, of an area of the device or one of its components, in particular the specific component, a visual representation of a component, in particular the specific component, or a combination of several of the above options.

An information board can be used advantageously to display text passages, such as a test report, or characteristic values of the device or one of its components.

A diagram can be used advantageously to represent a temporal progression of a quantity, such as an operating parameter, of the device or one of its components.

A highlight, in particular a border, can advantageously be used to identify a certain (in particular the specific) component of the device.

Alternatively or additionally, it can also be provided that the user support information is assigned to at least one of the following information classes: component identification, maintenance information, operating information, low priority, medium priority, high priority and/or process information.

Process information advantageously relates to a process that is at least partially carried out using the device. Process information can also relate to circumstances that lie outside the device itself. For example, process information can relate to information about a good processed using the device, such as bulk material. Process information can also relate to parts of the process in which the device is not involved, such as preceding or subsequent parts of the process.

Alternatively or additionally, it can also be provided that the user support information is a table-like overview with information on one or more operating parameters of the device or the specific component and/or on one or more process parameters of a process at least partially carried out with the device.

This allows the user of the augmented reality application to grasp the user support information particularly quickly.

A process parameter advantageously refers to a process that is at least partially carried out with the device. A process parameter can also refer to circumstances that lie outside the device itself. For example, a process parameter can relate to information about a parameter of a good processed with the device, such as a bulk material. A process parameter can also refer to parts of the process in which the device is not involved, such as preceding or subsequent parts of the process.

Alternatively or additionally, it can also be provided that the user support information is a graphical representation, such as a diagram, a chart, an instrument display and/or a pictogram, of one or more operating parameters of the device or the specific component or of one or more process parameters of a process at least partially carried out with the device.

Alternatively or additionally, it can also be provided that at least one operating parameter is selected from the following parameters: delivery rate, performance, energy consumption and/or target/actual comparison of a device size.

Alternatively or additionally, it can also be provided that at least one process parameter is information, such as a designation or a property, about a bulk material processed with the device within the process.

Processing a bulk material can include, for example, handling, processing and/or which involves the processing of the bulk material. For example, the bulk material is processed when it is ground, mixed, sieved or crushed.

Alternatively or additionally, it can also be provided that the user support information is a graphic illustration of the specific component (for example, screw, spiral, intelligent feeder) and wherein preferably the specific component is not visible in the recorded image or in the field of vision of the user.

This makes it particularly advantageous to display the specific component that is installed in the device but is not visible in the recorded image (because it is covered by other parts of the device, for example). This also makes it possible to display hidden components without having to open the device and therefore without possibly having to interrupt ongoing operation of the device. This allows users of the augmented reality application to quickly and easily gain an overview of the device and its installed components.

Preferably, the user support information displayed as a graphic illustration of the specific component is shown in the image in the orientation that the specific component also has in the device and/or offset from its installation location. The user support information displayed can thus show the specific component offset from its installation location, as in an exploded view.

Alternatively or additionally, the user support information may also be an overview with documentation on the device or on the specific component or a reference to such an overview or to such documentation.

This allows quick access to the relevant documents. It is also possible to assign the documents to the correct component, thus preventing them from being mixed up.

Alternatively or additionally, it can also be provided that the user support information is a highlighting, in particular a border, of an area occupied by parts of the device or by the specific component and/or for highlighting a contour of parts of the device or the specific component, in each case in the image or in the field of vision of the user.

This allows the attention of the user of the augmented reality application to be directed particularly reliably to a specific area of the device or to a specific (in particular the specific) component. For example, a component to be replaced (in particular the specific) can be marked with it.

This means that, especially when there are several similar or identical components, mixing up a component, for example during maintenance work, can be reliably avoided.

Alternatively or additionally, it can also be provided that the user support information is a visualization of at least one measuring section and/or at least one measuring location in the image or in the field of vision of the user.

This allows the user of the augmented reality application to be shown particularly reliably at which location (or at which several locations), for example, one or more measuring sensors must be provided to carry out a measurement, such as a voltage measurement, a vibration measurement or a temperature measurement, or between which locations a measurement must be carried out, such as a distance measurement.

For example, after installing a new component, the user of the augmented reality application can reliably measure the distances displayed by the visualization on the device or its components.

Alternatively or additionally, it can also be provided that the user of the augmented reality application is guided step by step through at least one work process, in particular through at least one maintenance process and/or through at least one measurement process, and for this purpose, in at least some, preferably in every, step, updated user support information is displayed in the image or in an at least partially newly recorded image or in the user's field of vision.

This allows the user of the augmented reality application to highlight different components at each step, display different graphic illustrations, different measurement path visualizations and/or different measurement location visualizations. This makes the work process more efficient, safer and more reliable.

Alternatively or additionally, it can also be provided that a pixel or image area in the image is selected by the user, in particular by a selection via a touch-sensitive screen, and the specific component is selected on the basis of this user selection.

This provides an advantageous and particularly simple way to request information about a specific component by selecting the respective component in the image and then displaying the requested information as user support information in the image or in the field of view.

Preferably, the user support information is associated with the selected component as previously described.

Alternatively or additionally, it can also be provided that the pixel or image area of the image selected by the user selection is assigned to a component of the device via an assignment, in particular on the basis of the data-based collection, and that this component is selected as the specific component.

The assignment can advantageously be carried out by taking into account the data-based collection, the reference coordinate system and/or the coordinate system of the device.

Advantageously, the selected pixel or image area of the image is synchronized via the respective coordinate system with the information stored in the data-based collection, in particular geometry information, of the individual components.

For example, for the selected pixel or image region of the image, a location or a multidimensional region in the reference coordinate system and/or in the coordinate system of the device can be determined and, in particular at least partially based on the data-based collection, a component of the device provided at the location and/or at least partially in the region can be selected as the specific component.

Alternatively or additionally, it can also be provided that different components of the device are assigned to different pixels or image areas in the image.

Here too, the assignment can advantageously be carried out by taking into account the data-based collection, the reference coordinate system and/or the coordinate system of the device, as previously described.

Alternatively or additionally, it can also be provided that a plurality of images, in particular as frames of a video stream, are recorded and for each image the display position and/or the perspective adjustment of the user support information is/are newly determined or carried out and the user support information is displayed accordingly in the respective image or in the user's field of vision.

For example, the individual images of the plurality of images are recorded one after the other. For example, a user can move at least partially along or around the device and repeatedly record an image.

Preferably, the same marker is identified and evaluated in all images. However, it can also be advantageous for at least partially different markers to be identified and evaluated in different images. Preferably, in the case of several different markers, the individual markers have at least partially different patterns and/or marker information. Different patterns and/or marker information can, for example, be used to clearly identify the device and/or determine information about the side of the device shown (for example, front, back, left, right).

For example, if a device with the unique identifier "123" has a different marker on each of its four sides, the marker information of the markers located on the right, left, front, and back of the device could be "123-right," "123-left," "123-front," and "123-back," respectively.

Alternatively or additionally, it can also be provided that the device is a device from the field of bulk material handling, bulk material processing and/or bulk material processing.

For example, the device may be a conveying, measuring, weighing, grinding, mixing, screening, drying and/or dosing device for bulk material.

The object is achieved by the invention according to a second aspect in that an augmented reality device for providing at least one user support information in relation to at least one device or a its components to a user, having the device
at least one detection unit which is configured to record at least one image of a section of the environment in which the device is at least partially visible,
at least one display unit which is configured to receive the user support information to be provided and to display it in the image or in a field of view of the user of the augmented reality device which at least partially contains the environmental section,
at least one evaluation unit which is configured to identify a marker associated with the device in the image and to evaluate the identified marker, and
at least one processing unit which is configured to determine, at least in part based on at least one result of the evaluation, (i) a position at which the user support information is displayed in the image or in the field of view of the user as a display position and/or (ii) to carry out a perspective adjustment of the user support information or parts thereof.

All advantages that have been described in relation to the method according to the first aspect of the invention also apply accordingly to the device according to the second aspect of the invention. Therefore, reference can be made to the previous explanations at this point.

The features that have been described in relation to the method according to the first aspect of the invention can also be provided accordingly in the device according to the second aspect of the invention, individually and in any combination, unless the context indicates otherwise. For this purpose, the device and/or the detection unit, the display unit, the evaluation unit and the processing unit can each be set up to carry out corresponding method steps.

The recording unit, the display unit, the evaluation unit and the processing unit can each be implemented, for example, in software, in hardware or a combination of both. The respective unit can be a device for data processing. The respective unit can alternatively or additionally have a memory, a processor, a receiving device, a transmitting device or any combination thereof.

The following implementations can advantageously be provided individually and in any combination: The detection unit can be or have a camera. The display unit can be or have a screen, in particular with a receiving module that is set up to receive the user support information. The evaluation unit can have means of digital image recognition and/or processing and preferably apply these to the image in order to identify the marker in the image and evaluate the identified marker. The processing unit can be or have a computing unit. The units mentioned can advantageously exchange data with each other at least in part.

Alternatively or additionally, the device may be designed as, or be part of, a smartphone, a tablet, a head-up display or glasses.

Short description of the drawings

Further features and advantages of the invention will become apparent from the following description, in which preferred embodiments of the invention are explained using schematic drawings.

• 1a eine schematische Querschnittsdarstellung einer Dosiervorrichtung für Schüttgut;
• 1b eine schematische Frontansicht der Dosiervorrichtung aus 1a;
• 2 eine Augmented-Reality-Einrichtung gemäß dem zweiten Aspekt der Erfindung;
• 3a ein aufgenommenes Abbild der Dosiervorrichtung aus 1b;
• 3b das Abbild aus 3a mit einer darin eingeblendeten ersten Benutzerunterstützungs-Information;
• 3c das Abbild aus 3a mit einer darin eingeblendeten zweiten Benutzerunterstützungs-Information;
• 3d das Abbild aus 3a mit einer darin eingeblendeten dritten Benutzerunterstützungs-Information; und
• 4 ein Ablaufdiagramm eines Verfahrens gemäß dem ersten Aspekt der Erfindung.

The following show:

• 1a a schematic cross-sectional view of a dosing device for bulk material;
• 1b a schematic front view of the dosing device 1a ;
• 2 an augmented reality device according to the second aspect of the invention;
• 3a a recorded image of the dosing device 1b ;
• 3b the image from 3a with initial user support information displayed therein;
• 3c the image from 3a with a second user support information displayed therein;
• 3D the image from 3a with a third user support information displayed therein; and
• 4 a flow chart of a method according to the first aspect of the invention.

Description of the embodiments

1a shows a schematic cross-sectional view of a dosing device 1 for dosing bulk material. 1b shows a schematic front view of the dosing device 1 from 1a .

A bulk material 3 to be dosed is fed to the dosing device 1, which is received there in a receiving container 5 of the dosing device 1 and forms a bulk material surface 7. The bulk material 3 passes from the receiving container 5 via a bottom opening 9 into a discharge element 13 arranged such that it can rotate within a horizontal section 11, by means of which the bulk material 3 is then discharged from the dosing device 1 in a manner known per se and leaves the dosing device 1 via a vertical discharge 15. The discharge element 11 is coupled to a motor 17 and can be rotated at an adjustable, variable speed, controlled by a motor control of the motor 17. During the material discharge, a change in the weight of a system of the dosing device 1 weighed with two weighing units 19 and 21 is recorded and used to regulate the speed of the discharge element 11. The weighing units 19 and 21 are supported on a support frame 23.

The bulk material 3 can be processed both before and after the dosing process in further process steps, whereby the necessary elements are in 1a are not shown.

On the dosing device 1 on the (in 1b visible) frontal outside of the horizontal section 11 a marker 25 with a defined stripe pattern is arranged ( 1b) The marker 25 is arranged within a coordinate system 27 of the device 1 at a defined location and with a defined orientation on the dosing device 1. The coordinate system 27 is a purely definitional construct and not a physical part of the device 1 and is shown in the figures only for illustration purposes.

The device 1 and its components are represented by a virtual model defined within the coordinate system 27, which is stored in a data-based collection. Position, orientation and extension information is stored for each component. Using the data-based collection, a volume of space occupied by this component within the coordinate system 27 can be determined for each component. Conversely, using the data-based collection, the component intended for a specific volume of space can be determined there. The level of detail with which the components of the device 1 are stored in the data-based collection can vary. For example, a component can be approximated by just a single cuboid, or the external contours of a component can also be recorded in detail using a large number of voxels (volume pixels; 3D pixels).

2 shows an augmented reality device 29 according to the second aspect of the invention.

The augmented reality device 29 is designed in the form of a smartphone and has a detection unit in the form of a camera 31, a screen 33 as well as a display unit 35, an evaluation unit 37 and a processing unit 39. The camera 31 can record an image of a section of the environment in which the device 1 is at least partially visible. An image of a device is shown on the screen 33 only in a purely schematic manner.

3a shows an image 41 of an environmental section taken with the augmented reality device 29, in which the dosing device 1 from 1b The perspective of image 41 is based on a frontal view (as shown in 1b is selected) is directed to the left and upwards towards the device 1. The perspective is also visible in the rotated coordinate system 27 of the device 1, which is also shown for illustration purposes. In the image 41 shown, perspective effects are only indicated schematically; however, for reasons of simplicity of representation, parallel lines, for example, are still all parallel, even if they would actually run obliquely in the image 41 due to the selected perspective due to a perspective distortion during the imaging process.

In order to be able to provide a user of the augmented reality device 29, for example a service employee, with targeted information on the individual components of the device 1 (for example on the discharge element 13, the motor 17 or one of the weighing units 19, 21), user support information can be displayed in the image 41.

For this purpose, the marker 25 is identified in the recorded image 41 and the identified marker 25 is evaluated. The representation of the marker 25 in the image 41 is evaluated. For example, perspective effects of the marker 25 (such as a non-parallelism of edges of the marker 25 that are parallel in reality that occur in the image) and/or the representation of the known pattern of the marker 25 in the image 41 is evaluated (for example, a comparison between a distorted representation of the pattern in the image and the (known) undistorted pattern can be carried out at least as part of the evaluation). Based on this information, a reference coordinate system 43 can be determined in relation to the marker 25, which is viewed under the perspective selected for the image 31. define. The reference coordinate system 43 is defined in three-dimensional space. If the reference coordinate system 43 is defined, the perspective under which the image 41 was taken can also be determined. The reference coordinate system 43 has its origin in the middle of the marker 25 and the marker surface lies within the YZ plane of the reference coordinate system 43. The reference coordinate system 43 is shown in 3a at the location of the marker. The reference coordinate system 43 does not necessarily have to appear, but in principle it is sufficient to further process the information determined with which the reference coordinate system 43 can be set up.

Since the front side of the horizontal section 11, on which the marker 25 is arranged, lies in a plane parallel to the YZ plane of the coordinate system 27, the X, Y and Z axes of the two coordinate systems 27 and 43 are parallel to each other and only the zero points of the two coordinate systems 27 and 43 are offset from each other (with a known displacement vector). In principle, however, the reference coordinate system 43 could be rotated, inclined and offset from the coordinate system 27 as desired, depending on how the surface of the marker 25 runs within the coordinate system 27. In any case, the coordinates of a location that are specified in coordinates of the reference coordinate system 43 can be converted into coordinates of the coordinate system 27 and vice versa.

This means that each pixel of the image 41 that shows a part of the device 1 can be assigned a corresponding location on/at the device 1 both in coordinates of the reference coordinate system 43 and in coordinates of the coordinate system 27. This is possible because the orientation of the reference coordinate system 43 is known or can be determined using the information determined (which is necessary for defining the reference coordinate system 43) and thus the virtual model of the device 1 described above can be rotated and tilted in such a way that it is viewed from the same perspective as the device 1 is recorded in the image 41. The parts of the model that are then visible under the respective perspective can be placed on the image plane (i.e. the drawing plane of the 3a) be projected, whereby an association between locations on/in the device 1 and pixels of the image 41 can be determined.

The pattern on the marker 25 is a 2D code (such as a QR code or an April day) in which a unique identifier of the device 1 is encoded. The unique identifier is read out of the image 41 as marker information from the marker 25 by decoding the 2D code. Using the unique identifier, user support information is requested from a remote computer (for example a cloud computer). For this purpose, the marker information can be sent to the remote computer, whereupon the user support information is received by the remote computer. In addition to the marker information, a component of the device 1 for which user support information is desired can also be specified. For example, an image area in the image 41 shown on the screen 33 can be marked by touching and in this way (by including an assignment of different image areas to the respective components of the device 1) a component of the device 1 for which user support information is desired can be selected.

The user support information is assigned to both a component of the device 1 and an information class. The assignment of the user support information to a component of the device 1 makes it possible to determine a display position for the user support information on a component-specific basis, for example by including the data-based collection. This allows the user support information to be displayed in the image 41 in such a way that it is placed in the image near the location of the respective component or with reference to it. An output position that is specified in coordinates of the coordinate system 27 or the reference coordinate system 43 can be converted into coordinates of the image 41, just as described above for the opposite direction. By assigning the user support information to an information class, different types of user support information can be displayed differently.

In the following, three scenarios are used to describe different possibilities for supporting a service employee as a user of the augmented reality device 29 during maintenance of the device 1.

In a first scenario, for example, a check of the engine 17 can be carried out and the user support information for this can be a speed of the engine 17. The user support information can then be assigned to the engine 17 (engine component) and the information class “operating parameters”.

In the data-based collection, an output position for the user support information of the information class “operating parameters” is stored for the motor 17 in the coordinate system 27 of the device 1. This output position is transferred to the reference coordinate system 43 using the information determined by the marker evaluation and thus the fade-in position in the image 41 is determined. In addition, a perspective adjustment of the user support information is carried out using the determined information.

3b shows an image 45, which is the image 41 from 3a with the user support information 47 displayed therein for the first scenario. Due to the information class, the user support information 47 is selected as an instrument display with a pointer position corresponding to the current engine speed. The instrument display is also visually assigned to the engine 17 in the image 45 by a reference line and is therefore easy for the user to understand. In addition, the user support information 47 is displayed in an adjusted perspective, with the instrument display being perceived by the user of the device 29 as being parallel to a plane of the marker surface when viewing the image 45.

In a second scenario, for example, the weighing unit 19 may be defective and the user support information may highlight the defective weighing unit 19.

Then the user support information can be assigned to the weighing unit 19 (component weighing unit) and the information class “Highlighting”.

In the data-based collection, an output position for the user support information of the information class “highlighting” is stored in the coordinate system 27 of the device 1 for the weighing unit 19. This output position is transferred to the reference coordinate system 43 using the information determined by the marker evaluation and the fade-in position in the image 41 is thus determined. In addition, a perspective adjustment of the user support information is carried out using the determined information.

3c shows an image 49, which is the image 41 from 3a with the user support information 51 displayed therein for the second scenario. The user support information 51 is selected as a three-dimensional border of the weighing unit 19 due to the information class. The border is placed directly at the location of the weighing unit 19. In addition, the user support information 51 is shown in a perspectively adjusted manner. This can provide the user with helpful support during maintenance, for example, so that the correct weighing unit 19 (and not the identical weighing unit 21) is reliably maintained.

In a third scenario, for example, a check of the discharge organ 13 can be carried out and the user support information can be a visual representation of the discharge organ 13 (not visible during operation of the device 1). The user support information can then be assigned to the discharge organ 13 (discharge organ component) and the information class “visual representation”.

In the data-based collection, an output position for the user support information of the information class “visual representation” is stored in the coordinate system 27 of the device 1 for the output device 19. This output position is transferred to the reference coordinate system 43 using the information determined by the marker evaluation and the fade-in position in the image 41 is thus determined. In addition, a perspective adjustment of the user support information is carried out using the determined information.

3D shows an image 53, which is the image 41 from 3a with the user support information 55 displayed therein for the third scenario. The user support information is selected as a visual representation of the discharge organ 19 due to the information class. The visualization is also assigned to the location where the discharge organ 19 is located in the device 1 (i.e. within the horizontal section 11) by means of dashed reference lines. This makes the position of the component easy for the user to understand. In addition, the visualization is shown in a perspective-adjusted manner and the impression of the spatial course of the visualized discharge organ also corresponds to the true course of the discharge organ 13. Although the installed discharge organ 13 is not visible in the image 41, this component 13 can still be made visible by the user support information 55 and thus helpful support can be given to the user.

In an alternative embodiment, the user support information could also be displayed in a field of view of the user that at least partially contains the environmental section. For this purpose, the user could use a modified augmented reality device instead of the augmented reality device 29, for example in the form of glasses that have a camera and a projection unit for displaying user support information.

4 shows a flow chart 100 of a method according to the first aspect of the invention.

In 101, an image of a section of the environment in which a device is at least partially visible is recorded. This can be done, for example, by means of the camera 31 of the augmented reality device 29. The device can 1a and 1b described device 1.

In 103, a marker associated with the device is identified in the image and the identified marker is evaluated. This can be done by means of the evaluation unit 37 of the augmented reality device 29.

In 105, user support information to be displayed in the image is obtained. For example, marker information can be read from the marker or determined using the marker and the user support information can be requested using the marker information.

In 107, based at least partially on a result of the evaluation of the marker, a position at which the user support information is displayed in the image or in the field of view of the user is determined as the display position and a perspective adjustment of the user support information or parts thereof is carried out. This can be done, for example, by means of the processing unit 39 of the augmented reality device 29.

In 109, the user support information is displayed in the image or in a field of view of a user of the augmented reality application that at least partially contains the environmental section. This can be done, for example, by means of the display unit 35 of the augmented reality device 29. The recorded image with the user support information displayed therein can then be output on the screen 33.

The method is therefore advantageously suitable for displaying user support information relating to at least one device or one of its components in an augmented reality application. For example, the information relating to 3b-3d types of user support information described above may be displayed.

The features disclosed in the foregoing description, in the drawings and in the claims may be essential to the invention in its various embodiments both individually and in any combination.

list of reference symbols

1

dosing device

3

bulk goods

5

receiving container

7

bulk material surface

9

floor opening

11

horizontal section

13

discharge organ

15

vertical drop

17

Motor

19

weighing unit

21

weighing unit

23

support frame

25

marker

27

coordinate system of the device

29

augmented reality device

31

camera

33

Screen

35

display unit

37

evaluation unit

39

processing unit

41

image

43

reference coordinate system

45

image

47

user support information

49

image

51

user support information

53

image

55

user support information

100

flow chart

101

Taking an image of a section of the environment in which a device is at least partially visible

103

Identifying a marker associated with the device in the image and evaluating the identified marker

105

Obtaining user support information to be displayed in the image

107

Determining a display position of the user support information in the image or in the field of view of a user and making a perspective adjustment of the user support information or parts thereof, each at least partially based on a result of the evaluation of the marker

109

Displaying the user support information in the image or in a field of view of the user that at least partially contains the surrounding area

X, Y, Z

coordinate axis

Claims (13)

Method for displaying at least one item of user support information relating to at least one device or one of its components in an augmented reality application, wherein at least one image of a section of the environment in which section of the environment the device is at least partially visible is recorded, wherein the user support information to be displayed is obtained and displayed in the image or in a field of view of a user of the augmented reality application that at least partially contains the section of the environment, wherein a marker associated with the device is identified in the image and the identified marker is evaluated, and where, at least partially based on at least one result of the evaluation, (i) a position at which the user support information is displayed in the image or in the field of view of the user is determined as the display position and (ii) a perspective adjustment of the user support information or parts thereof is carried out. procedure according to claim 1 , wherein the display position is also determined depending on the user support information to be displayed, in particular a categorization of the user support information to be displayed, and/or the component to which the user support information to be displayed is assigned. Method according to one of the preceding claims, wherein at least some, preferably all, components of the device are stored together with information associated with them in a data-based collection and the data-based collection is included in the determination of the fade-in position and/or in the perspective adjustment, wherein preferably the information associated with the components each represents or makes it possible to determine at least one fade-in position for a user support information associated with the respective component. Method according to one of the preceding claims, wherein the marker contains readable and/or decodable information, such as a unique identifier of the device, as marker information and the marker information is read and/or decoded from the marker and the user support information is requested at least partially on the basis of the marker information, in particular from a memory and/or from a remote computer. Method according to one of the preceding claims, wherein information on the geometry of the device and/or its components is stored for the marker, in particular in the data-based collection, and the geometry information is included in the determination of the fade-in position and/or in the perspective adjustment. Method according to one of the preceding claims, wherein the user support information is associated with a specific component of the device, wherein preferably the specific component is a component of the device with a defect or impending maintenance and/or the device has a plurality of identical components as the specific component. Method according to one of the preceding claims, wherein the evaluation of the marker comprises determining, at least partially based on the representation of the marker in the image, at least some or all of the necessary information for defining a, in particular three-dimensional, reference coordinate system in relation to the marker, in particular viewed under the perspective selected for recording the image, and wherein the determined information is used to determine the display position of the user support information and/or to adjust the perspective of the user support information is used at least partially in each case, wherein preferably the reference coordinate system is a coordinate system of the device and/or the reference coordinate system can be converted in a previously known manner into a coordinate system of the device and vice versa. procedure according to claim 7 , wherein the user support information and/or the specific component is assigned an output position which indicates a position within the reference coordinate system or which is converted into a position within the reference coordinate system, and wherein the display position of the user support information is determined based on the position within the reference coordinate system. Method according to one of the preceding claims, wherein the user support information is assigned to at least one information class of several possible information classes and the form of presentation of the user support information is selected at least partially depending on the information class. Method according to one of the preceding claims, wherein the user support information is (i) a table-like overview with information on one or more operating parameters of the device or the specific component and/or on one or more process parameters of a process at least partially carried out with the device, (ii) a graphic representation, such as a diagram, a chart, an instrument display and/or a pictogram, of one or more operating parameters of the device or the specific component or on one or more process parameters of a process at least partially carried out with the device, (iii) a graphic illustration of the specific component and wherein preferably the specific component is not visible in the recorded image or in the field of vision of the user, (iv) an overview with documentation on the device or on the specific component or a reference to such an overview or to such documentation, (v) a highlight, in particular a border, of an area occupied by parts of the device or by the specific component and/or to highlight a contour of parts of the device or the specific component, in each case in the image or in the field of vision of the user, and/or (vi) a visualization of at least one measuring section and/or at least one measuring location in the image or in the field of vision of the user. Method according to one of the preceding claims, wherein the user of the augmented reality application is guided step by step through at least one work process, in particular through at least one maintenance process and/or through at least one measurement process, and for this purpose, in at least some, preferably in every, steps, updated user support information is displayed in the image or in an at least partially newly recorded image or in the field of vision of the user. Method according to one of the preceding claims, wherein a pixel or image region in the image is selected by the user, in particular by a selection via a touch-sensitive screen, and the specific component is selected based on this user selection. Preferably, wherein (i) the pixel or image region of the image selected by the user selection is assigned to a component of the device via an assignment, in particular based on the data-based collection, and this component is selected as the specific component and/or (ii) different components of the device are assigned to different pixels or image regions in the image. Augmented reality device for providing at least one item of user support information relating to at least one device or one of its components to a user, the device comprising at least one detection unit which is set up to record at least one image of a section of the environment in which section of the environment the device is at least partially visible, at least one display unit which is set up to receive the user support information to be provided and to display it in the image or in a field of view of the user of the augmented reality device which at least partially contains the section of the environment, at least one evaluation unit which is set up to identify a marker assigned to the device in the image and to evaluate the identified marker, and at least one processing unit which is set up to, at least partially based on at least one result of the evaluation, (i) determine a position at which the user support information is displayed in the image or in the field of view of the user as the display position and (ii) perform a perspective adjustment. version of the user support information or any part thereof.

Images